If you are measuring a long(er) running project with velocity to get an idea when features or milestones will be reached it is a good idea to not estimate defects. Defects are work that should have been completed in other points already in the past. If you have a lot of defects now, probably you will have the same amount in the future, slowing your whole project down. It would give a false sense of speed if you give non-valuable work story-points. Velocity goes down when you have defects or technical-debt.
An increase in defects or decrease in velocity should always result in a good root-cause analysis during a retrospective to find ways to prevent similar mistakes or improve the process. If the cause is really your QA process is something you have to find out, as it greatly depends on the situation.
Some metrics I like to use to signal issues:
- Average velocity over time (should go up or stay stable)
- Defect count vs complete features count (feature side should go up)
- Time spend on defects (should go down or stay stable)
These should be pretty stable or going down/up, depending on the metric.
Now from experience the root-cause of most defects is most often not the lack of a QA process, but an architecture or design that has a lot of coupled dependencies, making it hard to maintain and extend the code base. Often it is hard to create test-automation for it and therefor it hard to refactor and clean it. I always suggest to watch the first episode of Clean Code to get an idea of common issues with code.