Routine refactorings should be part of your story estimates. Deliberate changes should be estimated as stories. No work should ever be done "outside" of the Sprint Goal or Backlog.
Routine refactorings are a normal by-product of working on code. Code refactoring is the art of changing internal implementation without changing external behavior, and is often necessary overhead for adding new features. This overhead should be included in your story estimates.
If you are planning deliberate changes to reduce technical debt or improve non-functional requirements, then these changes should be user stories that are prioritized on the Product Backlog. They are project goals and consume team resources, and must therefore be scheduled and estimated like any other deliverable.
No Invisible Work, Ever
In Scrum, you should never be performing work that isn't directly related to a Sprint Goal. That means that team capacity should never be consumed by work that isn't on the Product or Sprint Backlog, and that explicit allocation of team effort trumps implicit estimation every time.
That doesn't mean that implicit work (such as routine refactorings) can't be included in your team's estimates or part of your definition of done. However, it does mean that there's a huge difference between embedded assumptions about level-of-effort and "off the books" requirements. For example:
Adding a new Widget#foo method might require refactoring the Widget class and the Widget#bar method.
These are expected activities, and should be explicit tasks related to the user story. This work should certainly be included in the story/task estimates.
Fixing Widget#implode requires refactoring the unit test.
This should probably be part of your definition of done. It should be accounted for in the estimate, but probably doesn't need to be made explicit unless your team isn't experienced with TDD. Your mileage may vary on this one.
Add a Widget#get_rich_quick feature, but don't track all the work done on Widget#steal_from_poor to make this feature work.
This would be bad on many levels: off-record work, invisible processes, non-inclusive estimates, and lack of project transparency. If your project is okay with stealing from the poor, then that effort needs to be visible and tracked, rather than swept under the rug.
What can be implicit and what needs to be explicit will vary from team to team, and often depends a great deal on the maturity of your team processes. However, there's often a very clear delineation between implicit work and untracked work; never cross that line deliberately, and leverage your retrospectives to make sure you haven't crossed the line accidentally.