You should define "big impact" - what metrics are important to you? Which improvement gives the biggest customer value? You might apply the same measures, that are used to priorize features in the backlog (for example: [Backlog Prioritization techniques][1]). 

Try to estimate what the impact of these improvements in a long term will be. Are you able to deliver features faster and with better quality? 

If there are critical issues / bugs that might break the whole application, they should be fixed as soon as possible!

Dealing with technical dept can be a thin line, where there is sometimes no right or wrong - but there are approaches to do so: 

* [11 Strategies for Dealing With Technical Debt][2]
* [Escaping the black hole of technical debt][3]

In general the agile way is iterative, maybe you try to split the big problems into smaller ones and improve your process / code base in small steps (Although I know that some refactorings have to be done in one big step).

  [1]: http://agileworld.blogspot.de/2010/06/backlog-prioritization-techniques.html
  [2]: http://www.disciplinedagiledelivery.com/technical-debt/
  [3]: https://www.atlassian.com/agile/technical-debt