The problem comes with the impact of an unfocused stream of refactors to key code.

For example, there is much less long term developer impact if refactoring were _accelerated_, scheduled to be performed in a one-week sprint.  There is a lot of breakage, yes, but after that week the average level of downstream patch breakage is significantly lower.  A "rip the band-aid off quickly rather than slowly" approach.