House uprgrading when negative impact production building are on pause

Hello,

I really like the min max system for production but it can lead to house beeing deserted. When a building as a negative impact on a house, when production is paused, the house will uprgade. Then production is unpaused and house deserted. Could it be an idea to keep the negative impact from production buildings active even if production is paused ?