When selecting a villager by clicking them in the building UI screen movement is locked.
I frequently do this to select hunters inside their cabin, or out on a hunt, to defend against predators, but my workaround is to first give a move order to have them leave the building then select them directly to enable screen movement (or deselecting them and reselecting them if they’re out on a hunt).
Allowing screen movement after selecting a villager using the building UI would provide a better experience in defending against predators through the early years.