Similar to Fertility or Desirability hot maps, there could be “Danger zone” mapping.
Depending on danger level, Villagers could refuse to go into completely (red level), or they could work with decreased Happiness/efficiency (yellow level).
Danger increasing factors: Raiders, wild animals, unexplored terrain, etc.
Danger decreasing factors: Barracks, Towers, Town Center, gates and walls(?), nearby soldiers, etc.