Regarding shelter

Is it normal for hunters, foragers and fishing folk to leave the town and seek shelter in their huts and shacks? I click on the shelter icon in the Town Centre and next thing I know I’m getting multiple messages about villagers dying in combat, and my barracks aren’t even full. I’m doing vanquisher but with these losses, and immigration taking three or more turns even with food and housing, there’s almost no way to keep my Tier 3 town alive.
When the seek shelter icon is turned on villagers in the field should run towards town, not away.