Built a few shelters near a market with 85%+ desirability. They’re stuck as shelters because most residents are soldiers. As such, the shelters never get stocked with food or herbs to progress beyond being shelters. This probably applies to guards too, but haven’t been able to confirm that.
The non-soldier residents are foragers, fishermen and hunters.