Herbs and medicinal roots can only be collected by Foragers from the location where this resource spawns when the map is rolled.
When your village / city expands, it will usually engulf these grow sites and players need to build around herbs / roots spawnpoints (or lose them).
Perhaps the Forager Shack could get an upgrade to Gardening Shed, allowing players to cultivate an area like the Forester building does, through a planting function.
Ideally, Willow would also be cultivatable, provided a body of water is nearby