Not sure if this is considered a bug, hence the question mark.
Workers are being assigned to jobs far away from their houses. This becomes very frustrating and limits the ability of the town to work efficiently as it gets large. I set up a test example to demonstrate what’s happening.
I created a town with three work sites (forager, hunter, firewood), all spread out from each other. I set one shelter at each work site.
The work sites got built first, so workers were assigned before the houses existed. As each family moved into the houses, it became clear that one family contained all the assigned workers. They moved into the last house built, which happened to be the one near the hunter cabin.
I subtracted the worker from the forager shack and then added a worker back in. The game chose a villager from that one house by the hunter’s cabin rather than choosing a worker from the house right next to the work site. When I added a second worker to the firewood splitter, the game chose a worker from the house by the hunter’s cabin again. These workers did not appear to be closer to the work site than other laborers when they were chosen.
It seems to me that the game has an internal list of who needs work, and it always picks the laborer from the top of that list, irrespective of how far that laborer will have to travel to work. And of course, this causes huge efficiency issues in late game.
For me, efficient worker assignment and believable worker behavior is so important for game enjoyment, so I hope there’s a way to beef up the algorithm. 
This is an awesome game so far and I can’t wait to see where you all are able to take it!
Here’s the experimental setup if it helps:
Lyonesse.zip (1.8 MB)