If I’m not mistaken, right now you can’t really choose who is going to work in what place as long as you have more than enough ppl in village (the ui is only shown to player when he wants to add another worker to i.e: mine, but there is no available man power, so you are prompted to replace someone who already have a job) and that was an issue for my playthrough.
I was setting up a small mining town on the other side of the map and I built there some settlements too, but for mines and towers AI selected workers from my main town, instead of taking them from the buildings right beside them… so they had a long way to go through, in fact it was the whole map 