While I really appreciate the option to relocate a building, I would suggest that it should be some cost (beyond the labour) associated with moving the building.
The reasoning behind my proposal, is simply that there’s very little downside to not planning ahead as it is.
Well, moving the town centre and graveyards do have a cost already.
Good! That means that my suggestion is not that far-fetched.
Plus level II Temple.
There’s not just the labour cost, there’s also the opportunity cost - a building isn’t working while it’s being rebuilt somewhere else. The problem is also magnified when you want to rearrange multiple buildings in the same area - you first have to move them out to clear space because you can move them back in. When dealing with upgraded shelters, this can take long enough that they devolve from lack of desirability.
None of these costs are huge, but they are not nothing. And, of course, labour itself might be the most precious resource in the game.
Having costs to move would certainly be possible, and make the game significantly harder. But, for me, it would remove a lot of the joy. Being able to experiment with different layouts and rearrange things as I grow so that I can make my town just right is a big part of the attraction. Larger movement costs would restrict that sense of freedom and playfulness considerably IMO. Although I can see how the additional rigidity would be a fun challenge for some players. So maybe it would make for a good optional setting?
Optional only, if it’s the case. As per Olleus, many of us are used to rearrange a lot of stuff around, and this has heavy hidden costs already. Having to pay also gold would mean depleting the town vault for nothing.
Personally I love the fact that you can relocate things without extra cost - this adds to strategy and planning and therefore fun. Unless you have a crystall ball or save game immediately - plan - then build your city - its actually nice mechanics that once you found better opportunity you can relocate the building. Especially that you have different needs at the start of the game than at the later stages so I really hope the devs will not change that.
Those are certainly valid concerns. I’ll readily admit that I did take the labour cost into account. Generally, my experience in Farthest Frontier resources are the primary bottle-neck, and the actual labouring seems in most cases almost like an afterthought.
I’ll also confess that when first I found out that this game allows the player to just move buildings (rather than having to go through the tedium of building a new building and then demolishing your old one) I thought it was extremely neat.
I can also see how if the main attraction is really building a beautiful city rather than playing a town manager simulation, my proposal is not all appealing. Having it an optional setting seems a good compromise.
I wasn’t thinking of adding a gold cost. (To be honest I think gold doesn’t really make sense in this game.) What I had in mind would be a percentage of the costs of initially placing the building - say one log to move a shelter.
I would go as far as to say that the opposite is the case - The easier it is to change your mind, the less important it is to plan ahead. But again, I too really like the relocate option. Also, as Olleus suggested above, it could be an optional thing.
I can see your point of view - you thinking of importance of planning ahead. I see the planning as constant progress - at the beginning its important to place everything near by. Later your needs change so now you need to plan how to rearrange things, as things keep changing so does your planning - personally I simply enjoy this aspect of the game but I also know that different players like different things 
In construction, the labour cost isn’t very big compared to the resource cost. But, in the game as a whole, labour cost is a big deal. When you’re making, eg, planks, the villager-seconds (aka, man hours) it takes to manufacture them is a more important resource than the logs themselves IMO. In fact, you could convert almost everything into villager seconds in the game: and for most buildings the villager-seconds taken to manufacture the resources are far greater than the villager-seconds spend in construction work. By a couple of orders of magnitude (which is potentially a balance problem in its own right, but that’s a different topic).
That said, if you’re constantly moving buildings around it will wear down your general efficiency. The villagers doing that aren’t doing moving items, manufacturing things, or running services. So you’re already incentivised to move as rarely as possible, so both planning for now and planning ahead is rewarded. But not so much that changing your mind, or adapting to a new situation, feels like a punishment.
Which I think is important. Different people get different things out of it, but Farthest Frontier is more oriented towards building your perfect village (in the face of internal and external pressures), rather than a hardcore survival or tower defence game. Or, to put it another way, when I play with Lego, I like to be able to take them apart and put them back together again, and I would hate it if they were glued together (even if I could dissolve the glue with extra effort).
But, then again, if the devs doubled or tripled the labour cost of construction and moving buildings, I wouldn’t care so much. I suspect most people would barely notice.
I think you really get to the core of the matter here - The thing that takes ‘villager-seconds’ (Lovely turn of phrase! 
) in construction is the production of resources at not actually the labour associated with the ‘building phase’. In general if that was increased, I think I would be pretty well pleased too. As it is now, when building new buildings, it seems that from the time the resources are delivered, the building is as good as build.