In construction, the labour cost isn’t very big compared to the resource cost. But, in the game as a whole, labour cost is a big deal. When you’re making, eg, planks, the villager-seconds (aka, man hours) it takes to manufacture them is a more important resource than the logs themselves IMO. In fact, you could convert almost everything into villager seconds in the game: and for most buildings the villager-seconds taken to manufacture the resources are far greater than the villager-seconds spend in construction work. By a couple of orders of magnitude (which is potentially a balance problem in its own right, but that’s a different topic).
That said, if you’re constantly moving buildings around it will wear down your general efficiency. The villagers doing that aren’t doing moving items, manufacturing things, or running services. So you’re already incentivised to move as rarely as possible, so both planning for now and planning ahead is rewarded. But not so much that changing your mind, or adapting to a new situation, feels like a punishment.
Which I think is important. Different people get different things out of it, but Farthest Frontier is more oriented towards building your perfect village (in the face of internal and external pressures), rather than a hardcore survival or tower defence game. Or, to put it another way, when I play with Lego, I like to be able to take them apart and put them back together again, and I would hate it if they were glued together (even if I could dissolve the glue with extra effort).
But, then again, if the devs doubled or tripled the labour cost of construction and moving buildings, I wouldn’t care so much. I suspect most people would barely notice.