There’s not just the labour cost, there’s also the opportunity cost - a building isn’t working while it’s being rebuilt somewhere else. The problem is also magnified when you want to rearrange multiple buildings in the same area - you first have to move them out to clear space because you can move them back in. When dealing with upgraded shelters, this can take long enough that they devolve from lack of desirability.
None of these costs are huge, but they are not nothing. And, of course, labour itself might be the most precious resource in the game.
Having costs to move would certainly be possible, and make the game significantly harder. But, for me, it would remove a lot of the joy. Being able to experiment with different layouts and rearrange things as I grow so that I can make my town just right is a big part of the attraction. Larger movement costs would restrict that sense of freedom and playfulness considerably IMO. Although I can see how the additional rigidity would be a fun challenge for some players. So maybe it would make for a good optional setting?