There are currently a few buildings that don’t have a meaningful function; the town center, the graveyard, and the shrine. I think they could be made actually interesting with the addition of tradition as a resource (similar to gold except immaterial), that is gathered by otherwise idle villagers at tradition builds which can then be spent on meaningful choices to upgrade your village.
The TC could be where you manage and make the decisions, and earn a low base amount of tradition. By meaningful decision I mean forced choices with each having a positive and a negative. Ie, troops fight better in melee but eat more food, or the fight better with arrows but waste them while practicing. Or, declare veganism, banning meat but lowering the number of types required to upgrade. Etc etc, just avoid the boring kind of tech tree where you can unlock everything.
When villagers are idle they will occasionally visit the graveyard to earn a bit of tradition. Also maybe allow investment in servicing to give it a desirability bonus.
The shrines could also be visited too. When upgraded allows you to pick a bonus that is applied in its area. (One boost could be desirability, another workrate, etc).