I decided to try out the Graveburg setup (same seed, small map, vanquisher), and when I my people finished building the town center it flattened a bunch of the hill which definitely was not under my TC:
It’s that bit down and to the left of the TC itself. (The lake is to give a geographic reference point.)
I don’t know if this happens for any other buildings, or if it happens reliably, but it was very weird, so I wanted to report it. I can provide a save if it’s helpful, but I suspect if you can get it to reproduce reliably, that is more useful.