Well, for starters, the feckless of the walls is disproportionately low in comparison with the costs of their construction - There is no point in building stone walls if they can be destroyed by a homeless person with a club.
those. Walls have no value at all in defense.
I suggest doing this.
- the walls cannot be destroyed by infantry, only with the help of a ram. The raiders are forced to storm the gate.
- A platform appears on the wall that allows soldiers to move along the top of the wall and, in the event of an attack, begin to fire at the attackers with bows or crossbows. Also, in my opinion, there should be battlements and loopholes that give the defenders a bonus to defense.
reference - Fortress walls in Stronghold.