You are limited how you can react to an incoming raider attack. Out of about 14 turrets in my town maybe 4 will be usefull at defending. So whats about the 20 people being watchers in the other towers.
My solution would be : " You are able to set a defense area/location for a tower to make its watcher go there and shoot at the enemys. Similar to the option from the barracks "
That would be an easy way to improve your defenses and you can react to an incoming attack and make use of your people.
How are your thoughts about the defense situation?