You mentioned that the raiders seem to consistently be breaching your walls. Are they getting through in areas that your towers don’t cover? If you have an area not covered by towers, a large number of raiders will likely swarm there seeing it as a safe, unprotected place. Stone walls will take quite some time to breach by raiders, but even with only one or two towers, especially with a single guard in each, a group of forty or more raiders can take that time and still get through before being whittled down.
Also, do you have one or two people in each tower? I don’t know that there really is a perfect ratio, as my experience has been that the number of towers has less to do with population size, and more to do with how physically large your settlement is and the area you need to protect.
My general strategy seems pretty consistent with what has already been suggested. I start with towers, even well before walls, but I always keep two guards in each tower (after all, you pay by the tower, not the guards, so if you have the gear and bodies, why not?), and make sure that all towers overlap their field of fire. I find crossbows are more effective, so I try to get my tower guards armed with them as early as I can. Soldiers and garrisons I add at a later point.
I tend to design my settlements in a sort of neighborhood or district layout, and as I place things, I’m already planning on where walls and towers will go. As @red1 mentioned, try to have the first towers cover things like the town centre and market. I have had others tell me that they’ve encountered raids where the raiders sack outlying buildings, mines, and in one case even a night soil collector. But my experience has typically been that raiders are drawn to two things. First, they aim for places where they expect to get good loot. And it makes sense. I’ve seen them run right past an unprotected stockyard to attack a market, because the market has higher value, more easily transported items than the logs and stones in the stockyard. The other is they tend to respond to whoever is attacking them, so they will go for your towers and town centre. This is where overlapping field of fire makes a big difference. If they split up and attack every tower, the number of raiders per tower is pretty low and the two guards can usually deal with it. If they attack a single tower en masse, the neighboring towers can help reduce the attackers before the tower falls. Hunters can be useful here to lead raiders where you want them, but be careful. I’ve seen a hunter run out of arrows and try to engage a group of over thirty raiders alone in melee combat.
Once I start walling neighborhoods, I usually leave the one with my town centre until a few others are already secured. This is because with a good amount of villagers inside it, and a couple of towers with overlapping field of fire on it, it can do a decent job of defending itself. This is also usually when I’ll start using soldiers. As was mentioned, be sure to equip them well. From my own experience, armour and shields do make a pretty big difference. But as @machin mentioned, well equipped soldiers can easily face far superior odds with a good chance of success. Even with just hauberk, sword, and shield, I would say a 3:1 or even 4:1 is a pretty safe bet. But once you have the soldiers outfitted in plate armour, heavy weapons and shields, and maybe even a crossbow to boot (which is what I usually set them up with), 5:1 or possibly even greater odds are not much of a challenge for them.
When I start to focus more on soldiers, I use them to move to less defended areas. Just remember, soldiers don’t move instantly, so make sure to place the barracks close to areas you think will need the most support. And also remember, barracks double as a pretty high powered tower as well. With range weapons, soldiers can fire out of the garrison and nicely soften up larger raider forces before even getting into combat directly. Once in direct combat, you can have your soldiers fall back to areas covered by towers for the added support as well. The raiders will stay on them, and get drawn right into the towers’ fire.
Of course, all this may need some tweaks and changes with the upcoming patch, as specialized troop types, advanced raiders, etc., may change any or all of it. For now though, it has worked well for me. But as you can see from the various suggestions made, there really is no one size fits all answer. Whether you have lots of towers, or no towers at all, a single garrison with twelve soldiers, or three garrisons with four soldiers each, or any combination you choose, seems more determined by your own settlement layout and what you need to protect, as well as your preferred style of play and defense, than any hard and fast rule or ratio.
Hope the suggestions help, and best of luck with your settlement.
Cheers
Anyway…
… No threat.
