Especially early in the game, I remove my guards from towers which aren’t going to be doing any fighting and move them to a location I may have left undefended. Okay so far but why not have a rally point in the Tower menu? Not only will my guards go there but they’ll probably stay there too.
Also setting a CTRL+number does not work for guards.