yes, only way is to go to a tower, remove everybody there and put back people (hopefully not the same).
(I’m not sure what happens with the weapons but I guess they magically go back into storage).
Update: This house is currently on its way to becoming a large house … and probably a manor shortly afterwards. So is another one that suffered the same problem.