Completing set unsummons player-scaled pets

I’ve been tinkering with weapon swap with this build:

And every time I change weapon set, my guardians vanish. I figured it was the blazeheart, so I removed it… in fact I also removed everything on swap.

Adding stuff after summoning the guardians is fine, and removing things un-summons them (as it should) …but as soon as I complete the virtue set, they disappear.

I tried removing other pieces of gear too, and the same thing happens: I summon them, then start putting my gear back on, and as soon as I put on the last piece, they are gone no matter which piece of the set is the last one in.