I am using a Will of Bysmiel to get an extra Familiar. When I unequip that amulet, the extra familiar does not unsummon. I can equip my Heart of the Mountain and get an extra Flappy and Chilldude at the same time.
The same is true the extra Briarthorn from the Beastcaller set.
Just my opinion, but this seems like a bug.