At the minimum, you want to farm up a Korvaak Burning Blade to convert Briarthorn damage into fire. Other than that, you should be able to more or less respec into the setup from the get go since the idea is just fire pets. May want to get Familiar as well though since without Beastcaller set, you will be low on pets.
That said, this is still going to be my recommendation for any pet conjurer before you have an endgame set - Dark Rose - Lazy Dark One’s Set Budget Pet Conjurer (but that has to do with my obsession with laziness)
It does not need the double rares per se, but the build was created just for Uber Nerfatal and without the bonuses from those MIs, that particular boss will be harder.
But if you ignore its existence, can even do without any MIs.