Get Pet Aether res to atleast 80%. Overcapping recommended. Then see if you can get more Pet Phys res too as Aether res alone won’t help. Other than that, use the Pet Attack command to better position your pets and don’t fight near his Aether Crystals.
On the topic of your pets getting destroyed by his meteor, this is my own Bleed Pet Conjurer: Sanguine Lupus - Pet Conjurer which boasts much better defenses and I still wouldn’t recommend letting your pets (or yourself) face tanking his meteors if you can. It just hits hard 