Without seeing OP’s Grimtools link, I can’t do much, but you definitely don’t need a double rare MI green pants to make pet builds work.
If you are interested in a Pet Cabalist, I’d recommend either mine or Sigatrev’s builds. Atleast until OP updates the thread.
All my Pet builds can be found here: The Carnival - A Guide to Pets
As for Sigatrev’s, look here: SkeleWraith Lost Souls Cabalist / SkeleWraith Lost Souls Cabalist
There is also another one by Squib that has an interesting take with Dual Wielding: [CABALIST PET] MAXIMUM AMOUNT OF SUMMONED SKELETONS POSSIBLE BUILD - dual wield to squeeze out more!