As ya_ said, Dying God + Aeon’s Hourglass is a better devotion setup for Witching Hour pet builds. Time Dilation gives you better uptime for Witching Hour’s active skill (and the actives on Beastcaller’s Cowl and Sovereign Ruby of Domination), and Dying God has extra chaos bonus damage (the non-pet bonuses apply to pets too because it’s an aura). Time Dilation also helps you resummons skeletons faster when you need to.
Manticore’s resistance reduction doesn’t stack with Witching Hour. Things that say “X reduced target’s resistances” don’t stack, where as things that say “-x% resistance” do.
I disagree with ya_ about the augments. Mender’s Powder is really good for helping keep skeletons alive. Aether and Chaos resistance are really important.
Eldritch Mirror is the preferred component for both helm and chest armor on pet builds, they do stack when you have two of them. Probably use Silk Swatch on the pants, and Sacred Plating on shoulders, and probably use Black Tallow on the medal (if you still need more chaos after changing devotion setups). Also use Seal of Ancestry on the Amulet if you can.
If you have it, Mythical Fiendflesh is really really good for that kind of a build. It provides massive pet resistances which will do a lot to help keep them alive, and also provides you with good resistances and some absorption.
I would recommend heavy investment into a second pet, either Familiar or Blight Fiend (or both). Use Circlet of the Great Serpent and Unstable Anomaly if you want to go Blight Fiends (Blight Burst is the most important skill in the Blight Fiend tree, btw), or use Mythical Will of Bysmiel if you want to go with Familiar. There are a lot of skill points you could free up, here is an example of where you can take them from (your build with unnecessary points removed): https://www.grimtools.com/calc/eVLDmqLV
Mogdrogen’s Ardor is probably the relic you’d want to use, the resistances are pretty important.
I don’t know whether you’re using them or not because you didn’t say, but the crucible buffs make a HUGE difference. It’s very very difficulty to clear Gladiator on a skeleton build without them.
You can look at these threads for some ideas of how to balance all of what I mentioned above.
http://www.grimdawn.com/forums/showthread.php?t=77518
http://www.grimdawn.com/forums/showthread.php?t=80282