Except there’s a lot more physical resist reduction sources than there’s Chaos resist reduction sources. This couple with the fact that one of the main enemy factions heavily resist it makes it one of the worst damage types. Aether suffers the same exact issue. And it’s not just a few more hits, it can make traversing several areas in the game annoying because how much longer it takes to kill the enemies.
This is why physical damage builds are so popular these days because it’s so easy to heavily reduce their physical resist.