[SPOILER] Faction benefits.

Title is a bit misleading, but I didn’t want to spoiler anything.
But here comes a Spoiler now:
I’m closing in to complete Elite. Now I have already decided what the smaller decisions concerning NPCs will be in my Ultimate run. I am trying to prepare for endgame as best as I can to grind some gear and maximize my build.
What I cannot decide however is which of the factions to join when I get to make that decision.
For now I have joined the Order in both normal and elite. I guess I can still swap over to the Chosen in Ultimate. Should I do this?
In normal I joined the Outcast and in elite I killed the witch. I don’t see what the consequences are in the long run. What would you suggest for me to do?
I’m playing hardcore and I’m playing a Blademaster with Bleed, Pierce and Cold.