Outcast and Barrowholm you both get extra Nemesis. But it’s slow to reach that status and you lose some experience from not doing their quests. Furthermore Barrowholm provides excellent augments. So unless you dislike them in terms of story, stay friendly on all difficulties.
As for FG cults, I personally skip FG content on elite, cause mobs are scaled and deal damage (especially bosses) and I think there are faster ways to level. So not sure if its best to stick to single one throughout the game. Also there’s faction booster, so you might reach revered status on earlier difficulty and choose different faction.