Do both effects cancel each other out? Or is there an order in the effects?
First thing to understand is damage can only be converted once. Converted damage cannot be converted again.
What mamba has said is the basic way of understanding conversion but it goes slightly deeper than that:
[spoiler]Yes there is an order to conversion but it is in terms of where that conversion is found. According to the Game Guide, it goes as follows:
Base Skill > Skill Modifiers > Conversion on the Skill or Transmuter > Conversion on Equipment and Buffs > Equipment, Auras and Passives
Within every step, each individual conversion that falls into that step is tallied up and executed at the same time. This is what mamba means.
In your example, the Physical to Vitality to Reaping Strike/Markovian’s Advantage conversion on Mythical Edge of Death are Skill Modifier’s whereas the conversion on Mythical Gladiator’s Distinction is a source of conversion on equipment.
It can be important to note when conversions occur at different stages. One reason being that every step considers the pool of damage as a total out of 100%. For example, if I were to take the transmuter for Grenado that converts 80% of it’s Physical to Lightning, the remaining 20% after this step would be treated as 100% for things like Stormcaller’s Pact (15% Physical to Lightning) or Seal of Corruption (10% Physical to Lightning) etc.
As another example, since Mythical Edge of Sanity does 100% Physical->x conversion to Reaping Strike, this means that any conversion in subsequent steps that is Physical->x that would apply to Reaping Strike does nothing as there is no more Physical left to convert. If another Skill Modifier that did Physical->y to Reaping Strike was introduced, it would be worked out at the same time as the conversion on Mythical Edge of Sanity and compete with it by following a weighted system as conversions do normally.
That’s how I’ve come to understand it from others on the board anyway. You might be interested in looking over Halcyon’s topic on conversion as well. It goes over a few other things like armour piercing.