I’ve always thought this was something of a plot hole, and justification for more bridge repair / road clearing in Act II.
Either the rifts are usable without the Taken present, or the food shipment wagoners are pros at making it through some of the most perilous underground passages and dungeons.