The only downside is that you will start over-leveled and will have a boring time until you reach Burrwitch where the experience starts getting decent again. Besides that, nah, there aren’t any downsides.
Although it trivializes Act 1 up to the Warden, it’s worth it in my opinion for the levels, equipment, and money, but most important is the easy to get 10 Devotion points meaning you can cap your Devotions by the end of Elite if you don’t do the expansion as well (mid-Elite if you do the Expansion).