Only part about descending to the Portal from the summit: that requires a SECOND permit, the Trail Crest Exit permit. There are only 25 available for each day, and most of them will be taken when you show up to get your Horseshoe Meadows permit. Your plan B would need to be to return to your car the same way you hiked in.