Jammer, only the trip leader needs to be at the Visitor Center.
Your comments about camping outside the Whitney Zone: You must still have an overnight wilderness permit to camp even at Lone Pine Lake. Camping at Whitney Portal is the only place a wilderness permit is not required, since that is more a car-camping mode.
Best of luck getting everything worked out! I hope it goes smoothly.