As per the Inyo National Forrest website, "Everyone in your group must start on entry date and trail listed on your permit".

https://www.fs.usda.gov/main/inyo/passes-permits/recreation