The only reason to leave the summit that early would be foul weather coming in. You're going to hit softer snow in the afternoon no matter what -- just plan on it.

But it is down-hill, so shouldn't be too bad.