Wildrose is a drive off the main road, but I would think getting up to 4k then would be tolerable, especially if you weren't spending the mid-day. Not sure how full it is in summer.
If you have a place to stay in Yosemite, and you (or the kids) have never been, that is really good.
HM: Yes, it is designated one night only. It is almost never full. Packing up for the day, then spending a second night would probably work. There are also the walk-in hiker sites at Whitney Portal, just below the trail head.
