I assume by "portal car campground" you mean the federal Whitney Portal Campground. It's about 1/2 mile from the Mt Whitney trail head. No need to walk, you can drive to and park at the Portal. (If you're not camping a second night after your hike, you'll have to break camp, pack up, and move your car anyway).
I've camped there twice - once starting my hike at 3:30AM and once starting at 7AM. Never have I noticed any noisy activity. There will be late arrivals and early departures but my experience is that the campers are aware and considerate.
According to the Rec.gov website the Lone Pine Campground is 7 miles from the trail head. I've never camped/been there. It appears to be some distance from the Whitney Portal Road.
Another option is the walk-in campsites right at the Portal. There you might notice more activity because there are many hikers coming and going who are not camping there. The sites there are very convenient - always available, very close to trail head, Portal Store, restroom, water faucet, garbage cans, and bear lockers. I have camped there a couple of times and experienced no problems. I think the fee is $12 per night (1 night only) and the sites are small only big enough for one tent (but there are plenty)
