I first went in 1994 for my 18th and it was awesome. I have been 3 times since in 1997, 1998 and 2009 and progressively it has become a far more professional experience (although it has lost some charm and quirks along the way). There is no comparison to a Grand Prix (save Monaco and Monza imo) as Le Mans is much better.
The drive, as LFA said, is one of the most enjoyable parts of the experience. This is what I followed on the way there in 2009. I was busting my balls just to keep it in sight!
The race is only part of the experience as the food, the atmosphere (150,000+ Brits), the ability to get close to the drivers and cars, Mad Friday, Night time qualifying, the sheer amount of supercars in the car park, the museum...
More recently I have always rented a small mobile chalet 4km from the circuit as the toilets/showers are not wonderful.