My educated lie* about the origin of the term comes from the Navy SEALs who fought in Vietnam (and possibly their jungle-warfare predecessors). When you're running around a hot, sweaty jungle environment for days on end, the last thing you want is to have any more clothing on than is necessary--especially an article that is likely to get funky, and stay that way, for the duration of your trip.
In fact, now that I think on it, my Dad mentioned that one of the rituals that his air wing would subject the newbies to was to check to see if they were wearing skivvies on arrival. If they were, they would grab the waistband and forcibly rip them off the soon-to-be-clutching-his-groin-in-pain-newbie's body.
This is also recommended during the hot, sticky summer months in NYC. Although be warned: a colleague of mine recently confided that not only had he (and others in the department) noticed my, um size, but "also my religion".
* similar to an educated guess