Not really a genre, or even a sub-genre of hip-hop, or rap; so much as a term used to designate the dense, heavily artistic, neurotic, verbose, and above all intelligent works of artists like MC Paul Barman, Buck65, Del tha Funky Homosapian or, hell, even Kool Keith to some extent.
The subject matter of nerd rap tends towards what's usually on the minds of nerds themselves: sex (from the point of view of someone who is NOT themselves getting any), avant garde cinema, baseball, etc.