Playing 'the dozens' refers to the act of trading witty insults. Theoretically, any witty insult will do, but as this is a bit of African-American slang, it most often refers specifically to insults popular in African-American communities. 'The dozens', as a slang term, has been around since at least the early 1900s, but it gained widespread use across the nation and across racial and cultural boundaries in the 1960s and 70s.
Say man, what's that, boy?
I want to tell you 'bout your girlfriend.
What about my girl?
Well, you don't look strong enough to take the message.
--Say Man by Bo Diddley (1959)
The dozens are an important part of African-American cultural history, and were one of the precursors to the battle rapping that strongly influenced rap and hip-hop music. Two young men would trade insults at a rapid but carefully measured pace, usually formulated into a simple rhyming scheme. The game was all about insulting your opponent to the point that he would concede defeat (or, at least, run out of clever insults), without insulting him so badly that he would decide to break your face.
"Alan Dundes found that the social and artistic are infused in the Afrodiasporic practice of the dozens, which he notes functions both as an assertion of masculinity and as a rite of passage for the secular mastery of words. The dozens not only establishes a framework for verbal creativity; children also use them to determine a social hierarchy. A good dozens player not only cooly withstands merciless insults to his family; he also twists memorized insults quickly to suit the opponent at hand."
-- Ali Colleen Neff, Let the World Listen Right (2009).
The first written use of the phrase 'the dozens' is from 1915, but it was surely used before that. There are records of insult battles of some sort between the plantation slaves working in the field and those working in the masters house, although how serious these might have been and what they were called is unknown. While we will never know how old the term is, we do know that the custom of trading clever insults goes back to the time of Methuselah; just ask your mother, she'll remember.
Painter: "Y'are a dog."
Apemantus: "Thy mother's of my generation. What's she, if I be a dog?"
-- Timon of Athens, I.i by William Shakespeare.
Speaking of your mamma... Yo' Mama jokes are a type of the dozens, but they are by no means the most traditional. In fact, the common form, "Yo mama so fat, her school picture was an aerial photograph", is a comparatively new invention, emerging in the 1970s and quickly taking the country by storm. It has the benefit of being quick and snappy, and wittier than the more traditional rhyming couplet. Yo mama jokes also tend to be much much cleaner than most of the traditional rhyming dozens -- and therefor spread faster and through a wider audience.
I ****ed your mama
till she went blind.
Her breath smells bad,
But she sure can grind.
--Die Nigger Die by H. Rap Brown (1969)
As I mentioned, the origins of the term is clouded in mystery; however, there are a number of interesting theories. One of the more popular explanations is that back in the olden days weak, deformed, or elderly slaves were sold in job lots of 12, and hence feeble negros became known as 'dozens'. Or, it might be a shortened form of 'bulldoze'; originally bulldose meant a whipping, literally a 'dose fit for a bull'. The word was coined during the 1876 US Presidential Election, when black voters were intimidated into voting Democrat through means of severe beatings.
I saw yo' mama yesterday on the welfare line
lookin' like she done drank some turpentine.
--Anon.
There are other theories as to the origins; Dan Burley, author of The Dirty Dozen, believes that the term comes from an anonymous blues song called 'The Dirty Dozen', which was so crude that it was never written down. It enjoyed a vigorous word-of-mouth renascence in the days of barrel-houses and rent parties, where new verses might be invented at a whim. Author Roger D. Abrahams speculated that the 'dozen' in question refers to either an unlucky roll of twelve in craps; a corruption of "doesn't" (as in "at least my mother doesn't"); or from the 1700s definition of 'dozen' meaning 'to stun' or 'stupefy' (From Deep Down in the South).
"You wanna play the dozens?
Well the dozens is a game
But the way I **** your mothers
Is a goddamned shame.
--George Carlin, Occupation: Foole (1973)
Doin' the dozens may be confounded with wolf tickets (or woofin') and signifying. All of these terms are rather vague in scope, and in any case more than one of these may be a 'correct' term; moreover, many of the people using these terms don't care much about precise terminology. If you are dealing with direct and open insults given with both humor and the intent to stand firm in the face of a cutting retort, then it is probably safe to call it a game of the dozens.