An action figure is a small toy person or other entity, usually made of plastic, that can move body parts in order to be put in various poses. Action figures first became prominent in the United States in the late 1970s/early 1980s, when toy lines were released such as Star Wars, including nearly every on-screen character from the Star Wars trilogy, the 3 + 3/4 inch G.I. Joe figures, and the company Mego's many lines of toys based on comic books/superheros, television series, and movies. Also influential in the early 1980s were several lines imported from Japan, including Micronauts (Microman in Japan), Transformers (Mostly Japanese Microchange and Diaclone toys at first), and Shogun Warriors (composed of toys of several "Chogokin", or "super alloy" robot characters, though that term usually applies to the diecast metal toys and not the Jumbo Machinders).
If your friend likes or collects action figures, and you call them dolls, he'll probably get mad at you (I say "he" because men are generally touchier about "feminine" words like doll, and because most female action figure collectors probably get enough static over it from the stereotype-minded that they could shrug off someone calling them dolls, if it bothered them at all).
As a general rule, if you can dress it up, it's a doll, unless it has a gun. If it has stereotypically feminine imagery (by which I mean pink, other bright pastel colors, flowers and cutesy things, to the exclusion of all else), it's also a doll. If it has a gun or looks like it could kill something, it's an action figure.
Of course, the above is hopelessly sexist, as much in line as it is with most people's thinking. It also fits with what toy companies will term an action figure or a doll, as it fits with their typical "girls' toys are this, and boys' toys are this, and there's no variation or in-between" marketing.
More useful is a more thoughtful, less physical definition, like the one put forth by Michael Crawford, of mwctoys.com*. He defines an action figure as a toy designed to allow children to emulate adult actions - therefore, the likes of a G.I. Joe, a Transformer, and a Lego man are all action figures, as they are designed to allow children to reenact such actions as defending the free world from terrorists, destroying one's fellow space robots to conquer the galaxy, and building one's own car out of giant bricks. Regardless of the realism of these actions, they are unquestionably adult tasks, and so these are action figures. By Crawford's definition, Barbie is also an action figure, as she enables children to recreate a whole panapoly of adult actions. That's fine by me, I own some 12" (though "1/6 scale" sounds so much more mature :P) military and character figures, and if any particular Barbie would go well with them, I'd buy it.
Anyway, Crawford seeks to distinguish between role-playing themed toys like Barbie/conventionally-defined action figures/etc and things like baby dolls, which are intended as a simulation not of an adult but a child - a child the intended owner's age or younger, to be taken care of. Things like the American Girl dolls, as high-quality a product as they may be, are not action figures because they do not represent a possible future or fantasy of adulthood for the child, but a fantasy of the current time, an alternate childhood for the owner to imagine (American Girl dolls, continuing this example, are a line of characters which each represent a young girl living in a particular era in the United States' history.). This definition leaves behind entirely the idea of "dress-up" as a criterion for action-figure-hood. After all, even some WWF figures have removable/replacable clothes.
Now, the above applies just fine to action figures marketed as children's toys, but what of those targeted to adult collectors? It's just as easy to tell the action figures from the dolls, because generally they follow the same delineations as those for children. But collectors' dolls and figures are usually more expensive, higher quality in terms of detail, sculpting, paint operations and realistic use of materials like cloth, and more fragile. Still, one can tell the plastic film characters in action poses from the porcelain dolls in fancy outfits. Generally, action figures targeted at adults aim for more adult subject matter - licensed likenesses from television, movies, sports or popular music, and subsets of the same that would not be appropriate for children's toys, such as horror movies and the recently infamous line of Playboy Playmate action figures. Also, adult-collector figure lines often have an original concept behind them, which, though their subject matter and aesthetics may be interesting, would not sell well to children because of the lack of playability, or lack of a cartoon tie-in. In contrast, collectors' dolls largely stick to the look and subject matter of children's dolls, pushing their artistry and craftsmanship as the main selling point, rather than their likeness or subject.
So we could say it is even easier to distinguish between action figures and dolls at the collectors' level than at the childrens' level. The question is, at what point does an adult-collector-targeted action figure become a plastic statue? There is a growing trend among companies like McFarlane Toys, N2 Toys, Mezco, and Art Asylum towards increasing detail and realism at the cost of articulation - Stan Winston Creations' "Realm of the Claw" figures, despite a better-than-average amount of joints, reportedly cannot stand up without their display bases, and seem to have been designed that way. Also, resin statues and mini-busts have become more popular among action figure collectors, so much that they've gotten coverage in toy collecting publications such as ToyFare.
Some companies are pushing back against the tide of statuesque action figures. McFarlane has recently put a surprising amount of articulation in several of its figure lines, and increased their durability, especially in the joints. Mezco seeks a compromise between appearence and posability that's better for both sides than McFarlane's old standard. The articulation and durability of child-targeted action figures that are also popular among collectors, such as many of Hasbro's, Bandai's, and Toy Biz's offerings, also does not disappoint. But though these products and practices are a good thing for action figure collectors who see a clear difference between their beloved figures and statues, they do not help to make the line between any clearer, as articulation becomes less of a concern for many companies and some collectors demand great sculpting and paint jobs at any cost.
Action figures versus statues, articulation versus detail - these arguments cause much controversy among collectors today, and it doesn't look like it'll get better anytime soon. Come on, guys, they're just toys - don't you care about playing with them? How can one collect children's toys and yet seem to have forgotten what it's like to be a kid?
*In the article "Action Figure Or Doll", December 19, 2001.
http://www.mwctoys.com/FEATURE_121901.htm
Thanks, Mike!