Recent findings in the nascent field of behavioral economics suggest that the Coase theorem does not accurately model how people act in certain conditions.
Underlying the Coase theorem is the assumption of perfect rationality from rational choice theory, the way standard economics expects individuals to make decisions when faced with uncertainty. Simply, this means people will get the most they can out of any situation. In practice, humans make decisions under conditions of bounded rationality, which means they don't always get the most they can. The Coase theorem, which depends on people getting all they can from an economic situation for its prediction of efficiency in any resource dispute between two parties with clearly defined property rights, falls right apart.
The Coase theorem is derived from two basic premises of standard economics. First, to an economist (and the individuals in their models), an out-of-pocket cost is no different from opportunity cost, and sunk costs are not considered in decision making. Second, resources tend to gravitate toward their most efficent use because market actors seize all profit opportunities. Together, these propositions give us the Coase theorem: given no wealth effects or transaction costs, the distribution of entitlements will not affect the eventual efficient distribution of resources between the parties.
However, if opportunity costs are undervalued when the actors make decisions, the Coase theorem's prediction that initial entitlements are irrelevant falls apart. Consider a farmer whose property abuts a rancher's land. The rancher's cow tramples the farmer's corn at a cost to the farmer of $200 dollars per year. The cost to the rancher to prevent the trampling is $300 per year. Coase's theorem would lead us to predict that, in the event of a lawsuit which clarified their property rights (which for our purposes costs nothing), regardless of who won, the cattle would still be allowed to trample the land. If the farmer won, the rancher would bribe him with $200.000000001 a year for the right to trample. The farmer, exhibiting perfect rationality, would realize that it was $.000000001 cheaper to accept the bribe than maintain untrampled crops and would act accordingly. Conversely, the farmer would not bribe if the rancher won, because he would have to offer $300 to avoid a $200 dollar cost. Therefore, the argument goes, both parties' pursuit of their own self-interest serendipitously corresponds with the most efficient use of resources from a societal point of view (minimizing the total costs for the same resource production).
However, if the farmer does not consider out-of-pocket and opportunity costs as equivalent, he might not accept the $200 bribe. In fact, it might take a $400 dollar bribe to get the farmer to allow trampling. This is too much for the rancher to pay, as it only costs him $300 to prevent trampling. The end result is the rancher pays $300 for a $200 cost, yielding a net inefficiency of $100. What is going on here? The farmer is behaving "irrationally", and underestimating the opportunity cost of failing to take a bribe. His actual cost for trampling is $200, but he requires $400 to allow the trampling at an opportunity cost of $200. Nobody could behave so irrationally, you might scream. Research shows they do.
Behavioral economists have found, contrary to Coase predictions, in markets where the price of goods is not exogenously defined (as prices are in stock markets, e.g.) entitlements matter. People systematically overvalue rights and goods they possess, a phenomenon known as "the endowment effect". This makes the initial distribution of resources important because they tend to be "sticky", i.e. they affect future dealings. In contrast, the standard model of economics assumes bygones are bygones when people make choices. The end result is often an inefficient distribution of resources -- the cows in the above example are fenced.
One of many experiments that demonstrate the "endowment effect" was performed at Cornell university with 44 students in a law and economics class. 22 of them at random were given coffee mugs. The other half of the students were given $6. The researchers then held a market for the mugs, eliciting from the mug-holders their lowest selling price, and from the mug-buyers their highest buying price and then clearing the market.
It is clear what the Coase theorem predicts in this situation. There are no transaction costs and no wealth effects, and the property rights are clearly defined. Therefore, the mugs should be redistributed efficiently, which means people who value them most end up with them (if a mug-holder values the mug more than average, he will not find a buyer; if a buyer values the mug more than average, he will find a seller.) Because the mugs were randomly distributed, we would expect that half would sell their mugs and half would not (as random distribution would lead about half of the people who value mugs more than average to get them).
However, people who were given mugs offered to sell them at prices which were, on average, twice as high as the average buying price from non-mug-holders. Instead of 50% of the mugs changing hands, only about 15% did. People overvaluing what they have (or undervaluing opportunity costs) has also been shown in many real-world cases. For example, in a survey of lawyers on nuisance cases, researchers found that not one of twenty cases involved any post-judgment bargaining, even when transaction costs were low and there were clearly existing, mutually-advantageous deals to be made. Tellingly, the lawyers also said that, even if the judgment had been reversed, the parties would still not negotiate. It appears that acrimony and/or the overvaluation of what one has compared to what one could get in a sale is preventing these parties from negotiating for the most efficient outcome. As the Coase theorem and standard economics is very influential to policymakers, especially concerning nuisance law, this finding is important because it suggests damages, even if relatively imprecise, might be a better approximation of efficiency between the parties than the injunction a Coasian would suggest (allowing parties to bargain between themselves). Because people are people and not prefectly rational utility-maximizers (gasp!), laissez-faire does not always lead to efficiency, even with clear property rights and without transaction costs.
Sources
Jolls et. al., A Behavioral Approach to Law and Economics, 50 Stan. L. Rev. 1471 (1998).
Kahneman et. al., Experimental Tests of the Endowment Effect and the Coase Theorem, 98 J. Pol. Econ. 1325 (1990).