One of the difficulties involved in discussing anarchy is whether or not everyone is actually thinking about the same thing. Social, political, and economic interests are all involved, yet anarchists and authoritarians alike rarely have all these things in mind.
Some people are primarily concerned with the oppression by government, seeing mistreatment of citizens and bureaucratic inefficiency as primary sources of discontent. These anarchists tend to believe that people are inherently able to get along, much in the same way Karl Marx believed that the working class, regardless of nationality, could unite and form a Communist utopia. For a while, this seemed to work quite well, just as an anarchic revolution might at first seem to succeed. In any revolution, the people involved are willing to surrender certain luxuries to ensure their success, as efficient consumption of resources is a necessary part of winning a war.
Unfortunately, once the revolution is complete and the old power system overthrown, problems quickly arise. In the middle of a heated conflict, people are willing to believe that they must be content with what little they have, since they acknowledge that there is little to go around. Once peace is restored, there is more to go around, and people are not so so willing to give up their own luxuries. Everyone believes they are entitled to a share of the spoils (be it food, wealth, or land), and conflict immediately ensues. If left to their own devices, the conflicting parties will simply fight it out until one side is either subdued or killed. Human nature, however, dislikes conflict within its own group, be it an anarchic commune or otherwise. Almost immediately, a new power system takes place to quell these conflicts, and people learn to respect this power system, whether or not it is actually able to control them physically, because it helps everyone to survive.
As the group grows larger, which it inevitably will in prosperous times, the power system must also grow to accommodate it. It evolves a hierarchy, and becomes more and more like a government. Because it keeps people from killing each other, future generations will tend to respect it, and its influence may exceed its actual power. As it grows larger and more impersonal, the government develops inefficiencies and its subject may suffer because of it, but it is rarely overthrown because people generally prefer security to risking everything for a better quality of life. There is a saying, that every government is only two meals away from a revolution, and this is largely true. Revolutions tend only to occur when people truly believe their ability to survive is being threatened.
This tendency for people to obey the existing power is seen by some people as a weakness, like a shepherd-flock relationship. In our modern world, most of our basic needs are cared for, so we look to principles as our main concern. Those who value free will on principle may believe anarchy is the best way to preserve it, thinking that the government's only purpose is to oppress the governed for the benefit of the elite. The flaw in this reasoning is a neglect for the human element: people would not organize themselves into governed societies if there were no inherent value to them.
Natural selection speaks for itself in this matter: people survive and reproduce in greater numbers when they form organized societies. The survival of the society as a coherent group becomes synonymous with individual survival as long as the society provides for the individuals. Therefore, those who are willing to limit their personal freedom survive, and those who aggressively seek their own means of survival outside of the norm set by the society/government are condemned as criminals.
This is where anarchists and authoritarians both seem to have a blind spot. Both sides argue on the basis of principle, with little regard for the reality of the situation. Both sides tend to believe that they understand human nature completely, and that everyone will go along with their plan if it is given a chance. I don't pretend that I have exactly defined human nature in the above paragraphs, but it should be clear that neither free-for-all anarchy nor iron-fisted government can be perfect, and that a truly stable society must be based on realism over idealism.