As the name suggests, anecdotal evidence is evidence for a claim which is based on remembered anecdotes either first hand from the speaker or second hand from others. Such evidence is often used to justify empirically verifiable claims. This is fine and even appropriate in the case that the claim is of little importance or in the case that it is reasonable to believe that few observations with little precision are needed as basis for a claim. Often, however, anecdotal evidence is used as backing for sweeping claims about a wide class of things, like natural phenomena or the behavior of people, which are the sort of claims usually examined by science (natural science, life science, or social science). Below I will investigate the reasons why anecdotal evidence is unreliable and unacceptable evidence of scientific claims.

Sample size

The basic point is that you need enough evidence to make a credible claim, so you must justify how much data is "enough". In science this is done through statistical techniques and error analysis. You figure out mathematically if you take so much data, what is your probability of getting the wrong answer. Then you take enough data that that chance is very small, normally 1-5% or less, depending on the situation. This problem of sample size is particularly troublesome when you're making claims about people or society. It is one of the reasons why when people make claims about "every accountant" or "every Asian" they ever met, their conclusions are invalid; every person you have ever met is still only a miniscule portion of all people, so your data would have to be extremely precise and unbiased in order overcome the uncertainty from your small sample.

Reliability of Records

Anecdotal evidence is usually based on events people remember, but that makes it prey to many different problems. The main problem is that memory is not 100% accurate nor is it really static. You don't remember an event the way it was; you remember what you thought was important at the time; furthermore, this memory is continually reshaped by your later experiences and attitudes, so that the way different people remember the same event, or the way the same person remembers the same event at two different times, can be radically different. It is a commonly known and accepted fact in psychology that people will reshape their memories to fit their later circumstances or convictions. Memory is also selective. Again, you remember what you believe is, or believed at the time was, important, and if you believe a certain thing you are often more likely to remember instances in which that belief was validated rather than ones in which it was not. For example, it will seem to many people that there is always a long line when they go to the supermarket. It is the fact that you're unlikely to remember the time when you only waited in line for a few minutes, but you'll definitely remember when you waited for 45 minutes. And once you're already convinced the line is always long for you, then you are likely to get even more agitated when you have to wait, making those times even more likely to stick out in your mind. So, remembered information, without objective recording at the time data was taken, is simply unreliable even when people are being honest, leaving aside completely the possibility that they may intentionally bend the truth.

Measurement Precision

It is very important that one know the reliability of data and that that reliability be high enough to justify conclusions. Measurements should be objective and the uncertainty in them should be examined. This usually means that data should be quantitative and for any measurement the uncertainty in that measurement should be estimated based on statistics and examination of the theory being tested. Even if data is more qualitative, as in social science for example, still it can be generally categorized and analyzed statistically to estimate uncertainty. This is one of the most important parts of experimental science. In essence, you take data such that everyone who saw the particular event could agree on the data, and you know the answer won't be exactly right, so you want to know about how far off it is. If you know the uncertainty you have a measurement, otherwise all you've got is a wrong answer. Anecdotal evidence often employs subjective claims. One particular example is when someone says he took a medicine and "felt better". But you don't really know what that means. Essentially you might ask, "how much better?" because what one person considers "better" another may not. This is why you must use objective evidence, like quantitative measurements, so that you can compare on a case to case basis and everyone can agree.

Control

This is probably the main area in which anecdotal evidence is usually lacking. In general you are making the claim that X causes Y based on the fact that every time that X happens Y happens. However, you must make sure there's not a third thing Z that is the cause and just happens to be around at the same time as X. This is where control comes in. You want to have an experiment where you only change the thing you're interested in, and everything else is the same. This is usually done through calibration and often the use of a control group. In anecdotal evidence, however, one is usually drawing from experiences in which the circumstances are very different from one case to the next and many pieces of information that may be relevant are unknown. So, you may, for example conclude you get colds all the time because you're always congested, but a controlled experiment may show you actually have allergies, because you only become congested when certain allergens are present. Properly controlling an experiment is particularly hard in medicine and the social sciences (since you can't just do whatever you want with living people), but it can be done, often through statistically compensating for other factors.

Review

Finally, an integral part of any honest research is the ability for others to review what has been done. This partially rests on earlier points, because in order for others to review your work, you must have reliable records of objective data. This is at the core of true investigation precisely because we must be very careful about the way we collect and interpret data, so others (who do not necessarily share our views) must be able to look over all the facts and see if our methods and conclusions are reasonable. This is a tradition within science, carried out through peer reviewed journals and conferences. Also, the ability to review means the ability to compare the data of one person with that of another person to see if results are repeatable, and it means the ability for others to use the data later as part of a larger set of data to draw more reliable conclusions. Anecdotal evidence does not allow for this sort of review, because there is no record other than recollection, so you don't have a clear idea of the facts, just one interpretation, and there are no objective measurements against which to compare. In short, if someone tells you their conclusion but can't lay down the evidence for open inspection, then you can't be sure that they're not in error or lying.

The basic point being made here is not particularly abstract. It's basically that if someone is trying to convince you of something, they should show you stuff that you can verify yourself, and they should be able to show you their entire process so that you (or someone you trust knows what they're doing) can be sure they've done things correctly. If someone cannot or will not do this, they may well be lying, and even if they aren't, there's a good chance they've made mistakes.

So now the question arises, when is anecdotal evidence acceptable? Basically, you have to take into account its limitations. If it's a situation where you don't need a big sample size, the result is unambiguous, and basically you're not trying to draw a very far reaching conclusion, it's ok. If you're trying to describe the laws of nature, the human body, people, or society, then it's totally inadequate for the task and you need to use the scientific method.

Anecdotal evidence is often used to justify pseudoscience in a number of areas. These include

Some of the worst offenders I have heard are politicians, who use ridiculous anecdotal evidence to "prove" particular ideas about nations or about their policies, and proponents of alternative medicine. On the later point, I am not saying that there is nothing to alternative medicine, only that there are those within that community who wish to abandon the scientific method and that their conclusions are worthless. That does not invalidate the work of others who apply scientific practices in the same field. Also, it's not surprising that this should happen, after all, drug companies have been using such pseudoscientific arguments in their commercials for years with things like testimonials. Another thing that tends to creep up from this sort of sloppy reasoning is racism, which is also often based on pseudoscientific arguments both about the nature of race and the relationship between race and behavior.

It is very natural to use anecdotal evidence. Noticing things and inferring a pattern is a basic human behavior and the basis of the science; however, we must understand that while anecdotal evidence works well enough for everyday things, it is inadequate to tackle the big questions, which is why scientific methods were invented. So, when you hear people making sweeping claims about anything, nature, people, whatever, analyze their evidence and see if it really holds up to the demands on it. If it's just a collection of anecdotes, then explain to them why it doesn't prove anything.


I had some trouble organizing my ideas for this write-up into sections or headings, so if you have a good idea for a more clear organizational scheme /msg me. Also, any other suggestions are always appreciated.