display | more...

(This writeup inspired by the Prime Spiral node.)

It is human nature to try and discern patterns in everything we see. Pattern recognition is widely regarded in the field of artificial intelligence as being one of the very fundamentals of what makes intelligence, intelligence. Sometimes this works against us, and gives us the perception of a pattern where there really is none. This kind of occurrence is called statistical clustering. A very good test of whether a significant pattern is really present is to examine the real extent of the field in which a coincidence occurs, and to examine which knowledge is a priori and which is a posteriori.

If a conjecture is made a priori, and is borne out to be supported by the data, it is likely a true pattern with a true cause. If an observation is made a posteriori, the best test is to conjecture "things will be more normal from now on" (without dwelling on patterns of the past), and seeing how well that conjecture holds up in the future. If the conjecture passes, you have a case of statistical clustering - if the conjecture fails, there is a valid statistical link.

The field of interest in which a pattern is found is often difficult to recognise at first. For example, there may be an article in the newspaper that someone got dealt the perfect Bridge hand today - all 13 spades cards. The chances of that happening are tiny. Or that two drivers in a race got exactly the same time on the qualifying lap, to within a thousandth of a second. This seems newsworthy too (at least for the Sports section). But think about what else there might be, that would be just as unusual. This is somewhat difficult, because we think about what we expect ( - evaluating relevance is another core issue of intelligence). A perfect 147-point break in snooker. The finalists in a karate championship who have the same initials and the same birthday. Two ballet dancers who both trip in their hotel rooms before a big show. Two horses, whose names are anagrams of each other, in the same horse race. Very outlandish, yes - but there are so many outlandish things that it's really no wonder one or two crop up occasionally. And numerology, don't get me started, there are so many combinations of mathematical operators that some sequence of tricks will link several unrelated items together from a large pool of information.

See The Psychology of Randomness. We are taught that random is the opposite of order, the imposed absence of order, where a better conceptual model might be to think of random as being the absence of imposed order. 'Order', in the psychology of randomness node above, is perceived as being a long run of heads or tails. Imposed order would cause 5 heads in a row, but given enough tosses of the coin without any impositions (of perceived order or of perceived randomness), 5 heads in a row will occur naturally.

- However, just as statistical arguments can be abused, the statistical clustering argument can be abused too. Caution must be exercised in believing either claim. The only way to see through either type of abuse is to use the scientific method impartially (which is impossible for most of us who don't have access to the amounts of data required to do so, especially for events which are big enough to be newsworthy, like elections, so the next best thing we can do is put trust in someone else to use the scientific method and then report honestly to us).

Log in or register to write something here or to contact authors.