Why randomness doesn’t always look random


I got the lightning puzzle that I posed yesterday from the latest book by Steven Pinker The Better Angels of Our Nature: Why Violence Has Declined that I reviewed in December. He uses it to illustrate that our intuitive notions of randomness and probability can easily lead us astray.

Here is the problem again.

Suppose you live in a place that has a constant chance of being struck by lightning at any time throughout the year. Suppose that the strikes are random: every day the chance of a strike is the same, and the rate works out to one strike a month. Your house is hit by lightning today, Monday. What is the most likely day for the next bolt to strike your house?

Here is his answer followed by a discussion from pages 202-204 of the book.

The answer is “tomorrow,” Tuesday. That probability, to be sure, is not very high; let’s approximate it at 0.03 (about once a month). Now think about the chance that the next strike will be the day after tomorrow, Wednesday. For that to happen, two things have to take place. First lightning has to strike on Wednesday, a probability of 0.03. Second, lightning can’t have struck on Tuesday, or else Tuesday would have been the day of the next strike, not Wednesday. To calculate that probability, you have to multiply the chance that lightning will not strike on Tuesday (0.97, or 1 minus 0.03) by the chance that lightning will strike on Wednesday (0.03), which is 0.0291, a bit lower than Tuesday’s chances. What about Thursday? For that to be the day, lightning can’t have struck on Tuesday (0.97) or on Wednesday either (0.97 again) but it must strike on Thursday, so the chances are 0.97×0.97×0.03, which is 0.0282. What about Friday? It’s 0.97×0.97×0.97×0.03, or 0.274. With each day, the odds go down (0.0300 … 0.0291 … 0.0282 … 0.0274), because for a given day to be the next day that lightning strikes, all the previous days have to have been strike-free, and the more of these days there are, the lower the chances are that the streak will continue. To be exact, the probability goes down exponentially, accelerating at an accelerating rate. The chance that the next strike will be thirty days from today is 0.9729x0.03, barely more than 1 percent.

Almost no one gets this right. I gave the question to a hundred Internet users, with the word next italicized so they couldn’t miss it. Sixty-seven picked the option “every day has the same chance.” But that answer, though intuitively compelling, is wrong. If every day were equally likely to be the next one, then a day a thousand years from now would be just as likely as a day a month from now. That would mean that the house would be just as likely to go a thousand years without a strike as to suffer one next month. Of the remaining respondents, nineteen thought that the most likely day was a month from today. Only five of the hundred correctly guessed “tomorrow.”

Lightning strikes are an example of what statisticians call a Poisson process (pronounced pwah-sonh), named after the 19th-century mathematician and physicist Simeon-Denis Poisson. In a Poisson process, events occur continuously, randomly, and independently of one another. Every instant the lord of the sky, Jupiter, rolls the dice, and if they land snake eyes he hurls a thunderbolt. The next instant he rolls them again, with no memory of what happened the moment before. For reasons we have just seen, in a Poisson process the intervals between events are distributed exponentially: there are lots of short intervals and fewer and fewer of them as they get longer and longer. That implies that events that occur at random will seem to come in clusters, because it would take a nonrandom process to space them out.

The human mind has great difficulty appreciating this law of probability. When I was a graduate student, I worked in an auditory perception lab. In one experiment listeners had to press a key as quickly as possible every time they heard a beep. The beeps were timed at random, that is, according to a Poisson process. The listeners, graduate students themselves, knew this, but as soon as the experiment began they would run out of the booth and say, “Your random event generator is broken. The beeps are coming in bursts. They sound like this: “beepbeepbeepbeepbeep .. . beep … beepbeep .. . beepitybeepitybeepbeepbeep.” They didn’t appreciate that that’s what randomness sounds like.

This cognitive illusion was first noted in 1968 by the mathematician William Feller in his classic textbook on probability: “To the untrained eye, randomness appears as regularity or tendency to cluster.” Here are a few examples of the cluster illusion.

The London Blitz. Feller recounts that during the Blitz in World War II, Londoners noticed that a few sections of the city were hit by German V-2 rockets many times, while others were not hit at all. They were convinced that the rockets were targeting particular kinds of neighborhoods. But when statisticians divided a map of London into small squares and counted the bomb strikes, they found that the strikes followed the distribution of a Poisson process-the bombs, in other words, were falling at random. The episode is depicted in Thomas Pynchon’s 1973 novel Gravity’s Rainbow, in which statistician Roger Mexico has correctly predicted the distribution of bomb strikes, though not their exact locations. Mexico has to deny that he is a psychic and fend off desperate demands for advice on where to hide.

The gambler’s fallacy. Many high rollers lose their fortunes because of the gambler’s fallacy: the belief that after a run of similar outcomes in a game of chance (red numbers in a roulette wheel, sevens in a game of dice), the next spin or toss is bound to go the other way. Tversky and Kahneman showed that people think that genuine sequences of coin flips (like TTHHTHTTTT) are fixed, because they have more long runs of heads or of tails than their intuitions allow, and they think that sequences that were jiggered to avoid long runs (like HTHTTHTHHT) are fair.

The birthday paradox. Most people are surprised to learn that if there are at least 23 people in a room, the chances that two of them will share a birthday are better than even. With 57 people, the probability rises to 99 percent. In this case the illusory clusters are in the calendar. There are only so many birthdays to go around (366), so a few of the birthdays scattered throughout the year are bound to fall onto the same day, unless there was some mysterious force trying to separate them.

Constellations. My favorite example was discovered by the biologist Stephen Jay Gould when he toured the famous glowworm caves in Waitomo, New Zealand. The worms’ pinpricks of light on the dark ceiling made the grotto look like a planetarium, but with one difference: there were no constellations. Gould deduced the reason. Glowworms are gluttonous and will eat anything that comes within snatching distance, so each worm gives the others a wide berth when it stakes out a patch of ceiling. As a result, they are more evenly spaced than stars, which from our vantage point are randomly spattered across the sky. Yet it is the stars that seem to fall into shapes, including the ram, bull, twins, and so on, that for millennia have served as portents to pattern-hungry brains. Gould’s colleague, the physicist Ed Purcell, confirmed Gould’s intuition by programming a computer to generate two arrays of random dots. The virtual stars were plonked on the page with no constraints. The virtual worms were given a random tiny patch around them in which no other worm could intrude. They are shown in figure 5-5; you can probably guess which is which. The one on the left, with the clumps, strands, voids, and filaments (and perhaps, depending on your obsessions, animals, nudes, or Virgin Marys) is the array that was plotted at random, like stars. The one on the right, which seems to be haphazard, is the array whose positions were nudged apart, like glowworms.

I love this kind of stuff.

Comments

  1. Peter says

    Oh man -- that bit about the glow-worms is fascinating! I can’t believe I hadn’t heard of it before, as I thought I had read all of SJG’s stuff minus the text books…

    I also never thought about the idea that there would have to be a force causing things to migrate out of a clustered spacing. Really really interesting.

  2. sometimeszero says

    I’m still trying to digest his explanation with the lighting problem, as I still don’t understand how the lightning strikes example differs from an example trying to illustrate the gambler’s fallacy.

    Pinker’s latest book sounds quite good. I was going to pass on it since it’s long and violence isn’t really my area of interest, but I think I need to reconsider that and pick it up.

  3. Hans says

    Almost no one gets this right

    Statistically speaking, that is a meaningless remark. 😉

    Only five of the hundred correctly guessed “tomorrow.”

    That’s also a strange sentence. Did they guess or did they calculate and know? As you can see from my answer in the other post, I calculated and knew.

    The problem is that statistics is very often counter-intuitive which is why guessing usually gives you the wrong answer. Take the three-door problem for example. (After choosing one door is shown to have nothing behind it. Do you change your choice?) Without some introduction to statistics you will most likely not even be able to understand an explanation of why you should change your choice.

    The gambler’s fallacy is also such a counter-intuitive example. Just look at the mathematical background someone has to have before you can begin to explain that in a series of n coin tosses you can expect to see a streak of 2log n heads or tails.

  4. Jared A says

    It isn’t that different than the gambler fallacy. Consider the lightning problem written a different way:

    There have been three ‘heads’ tosses in a row. What is the most likely toss to be the next heads toss?

    As you know, the previous history is irrelevant. The next toss has a 50/50 chance of being heads. The following also has a 50/50 chance of being heads. But what is the probability that the 2nd toss is heads and the first is tails? 1/4

    Test it yourself.

  5. M.Nieuweboer says

    When I rely on intuition I always am wrong on statistics; that included the lightning problem. Still I am quite good at statistics, but only after the solutions have been explained to me.
    Another example is the famous Monty Hall problem.

  6. Donald Eckhardt says

    I once installed a wainscoted wall of pseudorandomly distributed pine boards. It couldn’t be truly random because I had equal numbers of three, five, and seven inch wide pine boards. I had to use them all up, yet give a random looking appearance to it -- including runs of repeated widths. It was a formidable challenge.

  7. LeftyGomez says

    Ollie: yes it is the geometric distribution if there can only be one strike each day, hence making the time discrete, i.e. each day is basically a coin flip with a coin weighted approximately 29 to 1.

    If it were really a Poisson process with continuous time, then the most likely day could be “the same day” if the given strike was, say, at 12:00:04 am.

    But the main point applies in either case, which is that the intervals between events have an exponentially decaying distribution and so when the first strike hits, a sooner (whole) day is more likely than a later day to be the day of the next strike.

  8. BillyJoe says

    Pinker:

    “I gave the question to a hundred Internet users, with the word next italicized so they couldn’t miss it. Sixty-seven picked the option “every day has the same chance.” But that answer, though intuitively compelling, is wrong. If every day were equally likely to be the next one, then a day a thousand years from now would be just as likely as a day a month from now.”

    BillyJoe:

    Except for that word “next”.
    By your reasoning, the NEXT strike is just as likely to occur in a million years time as the following day.
    That is clearly false.

    And I swear I haven’t read Pinker’s book!

  9. sailor1031 says

    Yeah -- well all that probability stuff doesn’t detract from the fact that you can still get your smilin’ ass fried by a second strike on Monday -- it’s just not very likely. So knowing the calculated probability may not be all that useful when crap happens!

  10. Uri says

    Doesn’t the answer depend on how far we are into Monday? If we’re less than 0.03 (or whatever day/month is) of the way in, it would seem that today, Monday, is the correct answer.

Leave a Reply

Your email address will not be published. Required fields are marked *