A Computer Scientist Reads EvoPsych, Part 3

[Part 2]

As a result of selection acting on information-behavior relationships, the human brain is predicted to be densely packed with programs that cause intricate relationships between information and behavior, including functionally specialized learning systems, domain-specialized rules of inference, default preferences that are adjusted by experience, complex decision rules, concepts that organize our experiences and databases of knowledge, and vast databases of acquired information stored in specialized memory systems—remembered episodes from our lives, encyclopedias of plant life and animal behavior, banks of information about other people’s proclivities and preferences, and so on. All of these programs and the databases they create can be called on in different combinations to elicit a dazzling variety of behavioral responses.[1]

“Program?” “Database?” What exactly do those mean? That might seem like a strange question to hear from a computer scientist, but my training makes me acutely aware of how flexible those terms can be. [Read more…]

What is False?

John Oliver weighed in on the replication crisis, and I think he did a great job. I’d have liked a bit more on university press departments, who can write misleading press releases that journalists jump on, but he did have to simplify things for a lay audience.

It got me thinking about what “false” means, though. “True” is usually defined as “in line with reality,” so “false” should mean “not in line with reality,” the precise compliment.

But don’t think about it in terms of a single thing, but in multiple data points applied to a specific theory. Suppose we analyze that data, and find that all but a few datapoints are predicted by the hypothesis we’re testing. Does this mean the hypothesis is false, since it isn’t in line with reality in all cases, or true, because it’s more in line with reality than not? Falsification argues that it is false, and exploits that to come up with this epistemology:

  1. Gather data.
  2. Is that data predicted by the hypothesis? If so, repeat step 1.
  3. If not, replace this hypothesis with another that predicts all the data we’ve seen so far, and repeat step 1.

That’s what I had in mind when I said that frequentism works on streams of hypotheses, hopping from one “best” hypothesis to the next. The addition of time changes the original definitions slightly, so that “true” really means “in line with reality in all instances” while “false” means “in at least one instance, it is not in line with reality.”

Notice the asymmetry, though. A hypothesis has to reach a pretty high bar to be considered “true,” and “false” hypotheses range from “in line with reality, with one exception” to “never in line with reality.” Some of those “false” hypotheses are actually quite valuable to us, as John Oliver’s segment demonstrates. He never explains what “statistical significance” means, for instance, but later on uses “significance” in the “effect size” sense. This will mislead most of the audience away from the reality of the situation, and in the absolute it makes his segment “false.” Nonetheless, that segment was a net positive at getting people to understand and care for the replication crisis, so labeling it “false” is a disservice.

We need something fuzzier than the strict binary of falsification. What if we didn’t compliment “true” in the set-theory sense, but in the definitional sense? Let “true” remain “in line with reality in all instances,” but change “false” from “in at least one instance, it is not in reality” to “never in line with reality.” This creates a gap, though: that hypothesis from earlier is neither “true” nor “false,” as it isn’t true in all cases nor false in all. It must be in a third category, as part of some sort of paraconsistent logic.

This is where the Bayesian interpretation of statistics comes from, it deliberately disclaims an absolute “true” or “false” label for descriptions of the world, instead holding them up as two ends of a continuum. Every hypothesis in the third category inbetween, hoping that future data will reveal that its closer to one end of the continuum or the other.

I think it’s a neat way to view the Bayesian/Frequentism debate, as a mere disagreement over what “false” means.

A Computer Scientist Reads EvoPsych, Part 2

[Part 1]

the concept of “learning” within the Standard Social Science Model itself tacitly invokes unbounded rationality, in that learning is the tendency of the general-purpose, equipotential mind to grow—by an unspecified and undiscovered computational means—whatever functional information-processing abilities it needs to serve its purposes, given time and experience in the task environment.

Evolutionary psychologists depart from fitness teleologists, traditional economists (but not neuroeconomists), and blank-slate learning theorists by arguing that neither human engineers nor evolution can build a computational device that exhibits these forms of unbounded rationality, because such architectures are impossible, even in principle (for arguments, see Cosmides & Tooby, 1987; Symons 1989, 1992; Tooby & Cosmides, 1990a, 1992).[1]

Yeah, these people don’t know much about computer science.

You can divide the field of “artificial” intelligence into two basic approaches. The top-down approach outlined modular code routines like “recognize faces,” then broke those down into sub-tasks like “look for eyes” and “find mouths.” By starting at a high level and dividing these things down into neat, tidy sub-programs, we can chain them together and create a greater whole.

It’s never worked all that well, at least for real-life problems. Take Cyc, the best example I can think of. It takes basic facts about the world, like “water is wet” or “rain is water,” and uses a simple set of rules to query these facts (“is rain wet?”). What it can’t do is make guesses (“are clouds wet?”), nor discover new facts on its own, nor handle anything but simple text. Thirty years and millions of dollars haven’t made a dent in those problems.

Meanwhile, the graphics card manufacturer NVidia is betting the farm on something called “deep learning,” one of several “bottom-up” approaches. You present the algorithm with an image (or sound file or object, the number of dimensions can be easily changed), and it maps it to a grid of cells. You toss a slightly smaller grid of cells on top of it, and for each new cell you calculate a weighted sum of the nearby values in the previous grid, weights that are random to start off with. Repeat this several times, and you’ll wind up with a single cell at the end. Assign this cell to an output, say “person,” then rewind all the way back to the start. Wash, rinse, and repeat until you get another single cell, then at least enough single cells to handle every possible solution. All of these single cells have a value associated with them, so that “person” cell might give the image 0.7 “person”s. Having cataloged what’s in the image already, you know there’s actually 1.0 “person” there, and so you propagate that information back down the chain. Prior cell weights which were pro-person are increased, while the anti-person ones are decreased. Do this right to the bottom, and for every input cell, then repeat the process for a new image.

It’s loosely patterned after how our own neurons are laid out. Biology is a bit more liberal with how it connects, but this structure has the virtue of being easy to calculate and massively parallel, quite convenient for a company which manufactures processors that specialize in massively parallel computations. NVidia’s farm-betting comes from the fact that it’s wildly successful; all of the best image recognition algorithms follow the deep-learning pattern, and their success rates are not only impressive but also resemble our own.[2]

Heard of the AI that could play Atari games? Emphasis mine:

Our [Deep action-value Network or DQN] method outperforms the best existing reinforcement learning methods on 43 games without incorporating any of the additional prior knowledge about Atari 2600 games used by other approaches … . Furthermore, our DQN agent performed at a level that was comparable to that of a professional human games tester across the set of 49 games, achieving more than 75% of the human score on more than half of the games […]

Indeed, in certain games DQN is able to discover a relatively long-term strategy (for example, Breakout: the agent learns the optimal strategy, which is to first dig a tunnel around the side of the wall allowing the ball to be sent around the back to destroy a large number of blocks; …). […]

In this work, we demonstrate that a single architecture can successfully learn control policies in a range different environments with only very minimal prior knowledge, receiving only the pixels and the game score as inputs, and using the same algorithm, network architecture and hyperparameters each game, privy only to the inputs a human player would have.[3]

This deep learning network has no idea what a video game is, nor is it permitted to peek at the innards of the game itself, yet can not only learn to play these games at the same level as human beings, it can develop non-trivial solutions to them. You can’t get more “blank slate” than that.

This basic pattern has repeated multiple times over the decades. Neural nets aren’t as zippy as the new kid on the “bottom-up” block, yet they too have had great success where the modular top-down approach has failed miserably. I haven’t worked with either technology, but I’ve worked with something that’s related: genetic algorithms. Represent your solutions in a sort of genome, come up with a fitness metric for them, then mutate or randomly construct those genomes and keep the fittest ones in the pool until you’ve tried every possibility, or you get bored. Two separate runs might converge to the same solution, or they might not. A lot depends on the “fitness landscape” they occupy, which you can visualize as a 3D terrain map with height representing how “fit” something is.

A visualization of three "evolutionary fitness landscapes," ranging from simple to complex to SUPER complex.That landscape has probably got more than three dimensions, but those aren’t as easy to visualize and they behave very similarily to the 3D case. The terrain might be a Mount Fiji with a single solution at the top of a fitness peak, or a Himalayas with many peak solutions scattered about but a single tallest standing above them, or a foothills where solutions are aplenty but the best solution is tough to find.

All of these take the “bottom-up” approach, the opposite of the “top-down” one, and work up from very small components towards a high-level goal. The path to there is rarely known in advance, so the system “feels” its way there via evolutionary algorithms.

That path may not go the way you expect, however. Take the case of a researcher, Dr. Adrian Thompson, who used an evolutionary algorithm to find the smallest computer processor that could sense the difference between two tones.

Finally, after just over 4,000 generations, the test system settled upon the best program. When Dr. Thompson played the 1kHz tone, the microchip unfailingly reacted by decreasing its power output to zero volts. When he played the 10kHz tone, the output jumped up to five volts. He pushed the chip even farther by requiring it to react to vocal “stop” and “go” commands, a task it met with a few hundred more generations of evolution. As predicted, the principle of natural selection could successfully produce specialized circuits using a fraction of the resources a human would have required. And no one had the foggiest notion how it worked.

Dr. Thompson peered inside his perfect offspring to gain insight into its methods, but what he found inside was baffling. The plucky chip was utilizing only thirty-seven of its one hundred logic gates, and most of them were arranged in a curious collection of feedback loops. Five individual logic cells were functionally disconnected from the rest— with no pathways that would allow them to influence the output— yet when the researcher disabled any one of them the chip lost its ability to discriminate the tones. Furthermore, the final program did not work reliably when it was loaded onto other FPGAs of the same type.

It seems that evolution had not merely selected the best code for the task, it had also advocated those programs which took advantage of the electromagnetic quirks of that specific microchip environment. The five separate logic cells were clearly crucial to the chip’s operation, but they were interacting with the main circuitry through some unorthodox method— most likely via the subtle magnetic fields that are created when electrons flow through circuitry, an effect known as magnetic flux. There was also evidence that the circuit was not relying solely on the transistors’ absolute ON and OFF positions like a typical chip; it was capitalizing upon analogue shades of gray along with the digital black and white.[4]

Evolutionary approaches are very simple and require no understanding or insight into the problem you’re solving, but they usually requires ridiculous amounts of computation or training merely to keep pace with the top-down “modular” approach. The fitness function may lead to a solution much too complicated for you to understand or much too fragile to operate anywhere but where it was generated. But the bottom-up approach may be your only choice for certain problems.

The moral of the story: the ability to do complex calculation can be built up from a blank slate, in principle and practice. When we follow the bottom-up approach we tend to get results that more closely mirror biology than when we work from the top-down and modularize, though this is less insightful than it first appears. Nearly all bottom-up approaches take direct inspiration from biology, whereas top-down approaches owe more to Plato then Aristotle.

Biology prefers the blank slate.

[Part 3]

[1] Tooby, John, and Leda Cosmides. “Conceptual Foundations of Evolutionary Psychology.” The Handbook of Evolutionary Psychology (2005): 5-67.

[2] Kheradpisheh, Saeed Reza, et al. “Deep Networks Resemble Human Feed-forward Vision in Invariant Object Recognition.” arXiv preprint arXiv:1508.03929 (2015).

[3] Mnih, Volodymyr, et al. “Human-level control through deep reinforcement learning.” Nature 518.7540 (2015): 529-533.

[4] Bellows, Alan. “On the Origin of Circuits • Damn Interesting.” Accessed May 4, 2016.

A Computer Scientist Reads EvoPsych, Part 1

Computer Science is weird. Most of the papers published in my field look like this:

We describe the architecture of a novel system for precomputing sparse directional occlusion caches. These caches are used for accelerating a fast cinematic lighting pipeline that works in the spherical harmonics domain. The system was used as a primary lighting technology in the movie Avatar, and is able to efficiently handle massive scenes of unprecedented complexity through the use of a flexible, stream-based geometry processing architecture, a novel out-of-core algorithm for creating efficient ray tracing acceleration structures, and a novel out-of-core GPU ray tracing algorithm for the computation of directional occlusion and spherical integrals at arbitrary points.[1]

A speed improvement of two orders of magnitude is pretty sweet, but this paper really isn’t about computers per se; it’s about applying existing concepts in computer graphics in a novel combination to solve a practical problem. Most papers are all about the application of computers, and not computing itself. You can dig up examples of the latter, like if you try searching for concurrency theory,[2] but even then you’ll run across a lot of applied work, like articles on sorting algorithms designed for graphics cards.[3]

In sum, computer scientists spend most of their time working in other people’s fields, solving other people’s problems. So you can imagine my joy when I stumbled on people in other fields invoking computer science.

Because the evolved function of a psychological mechanism is computational—to regulate behavior and the body adaptively in response to informational inputs—such a model consists of a description of the functional circuit logic or information processing architecture of a mechanism (Cosmides & Tooby, 1987; Tooby & Cosmides, 1992). Eventually, these models should include the neural, developmental, and genetic bases of these mechanisms, and encompass the designs of other species as well.[4]

Hot diggity! How well does that non-computer-scientist understand the field, though? Let’s put my degree to work.

The second building block of evolutionary psychology was the rise of the computational sciences and the recognition of the true character of mental phenomena. Boole (1848) and Frege (1879) formalized logic in such a way that it became possible to see how logical operations could be carried out mechanically, automatically, and hence through purely physical causation, without the need for an animate interpretive intelligence to carry out the steps. This raised the irresistible theoretical possibility that not only logic but other mental phenomena such as goals and learning also consisted of formal relationships embodied nonvitalistically in physical processes (Weiner, 1948). With the rise of information theory, the development of the first computers, and advances in neuroscience, it became widely understood that mental events consisted of transformations of structured informational relationships embodied as aspects of organized physical systems in the brain. This spreading appreciation constituted the cognitive revolution. The mental world was no longer a mysterious, indefinable realm, but locatable in the physical world in terms of precisely describable, highly organized causal relations.[4]

Yes! I’m right with you here. One of the more remarkable findings of computer science is that every computation or algorithm can be executed on a Turing machine. That includes all of Quantum Field Theory, even those Quant-y bits. While QFT isn’t a complete theory, we’re extremely confident in the subset we need to simulate neural activity. Those simulations have long since been run and matched against real-world data, the current problem is scaling up from faking a million neurons at a time to faking 100 billion, about as many as you have locked in your skull.

Our brains can be perfectly simulated by a computational device, and our brain’s ability to do math proves they are computational. I can quibble a bit on the wording (“precisely describable” suggests we’ve faked those 100 billion), but we’re off to a great start here.

After all, if the human mind consists primarily of a general capacity to learn, then the particulars of the ancestral hunter-gatherer world and our prehuman history as Miocene apes left no interesting imprint on our design. In contrast, if our minds are collections of mechanisms designed to solve the adaptive problems posed by the ancestral world, then hunter-gatherer studies and primatology become indispensable sources of knowledge about modern human nature.[4]

Wait, what happened to the whole “our brains are computers” thing? Look, here’s a diagram of a Turing machine.

An annotated diagram of a Turing machine. Copyright HJ Hornbeck 2016, CC-BY-SA 4.0.

A read/write head sits somewhere along an infinite ribbon of tape. It reads what’s under the head, writes back a value to that location, and moves either left or right, all based on that value. How does it know what to do? Sometimes that’s hard-wired into the machine, but more commonly it reads instructions right off the tape. These machines don’t ship with much of anything, just the bare minimum necessary to do every possible computation.

This carries over into physical computers as well; when the CPU of your computer boots up, it does the following:

  1. Read the instruction at memory location 4,294,967,280.
  2. Execute it.

Your CPU does have “programs” of a sort, instructions such as ADD (addition) or MULT (multiply), but removing them doesn’t remove its ability to compute. All of those extras can be duplicated by grouping together simpler operations, they’re only there to make programmers’ lives easier.

There’s no programmer for the human brain, though. Despite what The Matrix told you, no-one can fiddle around with your microcode and add new programs. There is no need for helper instructions. So if human brains are like computers, and computers are blank slates for the most part, we have a decent reason to think humans are blank slates too, infinitely flexible and fungible.

[Part 2]

[1] Pantaleoni, Jacopo, et al. “PantaRay: fast ray-traced occlusion caching of massive scenes.” ACM Transactions on Graphics (TOG). Vol. 29. No. 4. ACM, 2010.

[2] Roscoe, Bill. “The theory and practice of concurrency.” (1998).

[3] Ye, Xiaochun, et al. “High performance comparison-based sorting algorithm on many-core GPUs.” Parallel & Distributed Processing (IPDPS), 2010 IEEE International Symposium on. IEEE, 2010.

[4] Tooby, John, and Leda Cosmides. “Conceptual Foundations of Evolutionary Psychology.” The Handbook of Evolutionary Psychology (2005): 5-67.

Index Post: P-values

Over the months, I’ve managed to accumulate a LOT of papers discussing p-values and their application. Rather than have them rot on my hard drive, I figured it was time for another index post.

Full disclosure: I’m not in favour of them. But I came to that by reading these papers, and seeing no effective counter-argument. So while this collection is biased against p-values, that’s no more a problem than a bias against the luminiferous aether or humour theory. And don’t worry, I’ll include a few defenders of p-values as well.

What’s a p-value?

It’s frequently used in “null hypothesis significance testing,” or NHST to its friends. A null hypothesis is one you hope to refute, preferably a fairly established one that other people accept as true. That hypothesis will predict a range of observations, some more likely than others. A p-value is simply the odds of some observed event happening, plus the odds of all events more extreme, assuming the null hypothesis is true. You can then plug that value into the following logic:

  1. Event E, or an event more extreme, is unlikely to occur under the null hypothesis.
  2. Event E occurred.
  3. Ergo, the null hypothesis is false.

They seem like a weird thing to get worked up about.

Significance testing is a cornerstone of modern science, and NHST is the most common form of it. A quick check of Google Scholar shows “p-value” shows up 3.8 million times, while its primary competitor, “Bayes Factor,” shows up 250,000. At the same time, it’s poorly understood.

The P value is probably the most ubiquitous and at the same time, misunderstood, misinterpreted, and occasionally miscalculated index in all of biomedical research. In a recent survey of medical residents published in JAMA, 88% expressed fair to complete confidence in interpreting P values, yet only 62% of these could answer an elementary P-value interpretation question correctly. However, it is not just those statistics that testify to the difficulty in interpreting P values. In an exquisite irony, none of the answers offered for the P-value question was correct, as is explained later in this chapter.

Goodman, Steven. “A Dirty Dozen: Twelve P-Value Misconceptions.” In Seminars in Hematology, 45:135–40. Elsevier, 2008. http://www.sciencedirect.com/science/article/pii/S0037196308000620.

The consequence is an abundance of false positives in the scientific literature, leading to many failed replications and wasted resources.

Gotcha. So what do scientists think is wrong with them?

Well, th-

And make it quick, I don’t have a lot of time.

Right right, here’s the top three papers I can recommend:

Null hypothesis significance testing (NHST) is arguably the mosl widely used approach to hypothesis evaluation among behavioral and social scientists. It is also very controversial. A major concern expressed by critics is that such testing is misunderstood by many of those who use it. Several other objections to its use have also been raised. In this article the author reviews and comments on the claimed misunderstandings as well as on other criticisms of the approach, and he notes arguments that have been advanced in support of NHST. Alternatives and supplements to NHST are considered, as are several related recommendations regarding the interpretation of experimental data. The concluding opinion is that NHST is easily misunderstood and misused but that when applied with good judgment it can be an effective aid to the interpretation of experimental data.

Nickerson, Raymond S. “Null Hypothesis Significance Testing: A Review of an Old and Continuing Controversy.” Psychological Methods 5, no. 2 (2000): 241.

After 4 decades of severe criticism, the ritual of null hypothesis significance testing (mechanical dichotomous decisions around a sacred .05 criterion) still persists. This article reviews the problems with this practice, including near universal misinterpretation of p as the probability that H₀ is false, the misinterpretation that its complement is the probability of successful replication, and the mistaken assumption that if one rejects H₀ one thereby affirms the theory that led to the test.

Cohen, Jacob. “The Earth Is Round (p < .05).” American Psychologist 49, no. 12 (1994): 997–1003. doi:10.1037/0003-066X.49.12.997.

This chapter examines eight of the most commonly voiced objections to reform of data analysis practices and shows each of them to be erroneous. The objections are: (a) Without significance tests we would not know whether a finding is real or just due to chance; (b) hypothesis testing would not be possible without significance tests; (c) the problem is not significance tests but failure to develop a tradition of replicating studies; (d) when studies have a large number of relationships, we need significance tests to identify those that are real and not just due to chance; (e) confidence intervals are themselves significance tests; (f) significance testing ensure objectivity in the interpretation of research data; (g) it is the misuse, not the use, of significance testing that is the problem; and (h) it is futile to reform data analysis methods, so why try?

Schmidt, Frank L., and J. E. Hunter. “Eight Common but False Objections to the Discontinuation of Significance Testing in the Analysis of Research Data.” What If There Were No Significance Tests, 1997, 37–64.

OK, I have a bit more time now. What else do you have?

Using a Bayesian significance test for a normal mean, James Berger and Thomas Sellke (1987, pp. 112–113) showed that for p values of .05, .01, and .001, respectively, the posterior probabilities of the null, Pr(H₀ | x), for n = 50 are .52, .22, and .034. For n = 100 the corresponding figures are .60, .27, and .045. Clearly these discrepancies between p and Pr(H₀ | x) are pronounced, and cast serious doubt on the use of p values as reasonable measures of evidence. In fact, Berger and Sellke (1987) demonstrated that data yielding a p value of .05 in testing a normal mean nevertheless resulted in a posterior probability of the null hypothesis of at least .30 for any objective (symmetric priors with equal prior weight given to H₀ and HA ) prior distribution.

Hubbard, R., and R. M. Lindsay. “Why P Values Are Not a Useful Measure of Evidence in Statistical Significance Testing.” Theory & Psychology 18, no. 1 (February 1, 2008): 69–88. doi:10.1177/0959354307086923.

Because p-values dominate statistical analysis in psychology, it is important to ask what p says about replication. The answer to this question is ‘‘Surprisingly little.’’ In one simulation of 25 repetitions of a typical experiment, p varied from .44. Remarkably, the interval—termed a p interval —is this wide however large the sample size. p is so unreliable and gives such dramatically vague information that it is a poor basis for inference.

Cumming, Geoff. “Replication and p Intervals: p Values Predict the Future Only Vaguely, but Confidence Intervals Do Much Better.” Perspectives on Psychological Science 3, no. 4 (July 2008): 286–300. doi:10.1111/j.1745-6924.2008.00079.x.

Simulations of repeated t-tests also illustrate the tendency of small samples to exaggerate effects. This can be shown by adding an additional dimension to the presentation of the data. It is clear how small samples are less likely to be sufficiently representative of the two tested populations to genuinely reflect the small but real difference between them. Those samples that are less representative may, by chance, result in a low P value. When a test has low power, a low P value will occur only when the sample drawn is relatively extreme. Drawing such a sample is unlikely, and such extreme values give an exaggerated impression of the difference between the original populations. This phenomenon, known as the ‘winner’s curse’, has been emphasized by others. If statistical power is augmented by taking more observations, the estimate of the difference between the populations becomes closer to, and centered on, the theoretical value of the effect size.

is G., Douglas Curran-Everett, Sarah L. Vowler, and Gordon B. Drummond. “The Fickle P Value Generates Irreproducible Results.” Nature Methods 12, no. 3 (March 2015): 179–85. doi:10.1038/nmeth.3288.

If you use p=0.05 to suggest that you have made a discovery, you will be wrong at least 30% of the time. If, as is often the case, experiments are underpowered, you will be wrong most of the time. This conclusion is demonstrated from several points of view. First, tree diagrams which show the close analogy with the screening test problem. Similar conclusions are drawn by repeated simulations of t-tests. These mimic what is done in real life, which makes the results more persuasive. The simulation method is used also to evaluate the extent to which effect sizes are over-estimated, especially in underpowered experiments. A script is supplied to allow the reader to do simulations themselves, with numbers appropriate for their own work. It is concluded that if you wish to keep your false discovery rate below 5%, you need to use a three-sigma rule, or to insist on p≤0.001. And never use the word ‘significant’.

Colquhoun, David. “An Investigation of the False Discovery Rate and the Misinterpretation of P-Values.” Royal Society Open Science 1, no. 3 (November 1, 2014): 140216. doi:10.1098/rsos.140216.

I was hoping for something more philosophical.

The idea that the P value can play both of these roles is based on a fallacy: that an event can be viewed simultaneously both from a long-run and a short-run perspective. In the long-run perspective, which is error-based and deductive, we group the observed result together with other outcomes that might have occurred in hypothetical repetitions of the experiment. In the “short run” perspective, which is evidential and inductive, we try to evaluate the meaning of the observed result from a single experiment. If we could combine these perspectives, it would mean that inductive ends (drawing scientific conclusions) could be served with purely deductive methods (objective probability calculations).

Goodman, Steven N. “Toward Evidence-Based Medical Statistics. 1: The P Value Fallacy.” Annals of Internal Medicine 130, no. 12 (1999): 995–1004.

Overemphasis on hypothesis testing–and the use of P values to dichotomise significant or non-significant results–has detracted from more useful approaches to interpreting study results, such as estimation and confidence intervals. In medical studies investigators are usually interested in determining the size of difference of a measured outcome between groups, rather than a simple indication of whether or not it is statistically significant. Confidence intervals present a range of values, on the basis of the sample data, in which the population value for such a difference may lie. Some methods of calculating confidence intervals for means and differences between means are given, with similar information for proportions. The paper also gives suggestions for graphical display. Confidence intervals, if appropriate to the type of study, should be used for major findings in both the main text of a paper and its abstract.

Gardner, Martin J., and Douglas G. Altman. “Confidence Intervals rather than P Values: Estimation rather than Hypothesis Testing.” BMJ 292, no. 6522 (1986): 746–50.

What’s this “Neyman-Pearson” thing?

P-values were part of a method proposed by Ronald Fisher, as a means of assessing evidence. Even as the ink was barely dry on it, other people started poking holes in his work. Jerzy Neyman and Egon Pearson took some of Fisher’s ideas and came up with a new method, based on long-term prediction. Their method is superior, IMO, but rather than replacing Fisher’s approach it instead wound up being blended with it, ditching all the advantages to preserve the faults. This citation covers the historical background:

Huberty, Carl J. “Historical Origins of Statistical Testing Practices: The Treatment of Fisher versus Neyman-Pearson Views in Textbooks.” The Journal of Experimental Education 61, no. 4 (1993): 317–33.

While the remainder help describe the differences between the two methods, and possible ways to “fix” their shortcomings.

The distinction between evidence (p’s) and error (a’s) is not trivial. Instead, it reflects the fundamental differences between Fisher’s ideas on significance testing and inductive inference, and Neyman-Pearson’s views on hypothesis testing and inductive behavior. The emphasis of the article is to expose this incompatibility, but we also briefly note a possible reconciliation.

Hubbard, Raymond, and M. J Bayarri. “Confusion Over Measures of Evidence ( p ’S) Versus Errors ( α ’S) in Classical Statistical Testing.” The American Statistician 57, no. 3 (August 2003): 171–78. doi:10.1198/0003130031856.

The basic differences are these: Fisher attached an epistemic interpretation to a significant result, which referred to a particular experiment. Neyman rejected this view as inconsistent and attached a behavioral meaning to a significant result that did not refer to a particular experiment, but to repeated experiments. (Pearson found himself somewhere in between.)

Gigerenzer, Gerd. “The Superego, the Ego, and the Id in Statistical Reasoning.” A Handbook for Data Analysis in the Behavioral Sciences: Methodological Issues, 1993, 311–39.

This article presents a simple example designed to clarify many of the issues in these controversies. Along the way many of the fundamental ideas of testing from all three perspectives are illustrated. The conclusion is that Fisherian testing is not a competitor to Neyman-Pearson (NP) or Bayesian testing because it examines a different problem. As with Berger and Wolpert (1984), I conclude that Bayesian testing is preferable to NP testing as a procedure for deciding between alternative hypotheses.

Christensen, Ronald. “Testing Fisher, Neyman, Pearson, and Bayes.” The American Statistician 59, no. 2 (2005): 121–26.

C’mon, there aren’t any people defending the p-value?

Sure there are. They fall into two camps: “deniers,” a small group that insists there’s nothing wrong with p-values, and the much more common “fixers,” who propose making up for the shortcomings by augmenting NHST. Since a number of fixers have already been cited, I’ll just focus on the deniers here.

On the other hand, the propensity to misuse or misunderstand a tool should not necessarily lead us to prohibit its use. The theory of estimation is also often misunderstood. How many epidemiologists can explain the meaning of their 95% confidence interval? There are other simple concepts susceptible to fuzzy thinking. I once quizzed a class of epidemiology students and discovered that most had only a foggy notion of what is meant by the word “bias.” Should we then abandon all discussion of bias, and dumb down the field to the point where no subtleties need trouble us?

Weinberg, Clarice R. “It’s Time to Rehabilitate the P-Value.” Epidemiology 12, no. 3 (2001): 288–90.

The solution is simple and practiced quietly by many researchers—use P values descriptively, as one of many considerations to assess the meaning and value of epidemiologic research findings. We consider the full range of information provided by P values, from 0 to 1, recognizing that 0.04 and 0.06 are essentially the same, but that 0.20 and 0.80 are not. There are no discontinuities in the evidence at 0.05 or 0.01 or 0.001 and no good reason to dichotomize a continuous measure. We recognize that in the majority of reasonably large observational studies, systematic biases are of greater concern than random error as the leading obstacle to causal interpretation.

Savitz, David A. “Commentary: Reconciling Theory and Practice.” Epidemiology 24, no. 2 (March 2013): 212–14. doi:10.1097/EDE.0b013e318281e856.

The null hypothesis can be true because it is the hypothesis that errors are randomly distributed in data. Moreover, the null hypothesis is never used as a categorical proposition. Statistical significance means only that chance influences can be excluded as an explanation of data; it does not identify the nonchance factor responsible. The experimental conclusion is drawn with the inductive principle underlying the experimental design. A chain of deductive arguments gives rise to the theoretical conclusion via the experimental conclusion. The anomalous relationship between statistical significance and the effect size often used to criticize NHSTP is more apparent than real.

Hunter, John E. “Testing Significance Testing: A Flawed Defense.” Behavioral and Brain Sciences 21, no. 02 (April 1998): 204–204. doi:10.1017/S0140525X98331167.

Destruction of Justice

I’ve written about the “rape kit backlog” before; as a quirk summary, police departments are letting rape kits languish for decades, despite how easy they are to process and how effective they are at securing convictions.

Testing by Cleveland-area prosecutors linked more than 200 alleged serial rapists to 600 assaults. Statewide, Ohio Attorney General Mike DeWine’s effort to collect and test sexual assault kits has resulted in at least 2,285 CODIS hits so far.

In Houston, analysis of about 6,600 untested rape kits resulted in about 850 matches, 29 prosecutions and six convictions.

And, since the Colorado Bureau of Investigation began requiring police statewide to submit sexual assault kits for testing last year, more than 150 matches have been found.

But back then, I never thought of the dark side of the rape test backlog.

As scrutiny of disregarded rape kits mounted, a portrait of a more difficult to tally sort emerged – rape kits police destroyed. As with the rape kit backlog, there is no national tally of the kits police destroyed. But increasingly, local media have published reports of police destroying rape kits in states as disparate as Utah, Kentucky and Colorado. […]

In 2013, in Aurora, Colorado, police department workers derailed a prosecution when they destroyed a rape kit from a 2009 assault. The error was discovered when a detective got a hit on an offender DNA profile, went to pick up the rape kit and was told it no longer existed. Shortly thereafter, police stopped all evidence destruction while they investigated, and found workers destroyed evidence in 48 rape cases between 2011 and 2013.

In Salt Lake City, 222 of the 942 kits collected between 2004 and 2014 were destroyed. Of those, just 59 were tested and went to court.

In Hamilton County, Tennessee, sheriff’s employees destroyed rape kits with marijuana and cocaine from drug busts, angering the local prosecutor who said he wasn’t consulted.

In Kentucky, the state auditor discovered some police departments routinely destroyed rape kits after a year, even though the state had no statute of limitations for rape. The perpetrators could have been prosecuted as long as they were alive.

There was so little value placed on those kits, despite their track record of landing convictions, that the experts responsible for handling them saw no problem in their casual destruction. Criminals are allowed to walk freely, because the police bought into common myths about sexual assault.

It’s one more slice of rape culture.

The Monty Hall Problem, or When the Obvious Isn’t

“You blew it, and you blew it big! Since you seem to have difficulty grasping the basic principle at work here, I’ll explain. After the host reveals a goat, you now have a one-in-two chance of being correct. Whether you change your selection or not, the odds are the same. There is enough mathematical illiteracy in this country, and we don’t need the world’s highest IQ propagating more. Shame!”
– Scott Smith, Ph.D. University of Florida

There was a rather unusual convergence of feminism and skepticism in 1991. Over a thousand people with PhD’s and a few Nobel-prize winners sent angry letters to a puzzle author, insisting the answer to a particular puzzle was wrong. More than a few flashed their academic credentials as evidence in their favor, providing an excellent example of the Argument from Authority. Most seemed to ignore that the author had one of the highest recorded IQs in the world and talked down to her, providing an excellent example of how women’s credentials are frequently undervalued.

The flashpoint for it all was the Monty Hall problem. Here’s how Marilyn vos Savant originally described the problem:

Suppose you’re on a game show, and you’re given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what’s behind the doors, opens another door, say No. 3, which has a goat. He then says to you, “Do you want to pick door No. 2?” Is it to your advantage to switch your choice?

This wasn’t her invention, but nearly all the people who presented it before were men and I can’t find any evidence they were as heavily doubted. Even a few women shamed vos Savant for coming to the “incorrect” answer, that it’s to your advantage to switch.

On first blush, that does seem incorrect. One door has been removed, leaving two behind and only one with a prize. So there’s a 50/50 chance you picked the right door, right? Even a computer simulation seems to agree.

Out of 12 scenarios, you were better off staying put in 6 of them,
 and switching in 6 of them. So you'll win 50.000000% of the time if you
 stay put, and 50.000000% of the time if you switch.

Out of 887746 trials, you were better off staying put in 444076 of them,
 and switching in 443670 of them. So you'll win 50.022865% of the time if you
 stay put, and 49.977135% of the time if you switch.

But there’s a simple flaw hidden here.

A table of all the outcomes in the Monty Hall problem.You can pick one of three doors, and you have a one in three chance of picking the door with the car. On one of those lucky occasions, Monty Hall opens one of the other doors. Which one doesn’t matter, because it should be obvious that no matter what door he picks your best option is to keep the door you have.

Two-thirds of the time, you’ve picked a door to a goat. Hall can’t open your door yet, and he can’t open the door with the car behind it, so he’s forced to reveal the third door. If you switch, the only sensible choice is the door with the car. If you stay, as already established, you’ve lost.

So two-thirds of the time you should switch, while one-third of the time you’re better off staying put. If you switch all the time, your expected earnings are two-thirds of a car, if you stay put all the time it’s a third of a car, and any strategy that bounces between the two choices (save cheating) will pay off somewhere in between.

In short, always switch.

Still don’t believe me? I’ll modify one of vos Savant’s demonstrations, and show you how to verify this with a deck of cards. Toss out any jokers, then pull out just a single suit of cards and leave the bulk of the deck behind. Grab some way to track the score, while you’re at it, and a coin.

  1. Shuffle the 13 cards well.
  2. Deal out three cards in a horizontal line, face-up. In this game you’re Monty Hall, so there’s no need to hide things.
  3. Find the card with the lowest face value, as that one has the car.
  4. The player always picks the leftmost door.
  5. If they didn’t pick the lowest card, “show” them the other losing door by flipping it over. If they did, toss the coin to determine which of the two losing cards you’ll “show.”
  6. Mark down which strategy wins this round, and gather up the cards.
  7. Repeat from step one until bored or enlightened.

You’ll quickly realize that Hall’s precise choice is irrelevant, and start looping back after step four. After twenty or thirty rounds, you should see the “switch” strategy is superior. If you think I’m cheating by always having the player choose a specific door, you can easily modify the game to be two-player; if you’re suspicious of the thirteen card thing, use three (but shuffle really carefully).

So why did the program give the incorrect answers?

The probability space of the Monty Hall problem; note that not all outcomes are equally likely.While there are four distinct outcomes, they do not carry the same odds of happening. When you pick the door with the car, both choices that Hall can make occupy a third of the probability space in total, whereas both instances where Hall has no choice occupy two-thirds of the space. If you’re not careful, you can give all of them equal weight and falsely conclude that neither strategy has an advantage. If you are careful, you get the proper results.

Out of 9 scenarios, you were better off staying put in 3 of them,
 and switching in 6 of them. So you'll win 33.333332% of the time if you
 stay put, and 66.666664% of the time if you switch.

Out of 2000000 trials, you were better off staying put in 666131 of them,
 and switching in 1333869 of them. So you'll win 33.306549% of the time if you
 stay put, and 66.693451% of the time if you switch.

In hindsight the solution seems obvious enough, but this problem is unusually unintuitive.

Piattelli-Palmarini remarked (see vos Savant, 1997, p. 15): “No other statistical puzzle comes so close to fooling all the people all the time. […] The phenomenon is particularly interesting precisely because of its specificity, its reproducibility, and its immunity to higher education.” He went on to say “even Nobel physicists systematically give the wrong answer, and […] insist on it, and are ready to berate in print those who propose the right answer.” In his book Inevitable illusions: How mistakes of reason rule our minds (1994), Piattelli-Palmarini singled out the Monty Hall problem as the most expressive example of the “cognitive illusions” or “mental tunnels” in which “even the finest and best-trained minds get trapped” (p. 161). [1]

Human beings are only approximately rational; it’s terribly easy to fall for logical fallacies, and act in sexist ways without realizing it. Know thyself, and know how you’ll likely fail.

[1] Krauss, Stefan, and Xiao-Tian Wang. “The psychology of the Monty Hall problem: discovering psychological mechanisms for solving a tenacious brain teaser.” Journal of Experimental Psychology: General 132.1 (2003): 3.

A Slice of Rape Culture

There’s only so much you can cover in an hour. Early drafts of my lecture on sexual assault included a rant on rape kit testing, and it’s not hard to see why.

… police departments have been found to destroy records and ignore or mishandle evidence, which leads not only to undercounting but dismissal of cases. Many of the jurisdictions showing consistent undercounting are also, unsurprisingly, those with rape kit backlogs (there are more than 400,000 untested kits in the United States). Many cities and states don’t even keep accurate track of the number of rape exams or of kits languishing, expired or in storerooms—but when they do, the numbers improve. The arrest rate for sex assault in New York City went from 40 percent to 70 percent after the city successfully processed an estimated 17,000 kits in the early 2000s. However, it is only in the past year, after embarrassing and critical news coverage, that most departments have begun to process backlogs. After being publicly shamed for having abandoned more than 11,000 rape kits, the Michigan State Police began testing them, identifying 100 serial rapists as a result.

There’s some follow-up on that last item.

In Michigan, the Detroit kits make up the majority of those awaiting testing. To date, the largest backlog by far remains in Wayne County, where 10,000 of the 11,341 kits found in 2009 have been tested or are in the process of being tested. As of July 10, Detroit’s kit-testing initiative identified 2,478 suspects — including 456 serial rapists identified as of June 30 — and 20 convictions have been secured.

While it’s great to see justice served, take a step back and think about this. This one county managed to process 10,000 rape kits within a year or two; that means they’re quick to process. Those kits identified serial criminals and even in that short span generated 20 convictions; that means they are invaluable tools of law enforcement, an easy way to score convictions, keep the streets safe, and generate some good publicity.

But not only was this goldmine left to rot and grow since the 1980’s, it was discovered in 2009; in other words, even when they were aware of these kits and knew how valuable they were, they waited five years until they were embarrassed into action by the press.

This is one aspect of rape culture: the systematic devaluation of sexual assault victims, to the point that we blind ourselves to widespread injustice.

“This is not just an issue impacting Detroit or Wayne County,” [Shanon Banner, Michigan State Police manager of public affairs] said. “Everyone should care.”

[HJH 2015-07-19] USA Today was one of the first to break this story, and they have a follow-up too.

Testing by Cleveland-area prosecutors linked more than 200 alleged serial rapists to 600 assaults. Statewide, Ohio Attorney General Mike DeWine’s effort to collect and test sexual assault kits has resulted in at least 2,285 CODIS hits so far.

In Houston, analysis of about 6,600 untested rape kits resulted in about 850 matches, 29 prosecutions and six convictions.

And, since the Colorado Bureau of Investigation began requiring police statewide to submit sexual assault kits for testing last year, more than 150 matches have been found.

Despite those successes, many police agencies haven’t changed their policies.

In New York state, law enforcement agencies outside of New York City are under no legal requirement to test rape evidence. No state law exists requiring agencies to track how many untested kits are stored in their evidence rooms.

New York is one of 44 states with no law stipulating when police should test rape kits and 34 states that haven’t conducted a statewide inventory. […]

Interviews with law enforcement officials, and a review of police records obtained by USA TODAY, reveal sexual-assault-kit testing is often arbitrary and inconsistent among law enforcement agencies — and even within agencies.

In Jackson, Tenn., for example, notations in evidence records show contradictory reasons as to why rape kits were not tested. In some cases, the Jackson Police Department did not test evidence because the suspect’s identity was already known, records show. In 13 other cases since 1998, records show police decided not to test kits because there was “no suspect” or “no known suspect,” even though testing the kits could help identify a suspect. […]

Some government officials and researchers have faulted police for a predisposition to doubt survivors’ stories.

“The fact is that often rape kits are unsubmitted for testing because of a blame-the-victim mentality or because investigators mistrust the survivor’s story,” Illinois Attorney General Madigan told a U.S. Senate subcommittee at a hearing in May. “This outdated way of thinking must change.”

After more than 10,000 untested sexual assault kits were discovered in Detroit in 2008, a landmark study funded by the Justice Department faulted police for “negative, victim‐blaming beliefs.”

“Rape survivors were often assumed to be prostitutes and therefore what had happened to them was considered to be their own fault,” researchers from Michigan State University wrote in their analysis of Detroit’s rape investigations.

Welcome to rape culture.

A Statistical Analysis of a Sexual Assault Case: Part Three

[complications arise, as does simplicity]

In the last installment, we calculated the odds of nesting or attempted nesting at site 84744 M.S. to be 92%, based on Hugh’s claim. We also found that daufnie_odie’s claim made us 11% confident in nesting.

Hugh, though, was talking about a different point in time. Our original question only asked if the nesting site had seen a nest or attempted nest, without any other clear bounds. It’s similar to asking “will I ever see heads while flipping this coin;” the more distinct observations we have, the greater the chance of at least one head (or nesting attempt) appearing.

The obvious way to combine these two claims is to consider all the possibilities. If we have two independent events, A and B, then the odds of at least one happening is the sum of the first happening but the second not, the second happening but not the first, and both happening. That isn’t too annoying to add up when we have just two events, but if we use this technique for N events we’ll have to consider 2^N – 1 possibilities. Ouch.

Notice, though, that we’re calculating the probability of every possible observation combination, excluding one: that no events occurred. However, by definition the sum of all probabilities must be one. So if we calculate the odds of that single combination and subtract it from one, we know the sum of the odds for every other combination. We can accomplish 2^N – 1 calculations for the cost of one!

Putting this into practice with our numbers above, we calculate the odds of Hugh being wrong about the nest and the odds of daufnie_odie being wrong, then multiply and subtract that from one, and get 93%. A marginal improvement.

But hold on here; why did I multiply those two together? Let’s pull up a diagram:

Dividing the universe by the accounts of Hugh and daufnie_odieOur goal is to figure out A / (A + B + C + D). We can use a bit of algebra to rewrite that as

image

Oh, there’s our multiplication right there! In English, all we have to do is multiply the odds of daufnie_odie being wrong, by the odds of Hugh being wrong when we assume daufne_odie is wrong.

One problem: we don’t know the odds of the latter, just the odds of Hugh being wrong overall. If those two were dependent events, this could be a big problem, but thankfully they’re independent for our purposes; if we’re calculating the odds of no nest or attempt, we don’t care if two or more people are talking about the same event, we just need them to be wrong about whatever they’re talking about. That means that the vertical partition is exactly as it looks in the diagram, a straight cut across the entire probability space. In math terms, the ratio of A to B is the same as that of C to D, which leads to

image

So as long as we can be confident daufnie_odie’s claim is independent of Hugh’s, we can treat (A + B) / (A + B + C + D) as A / (A + B) and just multiply.

But when we take a closer look at daufnie_odie’s post, we realize we’re missing some key facts. They spoke up after reading another post by Pollock Myerson, wondering if the person who contacted them was the same as the one who contacted Myerson. Hopping over to Myerson’s post, we learn that he was introduced to someone claiming to have spotted a nest by Caroline Puppy, and that later on a third person contacted Myerson to validate the original tale. Again no names are mentioned, but Myerson, Puppy, and the third person make it clear that they know this nest claimant.

Scrolling back, we see someone named Bryant Tompsin claiming to know a witness to an attempted nest. This doesn’t look like the same person that contacted Puppy. There’s also a comment by someone who goes by “maryann”, who claims to have spotted at least an attempted nest; whether this is the same person that Tompsin, Puppy or daufnie_odie referred to isn’t clear, but it’s probably not Hugh under a different name.

Scrolling forward, we also find a few posts where Pauline Gray claims to have seen a Sexualis Asoltenti attempt to nest, but leaves out what nesting site she saw it at. Puppy reappears, claiming that she was told by someone named Dijai Gruthi that there was an attempted nesting at 84744 M.S., a fact she later confirmed with someone else who witnessed the same nesting. By comparing photos and accounts, it becomes probable that Pauline Gray was talking about 84744 M.S., that she saw it at the same time as Gruthi, and that Puppy’s other person is Gray. In the meanwhile, Tompsin reappears and also claims to have heard of the same attempted nesting from Gruthi.

As all that’s sinking in, we flip open the local birding magazine and find still more. Pauline Gray admits she really was talking about 84744 M.S. and that Gruthi was present for the attempted nest; the unnamed person of Myerson outs themselves as Ali Smyth, a local birder; and a well-respected person named Jim Grandie suggests he saw a nest or attempt at one but waved it off as horseplay, something birds do when drunk. Biff Jag confirms he was around shortly after Smyth’s nesting observation, and someone with the handle “skippingthem” mentions they know someone who was also a witness. daufnie_odie posts again, and confirms that the nesting Ali Smyth saw was not the one they were aware of. Finally, we can infer some information from the state of the nesting site; if it remains constant, that would suggest a nesting or attempt was unlikely, and if it shifted over time then it likely was nested in at some point. Myerson had a look at the long-term state of 84744 M.S., and indeed found evidence of shifting.

Working through all these combinations would be a nightmare. Fortunately, we don’t have to. As we only care if at least one nest or attempted nest happened, we can instead calculate the odds of no nesting occurring and then subtract that from one. This is a much simpler task, which we’ll accomplish in the next installment…

[HJH 2015-07-19: adding some missing links]