Keep on assessing science

Ugh. I got up at 5am and tried to read a statistics paper to put myself back to sleep, and it didn’t work. Dang numbers, stop being interesting! Anyway, this paper was a meta-meta-analysis to try and dig up the causes of bias that might be causal to the reproducibility crisis in the scientific literature. Here’s the abstract from the Fanelli, Costas, and Ioannidis (2017) paper; my emphasis on some of the key points.

Numerous biases are believed to affect the scientific literature, but their actual prevalence across disciplines is unknown. To gain a comprehensive picture of the potential imprint of bias in science, we probed for the most commonly postulated bias-related patterns and risk factors, in a large random sample of meta-analyses taken from all disciplines. The magnitude of these biases varied widely across fields and was overall relatively small. However, we consistently observed a significant risk of small, early, and highly cited studies to overestimate effects and of studies not published in peer-reviewed journals to underestimate them. We also found at least partial confirmation of previous evidence suggesting that US studies and early studies might report more extreme effects, although these effects were smaller and more heterogeneously distributed across meta-analyses and disciplines. Authors publishing at high rates and receiving many citations were, overall, not at greater risk of bias. However, effect sizes were likely to be overestimated by early-career researchers, those working in small or long-distance collaborations, and those responsible for scientific misconduct, supporting hypotheses that connect bias to situational factors, lack of mutual control, and individual integrity. Some of these patterns and risk factors might have modestly increased in intensity over time, particularly in the social sciences. Our findings suggest that, besides one being routinely cautious that published small, highly-cited, and earlier studies may yield inflated results, the feasibility and costs of interventions to attenuate biases in the literature might need to be discussed on a discipline-specific and topic-specific basis.

So, in part, the reproducibility problem is cause by new researchers scrambling to get a flashy result that will get them some attention, it’s worsened if they’re working in isolation rather than as part of a team, and there are a few scientists who are ethically compromised who have been spoiling the whole barrel of apples. That all makes sense to me.

It’s hard to police against individuals with little scientific integrity — rascals are present in every field. Catching them after the fact doesn’t necessarily help, because they’ve already tainted the literature with a flash-in-the-pan compromised paper.

Scientists who had one or more papers retracted were significantly more likely to report overestimated effect sizes, albeit solely in the case of first authors. This result, consistently observed across most robustness tests, offers partial support to the individual integrity hypothesis.

Catching a scientist who publishes bad data is already severely punished, so I don’t think that one is an avenue for improving the reliability of papers. It shouldn’t be ignored, obviously, but the other observations might lead to more improvement.

The mutual control hypothesis was supported overall, suggesting a negative association of bias with team size and a positive one with country-to-author ratio. Geographic distance exhibited a negative association, against predictions, but this result was not observed in any robustness test, unlike the other two.

Collaboration is good. In the days when I was in a large lab, it was always a little suspicious when someone suddenly plopped a whole, completed paper down in the lab meeting and announced that they’d finished the experiment, and by the way, would you like to be an author on the paper? I always turned those offers down, because a co-authorship ought to be the product of ongoing involvement in the work, not some attempt at fishing for external approval. But more cooperation and vetting of each other’s work ought to be a general hallmark of good science.

I’m not in a big research lab anymore, but I still try to get that across in student labs. There’s always someone who objects to having to work with those other students and wants to do their lab projects all by themselves, and I have to turn them down and tell them they have to work in teams. They probably think it’s so I’ll have fewer lab reports to grade (OK, maybe that’s part of it…), but it’s mainly because teamwork is an essential part of the toolkit of science.

And now I’m getting confirmation that it also helps reduce spurious results.

The biggest effect, though, is associated with small study size.

Our study asked the following question: “If we draw at random from the literature a scientific topic that has been summarized by a meta-analysis, how likely are we to encounter the bias patterns and postulated risk factors most commonly discussed, and how strong are their effects likely to be?” Our results consistently suggest that small-study effects, gray literature bias, and citation bias might be the most common and influential issues. Small-study effects, in particular, had by far the largest magnitude, suggesting that these are the most important source of bias in meta-analysis, which may be the consequence either of selective reporting of results or of genuine differences in study design between small and large studies. Furthermore, we found consistent support for common speculations that, independent of small-study effects, bias is more likely among early-career researchers, those working in small or long-distance collaborations, and those that might be involved with scientific misconduct.

More data! This is also helpful information for my undergraduate labs, since I’m currently in the process of cracking the whip over my genetics students and telling them to count more flies. Only a thousand? Count more. MORE!

The paper does end on a positive note. They’ve identified some potential sources of bias, but overall, science is in fairly good shape.

In conclusion, our analysis offered a “bird’s-eye view” of bias in science. It is likely that more complex, fine-grained analyses targeted to specific research fields will be able to detect stronger signals of bias and its causes. However, such results would be hard to generalize and compare across disciplines, which was the main objective of this study. Our results should reassure scientists that the scientific enterprise is not in jeopardy, that our understanding of bias in science is improving and that efforts to improve scientific reliability are addressing the right priorities. However, our results also suggest that feasibility and costs of interventions to attenuate distortions in the literature might need to be discussed on a discipline- and topic-specific basis and adapted to the specific conditions of individual fields. Besides a general recommendation to interpret with caution results of small, highly cited, and early studies, there may be no one-size fits-all solution that can rid science efficiently of even the most common forms of bias.


Fanelli D, Costas R, Ioannidis JPA (2017) Meta-assessment of bias in science. Proc.Nat.Acad.Sci USA doi: 10.1073/pnas.1618569114.

What’s Dan Dennett been up to lately?

He’s still battling dualism, as seen in this New Yorker profile. He’s still arguing with Chalmers, and he’s still going strong…with some exasperation.

Despite his affability, Dennett sometimes expresses a weary frustration with the immovable intuitions of the people he is trying to convince. “You shouldn’t trust your intuitions,” he told the philosophers on the Rembrandt. “Conceivability or inconceivability is a life’s work—it’s not something where you just screw up your head for a second!” He feels that Darwin’s central lesson—that everything in biology is gradual; that it arrives “not in a miraculous, instantaneous whoosh, but slowly, slowly”—is too easily swept aside by our categorical habits of mind. It could be that he is struggling with the nature of language, which imposes a hierarchical clarity upon the world that’s powerful but sometimes false. It could also be that he is wrong. For him, the struggle—a Darwinian struggle, at the level of ideas—continues. “I have devoted half a century, my entire academic life, to the project, in a dozen books and hundreds of articles tackling various pieces of the puzzle, without managing to move all that many readers from wary agnosticism to calm conviction,” he writes, in “From Bacteria to Bach and Back.” “Undaunted, I am trying once again.”

There’s something about this concept, that the mind is a product of the physics, chemistry, and biology of the brain, that some people cannot accept. But then I have an equally strong intuition that it is, so it’s hard to fault people for wanting to disbelieve it; I can still fault them for ignoring the growing evidence for the purely material basis of the mind, the absurdity and poor quality of the evidence for dualism, and the inability to come up with a mechanism, even an outline of an idea, for how dualism would work.

Breaking news: Racists want you to stop calling them racists

Sam Altman, a gullible Silicon Valley entrepreneur with no sociological qualifications, sent himself on a self-appointed mission to talk to Trump supporters. This was an interesting and helpful experience, he says, although I’m fed up with attempts to puzzle out what Trumpkins have to say, so it was the opposite of interesting to me, and he fails to explain what’s helpful about yet another set of rationalizations. In particular, his “TL;DR summary” of the various conversations is just self-serving extortion, and no, I neither accept this claim nor am I going to obey this suggestion.

“You all can defeat Trump next time, but not if you keep mocking us, refusing to listen to us, and cutting us out. It’s Republicans, not Democrats, who will take Trump down.”

We’ve been listening. We’ve been listening a lot. And it’s the same old crap that justifies mocking them. Like this quote:

“Stop calling us racists. Stop calling us idiots. We aren’t. Listen to us when we try to tell you why we aren’t. Oh, and stop making fun of us.”

But…they are racists! This is a racist comment:

“I’m so tired of hearing about white privilege. I’m white but way less privileged than a black person from your world. I have no hope my life will ever get any better.”

So is this:

“He is anti-immigration.” Note: This sentiment came up a lot. The most surprising takeaway for me how little it seemed to be driven by economic concerns, and how much it was driven by fears about “losing our culture,” “safety,” “community,” and a general Us-vs.-Them mentality.

We can hear them saying they’re not racist, and then making racist comment after racist comment. But then we’re supposed to not mention that what they’re saying is incredibly racist? Why not?

I think it’s way past time we stopped making nice with racists. If it hurts their feelings to point out that they’re racists, there are a couple of ways to resolve it: a) they could change and stop endorsing racist ideas, or b) we could obey their demands and be quiet about their racism.

Option B is no longer tenable, if it ever was. The ball is not in our court, but in theirs.

Then there is this: they say that the way Trump “talks about women is despicable”, but apparently not despicable enough to prevent them for voting for him, and then say stuff like this:

“He is anti-abortion.” Note: This sentiment came up a lot. A number of people I spoke to said they didn’t care about anything else he did and would always vote for whichever candidate was more anti-abortion.

They’re not only racist, but anti-woman. They don’t want us to say anything about Trump’s misogyny, or their implicit support for it.

Sam Altman isn’t very good at this cultural analysis stuff — he seems to accept everything the Trump voters say at face value, and then he turns around to chastise liberals for being mean to the Trumpkins. Not mean enough, I say: we need to make it crystal clear that their vile behavior is not acceptable.

Oh, right.

“The left is more intolerant than the right.” Note: This concept came up a lot, with real animosity in otherwise pleasant conversations.

Then stop doing intolerable things, and we can be tolerant. Otherwise, we’re done standing silent while they promote racism and misogyny.

Also, we’ll keep calling idiots who voted for a narcissistic, incompetent toddler idiots, thank you very much.

Sadistic Christianity

I always thought that one of the pleasures of parenting was helping kids grow up to be themselves — to develop to be independent people with their own interests and goals, which might be very different than my own. We’re about giving opportunities, not dictating how they should live their lives, and one of the advantages of that is that all of my children were relatively stress-free (kids are never totally stress-free) and have never caused us much in the way of problems — and I think part of the reason is simply that we did not force them to go against their natures. There were lots of moments where I didn’t understand their choices, or even disagreed with them, but I just had to remember that my parents didn’t quite understand what I was doing with my life, either, but they let me be me and we all ended up happier for it.

Some people just can’t do that, though. Authoritarians are all about control, and it can lead to catastrophic evil against children.

Today, in the United States, there is a multibillion-dollar industry for residential treatment—one that sells an illusory promise to desperate parents: Your children’s addictions and mental health problems can be cured with a relatively quick (and usually expensive) fix. Yet the potential danger of abuse and neglect is a real threat for many of the 200,000 to 400,000 young people trapped in the nation’s poorly monitored secular and religious “group care” facilities, “troubled teen” residential schools and unlicensed treatment programs. Too often, critics say, these programs profit off the misery of emotionally troubled kids, substance abusers or just misbehaving youth, as well as their parents, who struggle to deal with kids they can’t control. “These are throwaway children,” says Jodi Hobbs, the president of the nonprofit group, Survivors of Institutional Abuse. “They are looked at as dollar signs, not as individuals.”

One of the most common types of private programs for errant youths are the virtually unregulated religious schools, many of which push fundamentalist Christian beliefs and employ violently harsh discipline against enrollees. Inspired in part by the programs of a fiery Baptist radio preacher, the late Lester Roloff, purveyors of these programs have been exposed for whippings and beatings and accused of rape. Perhaps the largest alliance of such ultraconservative churches is the far-flung Independent Fundamental Baptist organization with thousands of churches nationwide and numerous boarding schools that cite the biblical importance of breaking the will of the child. “If you’re not bruising your child,” a pastor declared in a 2007 sermon captured by ABC News’s 20/20, “you’re not spanking your child enough.”

That’s part of the story of Restoration Youth Academy, a Christian boot-camp in Alabama that promises to straighten out those darned rebellious teenagers with discipline…which means solitary confinement, beatings, bloody whippings, and sexual abuse. Many of these kids do have serious problems — drug abuse and mental health issues — but a) those are problems that can be worsened by a miserable home life, and b) even if the parents are otherwise blameless, shipping them off to a violent, brutalizing incarceration isn’t going to make them better. The article doesn’t say, but I wonder what proportion of these kids weren’t actually serious problem children, but were just people who had a different sexual orientation or dissented from the religious views of their parents, and were sent off to be punished and re-molded into a different view. Given that many of the institutions discussed in the story were intolerant fundamentalist Christian horror shows, I suspect a lot.

The story is also about intransigent Alabama politicians who refused to take action and closed a blind eye to the evidence of child abuse going on, probably in part because they had a shield of immunity, that they were preaching Christianity. Among the problem characters was the Alabama attorney general, Luther Strange, who was in the news lately for a promotion.

Kennedy is equally outraged that former state Attorney General Luther Strange has been appointed a U.S. senator to replace Jeff Sessions, the new U.S. attorney general. “He [Strange] threw the children under the bus so he could grease the way for his political ambitions,” Kennedy says. “All these politicians have lined their pockets with the blood of children.”

And all of those churches.

Is there no religion Reza Aslan won’t pander to?

It seems to be his schtick. Religion is just plain good, and the only way to criticize it is to cherry-pick unrepresentative bad bits, he seems to argue, and he uses this argument to paper over a lot of truly horrific, deeply imbedded aspects of faith. And apparently, he has a show on CNN called Believer, which I haven’t seen, in which he does this repeatedly.

There’s an episode coming up in which he makes excuses for Scientology, of all things; he’s going to highlight small independent groups that have split away from the mainstream cult and are somewhat less toxic (in part because they also represent a way to get outside the controlling influence of the Church of Scientology, and can be a gateway to leaving the religion altogether), while ignoring the greater crimes of the much larger, main sect.

In the meantime, we’ll point out what we did a year ago, when CNN’s series was originally supposed to come out. On occasion, we are taken to task for focusing so much energy on such a small organization, the Church of Scientology, with its 20,000 members. We think Scientology, with its billions in assets, its ruthless legal tactics, and the way it treats children and families is worth keeping an eye on, even if we are just, for the most part, a sole proprietor with a single-subject website.

If some people, however, don’t think the Church of Scientology is worth paying attention to, what does it say that CNN, with its worldwide media reach, will be using its mighty resources to promote a “movement” of perhaps only a few hundred people doing something that is not really very controversial or that affects many other people at all?

“Aslan is clearly confused or deliberately trying to create a scenario to fit his preconceived story line,” Mike Rinder tells us. He points out, however, that even if Aslan is all wet, his show might accidentally be useful for people still stuck in the Church of Scientology to believe that there are alternatives to Miscavige’s brand of Hubbardism. “The idea that Scientology is only available in the church is something Miscavige and company try very hard to pretend is true.” The idea that there are alternatives, Rinder says, could be “beneficial.” But as for Aslan’s claims about the size and growth of independent Scientology?

“It just makes Aslan look uninformed and stupid,” Rinder says.

There was a time when I thought Aslan was mildly interesting as a counterbalance to the more extremist arguments against Islam, but it’s become clear that no, he’s just an apologist for inanity.

I never want to see another baby-eating joke about atheists

Yeah, sure, accuse atheists of eating babies. Do you know who actually consumes fetal tissue, though? Suburban new agers with a weird fetish for “natural” and “organic” BS.

I just learned about Minnesota Placenta, a place that does placenta encapsulation (pdf). It’s easy! After your baby is born, it comes with this hideous lump of fetal support tissue, the placenta, that looks like a lump of hamburger and a piece of raw liver got into a serious barroom brawl, and neither won. Scoop up that bloody sac slathered with slime and mail it off with about $250 and it will be steamed, chopped, ground, powdered, and packed into tidy pill capsules for you to consume at your leisure.

There are photographs of the process. The only thing that would make this more unappetizing would be if Guy Fieri were involved.*

Bonus! The company that charges $250 will also shape the umbilical cord into a short script message (“love”), and dry it down into a hard, leathery, mummified sign the color of old dried blood that you can hang on the wall and terrify your offspring with for years to come. I really missed out on this opportunity.

By the way, these outfits have lots of anecdotes about feeling more “energized” and “peppy” after consuming these discarded scraps of their baby (for a more entertaining version of this myth, see the movie Ravenous), but there is actually no evidence that it provides any benefit. No benefit. None at all. Lots of ick, though. Probably no worse than chowing down on calf’s liver, though.


*Would it perverse of me to say I really want to see what Fieri would do with placenta as an ingredient?

That’s a Minnesota kind of story

If only we had video of this daredevil stunt.

By daylight, it was clear what happened: The man blew through the stop sign at a T in the road, barreled through a yard and launched his car off a 35-foot to 40-foot embankment, clearing a span of open water on Lake Le Homme Dieu. before landing on the season’s remaining ice, Armstrong said.

The man, James Sundby, 38, of Wadena, had no drugs or alcohol in his system and he doesn’t remember what happened, said Alexandria Police Chief Rick Wyffels.

The guy then staggered into a nearby stranger’s house, turned on all the lights and the TV, and proceeded to relax until he was chased away.

I’m just impressed with how far his car had to fly to clear that open water.

alexandrialeap

(Alexandria, by the way, is the next big (pop 9000!) city to the north of Morris — we go there fairly often. Now I’m feeling challenged to try the Sundby Leap, though.)

More math!

Chris Dixon has written an excellent history of mathematics. When most of us think of math, we go “ugh” and call it boring and turn away, but really, it’s so fundamental that we should be far more excited about it. Most of the major turning points in my education involved math: it was geometry when I was in the 8th grade that sparked my first interest, and learning algebra and logarithms in high school chemistry got me focused on science. When I started teaching myself how to program computers (I was an inadequate teacher, and quickly signed up for courses in the CS department), I had to also teach myself basic Boolean logic, because in those ancient days when your only recourse was to learn assembly language, and ANDs, NANDs, NORs, and ORs were the name of the game. Transistors are just logic implemented in silicon.

I agree when Dixon writes,

Mathematical logic was initially considered a hopelessly abstract subject with no conceivable applications. As one computer scientist commented: “If, in 1901, a talented and sympathetic outsider had been called upon to survey the sciences and name the branch which would be least fruitful in [the] century ahead, his choice might well have settled upon mathematical logic.” And yet, it would provide the foundation for a field that would have more impact on the modern world than any other.

I would add that in the 1970s public education system, we wouldn’t have imagined that, either. I had teachers who thought math was stuff you only needed to know for business school — you know, accounting. You can still see that attitude when people wonder why they need to learn this algebra stuff, anyway — they’ll never use it. They’re wrong. You’ll just use it in unexpected ways, because what you’re being given is a creative toolbox for thinking about the world.

The historical context in this article is useful, though, for making a case that math isn’t just practical, it’s also a foundation for thought that belongs in the liberal arts canon. And also that it’s a significant part of philosophy, which too many scientific pragmatists also tend to dismiss.

Sperm with a Ph.D.

I once knew someone who was a contributor to the Nobel Prize sperm bank. He wasn’t a Nobelist, but he was a smart and accomplished scientist — he just had a dewar of liquid nitrogen next to his bed, where he’d make an occasional deposit (with the assistance of his wife, he assured me), and then the samples would be shipped off for processing and…insemination, I presume. It’s not something I’d ever do, and apparently very few Nobelists actually contributed to it.

The Repository for Germinal Choice was a real thing (it’s been discontinued since the death of its founder, Robert Graham), kind of the last gasp of the scientific eugenics movement. It’s premise is typical crankery. I’ve met a fair number of Nobelists and big name scientists, and I’m sorry, they’re just people, with the usual range from nice people to total assholes — and actually, I suspect that they’re enriched for the nasty end of the scale. Scientist sperm plucked out of a vat of self-selected donors is probably actually less valuable than sperm hand-picked from a donor you know and like. Since this vat also contains sperm from notorious racist William Shockley, you’re probably best off avoiding it altogether. Also note: all of the donors were white, because of course they were, and oh no, insisted Graham, he was not a racist.

Anyway, one of the offspring of the Repository tracked down his biological “father”. The result was disappointing and troubling. I’m more troubled by the idea that people still think there’d be some great advantage to having an absentee father who had an advanced degree.

While the Repository is defunct, there are still individuals, like this one, who advertise their willingness to inseminate people. I’d also be worried…what if extreme narcissism is a heritable trait?