How peer review works

Even if you’ve never been involved in scientific research, you’re probably aware that it involves a process called “peer review”. I want to take a minute to explain how this actually works. This is based on my personal experience, although I think much of it generalizes to other academic fields, including those outside of science.

1. Sending to referees

It starts with the submission of a manuscript to a journal. A lot of work has already gone into the manuscript, including input from collaborators and colleagues, but this is where peer review formally begins.

The journal assigns the manuscript to an editor, and then the editor chooses a few (usually 3) referees to look at the paper. Now, choosing referees can be quite difficult, because they need to be close enough to the field that they can understand and critique the manuscript. In fact, it’s common for referees to decline, because they think the manuscript is too far outside their field. And yet, referees can’t be so close that they’re direct competitors. Authors typically provide a list of competitors to the editor to avoid conflict of interest (or even worse, theft of ideas). But editors aren’t required to follow this advice, and authors never know because they don’t know the names of the referees.

[Read more…]

Paper: Attack of the psychometricians

Suppose that you want to demonstrate that baby boomers are more narcissistic than other generations, or that women are more agreeable and neurotic than men, or that people of different races have different amounts of intelligence. How do psychologists do that? Can they in fact do that?

Typically, the method is to come up with a bunch of questions that superficially appear to measure the intended characteristic. Then the questions are “validated”, for example, by making sure the questions all correlate with one another. Once the questionnaire is declared valid, psychologists can then measure a variety of different groups and make far-reaching claims about how our current political/social situation was caused all along by the thing that they happen to study.

If you find this methodology questionable, but aren’t sure exactly what went wrong, you might be interested in hearing about psychometrics, the field concerned with psychological measurement. According to psychometricians, part of the problem is that psychologists are failing to follow best practices. That is the subject of this paper:

Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71(3), 425–440.

[Read more…]

Christian Doubt

This is a repost of an article from 2014. Usually I like to repost articles that are related to my recent topics, but this is unrelated and just for fun.

When I grew up in Catholicism, I was never taught to think that doubt was a bad thing.  In fact, doubt was a good thing, ennobling even.  Doubts were something that everyone experiences.  Why then, is it said that Christianity is all about faith, dogma, and purging all doubt?  Where does this image come from?

Let me tell you what happened next.  I started doubting Catholicism.  And even though I was never taught that doubting was bad, I knew that the particular way I was doing it was bad.

What I was doing was reading on some arguments against Catholic beliefs, comparing them to the arguments for it.  I knew that changing my mind on so many things all at once was impossible, so I considered each issue independently, one at a time.  I worried about the consequences of deciding one way or the other, but I tried not to let that affect my judgment.  Finally, I collected my many thoughts and tried to draw some overall conclusions on Catholicism and God.

In my mind, this is more or less the proper way to deal with doubt, so why did I know in my gut I was running afoul of some rule of my religious upbringing?  The truth is that doubt was accepted in the Catholicism I grew up in, but only if the doubt fit into a specific narrative.  Doubt was not an epistemological tool, but a personal struggle to be overcome.  This is a fundamentally negative depiction of doubt.

[Read more…]

Boghossian is no Sokal

Peter Boghossian and James Lindsay (henceforth B&L) have an article titled “The Conceptual Penis as a Social Construct: A Sokal-Style Hoax on Gender Studies” on Skeptic.com.  The article describes a nonsense paper submitted, and accepted, to the journal Cogent Social Sciences.  The authors consider it an indictment of gender studies and pay-to-publish journals.

This being a Sokal-style hoax, it’s worth recapping some of the strengths and weaknesses of the original Sokal hoax.  First the weaknesses:

  • Sokal’s paper was accepted to The Social Text, which is a journal of only mediocre impact.
  • Peer review isn’t intended to weed out bad faith actors, but to enforce some minimum standard.  The real test is later, when the academic community cites (or ignores) the publication.
  • Sokal only had N=1. Distinguishing between good and bad papers is in general a difficult problem, and one expects that in the perfect balance, some good papers would be rejected, and bad papers accepted.

Now the strengths:

  • The Sokal hoax is immediately compelling to general public, even when people don’t look into the details.  There’s value in bringing the issue to popular attention.
  • When I did look into the details last year, I found the paper’s content to be a damning indictment of the entire field:

    It’s not simply that Sokal liberally salts his article with absurdities, it’s that he quotes plenty of postmodern academics doing the same damn thing.

    Even if Sokal’s paper were rejected, one would have to account for all the nonsense already published and respected within the field.

  • There was a clear way that The Social Text could have avoided being hoaxed, if anywhere in the review process they had asked someone in physics, biology, or math to glance at it.

B&L’s and  attempt at a hoax falls short of Sokal, having worse weaknesses, and missing important strengths.

[Read more…]

If you think there’s no atheist movement, tell me why

One of my pet peeves is when people say that there is no atheist movement. At many times, I’ve reacted angrily to the suggestion. I don’t understand how anyone could believe that, especially when I hear it from people who are involved in, or interact with atheist organizations.

Dear readers, help me understand. If you don’t believe there is an atheist movement, please explain your thinking in the comments. I will listen, and as long as you are polite to me I will be polite to you, setting pet peeves aside.

Here are some questions which you may use to guide your responses:

[Read more…]

Maybe literary fiction is just bad fiction

Earlier I mentioned this Gamasutra article which I disliked, and one thing I disliked about it was its discussion of literary fiction. According to the author, literary fiction was the epitome of cultural elitism, defining itself as simply better than “genre fiction”.

As someone who likes literary fiction and dislikes genre fiction, the discourse around literary fiction constantly annoys me. Hey, maybe I just like it because I like it, not because I think I’m better than you. Why does it always have to be about elitism? Why can’t it just be about differing tastes?

I am also complaining as an aspiring author of literary fiction. I do not consider myself to be very good at writing fiction. I have barely made it into writing the novel I started three years ago. I’ve encountered two obstacles: First, nobody I know likes to talk about literary fiction, so I don’t get the ideas I need. Second, there’s an expectation that literary fiction is “good” and that it’s hard to write. At this point I’m writing it for myself and don’t care if it’s good–why can’t I write “bad” literary fiction, what makes people think that’s a contradiction in terms?

But I realize that the elitist image of literary fiction often comes from lovers of literary fiction themselves. I wish to turn that on its head, by reframing literary fiction as bad fiction, bad fiction that I happen to like.

[Read more…]

Linkspam: May 12th, 2017

This is my monthly linkspam, which includes some articles I found interesting, and some brief commentary.

Video games are better without stories – Ian Bogost argues that video games just aren’t as good at storytelling as movies or literature.  Currently the best stories seem to be told through environmental storytelling, which falls quite short of the interactive storytelling often envisioned.  This is a point I’m sympathetic to, having played several games that were widely praised for their story, and being frankly disappointed at how poorly they compare to even mediocre literature.

As far as my personal calculations go, it’s not so much a question of what the medium of video games is capable of. To me, the question is, “Is it economically feasible for the medium to produce good stories that go beyond mainstream tastes?”  For me, movies fail this test because either they need to sell to a broad audience (which generally doesn’t include me), or they need to be poorly produced.  Video games are also in a bad position, but they may get better as development costs go down.

I found this link via Critical Distance, which paired Ian Bogost’s article with several critiques.  To be honest, I was really put off by that Gamasutra article, which started out by complaining that this whole debate had been dismissed by academics since 2005, and then immediately followed this with the complaint that Ian Bogost was being elitist.  Games critics, why are you bad?  Patrick Klepek makes good points though.

[Read more…]