There has been much discussion recently about computer software that seems to be able to grade students high school essays about as well as humans, but of course much faster.
[Human] Graders working as quickly as they can — the Pearson education company expects readers to spend no more than two to three minutes per essay— might be capable of scoring 30 writing samples in an hour.
The automated reader developed by the Educational Testing Service, e-Rater, can grade 16,000 essays in 20 seconds.
Sounds good, right? Except for course for those poor people who are trying to make a few extra bucks by doing the soul-killing job of assessing the quality of a student essay in just two minutes.
But there’s a catch.
A director of writing at MIT Les Perelman says that because these robo-graders work according to an algorithm, it is not hard to find out what it values and thus beat the system. He found that if you write long essays with big words, even if they are nonsensical, you will score high. The algorithm does not like short sentences or paragraphs or sentences that begin with ‘and’ or ‘or’ nor is it enamored of sentence fragments. In other words, all the little rules that good writers will break to create a particular effect will cause your essay to be marked down.
Perelman gives an example of how you can get a high score. The most interesting feature of the algorithm is that it doesn’t care about substance or even truth. It will ignore such trivialities as saying that the war of 1812 began in 1945, provided you say it grammatically.
The substance of an argument doesn’t matter, he said, as long as it looks to the computer as if it’s nicely argued.
For a question asking students to discuss why college costs are so high, Mr. Perelman wrote that the No. 1 reason is excessive pay for greedy teaching assistants.
“The average teaching assistant makes six times as much money as college presidents,” he wrote. “In addition, they often receive a plethora of extra benefits such as private jets, vacations in the south seas, starring roles in motion pictures.”
E-Rater gave him a [top score of] 6. He tossed in a line from Allen Ginsberg’s “Howl,” just to see if he could get away with it.
This got me thinking that the e-Raters are perhaps reflecting the zeitgeist because as I said in an earlier post, for some time we have been living a fact-free era.