RAVITCH ON ROBO-GRADING

EducationViews.org; Houston, Texas

Robo-Grading Student Essays is Garbage

July 15, 2018 by Diane Ravitch

—As someone who devotes her life to writing and trying to communicate meaning, I reject the idea of robo-grading as insulting to the craft. It demands garbage. It deserves to get what it asks for. Anyone complicit in this betrayal of educational values should be ashamed...

NPR ran an interesting story about robo-grading student essays. It didn’t use the headline that appears here, but there is no better way to describe the insanity or stupidity of asking a machine to judge a student essay. First, it shows disrespect for the student. Second, it diminishes the importance and value of language. Third, the machine can be easily fooled, as Les Perelmann of MIT has demonstrated conclusively with his studies of how easy it is to fool the machine. Basically, the machine can’t evaluate facts. Perelman showed that a student could write an essay declaring that the War of 1812 took place in 1945, and the machine would not recognize it as an error.

To demonstrate, he calls up a practice question for the GRE exam that’s graded with the same algorithms that actual tests are. He then enters three words related to the essay prompt into his Babel Generator, which instantly spits back a 500-word wonder, replete with a plethora of obscure multisyllabic synonyms:

History by mimic has not, and presumably never will be precipitously but blithely ensconced. Society will always encompass imaginativeness; many of scrutinizations but a few for an amanuensis. The perjured imaginativeness lies in the area of theory of knowledge but also the field of literature. Instead of enthralling the analysis, grounds constitutes both a disparaging quip and a diligent explanation.

“It makes absolutely no sense,” he says, shaking his head. “There is no meaning. It’s not real writing.”

But Perelman promises that won’t matter to the robo-grader. And sure enough, when he submits it to the GRE automated scoring system, it gets a perfect score: 6 out of 6, which according to the GRE, means it “presents a cogent, well-articulated analysis of the issue and conveys meaning skillfully.”

“It’s so scary that it works,” Perelman sighs. “Machines are very brilliant for certain things and very stupid on other things. This is a case where the machines are very, very stupid.”

Because computers can only count, and cannot actually understand meaning, he says, facts are irrelevant to the algorithm. “So you can write that the War of 1812 began in 1945, and that wouldn’t count against you at all,” he says. “In fact it would count for you because [the computer would consider it to be] good detail.”

Even human readers, who may have two minutes to read each essay, would not take the time to fact check those kind of details, he says. “But if the goal of the assessment is to test whether you are a good English writer, then the facts are secondary.” 
 
Perelman says his Babel Generator also proves how easy it is to game the system. While students are not going to walk into a standardized test with a Babel Generator in their back pocket, he says, they will quickly learn they can fool the algorithm by using lots of big words, complex sentences, and some key phrases—that make some English teachers cringe.

“For example, you will get a higher score just by [writing] “in conclusion,’” he says….

In places like Utah, where tests are graded by machines only, scampish students are giving the algorithm a run for its money.

“Students are geniuses, and they’re able to game the system,” notes Carter, the assessment official from Utah.

One year, she says, a student who wrote a whole page of the letter “b” ended up with a good score. Other students have figured out that they could do well writing one really good paragraph and just copying that four times to make a five-paragraph essay that scores well. Others have pulled one over on the computer by padding their essays with long quotes from the text they’re supposed to analyze, or from the question they’re supposed to answer.

But each time, Carter says, the computer code is tweaked to spot those tricks.

“We think we’re catching most things now,” Carter says, but students are “very creative” and the computer programs are continually being updated to flag different kinds of ruses.

—As someone who devotes her life to writing and trying to communicate meaning, I reject the idea of robo-grading as insulting to the craft.

It demands garbage. It deserves to get what it asks for.

Anyone complicit in this betrayal of educational values should be ashamed.

Source: Robo-Grading Student Essays is Garbage | 
Diane Ravitch’s blog