Bad Corrections, Hedges, and False Alarms in the Corrective Feedback of an ESL Grammar Checker

One distinction I like to make between the Virtual Writing Tutor and other online grammar checkers is that the Virtual Writing Tutor is an English second language grammar checker whereas all others are English first language grammar checkers. You might wonder what the difference is since good grammar is good grammar. For me, it is a matter of assumptions, frequency of occurrence, and catching more errors.

Bad Corrections

Whereas certain popular and well-advertised commercial grammar checkers will detect and flag verb form errors, they might not be able to propose the right correction because the wrong assumption has been made about the error. Here's how one very popular and well-developed online grammar checker responds to one of my student's sentences.

What has happened is that this grammar checker has used the auxiliary to guide the correction of the verb. Looking further into the sentence, it is clear that "am going" is a bad correction. This particular grammar checker seems to have assumed that the writer has full control over auxiliaries and simply used the wrong form for the participle. This assumption has led the developer of the error-detection rule to ignore the rest of the sentence and propose just one correction based the theory that the auxiliary is right but the lexical verb is in the wrong form.

There are other theories that might explain this error. My preferred theory for most errors is that a combination of chaos and the first language impose themselves upon the expression of the message in the second language. From my experience teaching English to French-speaking college students here in Quebec, learners lack control over both the participle and the auxiliary. They don't know when they need an auxiliary and they don't know how the auxiliary interacts with the lexical verb that follows it. When they don't know, they frequently turn to their first language for guidance.

Teachers familiar with the learner's first language are usually able to interpret the error and supply a correct form because of clues the learner provides. Learners tend to have a sense that if the event happened in the past, pastness needs to be conveyed either with the verb or with other words in the sentence like "yesterday" or "before." In many cases, how best to convey pastness remains a mystery for beginners and low-intermediates, so no assumption based one part of the verb or one part of the sentence is safe. In other words, as I attempt to write error detection rules for the Virtual Writing Tutor, the presence of an auxiliary won't always indicate how best to correct a verb form error.

In this case, a combination of chaos and first language word-for-word translation seems to be responsible for "I am went at New-York." French speakers might produce "I am went at New-York" simply by translating "Je suis allé à New York" word-for-word, plus a little confusion about what the past participle of "go" might be. Since students may be more familiar with the Past Simple form "went" and less familiar with the past participle "gone," the combination "I am went" would not surprise ESL teachers working in Quebec. The correction "I am going" would.

Grammar Checkers for ESL Teachers

Bad automatic corrections put ESL teachers off all automatic grammar checkers. One of my colleagues confessed to me that he has never used the Virtual Writing Tutor with his students simply because he cannot believe that a machine could ever be effective at correcting ESL writing. His limited experience with automatic correction has led him to believe that grammar checkers are terrible and only a human teacher can sort out the chaos of ESL writing.

I try not to press my grammar checker on my colleagues for two simple reasons: 1) I do not want to be a bore, and 2) I recognize that it is still very much a work in progress. Even so, I earnestly want teachers to incorporate automated quizzes and automatic grammar correction into their pedagogy because I want to free them from some of the drudgery of teaching ESL. I believe that the less time teachers spend on surface errors, the more time they can spend developing their own digital literacy and preparing really engaging lessons for their students.

A Work in Progress

I remain optimistic about the Virtual Writing Tutor because frankly the trend is good. My students can pop in a one-thousand word narrative that they have been working on and get 15-35 appropriate corrections in just a couple of seconds. If you consider that a teacher can correct a couple of surface errors in a minute, the time savings for an ESL teacher supplied with a free ESL grammar checker is potentially enormous. If my skeptical colleague spent just two minutes per student of surface error correction for each of his 150 students, that's two and half hours not spent reading students' papers for meaning and two and a half hours not spent developing greater proficiency with other pedagogical power tools

But who can blame the skeptics among us? Look at the effect of the bad correction advice on our learner's sentence. As you can see below, the corrected version of the learner's sentence based on the feedback offered by this popular grammar checker makes the sentence even less comprehensible than before.

Notice how no further errors are detected. Teachers who care about providing quality feedback to their students will roll their eyes and then roll up their sleeves to get back to correcting errors the old-fashioned way. When teachers lose confidence in automated grammar checkers, they won't introduce them to their students and one potentially powerful source of lifelong learning opportunities is denied to them. When students complete their ESL courses, their source of corrective feedback on grammar errors dries up and they may avoid writing in English in the future. Teachers need a better ESL grammar checker for their students for better instruction now and for lifelong learning in the future.

A Better ESL Grammar Checker

A better ESL grammar checker should avoid bad correction advice and detect more of the errors that learners make. Look how the Virtual Writing Tutor responds to the same sentence. It recognizes that it is the (#1) auxiliary that is the problem, not the lexical verb. Also, it detects an issue (#2) with "at New-York" and (#3) the hyphen in "New-York." It detects that (#4) two sentences have been joined with a coordinator but without a comma. Finally, it flags (#5) the use of the definite article before "shop." This last error is debatable.

Hedges

Since the Virtual Writing Tutor matches patterns at the sentence level in order to detect and correct errors, I can never be sure if the pattern and message I define for it will always anticipate the intended meaning of the author. False-alarms occur when an error-detection pattern matches a correct combination of words in an unanticipated context. When I suspect that a pattern could be correct, I hedge my bets by asking the writer to decide if the pattern has the intended meaning. I want to avoid giving such bad advice that it turns learners and their teachers off automated feedback forevermore. Correction #5 is one example of a hedge. I ask the learner to apply a rule "unless" dot, dot, dot.

Do not use the definite article before the word "shop" unless it is a shop you have mentioned before or there is only one shop, or it is a specific shop that everyone knows of. Did you mean "and I was lost in a shop"?. 

A human teacher wouldn't have to hedge in this situation. He or she would be able to look back at the sentences that came before to see if a particular shop had been mentioned earlier. The Virtual Writing Tutor can only look at the other words in the same sentence. Hedges that ask the writer to decide if there is only one shop or if there is a shop everyone can be expected to know runs the risk of making the feedback message overly complex for beginners to read and understand. Like bad corrections, overly complex feedback may put learners and teachers off also. Users can, however, try the Google Translation button when they feel really stuck.

Correct but Not Correct

Let's look at trickier problem for an ESL grammar checker. What do you do with a sentence which is grammatically correct but totally inappropriate for the learner's intended meaning? You hedge, of course. Here's an example of where I had to hedge. Consider the following three sentences: Our work place is a prison. It names Bordeaux Prison. It have three floors.

While this well-known grammar checker catches the conjugation error, in the context of the three sentences, a human should also recognize that "It names Bordeaux Prison" is not the right way to express the author's intended meaning in English. "It names" should be "it is called" or "it is named" even though the there is nothing wrong with the conjugation of the verb or pronoun choice. Ignoring this nonstandard phrase error is, in my opinion, a bad idea because it is an error an ESL teacher would ordinarily correct. 

Nevertheless, there are contexts in which "it names" makes sense. Checking Lextutor's concordance of a 14 million word corpus is encouraging because "it names" returns zero hits. See below. 

The standard phrase "it names" to express the meaning that a report names a particular individual or institution is relatively rare. But we can do better than even large corpora these days when looking for low-frequency phrases by doing a Google search. Here is the result of Google search with "it names" in quotes. Look what I found: a number of matches.

 While none of these instances of "it names" occurs at the beginning of a sentence they way it does in our learner text, restricting the error detection rule to only "It names" at the beginning of a sentence makes the rule less robust and there is no reason to believe that correct uses of "it names" could not also happen at the beginning of sentences. To avoid an outright false alarm in the future, I have chosen to hedge. 

Here is how the Virtual Writing Tutor ESL grammar checker detects, corrects, and hedges as it responds the errors in "Our work place is a prison. It names Bordeaux Prison. It have three floors."

Notice how correction #2 appropriately catches the non-standard phrase error. Also, notice how it is only by reading the preceding sentence "Our workplace is a prison" that we can be sure that the second sentence is indeed an error. Since the Virtual Writing Tutor cannot use words in other sentences to decide if a correction is warranted, a hedge is in order based on the assumption that "It names" has a higher probability of being used incorrectly in ESL writing. 

This is what the Virtual Writing Tutor says:

The phrase "It names Bordeaux Prison" will cause readers to pause and scratch their heads if you are using it to inform the reader of the name of someone or something. A more standard way is to say, ''It is called Bordeaux Prison''. The phrase "It names" can, however, be used in a sentence to tell the reader that an official report or legal judgement identifies a key individual. 

In my view, adding the hedge at the end should make the inevitable false alarm less upsetting to advanced ESL learners and their teachers. In the meantime,  this error detection rule can continue to catch the relatively high frequency inappropriate uses of "it names" + proper name in ESL writing.

Conclusion

So there you have it. I wanted to provide a little insight into the kinds of difficulties I face in writing error detection rules and correction messages. I also wanted to show how the Virtual Writing Tutor is aiming to become the best ESL grammar checker available by catching more errors--even if they are not strictly grammar errors--and how I am hedging my bets along the way. Finally, from time to time I like to reiterate my view that technology has the potential to liberate teachers from some of the time consuming drudgery of surface error correction and how a well-developed ESL grammar checker could help learners become lifelong learners of English writing. 

So what do you think? Leave a comment.