A Grammar Checker Fails the Test

The Virtual Writing Tutor Bombs Badly

A fellow ESL teacher recently sent me this message on Facebook about the VirtualWritingTutor, an English grammar checker for ESL learners.

I tried the VirtualWritingTutor with a text I ask my students to edit (English 100). It missed a lot of mistakes - especially with the present continuous verb forms. I think that you'll need to add even more error detection rules! 

Here's the text I tried out:

You have a pet? I do. I have a cats. It have five year old. I buy the animal for my childs. My son and daughter are very happy when they got Tommy. Tommy is the cat’s name. My son love the cat very much. He brings the cat in her bed at night and the cat sleep there. He is feeding the cat every day. Sometimes my daughter feed the cat. Last week, the cat go outside and don’t come back for two day. We not closing the door. Usually, we always close the door. This weekend, my friend is feed the cat because we going on a trip. We go at New York.

The Virtual Writing Tutor--a work in progress

Here was my reply

Thank you very much for this. Obviously, there are many more rules to write. The rules I am writing are based on authentic errors written by my own Francophone 100B and 101A students and based on the two student corpora on Lextutor.com . For some reason, the errors teachers submit to test the system almost always catch it off guard. I am not sure if it is how teachers reformulate errors from memory, the fact that different writing tasks produce a different range of errors, or just the sheer chaos of L2 writing.

In response to the text you posted, I have added a couple dozen rules to the system. Instead of catching 4 errors and throwing up false alarms, the system will now catch 17 errors in your text. Some errors I won’t try to catch simply because they involve understanding sentences that come before and after. This system is set up to look for overt sentence-level errors.

New Error Detection Rules Added

You have a pet?

I added a rule to catch the lack of an auxiliary with “You have a pet?” and a second rule for “You have pets?”

It have five year old.

I had a rule to catch “It have five years old” and many variations but not “it have five year old.” I modified the rule to catch the extra variation.

I buy the animal for my childs.

I created an exception for Childs in the proper name capitalization rule, reasoning that high frequency pluralization errors are more of a problem than low frequency family name capitalization errors. I will have to create a separate rule for the family name Childs. I created a rule to catch “childs” with the exception that it should be ignored if “childs” appears before another noun. In such a case, it will assume that the learner needs a possessive form. I won’t try to catch the “I buy” tense error. it is too broad of an assumption to expect all storytelling to use the past tense when dramatic stories can use the present simple also. More to the point, I have been focusing on sentence-level errors only.

My son and daughter are very happy when they got Tommy.

I created a rule to compare the verb at the beginning of the sentence with the verb at the end of the sentence. I have 36 of these rules so far. I’ll need many more, I figure.

My son love the cat very much.

I created an additional condition for the 3rd-person rule. It is easier to teach than to program because of the danger of false alarms. For one, you have to add an exception for the subjunctive. Also, we need an exception for all past forms, past participles, modals and infinitives that look like present simple forms.

I fixed the adjective order false alarm for “cat very much” by adding an exception that looks for a determiner before “cat”.

He brings the cat in her bed at night and the cat sleep there.

Now the system catches the third person singular error.

He is feeding the cat every day.

The system was able to catch “Every day, he is feeding the cat”. It is easier to program the system to catch aspect errors when the adverb of frequency is at the beginning or in the middle of the sentence. I have nevertheless added the new rule.

Sometimes my daughter feed the cat.

The reason the system didn’t catch “feed” is that it is also the low-frequency but nevertheless past tense of verb “to fee” which is excluded as being potentially correct. I had to create a separate rule for “feed”.

Last week, the cat go outside and don’t come back for two day.

This one was tough because of the two verb errors, but I have cracked it. I’ll need to develop more rules along this line.

This weekend, my friend is feed the cat because we going on a trip.

This was helpful for me. I set up a new group of rules to deal with future plans and arrangements. The second part was simply a case of catching a missing auxiliary after “because,” a case that is very unlikely to be a question with auxiliary-fronting

Stumping the Virtual Writing Tutor

Do you have any error-correction activities or sentences that stump the VirtualWritingTutor? If you do, please share them by adding them in a comment below. I will certainly add any missing rules in short order.