CHAPTER 11: IMPROVING DECISION MAKING AND MANAGING KNOWLEDGE
Case
Study : SHOULD A COMPUTER GRADE YOUR ESSAYS?
Question
1 : How “intelligent” is automated essay grading? Explain your answer.
The ability to communicate in natural
language has long been considered a defining characteristic of human
intelligence. Furthermore, we hold our ability to express ideas in writing as a
pinnacle of this uniquely human language facility—it defies formulaic or
algorithmic specification. So it comes as no surprise that attempts to devise
computer programs that evaluate writing are often met with resounding
skepticism. Nevertheless, automated writing-evaluation systems
might provide precisely the platforms we need to elucidate many of the features
that characterize good and bad writing, and many of the linguistic, cognitive,
and other skills that underlie the human capacity for both reading and writing.
Using computers to increase our under standing of the textual features and
cognitive skills involved in creating and comprehending written text will have
clear benefits. It will help us develop more effective instructional materials
for improving reading, writing, and other human communication abilities. It
will also help us develop more effective technologies, such as search engines
and questionanswering systems, for providing universal access to electronic
information. A sketch of the brief history of automated writing-evaluation
research and its future directions might lend some credence to this argument.
Question 2: How effective is automated essay grading ?
First,
virtually every automated system generates scores that correlate with the
ratings of human judges as closely as human judges agree with one another. The
high correlations might reflect the interrelatedness of different elements in naturally
occurring compositions writers who produce well-organized passages also use a
rich vocabulary and carefully revise mechanics. Experiments with test passages
in which the various elements are independently varied (that is, well organized
but with poor mechanics, or strong vocabulary but with lots of misspellings)
would show how the different systems in this issue respond to different
elements.
Second,
all three essays take for granted the reading–writing connection. In most non academic
settings, this connection undergirds writing tasks. For example, when an
engineer prepares an evaluation report on a new widget, she first learns about
the widget, then outlines the report’s main points (often using a model),
studies other documents for background, and finally prepares a draft. Reading
and writing intertwine continuously. In school, however, traditions separate
reading and writing in all but a few settings, mostly in the later grades and
college-bound tracks.
Question 3 : What are the benefits of automated essay
grading? What are the drawbacks?
The two benefits of automated essay grading
are speed and cost. The e-rater program can score 16,000 essays in 20 seconds.
Many of the AES programs are free or share revenues with the universities and
colleges that use them.
The drawbacks include the inability
of AES programs to distinguish fact from fiction. The programs put more
emphasis on length of essays rather than quality. If the use of AES programs
continue to proliferate, writing instruction may be dumbed down to meet the
limited and rigid metrics machines are capable of measuring rather than
increasing the quality of writing.
Another major drawback may come from
the number of teaching positions that could be eliminated if the programs are
more widely used on campuses or in online educational courses.
Question 4: What management,
organization, and technology factors should be considered when deciding whether
to use AES?
Management: professors, instructors, and teachers in high schools, colleges,
and universities must have their say in how the programs are used, to what
extent they are incorporated in courses, and whether the programs are allowed
to supplant humans or supplement them. Students must be given the opportunity
to learn proper writing skills and not be subject to a simple pass/fail system.
Organization: universities and colleges must ensure that faculty is given the
greatest amount of consideration and not just the bottom line of profit and
loss. Economic motives to use AES systems to cut costs must not outweigh the
desire and requirement to provide students with first-rate quality instruction.
Educational standards must continue to be more important than the bottom line.
Technology: limitations in the program construction must be recognized and
taken into account when the systems are used. The knowledge base and inference
engines must continually be improved and proper “learning” techniques must be
included in the programs.
Question 5: Would you be suspicious of a low grade you received on a paper graded by AES
software? Why or why not? Would you request a review by a human grader?
Will probably be suspicious of a low grade they receive on a paper
graded by AES software because of the lack of human understanding that’s
inherent in the programs. Human grader if they receive a low grade. The problem is actually on the flip
side if they receive a high score from an AES program they will not request a
review and will assume their writing is excellent when in reality it may not
be.