CHAPTER 11: IMPROVING DECISION MAKING AND      MANAGING KNOWLEDGE

Case Study : SHOULD A COMPUTER GRADE YOUR ESSAYS?

Question 1 : How “intelligent” is automated essay grading? Explain your answer.

The ability to communicate in natural language has long been considered a defining characteristic of human intelligence. Furthermore, we hold our ability to express ideas in writing as a pinnacle of this uniquely human language facility—it defies formulaic or algorithmic specification. So it comes as no surprise that attempts to devise computer programs that evaluate writing are often met with resounding skepticism. Nevertheless, automated writing-evaluation systems might provide precisely the platforms we need to elucidate many of the features that characterize good and bad writing, and many of the linguistic, cognitive, and other skills that underlie the human capacity for both reading and writing. Using computers to increase our under standing of the textual features and cognitive skills involved in creating and comprehending written text will have clear benefits. It will help us develop more effective instructional materials for improving reading, writing, and other human communication abilities. It will also help us develop more effective technologies, such as search engines and questionanswering systems, for providing universal access to electronic information. A sketch of the brief history of automated writing-evaluation research and its future directions might lend some credence to this argument.

Question 2: How effective is automated essay grading ?

First, virtually every automated system generates scores that correlate with the ratings of human judges as closely as human judges agree with one another. The high correlations might reflect the interrelatedness of different elements in naturally occurring compositions writers who produce well-organized passages also use a rich vocabulary and carefully revise mechanics. Experiments with test passages in which the various elements are independently varied (that is, well organized but with poor mechanics, or strong vocabulary but with lots of misspellings) would show how the different systems in this issue respond to different elements.

Second, all three essays take for granted the reading–writing connection. In most non academic settings, this connection undergirds writing tasks. For example, when an engineer prepares an evaluation report on a new widget, she first learns about the widget, then outlines the report’s main points (often using a model), studies other documents for background, and finally prepares a draft. Reading and writing intertwine continuously. In school, however, traditions separate reading and writing in all but a few settings, mostly in the later grades and college-bound tracks.

Question 3 : What are the benefits of automated essay grading? What are the drawbacks?

          The two  benefits of automated essay grading are speed and cost. The e-rater program can score 16,000 essays in 20 seconds. Many of the AES programs are free or share revenues with the universities and colleges that use them.

The drawbacks include the inability of AES programs to distinguish fact from fiction. The programs put more emphasis on length of essays rather than quality. If the use of AES programs continue to proliferate, writing instruction may be dumbed down to meet the limited and rigid metrics machines are capable of measuring rather than increasing the quality of writing.

Another major drawback may come from the number of teaching positions that could be eliminated if the programs are more widely used on campuses or in online educational courses.

Question 4: What management, organization, and technology factors should be considered when deciding whether to use AES?

Management: professors, instructors, and teachers in high schools, colleges, and universities must have their say in how the programs are used, to what extent they are incorporated in courses, and whether the programs are allowed to supplant humans or supplement them. Students must be given the opportunity to learn proper writing skills and not be subject to a simple pass/fail system.

Organization: universities and colleges must ensure that faculty is given the greatest amount of consideration and not just the bottom line of profit and loss. Economic motives to use AES systems to cut costs must not outweigh the desire and requirement to provide students with first-rate quality instruction. Educational standards must continue to be more important than the bottom line.

Technology: limitations in the program construction must be recognized and taken into account when the systems are used. The knowledge base and inference engines must continually be improved and proper “learning” techniques must be included in the programs. 

Question 5: Would you be suspicious of a low grade you received on a paper graded by AES software? Why or why not? Would you request a review by a human grader?

         Will probably be suspicious of a low grade they receive on a paper graded by AES software because of the lack of human understanding that’s inherent in the programs. Human grader if they receive a low grade. The problem is actually on the flip side if they receive a high score from an AES program they will not request a review and will assume their writing is excellent when in reality it may not be.