Intelligent Tutoring Systems for Ill-Defined Domains - Philippe ...

5 Walker, Ogan, Aleven & Jones
2.3 Assessing post quality
In order to develop an adaptive support system to deliver feedback based on this
model, we needed to assess where student posts required assistance. We explored two
methods of assessing post quality, both based on the model described above. First, we
attempted to design an individual support system to detect student performance on the
model dimensions. Because building an entire natural language processing system for
a new domain is very difficult and perhaps unnecessary, the goal for our prototype
was to use simple algorithms to rate the posts on the five dimensions. With only a
keyword analysis based on words which appeared in the corpus and were highly correlated
with human coding of the dimensions, we achieved reasonable accuracy
(kappa values between .5 and .9) on three of the five dimensions: on topic, good argument,
and multiple perspectives. We also took a first step towards assessing the
correct facts dimension of the model; the keywords were used to characterize the particular
topic the students were discussing, based on several pre-defined topics for the
assignment.
Because automated assessment is not as complete as human assessment can be, we
also explored the use of student moderators from the class to assess the discussion
posts. After reading the board and choosing a post to address, moderators were asked
to answer 4 yes/no questions, mapping to the first four dimensions of the model.
Moderators were also asked to select the topic of the facts that appeared in the post,
and indicate whether those facts were correct (see Figure 2). The moderator then
submitted the ratings to the moderator support system.
2.4 Providing Feedback
Figure 2. Posting interface for the moderator (translated into English)
We then used the assessment of the post to provide feedback to students, again exploring
the individual support and moderator support approaches. In the individual