Intelligent Tutoring Systems for Ill-Defined Domains - Philippe ...
Intelligent Tutoring Systems for Ill-Defined Domains - Philippe ...
Intelligent Tutoring Systems for Ill-Defined Domains - Philippe ...
Create successful ePaper yourself
Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.
5 Walker, Ogan, Aleven & Jones<br />
2.3 Assessing post quality<br />
In order to develop an adaptive support system to deliver feedback based on this<br />
model, we needed to assess where student posts required assistance. We explored two<br />
methods of assessing post quality, both based on the model described above. First, we<br />
attempted to design an individual support system to detect student per<strong>for</strong>mance on the<br />
model dimensions. Because building an entire natural language processing system <strong>for</strong><br />
a new domain is very difficult and perhaps unnecessary, the goal <strong>for</strong> our prototype<br />
was to use simple algorithms to rate the posts on the five dimensions. With only a<br />
keyword analysis based on words which appeared in the corpus and were highly correlated<br />
with human coding of the dimensions, we achieved reasonable accuracy<br />
(kappa values between .5 and .9) on three of the five dimensions: on topic, good argument,<br />
and multiple perspectives. We also took a first step towards assessing the<br />
correct facts dimension of the model; the keywords were used to characterize the particular<br />
topic the students were discussing, based on several pre-defined topics <strong>for</strong> the<br />
assignment.<br />
Because automated assessment is not as complete as human assessment can be, we<br />
also explored the use of student moderators from the class to assess the discussion<br />
posts. After reading the board and choosing a post to address, moderators were asked<br />
to answer 4 yes/no questions, mapping to the first four dimensions of the model.<br />
Moderators were also asked to select the topic of the facts that appeared in the post,<br />
and indicate whether those facts were correct (see Figure 2). The moderator then<br />
submitted the ratings to the moderator support system.<br />
2.4 Providing Feedback<br />
Figure 2. Posting interface <strong>for</strong> the moderator (translated into English)<br />
We then used the assessment of the post to provide feedback to students, again exploring<br />
the individual support and moderator support approaches. In the individual