Intelligent Tutoring Systems for Ill-Defined Domains - Philippe ...

7 Walker, Ogan, Aleven & Jones
3 Preliminary Evaluation
3.1 Individual Support System
We evaluated the individual support system in a small pilot study in an upper level
French language class. Participants were 8 students who were relatively fluent in
French. The study took place over the course of a week, during which students
watched the film clip and posted to the discussion board. Students were asked to post
in French at least three times over the course of the week (specifically on a Monday,
Wednesday, and Friday), and to read the other posts before posting. The 8 students
posted a total of 25 times, which was slightly more than the 3 posts per student which
was required in the assignment. Posts on average were around 48 words long. Two
coders then rated all posts on the five dimensions of our model based on a coding
manual and resolved differences through consensus.
Our first research question was whether the feedback that students received from
the adaptive system was appropriate. It is clear that students needed feedback, as
posts submitted to the system were at a fairly low level. Almost half the posts were
missing all dimensions of good discussion other than being on-topic, and no post contained
all five dimensions. When students submitted their posts, the feedback provided
by the individual support system was fully relevant in 64% of cases, in that for
those posts the model generated an assessment that matched human ratings. For the
other cases, the feedback may not have been as accurate. However, feedback was
worded tentatively, so when the assessment was faulty it still appeared to offer a valid
suggestion.
The next step was to examine whether student posts tended to improve as a result
of receiving feedback (note that numbers here are not significant due to low sample
size in this exploratory pilot, but the trends reported warrant further study). Almost a
third of student posts improved their score on at least one dimension as a result of
their revisions, and several addressed the feedback directly in their posts. The most
improvement tended to be on the correct facts and good argumentation dimensions,
possibly because of the additional facts students received (see Table 1). Adding facts
can provide supporting evidence for a conclusion. For example, the following post
received appropriate feedback and improved on those two dimensions:
Version 1: There is tension because of the difference between cultures, but I
think that M Ibrahim wants to reduce the tension by being familiar with
Momo
Version 2: I think that there is tension between cultures because of the difference
in cultures, but M Ibrahim wants to reduce the tension by being familiar
with Momo. I don't think that the tension is because of hate, and this is explained
because Momo still does his shopping at M Ibrahim's
Another third of posts made non-trivial changes upon receiving feedback but did not
improve on the ratings, such as this post:
Version 1: What religion is Momo? Because he is named Moses