Intelligent Tutoring Systems for Ill-Defined Domains - Philippe ...

9 Walker, Ogan, Aleven & Jones
plate we provided and wrote their own feedback. While the support was underutilized,
it did appear to have a positive effect on the moderator posts. The 4 moderator
posts in this condition communicated elements of the model, like multiple perspectives,
prior personal experience, and good argumentation. For example, one moderator
chose to respond to the following post:
“Do you think that, at the end, he made a little joke about the word Arab? For
me, it's like he is commenting on the big problem of stereotypes of race and
religion in France; that all the Arabs have stores and work all the time”
This post was rated as being on-topic and having multiple perspectives, due to the
recognition of the stereotypes existing in France. The moderator replied:
“It is not a joke, it seems to me, but a stereotype that comes from the manner
in which the French think. Here, in the US, it's the people who call storekeepers
Koreans. Do you think for the same reason? Also, for the Catholics, Sunday
is a sacred day, for God, but the Arabs are a different religion. Is Sunday
also exceptional? If yes, why is it that they work?”
In the first line, the moderator acknowledged a conclusion the poster made and
clarified the point. The moderator then compared a US perspective to the French perspective,
using prior knowledge about the US perspective. The moderator also
brought up an important point for understanding the movie, which is that Sunday is a
sacred day for Catholics but possibly not for Muslims. Finally, the moderator asked
thought-provoking questions to keep the discussion going. Although the moderator’s
understanding of the situation was not perfect, the end result was an improvement in
the overall quality of the discussion.
Next, we attempted to look at the quality of the regular student posts before and after
they received feedback given by the moderators, a difficult task due to the low
number of students who posted. We divided posts into two categories based on student
exposure to moderator feedback. We placed posts that were made before any
moderator feedback appeared in the thread in one category, and posts that were made
after feedback appeared in the thread in a second category. Posts on the moderator
board on average across students rated fairly low at the outset, but had a trend towards
improvement after feedback from the moderator. While this result is initially
promising, more investigation is necessary due in part to the low number of participants.
6 Discussion
In this exploratory work, we compared the relative advantages and disadvantages of
two approaches for supporting discussion in an ill-defined domain. The first question
in an assessment of whether these interventions could be successful is whether the
feedback provider (human or computer) understands what is occurring in the discussion.
In the individual support condition, the ratings generated by the system matched
human raters in the majority of cases, even though the technology used to process the
post was relatively simple. However, the system only made ratings on a subset of the
dimensions. In the supported moderator condition, only one of the two moderators
used the ratings appropriately, but those ratings matched our codes for the posts. The