Intelligent Tutoring Systems for Ill-Defined Domains - Philippe ...

5 Discussion
Using Expert Decision Maps 78
The experimental group was able to view and interact with the expert decision maps
with minimal instruction and was better able than the control group to arrive at the
correct diagnosis. It should be noted, however, that the experimental group
outperformed the control group for Case 1 (before viewing any of the maps). It is
thus not clear how much of their superior performance can be directly attributed to
their interactions with the maps. Use of the maps did however seem to be associated
with increased confidence in a correct diagnosis. For correct diagnoses, the
experimental group became more confident over time, while the control group was
less confident for Case 3 than for Case 2. There was only one incorrect diagnosis
from an experimental group student for either Cases 2 or 3, which was submitted with
low confidence (.40), while there were 4 incorrect diagnoses from the control group
for these two cases. These 4 diagnoses were submitted with relatively high average
confidence levels (.75 for Case 2 and .72 for Case 3), suggesting that the control
group students were less able to assess their own performance. Further analysis of
how often students who submitted the 'incorrect' diagnosis updated their confidence
levels may assist us in identifying specific misconceptions, which can then be
addressed by providing appropriate feedback in the context of the activity.
The case questionnaire responses demonstrated a great deal of variability in the
students' baseline reflection and self-assessment abilities. Previous research has
shown that it is difficult for students to reliably assess their current level of
knowledge and skill [37, 38], but also that these capabilities can be developed with
appropriate training [39]. In general, the response to the case questionnaire after
Cases 2 and 3 show less evidence of reflective comments than the Case 1
questionnaire. We suspect that the combination of solving three cases and completing
three case questionnaires was overly ambitious within a two hour time period.
Additionally, minimal instruction was provided to the students as to what types of
self-reflective comments were expected. This issue is confounded, however, by the
fact that Cases 2 and 3 were perceived by both groups as markedly less difficult than
Case 1. Perhaps these students were not as motivated to perform self-reflection
activities when they were more confident in their performance. Further analysis of
the relationship between confidence, performance and quality of self-assessment may
help us to untangle these issues, as well as streamline our experimental design for
future studies.
The visual representations seemed to make some students in the experimental
group more aware of diagnostic reasoning as a process rather than an outcome.
Though there were number of non-responses to this question, this may be attributed to
the fact that the interaction with the expert decision maps was an additional task that
the experimental group performed, as compared to the control group. This task took
both additional time and required additional cognitive resources, both of which may
have impacted on the group's ability to provide evidence of ‘deep’ reflection within
the case questionnaire. In addition to communicating expectations to both groups
about the amount and level of reflection expected, future studies will need to ensure
that tasks for both conditions are designed to be equivalent in terms of both time and
difficulty. Further studies can then investigate how self-assessment and reflection can
be better scaffolded within the environment itself.