Intelligent Tutoring Systems for Ill-Defined Domains - Philippe ...

Recommendations

Info

What Do Argument Diagrams Tell Us About Students’ Aptitude Or Experience? A Statistical Analysis In An Ill- Defined Domain∗ Collin Lynch 1 , Niels Pinkwart 2 , Kevin Ashley 1,3 and Vincent Aleven 4 1 University of Pittsburgh, Intelligent Systems Program, Pittsburgh, PA, USA 2 Clausthal University of Technology, Department of Informatics, Germany 3 University of Pittsburgh, School of Law, Pittsburgh, PA, USA 4 Carnegie Mellon University, HCI Institute, Pittsburgh, PA, USA Abstract. In ill-defined domains, argumentation skills are essential in order to define problems and to present, justify, and evaluate solutions. In well-defined domains there exist accepted methods of characterizing student arguments as good or bad. This is not always possible in ill-defined domains, where competing arguments are often acceptable. In this paper, we use a set of statistical analysis methods to investigate whether, despite the lack of an “ideal solution,”, student-produced argument diagrams can be diagnostic in that they can be used to reliably classify students into novices and experts or high and low aptitude. Our analysis, based on data collected during three studies with the LARGO ITS, suggests that indeed, argument graphs created by different student populations differ considerably, particularly with respect to the completeness and “connectedness” of graphs, and can thus potentially be used to adapt the system to a particular student’s needs. Keywords: ill-defined domains, assessment, feedback, graph comparison 1 Introduction Argumentation is a fundamental mode of reasoning and analysis in many domains, such as law, ethics, public policy, and science, that typically involve illstructured problems. Ill-structured problems are characterized by an incompletely stated goal, relevant constraints not stated in the problem, and competing, possibly inconsistent, reasonable solutions [19]. By contrast, well-structured problems have clearly stated goals and a strong well-supported domain theory that may be used to validate solutions [19]. In addressing ill-structured problems, argumentation is essential in order to define the problem and to present, justify, and evaluate solutions. The solver must frame the problem, refining the goal and inferring constraints [5]. Solvers may frame ∗ This research is supported by NSF Award IIS-0412830.
57 Lynch, Pinkwart, Ashley, & Aleven the problem in different ways depending on their knowledge, values, and interests, and often there is less consensus on the “right” way to frame the problem [19]. A solution “usually is justified by verbal argument that indicates why the solution will work, and provides a rebuttal by attacking a particular constraint or barrier to the solution or by attempting to refute an anticipated opposing position.” [19]. Argumentation skills are therefore essential for students to learn how to address illstructured problems not only in domains like law or ethics. Although mathematics and science are typically taught using well-structured problems, when it comes to discovering new knowledge, the problems are ill-structured. Skilled mathematicians and scientists engage in argument schema that are similar to those used by attorneys or ethicists, such as evaluating a proposed definition or decision rule by testing how it works on hypothetical examples as described below [6,7]. Even in solving a wellstructured problem, a skillful student can state an argument to justify his solution approach. One expects such arguments to be more uniform, however, because the goal and constraints are stated in the problem and there is more of a consensus about how to proceed. Increasingly, argument diagrams are used to help students acquire argumentation skills. Argument diagrams are graphical formats that reify aspects of the arguments' structure [1]. They can be a simple tree, whose root contains the argument’s conclusion, each child of which can be supported by additional nodes that represents an argument premise. [13]. Often, argument diagrams are based on Toulmin’s model of argumentation in which the argument structure represents data moving through a warrant to support a claim [15]. As described in [13] and at http://www.phil.cmu.edu/projects/argument_mapping/, other schemes for diagramming arguments have been developed, some of which have been employed in teaching. The use of argument diagramming appears to be most widespread in courses on critical thinking and in philosophy [4,16,18]. To a lesser extent, they have been applied in teaching legal and scientific reasoning [2,11,14]. Students use them to represent their own arguments or to reconstruct arguments in some source text [18] such as, in our present research, transcriptions of oral arguments before the U.S. Supreme Court. Whatever the diagramming scheme, software is often used to assist students in constructing the diagrams [4,13,18]. Initially, the diagrams served primarily as representational tools; the important thing was the diagramming process and its impact on learning, not the details of the resulting diagrams. In a software environment, however, the diagrams can also convey diagnostic information about the student's understanding. A small but increasing number of intelligent tutoring systems (ITS) focus on argumentation skills, employ diagrams, and even evaluate studentgenerated diagrams in order to provide feedback [11,14]. Automatically evaluating argument diagrams for providing feedback necessarily presents a challenge, especially when problems are ill-structured. For reasons discussed above, arguments supporting correct solutions to a well-structured problem are unlikely to diverge widely. The problem constraints are clearly specified, and the solution is deterministic and may be derived by more or less algorithmic means. As a result, such arguments can take relatively few plausible paths and one would expect to see detailed similarities across the alternatives (e.g., invoking the same physics
Page 1 and 2:
Intelligent Tutoring Systems for Il
Page 3 and 4:
iii Aleven, Ashley, Lynch & Pinkwar
Page 5 and 6:
v Ashley Intelligent Tutoring Syste
Page 7 and 8:
vii Ashley Methods for Assessment a
Page 9 and 10:
Two Approaches for Providing Adapti
Page 11 and 12:
3 Walker, Ogan, Aleven & Jones 2 Ad
Page 13 and 14: 5 Walker, Ogan, Aleven & Jones 2.3
Page 15 and 16: 7 Walker, Ogan, Aleven & Jones 3 Pr
Page 17 and 18: 9 Walker, Ogan, Aleven & Jones plat
Page 19 and 20: 11 Walker, Ogan, Aleven & Jones ond
Page 21 and 22: Interactive Narrative and Intellige
Page 23 and 24: 15 Hodhod & Kudenko components, con
Page 25 and 26: 17 Hodhod & Kudenko actions and re-
Page 27 and 28: 19 Hodhod & Kudenko dollars. That S
Page 29 and 30: 21 Hodhod & Kudenko deeper understa
Page 31 and 32: 23 Pino, Heilman, & Eskenazi with a
Page 33 and 34: 25 Pino, Heilman, & Eskenazi 2 Cloz
Page 35 and 36: 27 Pino, Heilman, & Eskenazi Fig. 1
Page 37 and 38: 29 Pino, Heilman, & Eskenazi Table
Page 39 and 40: 31 Pino, Heilman, & Eskenazi Table
Page 41 and 42: 33 Pino, Heilman, & Eskenazi opinio
Page 43 and 44: Generating and Evaluating Object-Or
Page 45 and 46: 37 Moritz & Blank using them to der
Page 47 and 48: 39 Moritz & Blank searching for phr
Page 49 and 50: 41 Moritz & Blank as a related term
Page 51 and 52: 43 Moritz & Blank to match on data
Page 53 and 54: 45 Moritz & Blank 6. Liu, H., ieber
Page 55 and 56: 47 Fournier-Viger, Nkambou, & Mephu
Page 63: 55 Sequential Pattern Mining/Partia
Page 67 and 68: 59 Lynch, Pinkwart, Ashley, & Aleve
Page 77 and 78: 69 Gauthier, Naismith, Lajoie, & Wi
show all

Intelligent Tutoring Systems for Ill-Defined Domains - Philippe ...

Create successful ePaper yourself

Delete template?

Save as template?