Reading Comprehension: Reading, Learning, Teaching - Mitre
Reading Comprehension: Reading, Learning, Teaching - Mitre
Reading Comprehension: Reading, Learning, Teaching - Mitre
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
<strong>Reading</strong> <strong>Comprehension</strong>:<br />
<strong>Reading</strong>, <strong>Learning</strong>, <strong>Teaching</strong><br />
Lynette Hirschman<br />
781-271-7789 • lynette@mitre.org<br />
MITRE Sponsored Research<br />
© 2003, 2003 The MITRE Corporation
Problem<br />
� Can we define a new research paradigm for<br />
natural language processing by testing systems<br />
with measures for human comprehension?<br />
– <strong>Reading</strong>: Can we build a natural language<br />
processing system that can pass a 3rd grade<br />
reading comprehension test (> 65% score)?<br />
– <strong>Learning</strong>: Can we create a system that learns<br />
as it “reads” -- new words, concepts, facts?<br />
– <strong>Teaching</strong>: Can we create a reading<br />
companion that interacts with a student so<br />
that both system and student learn new<br />
things?<br />
© 2003, The MITRE Corporation
Background<br />
Climbers Reach Top of the World<br />
(Nepal, 1953) – Two mountain climbers have reached the top of the world’s tallest<br />
mountain. When they go to the top of Mount Everest, the two men had climbed<br />
29,028 feet.<br />
The men were helped by a team of climbers. First, they set up a base camp. This is<br />
where supplies are kept. Other camps were made higher up the mountain. Here, the<br />
climbers could get away from the wind. At times, snowstorms made it hard to climb.<br />
So they stayed in the camps until the weather got better.<br />
When the storms cleared, they moved on. Edmund Hillary was the best climber. He<br />
took along his guide, named Tenzing.<br />
After hours of fighting the snow and ice, they reached the top, called the summit.<br />
Eight teams of climbers tried and failed before them. But Hillary and Tenzing<br />
proved it can be done!<br />
1. Who reached the top of Mount Everest? Hillary and Tenzing<br />
2. What is another name for a mountain top? summit<br />
3. When did the climbers reach the top? 1953<br />
4. Where is Mount Everest? Nepal<br />
5. Why do you think it is so hard to reach the top? Because of bad weather and long<br />
distance<br />
<strong>Reading</strong> comprehension offers a new challenge<br />
and a human-centric evaluation paradigm for<br />
human language technology.<br />
What kinds of<br />
information are<br />
needed to<br />
answer<br />
questions?<br />
What makes<br />
some questions<br />
harder than<br />
others?<br />
Can humans<br />
guide machines<br />
toward<br />
understanding?<br />
The<br />
cat<br />
runs<br />
© 2003, The MITRE Corporation
Objective<br />
Establish an ambitious research agenda using<br />
“found” human test materials to test machines:<br />
� Level 1 <strong>Learning</strong> to read: pass reading<br />
comprehension tests of progressive difficulty<br />
� Level 2 <strong>Reading</strong> to learn: build a system that<br />
gains knowledge and improves performance<br />
through reading<br />
� Level 3 <strong>Teaching</strong> to learn: create humansystem<br />
team where the person and the system<br />
interact to provide improved performance<br />
© 2003, The MITRE Corporation
Activities<br />
�FY01-02: <strong>Reading</strong>, <strong>Learning</strong>, <strong>Teaching</strong><br />
�Interactive system and demo using<br />
reciprocal teaching (English,Spanish)<br />
�Architecture for “reading to learn” using<br />
linguistic info: parse, word sense, coreference<br />
�FY03:<br />
�Journal article (in preparation)<br />
“Methodology for the Analysis of Automated<br />
<strong>Reading</strong> <strong>Comprehension</strong> Test Taking”<br />
�Leading <strong>Learning</strong> Federation Workshop on<br />
Question Generation and Answering Systems R&D<br />
for Technology Enabled <strong>Learning</strong> Systems<br />
�Human-guided machine comprehension<br />
© 2003, The MITRE Corporation
Highlight: <strong>Teaching</strong> & <strong>Learning</strong><br />
ABCs processes a<br />
story<br />
ABCs interprets<br />
question, converting<br />
“who” to variable X<br />
ABCs tries to<br />
match (prove)<br />
question using story<br />
ABCs needs<br />
abduction for proof<br />
Who reached the top<br />
of Mt. Everest?<br />
Two mountain climbers<br />
reached the top…<br />
ABCs asks human Is a climber a kind of<br />
person?<br />
ABCs learns new<br />
fact & proof goes on<br />
reach(person(X), Y)<br />
& top(Y,‘mt_everest’)<br />
reach(climber(Z), V) &…<br />
is_a(climber(W), person(W))?<br />
Human answers: yes<br />
is_a(climber(W), person(W)).<br />
© 2003, The MITRE Corporation
Highlight<br />
Architecture<br />
Stories<br />
Questions<br />
Catalyst Language<br />
Processors<br />
Story<br />
Semantic<br />
Interpretation<br />
Question<br />
Semantic<br />
Interpretation<br />
Abductive<br />
Proof Procedure<br />
Knowledge<br />
Base Ontology<br />
Facts<br />
Rules<br />
Queries<br />
Processes<br />
Provided Data<br />
Derived Data<br />
Candidate<br />
Answers<br />
Answer<br />
Generator<br />
© 2003, The MITRE Corporation
Impacts<br />
� Released data sets for technology<br />
development and evaluation<br />
– in use at Edinburgh, Utah, Purdue, Brown,<br />
Texas<br />
� Developed methodology for the decomposition<br />
of reading comprehension system<br />
performance<br />
� Developed quantitative understanding of<br />
question difficulty and types of knowledge<br />
needed to answer questions<br />
© 2003, The MITRE Corporation
Future Plans<br />
<strong>Learning</strong> to<br />
read<br />
FY99<br />
FY00<br />
MSR Q3 FY98<br />
Stage 3: System+student<br />
form better translation<br />
team than either alone<br />
<strong>Reading</strong> to<br />
learn<br />
FY02<br />
<strong>Teaching</strong> to<br />
learn<br />
FY03<br />
FY04<br />
Stage 1: System<br />
passes 3rd grade<br />
Proposed MSR<br />
FY01<br />
FY05<br />
Stage 2: System reads<br />
a book & passes test<br />
© 2003, The MITRE Corporation
Change language