Papers in PDF format

(Uniform Resource Locators) are presented. The user can then select a document, which subsequently gets
retrieved from the appropriate server (e.g., HTTP, Gopher, FTP) at the provider site.
Post-retrieval Tools
The system is designed to support a wide range of post-retrieval tools to help users process information
after it is retrieved. The main services provided include document summarization and abstracting. These
services utilize a Part-of-Speech (POS) tagger based on the work of Eric Brill [Brill 93], which has been
modified by MITRE for better performance. Processing documents with a POS-tagger allows us to abstract and
summarize based on parts of speech. In the initial capability, a tagged document can be filtered for verb and
noun forms thereby reducing the word volume for the document. We eliminate common verb forms to avoid
biasing results. The remaining verb and noun forms can then be totaled to produce a vector for the document.
This vector can be used for clustering or passed forward to the summarization process. The summarization
process examines the verb and noun vector for the document and selects the most used words in the document.
Those words are then used to score each sentence of the document. The highest scoring sentences are then
presented as a summary in the order in which they appeared in the original document. The number of
sentences chosen is relative to the size of the document, but will not exceed a maximum threshold.
Experimental System
We have built a prototype MIDS to demonstrate our ideas and to evaluate and extend the system. The
system currently collects and processes approximately 20,000 documents from a number of servers at MITRE.
We currently have one Harvest Gatherer collecting information from three MITRE servers, four Harvest
Brokers providing the full-text search functionality, one GDS, and one BIS. Within the GDS classification
knowledge base, we have defined a number of fixed taxonomies covering the areas of business, government,
science, and education. Within the area of science, we have incorporated a subset of the ACM Computing
Reviews Classification System. [1]
Example Session
Our prototype system initially presents the top level screen shown in [Fig. 2] to the user. The screen
displays the high level topical categories as well as a list of tools including "Search" (full-text search). [Fig. 3]
shows a subset of the topics displayed as the result of selecting the "Science" topical area. The list box presents
the hierarchy of topics beginning with the selected root category. Indentation is used to show a topic, sub-topic
relationship. The number to the right of each leaf topic is the number of documents assigned to that area. A
user can select one or more topics from the list box, select the “View Documents” button, and receive a
breakdown of all documents in each area selected as shown in [Fig. 4]. The score associated with each
document in the list represents how similar the document is to the topical profile. Optionally, a user can
specify “List groups” under “Results Type” to receive a clustered listing of documents associated with selected
topics. [Fig. 5] shows a sample clustered view of documents. Each cluster of documents is preceded by a list
of the top discriminating terms, with the left-most terms being the highest discriminators. The score
associated with each document in a cluster, corresponds to how similar the document is to the discriminating
term list (cluster centroid). The user can then click on a document title to have it retrieved and displayed from
the provider site, with topical descriptors highlighted in the text (Fig. 6); or click on “Summary Information”
to see a summary of the document. The summary includes a list of other topics assigned to the document as a
result of the classification process, and an abstract of the document generated by processing it through the Brill
tagger software. [Fig. 7] shows a sample document summary. Within a topic list screen, a user can select the
“Search” function to perform a keyword search against all Harvest Brokers that manage information associated
with the selected topical area. The query routing service provided by the BIS determines the appropriate
Brokers to query, where results from different Brokers are combined into a single list similar to that shown in
[Fig. 4].
[1] The ACM Computing Classification System is Copyright (c) 1996 by the Association for Computing Machinery and is
included here with permission.