Papers in PDF format

Providers
Information
Servers
HTTP
Gopher
FTP
Gatherers
Meta-data
Extraction
SOIF
Gatherer
Dissemination
Service
Information
Classification
and
Dissemination
Brokers
Specialized
Information
Brokers
Broker
Information
Service
Abstraction
of Broker
Information
Space
Client
Interactive
Interface to
Information
Space
Figure 1: MITRE Information Discovery System (MIDS) Architecture
To augment the fixed taxonomy classification scheme, several clustering schemes (e.g., [Jain et al. 88]) are
under investigation to provide a topical decomposition of specific subject categories. After information gets
filtered into the fixed taxonomy, it is processed through a clustering algorithm which groups documents in
each topical area into related piles. This is done to provide a finer granularity of topics than is supplied by the
fixed taxonomy classification scheme. This is especially useful when a large number of documents are
assigned to a topical area as a result of the filtering process.
Information Organization
In addition to collecting and classifying information, the GDS distributes its processed information to a set
of Harvest Brokers and the BIS. Brokers register themselves with the GDS and have profiles, stored in a
knowledge base, that specify the topical categories of the kinds of documents to be received. Each time the
GDS classifies new information, Broker specific files containing SOIF records are written to an output queue
on secondary storage. Brokers periodically contact the GDS to receive new information and to index it. In
addition, information pertaining to objects and their classification categories is routed to the BIS.
Changes that occur to the document collections processed by the GDS, in terms of document deletions,
additions, and updates, are reflected throughout MIDS using a weak-consistency protocol [Downing et al. 90].
Since each MIDS subsystem is run only periodically (weekly), changes occurring to document collections are
not reflected until a new batch of information is processed through the system. After processing a new batch of
information, the GDS notifies Brokers and the BIS about any changes that had occurred so that they can
appropriately update their databases.
Information Discovery and Retrieval
Information discovery and retrieval in MIDS occurs through the client interface which interacts with
Harvest Brokers and BIS. The initial services provided by the system include topical browse, query routing,
and search. The BIS provides a topical browse service by managing the fixed taxonomy of topics as well as the
topics generated as a result of utilizing clustering techniques. The BIS also manages document summary
information, which had been generated as a result of the classification process in the GDS, and provides the
query routing capability by determining which Harvest Brokers to search as the user browses the topical
information space. This is accomplished by means of a table which lists the association between Brokers and
the topical categories they manage.
Harvest Brokers provide a full-text search capability, enabling a user to issue a query containing a Boolean
expression of keywords and then receive a list of documents.
All documents retrieved in the system reside at the information provider sites. Only metadata is processed
and stored within MIDS. After the user consults the system, a list of documents represented by titles and URLs