Papers in PDF format

Gatherers and Brokers. The Gatherer is a subsystem which collects metadata (indexing information) from a
set of providers (file servers). The Gatherer knows about different object types (e.g., HTML, text, PostScript)
and is able to parse and extract different metadata depending on the type of object being processed. A Broker
provides a search interface to information collected by one or more Gatherers or other Brokers. Harvest
provides much of the system-framework for handling distributed information. However, as discussed below,
the basic Harvest system does not address well requirements for content routing to multiple Brokers where
information cannot be a priori linked to a particular Broker, nor can it handle effectively query routing where
broker level metadata is needed to effectively assign user queries to appropriate Brokers. These areas and
others motivated a number of extensions to the basic Harvest system.
System Architecture
The MIDS system architecture is given in [Fig. 1]. The Gatherers are Harvest subsystems which extract
metadata from one or more providers (sources of information). The metadata format used, called the Summary
Object Interchange Format (SOIF) is based on a combination of the Internet Anonymous FTP Archives (IAFA)
IETF Working Group templates [Deutsch et al. 94] and BibTex [Lamport 86]. It has an attribute-value format
which is easily parsed and yet sufficiently expressive to handle many kinds of objects. The Gatherer
Dissemination Service (GDS) periodically collects streams of metadata in SOIF format from a set of Harvest
Gatherers. Using the collected metadata, the GDS provides a content routing capability that classifies the
documents associated with the metadata into a set of topical areas, and builds a set of files containing streams
of SOIF records for Brokers that had registered themselves with the GDS. It also disseminates its processed
information to Brokers and the Broker Information Service (BIS). Brokers are Harvest subsystems that provide
a full-text search capability for MIDS. Each Broker manages a set of topical areas and is periodically updated
with SOIF records from the GDS based on the information type it desires. The BIS provides a topical browse
service, enabling a user to retrieve documents based on topic selection. It also provides a query routing
[Sheldon et al. 94] capability by determining which Brokers to search for a particular topical area. The client
interface provides a user interface to enable effective navigation, browsing, and searching of information.
Information Collection
Information is collected into MIDS by utilizing Harvest Gatherers. The GDS contains a list of Gatherers
that it periodically contacts for streams of metadata. This information is stored in an input queue on disk for
later processing by the classification engine which is part of the GDS. For better network utilization
(bandwidth), it is more efficient to run Gatherers at each provider site, although Gatherers can access data
remotely as well. Running a Gatherer locally enables it to collect metadata by directly accessing documents via
file system I/O, and then have it ship one compressed file of its processed information to the GDS. Running a
Gatherer remotely requires that it obtain each document through the HTTP, Gopher, or FTP protocol, which
will incur a much greater performance penalty.
Information Classification
There are two methods used currently to classify information within the GDS. To provide for a tailorable
classification scheme, tools are provided for a knowledge engineer to define a knowledge base of classification
rules based on a fixed taxonomy of topics. The knowledge engineer will define the relationship between topics
of interest as well as descriptors used to map documents into the topical hierarchy. The classification engine
used to process information into the fixed taxonomy is a SIFT [Yan et al. 95] filtering engine, modified to
provide for a more expressive query language, and also to process streams of SOIF records as opposed to
regular full-text documents. SIFT's output processing logic was also modified to generate more classification
results, and also to generate Broker specific output files containing streams of SOIF records matching Broker
interest profiles.