Papers in PDF format

Introduction
MITRE Information Discovery System
Daniel J. Helm, Raymond J. D'Amore, Puck-Fai Yan
Digital Libraries Group
The MITRE Corporation
1820 Dolley Madison Blvd., McLean, VA 22102-3481, USA
{dhelm, rdamore, pyan}@mitre.org
Abstract: The MITRE Information Discovery System (MIDS) is a baseline system for integrating
advanced processing tools for information discovery and retrieval in large-scale distributed
environments. The system is built on a modular, extendible architecture that allows for system-level
decoupling and allocation of component processing tools across network nodes to provide for efficient
processing in distributed environments. At one level, the system provides for multi-platform user
access to HTTP, Gopher, FTP, and news servers using an HTML based client interface. However,
more significantly, the system provides advanced tools for metadata generation from disparate
network objects, and a content routing mediation layer for classification of metadata into appropriate
information brokers. This bottom-up layered information organization approach supports a wide range
of information retrieval and browsing strategies.
The MITRE Information Discovery System (MIDS) project is a multi-year MITRE-sponsored research
program to develop a set of multi-faceted capabilities for collecting, categorizing, organizing, and discovering
digital information in a distributed environment. MIDS tools not only support information management and
retrieval, but post-retrieval operations for summarizing and presenting information to the user.
A white pages directory organizes items by name (e.g., Internet user names), while a yellow pages
organizes information by attributes. MIDS is intended to support a dynamic yellow pages capability; that is,
information is organized by "topic," and can be dynamically reorganized as information collections change or
new collections are added. The system consists of an information organizer that groups multiple collection
contents topically, and a multi-broker network that uses object summaries to create a type of adaptable subject
catalog that is used to support browsing and searching specific collections. The organizer and broker are the
primary components which provide the foundation for the MIDS architectural framework.
The main emphasis of the program so far has been on the effective integration of advanced information
retrieval and natural language processing capabilities since there are a number of important system integration
issues and effectiveness and efficiency concerns that are critical to developing even a rudimentary pilot system.
While the short-term objective is not to support a large-scale system evaluation such as currently being
supported by ARPA's Tipster program, the current work will be focused on selected operational-based, enduser
assessments of system effectiveness and efficiency.
MITRE Information Discovery System
Essentially, MIDS is organized around several key functions: collection, indexing, routing, organization,
storage, retrieval and browsing. These functions are typically handled as tightly integrated components, in
traditional commercial off-the-shelf systems; however, MIDS provides these functions as separate modules that
can communicate within a networked environment. This distributed, modular approach provides flexibility in
terms of the methods employed and in defining extensions to the baseline architecture.
The MIDS project leverages heavily of both MITRE-developed technologies and tools available in the
public domain. As a result the system incorporates technologies for metadata extraction, clustering, filtering,
retrieval, and document summarization with new development focused on specialized problems not readily
addressed by off-the-shelf capabilities. In a sense, MIDS is both an integration framework and a set of tools.
As an example, the integration framework is centered around the Harvest [Bowman et al. 94] system.
Harvest is a scalable, customizable discovery and access system developed by the Internet Research Task
Force Research Group on Resource Discovery (IRTF-RD). Major Harvest components utilized include