Sharing Knowledge: Scientific Communication - SSOAR

162 Judith Plümer
The summarizers generate SOIF records for each document. SOIF was designed
for the storage and exchange of the summaries of documents which may originate
in different formats. It was built as a part of the harvest software. The
structure of SOIF is simple and effective. A SOIF object consists of multiple
attribute-value pairs. For example
author{16}: Erwin Mustermann
is such a pair, the number in brackets gives the number of characters of the value.
There are no restrictions on the number and names of attributes.
This principle of attribute-value pairs makes SOIF equivalent to the use of the
HTML2.0 META tag with its NAME-CONTENT pairs. But it can also be used
to store information from the body of a HTML document or from other formats.
For example, the HTML summarizer of harvest assigns all words between
and to the header attribute.
Obviously, it makes sense to build such attribute-value pairs from any markup
language like HTML, SGML, XML or TeX/LaTeX.
The summarizer for documents in a typeset format as .dvi, .ps or .pdf work
differently, they only store the first 100 words of these documents as keywords
because the text is not semantically structured. One could think of criteria to extract
more structure out of these unstructured documents via methods of artificial
intelligence, heuristics or the use of thesauri (see below).
The summarizers of harvest also store metadata such as the mimetype of the
original document, its URL, URLs of links contained in the document, name
and host of the gathering agent, time of generation, update time and time to live.
All these pieces of information play an important role for the system because the
information is not constant. They are used to control the expiration times.
Collection of documents and problems of heterogeneity
Since SOIF objects are just ASCII files, they can be transferred by any protocol
as SOIF streams. Harvest builds a database or an index of SOIF objects by using
Glimpse which makes the attribute-value pairs searchable.
Now the problem arises when upgrading to HTML4.0 or to XML with metadata
in RDF (Resource Description Framework). In MPRESS we currently store
SCHEME information of HTML4.0 META as a qualifier. For example
is mapped to
DC.subject.msc{5}: 19D10