Sharing Knowledge: Scientific Communication - SSOAR

MPRESS - transition of metadata formats 161
The gathering part of the Harvest [Hardy, 1996, Technical Report] software
which is used in Math-Net is responsible for this collecting process. It is a configurable
robot. Now the gathering of the documents does not mean making copies
and keeping them somewhere. It means taking a temporary copy, extracting
the relevant information from this copy, storing this information in a database
and deleting the temporary copy.
One gathering agent is determined by several configuration files that have to
be created by an administrator. There is one configuration file that contains the
URLs that the agent should gather and evaluate together with a lot of parameters
which say whether he should run recursively, at which depth the recursion
should stop, how many documents should be gathered at most, which documents
should be included or excluded during the recursion, how many different
hosts may be visited during the recursion and which types of protocols should be
used (ftp, http).
The other configuration files can be used in their default form or be modified,
too. As there are configuration files which say how to handle different mimetypes
and formats, how to interpret suffixes, and how to summarize different
formats.
The gatherer pipes the retrieved document through the essence machine
[Hardy, 1996, Trans. Comp. Sci] that generates summaries of the documents in
SOIF [Hardie, 1999] (summary object interchange format, see below). The
SOIF documents are stored by the gatherer and the original resources are deleted.
The SOIF documents of one agent reside in a gnu-zipped ASCII file with
some additional information. For example, there is a small databases storing the
MD5 [Rivest, 1992] numbers of the original files to avoid multiple copies of one
document. Internally the MD5 number is also used to check whether a document
has been changed or not in case the agent does not visit a site for the first time.
(The agents of MPRESS run every other week.) If the document has not been
changed, the essence process does not have to be run again on the same document.
In this case only the time-to-life (TTL) of the respective SOIF-record is
modified. This saves local computing power, but it does not save netload, which
would be desirable. So what we wanted was an incrementally running agent.
The original harvest software was modified in this way so that it runs incrementally
for MPRESS.
There are summarizers for a variety of formats. The pages in the Web that
contain mathematically relevant information are usually stored in PostScript,
PDF or HTML. The summarizers of the harvest software do not handle 8-bit and
unicode characters correctly. So we had to modify the PostScript and HTML
summarizer to interpret at least Umlauts and „ß“ for German needs in order to
allow correct responses when querying for German names. This problem is solved
with X-Harvest and HyREX because they are able to handle Unicode
characters (see below).