Rome Wasn't Digitized in a Day - Council on Library and Information ...

113
uncerta<strong>in</strong>ty regard<strong>in</strong>g dates <strong>in</strong> the legal texts <strong>in</strong> Volterra. To <strong>in</strong>tegrate HGV and Volterra, they created
annotations databases for each project or “randomly-generated values associated with each record <strong>in</strong>
the orig<strong>in</strong>al databases” so they could “demonstrate cross-database jo<strong>in</strong>s and third-party annotations”
(Jackson et al. 2009).
The project used OGSA-DAI 372 for data <strong>in</strong>tegration because it was considered a de facto standard by
many other e-science projects for <strong>in</strong>tegrat<strong>in</strong>g heterogeneous databases, it was open source, and it was
compliant with many relational databases, XML, and other file-based resources. OGSA-DAI also
supported the exposure of data resources on to grids (Bodard et al. 2009). Most important, <strong>in</strong> terms of
data <strong>in</strong>tegration:
… OGSA-DAI can abstract the underly<strong>in</strong>g databases us<strong>in</strong>g SQL views and provide an
<strong>in</strong>tegrated <strong>in</strong>terface onto them us<strong>in</strong>g distributed query<strong>in</strong>g. This fulfils the essential requirement
of the project to leave the underly<strong>in</strong>g data resources untouched as far as possible (Jackson et al.
2009).
One goal of LaQuAT was to be able to support federated search<strong>in</strong>g of a “virtual database” <strong>in</strong> order that
the underly<strong>in</strong>g databases would not have to undergo major changes for <strong>in</strong>clusion <strong>in</strong> such a resource.
“The ability to l<strong>in</strong>k up such diverse data resources, <strong>in</strong> a way that respects the orig<strong>in</strong>al data resources
and the communities responsible for them,” Bodard et al. (2009) asserted, “is a press<strong>in</strong>g need among
humanities researchers.”
A number of issues complicated data <strong>in</strong>tegration, however, <strong>in</strong>clud<strong>in</strong>g data consistency and some
specific features of OGSA-DAI. To beg<strong>in</strong> with, some of the orig<strong>in</strong>al data <strong>in</strong> the HGV database had
been “contam<strong>in</strong>ated by control characters,” a factor that had serious implications for the OGSA-DAI
system s<strong>in</strong>ce it provided access to databases via web services, which are based on the exchange of
XML documents. S<strong>in</strong>ce the use of control characters with<strong>in</strong> an XML document results <strong>in</strong> an <strong>in</strong>valid
XML file that cannot be parsed, they had to extend the system’s “relational data to XML conversion
classes to filter out such control characters and replace these with spaces.” The Volterra database also
presented its own unique challenges, particularly <strong>in</strong> terms of database design, s<strong>in</strong>ce not all tables had
the same columns and some columns with the same <strong>in</strong>formation had different names. A second major
challenge was the lack of suitable database drivers, and the data from both Volterra and HGV were
ported <strong>in</strong>to MySQL to be able to <strong>in</strong>teract with OGSA-DAI. Other issues <strong>in</strong>cluded need<strong>in</strong>g to adapt the
way the OGSA-DAI exposed metadata and hav<strong>in</strong>g to alter the way the system used SQL views
because of the large nature of the HGV database. In the end, the project could use only a subset of the
HGV database to ensure that query time would be reasonable. Despite these and other challenges, the
project was able to develop a demonstrator that provided <strong>in</strong>tegrated access to both HGV and
Volterra. 373
The LaQuAT project had orig<strong>in</strong>ally assumed that one of the most useful outcomes of <strong>in</strong>tegrat<strong>in</strong>g the
two databases would be where data overlapped (such as <strong>in</strong> terms of personal and place names), but
they found <strong>in</strong>stead that clear-cut overlaps were fairly easy to identify. A far more <strong>in</strong>terest<strong>in</strong>g question,
they proposed, was to try to automatically recognize “the co-existence of homonymous persons or
372 While the technical details of this software are beyond the scope of this paper, Jackson et al. expla<strong>in</strong> that “OGSA-DAI executes workflows which can
be viewed as scripts which specify what data is to be accessed and what is to be done to it. Workflows consist of activities, which are well-def<strong>in</strong>ed
functional units which perform some data-related operation e.g. query a database, transform data to XML, deliver data via FTP. A client submits a
workflow to an OGSA-DAI server via an OGSA-DAI web service. The server parses, compiles and executes the workflow.”
373 For more on the <strong>in</strong>frastructure proof of concept design, see Jackson et al. (2009). This demonstrator can be viewed at
http://doma<strong>in</strong>001.vidar.ngs.manchester.ac.uk:8080/laquat/laquatDemo.jsp