Cost-Based Optimization of Integration Flows - Datenbanken ...

2.1 Integration Flows
to automatically specify the semantics of ETL jobs in a platform-independent manner.
The Orchid project is restricted to the generation of ETL flows. In addition, Jörg and
Deßloch addressed the generation of incremental ETL jobs [JD08, JD09] and subsequently,
they also investigated the optimization of these incremental ETL jobs [BJ10].
In contrast, the METL project by Albrecht and Naumann focuses on the combination of
the generation and model management of ETL flows [AN08]. There, platform-independent
and tool-independent operators for model management were introduced. In contrast to
generic model management [BM07] for schema mappings, the order, type, and configuration
of data transformation steps must be taken into account. These requirements are
addressed with specific high-level operators and an ETL management platform [AN09].
In addition to these ETL flow approaches, our GCIP framework (Generation of Complex
Integration Processes) [BHLW09a] addresses the modeling of platform-independent
integration flows [BHLW08e], the flow generation for arbitrary integration platforms (e.g.,
FDBMS, EAI, ETL) as well as the application of optimization techniques [BHLW08f,
BHLW08g, BBH + 08a, BBH + 08b] during model-driven generation and finally, the deployment
of these generated integration flows [BHLW09b]. Therefore, a hierarchy of platformindependent,
platform-specific, and tool-specific models is used.
While all of these approaches address at most two aspects of integration flows, namely
the flow specification and schema definitions, Mazon et al. defined the multi-dimensional
model-driven architecture for the development of data warehouses [MTSP05]. There, the
aspects (1) data sources, (2) ETL flows, (3) multi-dimensional data warehouse design, (4)
customization code, and (5) application code as well as their inter-influences are taken
into account. This general framework was recently extended by data merging and data
customization aspects for data mining [KZOC09].
D. Declarative Integration Flow Modeling
In contrast to the usually used imperative integration flows that are modeled in a prescriptive
manner, the Demaq project [BMK08] models declarative integration flows in
a descriptive manner by using dependable XML message queue definitions (basic, timebased,
gateway), where the processing logic is described in terms of a declarative rule
language. As a foundation, a declarative Queue Definition Language (QDL) and a Queue
Manipulation Language (QML) based on the XQuery update facility are used [BKM07].
Furthermore, Demaq introduced the concept of slices that are specifications of logical
message groups in the form of virtual queues. Despite the concept of dependable and
time-based queues, complex procedural aspects, i.e., temporal dependencies, and complex
control-flows are hard to model. However, for message-centric integration flows, reasonable
performance, scalability and transactional properties are achieved [BK09].
While Demaq describes the overall flow with dependencies between queues, the Orchid
project [DHW + 08] uses declarative schema mappings between source and target schemes
and generates imperative ETL flows. To this extent, Orchid also uses declarative modeling
semantics despite the resulting imperative ETL flow specifications.
Finally, the modeled integration flows are deployed into the execution environment
using pre-defined exchange formats (e.g., XML). From this point in time on, the deployed
integration flows are identified by logical names and executed many times. However, there
are several side-effects from modeling to execution.
13