Cost-Based Optimization of Integration Flows - Datenbanken ...

3 Fundamentals of Optimizing Integration
Flows
In this chapter, we introduce the fundamentals of a novel optimization framework for
integration flows [BHW + 07, BHLW08f, BHLW08g, BHLW09a] in order to enable arbitrary
cost-based optimization techniques. This framework is tailor-made for integration
flows with control-flow execution semantics because it exploits the major integration-flowspecific
characteristic of being deployed once and executed many times. Furthermore, it
tackles the specific problems of missing statistics, changing workload characteristics, and
imperative flow specifications, while ensuring the required transactional properties as well.
The core idea of the overall cost-based optimization framework for integration flows is
incremental statistics maintenance in combination with asynchronous, inter-instance plan
re-optimization. In order to take into account both, data-flow- and control-flow-oriented
operators, we specify the necessary dependency analysis as well as a novel hybrid cost
model. Furthermore, we introduce the periodical re-optimization that includes the core
transformation-based optimization algorithm as well as specific approaches for search space
reduction (such as a join reordering heuristic), influencing workload adaptation sensibility,
and handling of correlated data. Subsequently, we present selected concrete optimization
techniques (such as the reordering/merging of switch paths, early selection application,
or the rewriting of sequences and iterations to parallel flows) to illustrate the rewriting
of plans. Finally, the evaluation shows that significant performance improvements are
possible with fairly low optimization overhead.
3.1 Motivation and Problem Description
The motivation for designing a tailor-made optimization approach for integration flows
is the specific characteristic of being deployed once and executed many times that can
be exploited for efficient re-optimization. Moreover, integration flows are specified with
control-flow semantics (imperative) in order to enable the execution of complex procedural
integration tasks.
Problem 3.1 (Imperative Integration Flows). When rewriting imperative flow specifications,
the data flow and the control flow (in the sense of restrictive temporal dependencies)
must be taken into account in order to ensure semantic correctness. Here, semantic correctness
is used in the sense of preventing the external behavior (data aspects and temporal
order) from being changed.
The majority of existing flow optimization approaches [LZ05, VSS + 07, BJ10, BABO + 09]
apply rule-based optimizations only (optimize-once) using, for example, algebraic equivalences.
There, rewriting decisions are statically made only once during the initial deployment
of an integration flow. However, this is inadequate due to the following problem:
33