Cost-Based Optimization of Integration Flows - Datenbanken ...

3.1 Motivation and Problem Description
they do not consider any data-intensive operators (such as Join, Groupby, etc) and related
optimization techniques as well as they are typically not aware of interactions with external
systems. Second, existing cost-based optimizers for DBMS or DSMS optimize data-flow
graphs rather than imperative control-flow graphs. Similarly to existing approaches of
integration flow optimization, both categories use the optimize-once or optimize-always
models and thus, do not take into account the major characteristic of integration flows in
the form of being deployed once and executed many times.
We use the introduced system architecture of typical integration platforms (see Subsection
2.1.2) in order to sketch the required architectural extensions for enabling the
cost-based optimization of integration flows. Figure 3.1 illustrates this extended reference
system architecture including the novel cost-based optimization component.
Modeling
Analyze
Optimization
Flow
Designer
Flow
Execution
Statistics
Monitor
Cost-Based
Optimizer
Plan
Optimizer
Rule-Based
Optimizer
External
System
External
System
External
System
Inbound
Adapter 1
...
Inbound
Adapter n
sync
async
Execute
Plan
Process Engine
Scheduler
Outbound
Adapter 1
...
...
Outbound
Adapter k
External
System
External
System
External
System
External
System
Temporary Datastore
Execution
Figure 3.1: Extended Reference System Architecture
In order to address the problem of imperative integration flows, the deployment process
is modified such that a dependency analysis of operators is executed during the initial
deployment. Furthermore, the rule-based optimization, where we do not require any
execution statistics, is executed once during this deployment as well. We described several
rule-based optimization techniques for integration flows [BHW + 07], but in this thesis,
we omit the details for the sake of clarity of presentation. From this point in time, the
plan is executed many times and the optimizer is used to continuously adapt the current
plan to changing workload characteristics. The goal of plan optimization is to rewrite
(transform) a given plan into a semantically equivalent plan that is optimal in the average
case with regard to the estimated costs. Therefore, we use a feedback loop according to
the general MAPE (Monitor, Analyze, Plan, Execute) concept [IBM05a]. During plan
execution, statistics are gathered (Monitor). Then, the optimizer periodically analyzes
the given workload (Analyze) in order to optimize the current plan. Subsequently, the
rewritten plan is deployed (Plan) and used for execution (Execute).
We use this general feedback loop as the overall structure of this chapter. First, we
present a self-adjusting cost model for integration flows and explain the cost estimation of
plans, using this cost model (Monitor → Analyze; Subsection 3.2.2). Second, we illustrate
the optimization problem as well as the overall periodical re-optimization algorithm (Analyze
→ Plan; Section 3.3). Third, we demonstrate the rewriting of plans using selected
35