Cost-Based Optimization of Integration Flows - Datenbanken ...

3 Fundamentals of Optimizing Integration Flows
Optimization Algorithm
According to the defined integration flow optimization problem, we now explain the overall
optimization algorithm including the two aspects of (1) when and how to trigger reoptimization
of a plan and (2) how to re-optimize the given plan using the set of available
cost-based optimization techniques. The naïve algorithm for solving the P-PPO comprises
three subproblems: (1) the complete creation of alternative plans (the search space), (2)
the periodical cost evaluation of each created plan (search space evaluation), and (3) the
choice of the plan with minimal costs. In contrast to this generation-based approach, we
exploit the specific characteristic of an initially given imperative plan, by using an iterative
(transformation-based) optimization algorithm. In the following, we describe in detail how
to trigger re-optimization and how to re-optimize the given plan.
Algorithm 3.1 Trigger Re-Optimization (A-TR)
Require: plan identifier ptid, optimization interval ∆t, workload window size ∆w,
aggregation method method, optimization algorithm algorithm
1: while true do
2: sleep(∆t)
3: P ← getPlan(ptid)
4: DG ← getDependencyGraph(ptid)
5: Estimator.aggregateStatistics(P , ∆w, method) // see Subsection 3.3.3
6: ret ← Optimizer.optimizePlan(P , DG, algorithm)
7: if ret.isChanged() then
8: P ← ret.getPlan()
9: putPlan(ptid, P )
10: putDependencyGraph(ptid, ret.getDG())
11: P arser.recompilePlan(ptid, P )
12: Runtime.exchangePlans(ptid, P )
Algorithm 3.1 4 illustrates when and how re-optimization is triggered. Essentially, this
algorithm is started as a background thread for each deployed integration flow and periodically
issues plan re-optimization with period ∆t (line 2). Therefore, monitored execution
statistics are aggregated with a certain aggregation method (line 5) and re-optimization
is initiated with one of our optimization algorithms (line 6). If the current plan has been
changed during this re-optimization, we recompile the logical plan into an executable physical
plan (line 11) and exchange the plan at the next possible point between two subsequent
plan instances (line 12). When triggering re-optimization, the optimization algorithm is
selected. There, patternMatchingOptimization (A-PMO) is the default algorithm, while
several additional heuristic algorithms can be used for search space reduction.
Algorithm 3.2 illustrates our transformation-based optimization algorithm A-PMO. This
algorithm is invoked for the complete plan, where it recursively iterates over the hierarchy
of operator sequences (of the current plan) and applies optimization techniques according
to the operator types (the included comments show the abbreviations of applied optimization
techniques, which we partly discuss in Section 3.4). There are four types of
optimization techniques. First, we apply all techniques, which need to be executed on top
level of a plan and before all other optimization techniques (line 2). For example, the join
4 Similar to the naming scheme of problems, we use the prefix A- to indicate names of algorithms.
48