Cost-Based Optimization of Integration Flows - Datenbanken ...

3.2 Prerequisites for Cost-Based Optimization
concept is to use an empirical cost model. There, physical operators are empirically analyzed
(static or dynamic/leaning-based) with regard to the execution costs when varying
the input parameters such as the input cardinalities. The results of these experimental
measurements are then used in order to determine continuous cost functions that approximately
describe the cost of the evaluated operator. This is done for all physical operators
and then, it is generalized to cost functions for logical operators. These continuous cost
functions are tailor-made for a given hardware platform or they include parameters to
fine-tune and adjust the cost models to different hardware platforms, respectively. Empirical
cost models are known to produce precise results in local settings, where operators
and data structures are known in advance and where all operations are executed locally.
For example, such empirical cost models exist for object-relational DBMS [Mak07], native
XML DBMS [WH09, Wei11], and computational-science applications [SBC06]. Unfortunately,
due to the integration of heterogeneous systems and applications, an empirical
cost model is inadequate for integration flows or distributed data management in general
because the dynamic adaptation to external costs (execution times of external queries
or remote data access) is required [ROH99, JSHL02, NGT98, GST96, RZL04, SW97].
Furthermore, learning-based approaches can lead to non-monotonic cost functions.
In contrast to empirical cost models in DBMS, our double-metric cost model relies on a
fundamentally different concept. The core idea is twofold. First, we define abstract cost
functions according to the asymptotic time complexity of the different operator implementations
using the monitored cardinalities—similar to empirical cost models—as input
parameter. Second, we additionally use the monitored execution times in order to weight
the computed costs. This ensures—similar to cardinality estimation in the Leo project
[SLMK01, ML02, AHL + 04]—that the cost model is self-adjusting by definition, which
means that it does not require any knowledge about the involved external systems, and
that we are able to compare costs of interaction-, control-flow- and data-flow-oriented
operators by a combined metric of cardinalities and execution times.
Statistics Monitoring and Cost Model
Execution statistics are monitored at the granularity of single operators in order to enable
our double metric cost estimation approach. Figure 3.3 illustrates the conceptual model
of operator statistics.
(|ds in1(o i)| … |ds in k1(o i)|)
|ds in(o i)|
|ds in1(o i)| |ds in2(o i)|
nid operator o i W(o i)
wait(o i)
(|ds out1(o i)| … |ds out k2(o i)|)
(a) Generic Operator
nid operator o i W(o i)
wait(o i)
|ds out(o i)|
(b) Unary Operator
nid operator o i W(o i)
wait(o i)
|ds out(o i)|
(c) Binary Operator
Figure 3.3: General Model of Operator Execution Statistics
An operator o i is described, in the general case (see Figure 3.3(a)), by cardinalities of
arbitrary input data sets |ds ini |, a node identifier nid, the total execution time W (o i ), the
subsumed waiting time wait(o i ) (e.g., time of external query execution, where local processing
is blocked) with 0 ≤ wait(o i ) ≤ W (o i ), and cardinalities of arbitrary output data
sets |ds outj |. Naturally, unary (see Figure 3.3(b)) and binary ((see Figure 3.3(c)) operators
with a single output are most common. With regard to efficient monitoring, we use the
39