Cost-Based Optimization of Integration Flows - Datenbanken ...

tegration flows. In detail, we make the following more concrete contributions that also
reflect the structure of this thesis.
• As a prerequisite, Chapter 2 analyzes existing techniques and introduces the notation
and examples used throughout this thesis. The literature survey is twofold. On
the one side, we review common integration approaches and, more specifically, we
discuss the modeling, execution and optimization of integration flows. On the other
side, we discuss adaptive query processing techniques to illustrate the state-of-theart
of cost-based optimization in other system categories such as DBMS (Database
Management Systems) or DSMS (Data Stream Management Systems).
• In Chapter 3, we explain the novel fundamentals for the cost-based optimization of
integration flows that enable arbitrary optimization techniques. The specific characteristics
of integration flows in terms of missing statistics, changing workload characteristics,
imperative flow specification (prescriptive) and transactional properties
have led to the need for fundamentally new concepts. In order to take into account
both data-flow- and control-flow-oriented operators, we explain the necessary dependency
analysis as well as a hybrid cost model. We introduce the inter-instance,
periodical re-optimization that includes the core transformation-based optimization
algorithm as well as specific approaches for search space reduction, workload adaptation
sensibility, and the handling of correlated data. Finally, we present selected
concrete optimization techniques to illustrate the rewriting of flows.
• Subsequently, in Chapter 4, we present the cost-based vectorization of integration
flows that is a tailor-made, control-flow-oriented optimization technique. Based on
the problems of low resource utilization and specific transactional requirements, this
technique computes the optimal grouping of operators to multi-threaded execution
buckets in order to achieve the optimal degree of pipeline parallelism with a minimal
number of buckets and hence, it maximizes message throughput. We present
context-specific rewriting techniques to ensure transactional properties and discuss
exhaustive and heuristic computation approaches for certain settings and constraints.
• As a tailor-made data-flow-oriented optimization technique, we introduce the concept
of multi-flow optimization in Chapter 5. It is based on the problem of expensive
access of external systems. The core idea is to horizontally partition the inbound
message queues and to execute flows for partitions of multiple messages. Due to
the decreased number of queries to external systems as well as cost reductions for
local operators, this results in throughput improvements. In detail, we introduce the
partition tree that is used as a physical message queue representation, an approach
for creating and incrementally maintaining such partition trees, and related flow
rewriting techniques. Furthermore, we extend the defined cost model and present
an approach to periodically compute the optimal waiting time for collecting messages
in order to achieve the highest throughput, while ensuring maximum latency
constraints of individual messages.
• In order to decrease the overhead for statistics monitoring and re-optimization but
to adapt to changing workloads as fast as possible, in Chapter 6, we introduce the
novel concept of on-demand re-optimization. This includes (1) to model optimality
of a plan by its optimality conditions using a so-called Plan Optimality Tree rather
than considering the complete search space, (2) to monitor only statistics that are
included in these conditions, and (3) to use directed re-optimization if conditions are
3