Cost-Based Optimization of Integration Flows - Datenbanken ...

More documents

Recommendations

Info

3 Fundamentals of Optimizing Integration Flows size of messages as good indicator for the cardinalities. Based on these monitored atomic statistics, derived more complex statistics such as the relative frequencies of alternative paths P (path i ) or operator selectivities sel(o i ) = |ds out1 (o i )|/|ds in1 (o i )| are computable. In order to allow cost estimation for integration flows with control-flow semantics, in the following, we define a double-metric cost model. Definition 3.2 (Double-Metric Cost Estimation). The costs of a plan P are defined as an aggregate of operator costs, where these operator costs are defined by two metrics: 1. Abstract costs C(o i ) are defined for all data-flow- and interaction-oriented operators in the form of their time complexity using cardinalities as the metric. 2. Execution times W (o i ) are then used as the second metric in order to weight the abstract costs. For control-flow-oriented operators, we only use execution times. Both types of input statistics (cardinalities and execution times) are used in the form of aggregates over the monitored atomic statistics of executed plan instances. For multiple deployed integration flows, it can be necessary to normalize the monitored execution statistics (in particular execution times) when aggregating them. We presented detailed cost normalization algorithms [BHLW09c] that we omit here for the sake of being focused on the core cost model. Based on Definition 3.2, Tables 3.1-3.3 show the double-metric costs of all operators of our flow meta model (see Subsection 2.3.1). In the following, we substantiate the different cost formulas according to the two steps of our cost estimation approach. 1: Abstract Costs: In a first step, we consider the mentioned abstract costs C(P ) that are based on the cardinality metric. Costs for interaction-oriented operators include the costs for the operators Receive, Reply and Invoke. The costs for the Receive operator are determined as |ds out |, i.e., by the cardinality of the received data set ds out and they comprise costs for transport, protocol handling, format conversion as well as decompression. The costs for the related Reply operator are similarly computed with |ds in |. Finally, the costs of the Invoke operator are computed by |ds in | + |ds out | because it actively sends and receives data. A data set |ds out | might be NULL (e.g., in case of pure write interactions); in that case, we assume a cardinality of 0. Control-flow-oriented operators have no abstract costs. An exception to this is the Switch operator because it evaluates expressions over input data sets, which requires costs of |ds in | for a single path expression and due to the if-elseif-else semantic total costs of ∑ n i=1 (P (path i) · i · |ds in |) for all n paths, where P (path i ) denotes switch path probabilities (relative frequencies). For computing the abstract costs of the data-flow-oriented operators, we partly adapted a cost model from a commercial RDBMS [Mak07] to the specific characteristics of integration flows. With regard to physical operator alternatives, we excluded hash-based algorithms because they Table 3.1: Double-Metric Costs of Interaction-Oriented Operators Operator Name Abstract Costs C(o i ) Execution Time W (o i ) Invoke |ds in | + |ds out | W (o i ) Receive |ds out | W (o i ) Reply |ds in | W (o i ) 40
3.2 Prerequisites for Cost-Based Optimization Table 3.2: Double-Metric Costs of Control-Flow-Oriented Operators Operator Abstract Execution Time W (o i ) Name Costs C(o i ) ⎛ ⎛ ⎞⎞ Switch |ds in | n∑ i∑ ⎝P (path i ) · ⎝ W ( ) ∑m i expr pathj + W (o i,k ) ⎠⎠ Fork - i=1 n max i=1 j=1 k=1 ⎛ ⎞ ∑m i ⎝ W (o i,j ) + i · W (Start Thread) ⎠ Iteration - r · ∑n i=1 W (o i) Delay - W (o i ) Signal - W (o i ) j=1 Table 3.3: Double-Metric Costs of Data-Flow-Oriented Operators Operator Name Abstract Costs C(o i ) Execution Time W (o i ) Assign |ds in | + |ds out | W (o i ) Translation |ds in | + |ds out | W (o i ) Selection |ds in | W (o i ) sorted:|ds in |/2 Projection π distinct : |ds in | · |ds out |/2 W (o i ) π all /sorted : |ds in | Join ⋊⋉ NL : |ds in1 | + |ds in1 | · |ds in2 | W (o i ) sorted:|ds in1 | + |ds in2 | Setoperation ∪, \, ∩ : |ds in1 | + |ds in2 | · |ds out |/2 W (o i ) sorted:|ds in1 | + |ds in2 | ∪ all : |ds in1 | + |ds in2 | Split |ds in | + ∑ n i j=1 |ds out j | W (o i ) Orderby |ds in | · log 2 |ds in | W (o i ) Groupby |ds in | + |ds in | · |ds out |/2 W (o i ) Window |ds in | · |ds out |/2 W (o i ) Validate |ds in | W (o i ) Savepoint |ds in | W (o i ) Action |ds in | + |ds out | W (o i ) do not introduce additional requirements on plan rewriting and they are not applicable without extensions for persistent messages that do not fit into main memory. Additionally to the relational operators, the operators Assign and Translation both exhibit abstract costs of |ds in | + |ds out | because they get an input message and transform it to an output message, where the output can be smaller or larger than the input. Furthermore, the Split operator decomposes a message into multiple smaller messages. Hence, the sum of cardinalities for the input message |ds in | and of the set of output messages ∑ n i j=1 |ds out j | is a representative indicator. Both the Validate and the Savepoint operator have costs of |ds in | because they do not create any output message. Integrating the Savepoint into the cost model enables, for example, to include the overhead for recoverability [SWDC10] 41
Page 1: Cost-Based Optimization of Integrat
Page 4 and 5: of traditional data management syst
Page 7 and 8: Contents 1 Introduction 1 2 Prelimi
Page 9: Contents 6.5 Experimental Evaluatio
Page 12 and 13: 1 Introduction for integration flow
Page 14 and 15: 1 Introduction violated. We present
Page 16 and 17: 2 Preliminaries and Existing Techni
Page 44 and 45: 3 Fundamentals of Optimizing Integr
Page 98 and 99: 4 Vectorizing Integration Flows •
Page 100 and 101:
4 Vectorizing Integration Flows exe
Page 102 and 103:
4 Vectorizing Integration Flows Ove
Page 104 and 105:
4 Vectorizing Integration Flows Alg
Page 106 and 107:
4 Vectorizing Integration Flows two
Page 108 and 109:
4 Vectorizing Integration Flows Inv
Page 110 and 111:
4 Vectorizing Integration Flows (a)
Page 112 and 113:
4 Vectorizing Integration Flows o 2
Page 114 and 115:
4 Vectorizing Integration Flows In
Page 116 and 117:
4 Vectorizing Integration Flows 2.
Page 118 and 119:
4 Vectorizing Integration Flows The
Page 120 and 121:
4 Vectorizing Integration Flows ord
Page 122 and 123:
4 Vectorizing Integration Flows 4.3
Page 124 and 125:
4 Vectorizing Integration Flows P
Page 126 and 127:
4 Vectorizing Integration Flows P
Page 128 and 129:
4 Vectorizing Integration Flows t1:
Page 130 and 131:
4 Vectorizing Integration Flows We
Page 132 and 133:
Page 134 and 135:
Page 136 and 137:
Page 138 and 139:
4 Vectorizing Integration Flows 4.7
Page 140 and 141:
5 Multi-Flow Optimization cannot be
Page 142 and 143:
5 Multi-Flow Optimization the query
Page 144 and 145:
5 Multi-Flow Optimization The incom
Page 146 and 147:
5 Multi-Flow Optimization example p
Page 148 and 149:
5 Multi-Flow Optimization not allow
Page 150 and 151:
5 Multi-Flow Optimization partition
Page 152 and 153:
5 Multi-Flow Optimization mention i
Page 154 and 155:
5 Multi-Flow Optimization • Case
Page 156 and 157:
5 Multi-Flow Optimization k ′ . F
Page 158 and 159:
5 Multi-Flow Optimization partition
Page 160 and 161:
5 Multi-Flow Optimization the waiti
Page 162 and 163:
5 Multi-Flow Optimization Execution
Page 164 and 165:
5 Multi-Flow Optimization Thus, for
Page 166 and 167:
5 Multi-Flow Optimization Thus, T L
Page 168 and 169:
5 Multi-Flow Optimization • P 5 :
Page 170 and 171:
5 Multi-Flow Optimization decreasin
Page 172 and 173:
5 Multi-Flow Optimization (a) Fixed
Page 174 and 175:
5 Multi-Flow Optimization reached,
Page 176 and 177:
5 Multi-Flow Optimization (2) plan
Page 178 and 179:
6 On-Demand Re-Optimization categor
Page 180 and 181:
6 On-Demand Re-Optimization present
Page 182 and 183:
6 On-Demand Re-Optimization stratum
Page 184 and 185:
6 On-Demand Re-Optimization o 3 o 4
Page 186 and 187:
6 On-Demand Re-Optimization For on-
Page 188 and 189:
6 On-Demand Re-Optimization 6.3.1 O
Page 190 and 191:
6 On-Demand Re-Optimization such th
Page 192 and 193:
6 On-Demand Re-Optimization Join En
Page 194 and 195:
6 On-Demand Re-Optimization f γ((
Page 196 and 197:
6 On-Demand Re-Optimization The res
Page 198 and 199:
6 On-Demand Re-Optimization project
Page 200 and 201:
6 On-Demand Re-Optimization (a) Sel
Page 202 and 203:
6 On-Demand Re-Optimization (a) Loa
Page 204 and 205:
6 On-Demand Re-Optimization ical re
Page 206 and 207:
6 On-Demand Re-Optimization evaluat
Page 208 and 209:
6 On-Demand Re-Optimization 6.6 Sum
Page 210 and 211:
7 Conclusions Existing approaches b
Page 212 and 213:
Bibliography [BBD05a] Shivnath Babu
Page 214 and 215:
Bibliography [BHP + 09b] [BHP + 11]
Page 216 and 217:
Bibliography [CM95] Sophie Cluet an
Page 218 and 219:
Bibliography [GZ08] [HA03] [Haa07]
Page 220 and 221:
Bibliography [IKNG09] [INSS92] [Ioa
Page 222 and 223:
Bibliography [LX09] [LZ05] Rubao Le
Page 224 and 225:
Bibliography [OMG07] OMG. XML Metad
Page 226 and 227:
Bibliography [Sto02] Michael Stoneb
Page 228 and 229:
Bibliography [ZRH04] Yali Zhu, Elke
Page 230 and 231:
List of Figures 3.27 Workload Adapt
Page 233:
List of Tables 2.1 Interaction-Orie
Page 237:
Selbstständigkeitserklärung Hierm
show all

Cost-Based Optimization of Integration Flows - Datenbanken ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?