automatically exploiting cross-invocation parallelism using runtime ...

after which we describe how SPECCROSS is applied for automatic parallelization.4.1 Motivation and Overview4.1.1 Limitations of analysis-based parallelizationFigure 4.2(a) shows a code example before parallelization. In this example, loop L1 updatesarray elements in array A while loop L2 reads the elements from array A and usesthe values to update array B. The whole process is repeated STEP times. Both L1 and L2can be individually parallelized using DOALL [1]. However, dependences between L1 andL2 prevent the outer loop from being parallelized. Ideally, programmers should only synchronizeiterations that depend on each other, without stalling the execution of independentiterations. If static analysis [50, 72, 78] could prove that each thread accesses a separatesection of arrays A and B, no synchronization is necessary between two adjacent loop invocations(Figure 4.2(b)). However, since arrays A and B are accessed in an irregular manner(through index arrays C and D), static analysis cannot determine the dependence patternbetween L1 and L2. As a result, this naïve parallelization may lead to incorrect runtimebehavior.Alternatively, if static analysis could determine a dependence pattern between iterationsfrom two invocations, e.g., iteration 1 from L2 always depends on iteration 2 fromL1, then fine-grained synchronization can be used to synchronize only those iterations. Butthis requires accurate analysis about the dependence pattern, which is in turn, limited by theconservative nature of static analysis. Instead, barrier synchronization is used to globallysynchronize all threads (Figure 4.2(c)). Barrier synchronization conservatively assumes dependencesbetween any pair of iterations from two different loop invocations. All threadsare forced to stall at barriers after each parallel invocation, which greatly diminishes effectiveparallelism. Figure 4.3 shows the overhead introduced by barrier synchronizationson eight programs parallelized with 8 and 24 threads. Barrier overhead refers to the total51