automatically exploiting cross-invocation parallelism using runtime ...

6050Non−ExclusiveExclusivePercentage (%)40302010027% 24% 16% 6% Job_ParallelismMessage_Passing9% 2% Threading7% 2% 3% 4% 4% Parallel_AnnotationGPU_based_ParallelismOthers34% NoneFigure 1.2: Types of parallelism exploited in scientific research programs: one third of theinterviewed researchers do not use any parallelism in their programs; others mainly use jobparallelism or borrow already parallelized programs.parallelism.Often, iterations from different loop invocations can execute concurrentlywithout violating program semantics. Instead of waiting, threads begin iterations fromsubsequent invocations.A simple code example in Figure 1.3(a) can be used to demonstrate the benefits ofexploiting cross-invocation parallelism. In this example, inner loop L1 updates array elementsin array A while inner loop L2 reads the elements from array A and uses the valuesto update array B. The whole process is repeated TIMESTEP times. Both L1 and L2 areDOALLable [1]. However, barriers must be placed between these two loops since datadependences exist across iterations of the two loops (e.g., iteration 1 of L2 depends oniteration 1 and 2 of L1).Figure 1.4(a) shows the execution plan for this program with barriers. Each block in5