On the Performance of Local Synchronization with Stochastic Task Times

Synchronization is often necessary in parallel computing, but it can create delays whenever the receiving processor is idle, waiting for the information to arrive. This is especially true for barrier, or global, synchronization, in which every processor must synchronize with every other processor. Nonetheless, barriers are the only form of synchronization explicitly supplied in OpenMP, and they occur whenever collective communication operations are used in MPI.

Many applications do not actually require global synchronization; local synchronization, in which a processor synchronizes only with a subset of other processors, is often adequate. However, the behavior of a system under local synchronization is more difficult to analyze, since processors do not start tasks at the same time. We show some exact analytic results and bounds, along with experimental simulation results, for a variety of synchronization models under various stochasticity conditions on the task times.