Sciweavers

PPOPP
2015
ACM

Stochastic gradient descent on GPUs

8 years 6 months ago
Stochastic gradient descent on GPUs
Irregular algorithms such as Stochastic Gradient Descent (SGD) can benefit from the massive parallelism available on GPUs. However, unlike in data-parallel algorithms, synchronization patterns in SGD are quite complex. Furthermore, scheduling for scale-free graphs is challenging. This work examines several synchronization strategies for SGD, ranging from simple locking to conflict-free scheduling. We observe that static schedules do not yield better performance despite eliminating the need to perform conflict detection and resolution at runtime. We identify the source of the performance degradation to be the structure of certain parts of the graph (dense vs sparse). This classification can be used to devise hybrid scheduling strategies which exploit different schedules for different regions of the graph to obtain better performance. We found that the best schedule for some problems can be up to two orders of magnitude faster than the worst one. To evaluate the performance of our...
Rashid Kaleem, Sreepathi Pai, Keshav Pingali
Added 16 Apr 2016
Updated 16 Apr 2016
Type Journal
Year 2015
Where PPOPP
Authors Rashid Kaleem, Sreepathi Pai, Keshav Pingali
Comments (0)