Julia has several packages for programming GPUs, each of which support various programming models. In this workshop, we will demonstrate the use of three major GPU programming packages: CUDA.jl for NVIDIA GPUs, AMDGPU.jl for AMD GPUs, and oneAPI.jl for Intel GPUs. We will explain the various approaches for programming GPUs with these packages, ranging from generic array operations that focus on ease-of-use, to hardware-specific kernels for when performance matters. Most of the workshop will be vendor-neutral, and the content will be available for all supported GPU back-ends. There will also be a part on vendor-specific tools and APIs. Attendees will be able to follow along, but are recommended to have access to a suitable GPU for doing so. Materials https://github.com/maleadt/juliacon21-gpu_workshop https://developer.nvidia.com/cuda-example Enjoyed the workshop? Consider sponsoring us on GitHub: https://github.com/sponsors/JuliaLang Contents 00:00 Welcome! ??? 00:24 Welcome ??? 01:20 Outline 02:44 JuliaGPU packages 04:08 JuliaGPU back-ends 05:34 GPU Architecture 07:25 Parallel programming models 08:55 Follow along and links to notebooks, JuliaHub 12:37 Start of tutorial with notebook 16:00 Array programming 28:20 Kernel programming 34:32 Parallel programming + questions 41:50 Question about `@cuprintln` 43:00 Question about threads in CPU vs GPU 45:08 Question: iterating over threads and blocks, ordering 48:03 Question of accuracy in CPU vs GPU 56:48 Question: array structure, "CUDA by Example" book recommendation 58:40 Profiling 1:01:50 Profiling: NVIDIA Nsight Systems: live example 1:11:00 Profiling: NVIDIA Nsight Compute: live example → optimize single kernel invocation 1:19:05 Common issues: unsupported array operations 1:21:50 Common issues: unsuppored kernel operations 1:25:45 Question: portability of optimizations 1:27:40 Parallel programming issues 1:31:55 Tour of accompanying GitHub repo 1:32:40 Case Study I: Image processing using AMDGPU 1:57:00 Break 2:01:30 Case Study II: Fun with arrays, Machine Learning 2:10:47 Case Study III: Random number generators 2:22:10 Kernel abstractions 2:42:10 Example: Solving heat equation with GPU 2:56:30 Sneak peek of Enzyme (automatic differentiation framework) 2:59:18 Questions and Future plans S/O to https://github.com/ignace-computing for the video timestamps! Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: https://github.com/JuliaCommunity/YouTubeVideoTimestamps Interested in improving the auto generated captions? Get involved here: https://github.com/JuliaCommunity/YouTubeVideoSubtitles