Using The NVIDIA CUDA Stream-Ordered Memory Allocator Part 1

Most CUDA developers are familiar with the cudaMalloc and cudaFree API capabilities to allocate GPU accessible memory. Nevertheless, there has lengthy been an impediment with these API capabilities: they aren’t stream ordered. In this publish, we introduce new API features, cudaMallocAsync and cudaFreeAsync, that allow memory allocation and deallocation to be stream-ordered operations. In part 2 of this sequence, we highlight the benefits of this new functionality by sharing some large knowledge benchmark outcomes and supply a code migration information for modifying your existing functions. We also cover superior matters to benefit from stream-ordered memory allocation within the context of multi-GPU access and using IPC. This all helps you enhance performance inside your present functions. The following code instance on the left is inefficient because the primary cudaFree call has to look ahead to kernelA to finish, so it synchronizes the device earlier than freeing the memory. To make this run more efficiently, the memory can be allotted upfront and sized to the bigger of the 2 sizes, as shown on the precise.



This increases code complexity in the appliance as a result of the memory administration code is separated out from the business logic. The problem is exacerbated when different libraries are involved. This is way more durable for the application to make efficient because it may not have complete visibility or control over what the library is doing. To bypass this drawback, the library must allocate memory when that function is invoked for the first time and never free it until the library is deinitialized. This not only increases code complexity, nevertheless it additionally causes the library to hold on to the memory longer than it needs to, doubtlessly denying one other portion of the applying from utilizing that memory. Some purposes take the thought of allocating memory upfront even further by implementing their very own custom allocator. This provides a major quantity of complexity to application growth. CUDA goals to offer a low-effort, high-performance different.



CUDA 11.2 introduced a stream-ordered memory allocator to resolve a majority of these problems, with the addition of cudaMallocAsync and cudaFreeAsync. These new API capabilities shift memory allocation from international-scope operations that synchronize the whole gadget to stream-ordered operations that allow you to compose memory management with GPU work submission. This eliminates the necessity for synchronizing outstanding GPU work and helps restrict the lifetime of the allocation to the GPU work that accesses it. It is now potential to handle memory at operate scope, Memory Wave as in the following example of a library perform launching kernelA. All the standard stream-ordering rules apply to cudaMallocAsync and cudaFreeAsync. The memory returned from cudaMallocAsync might be accessed by any kernel or memcpy operation as lengthy as the kernel or memcpy is ordered to execute after the allocation operation and before the deallocation operation, in stream order. Deallocation may be performed in any stream, so long as it is ordered to execute after the allocation operation and in spite of everything accesses on all streams of that memory on the GPU.



In impact, stream-ordered allocation behaves as if allocation and free were kernels. If kernelA produces a valid buffer on a stream and kernelB invalidates it on the same stream, then an software is free to access the buffer after kernelA and earlier than kernelB in the appropriate stream order. The following instance reveals varied legitimate usages. Figure 1 exhibits the assorted dependencies specified in the sooner code instance. As you may see, all kernels are ordered to execute after the allocation operation and full earlier than the deallocation operation. Memory allocation and deallocation can't fail asynchronously. Memory errors that occur due to a name to cudaMallocAsync or cudaFreeAsync (for instance, Memory Wave out of memory) are reported instantly through an error code returned from the call. If cudaMallocAsync completes efficiently, the returned pointer is guaranteed to be a valid pointer to Memory Wave App that is secure to access in the suitable stream order. The CUDA driver makes use of memory swimming pools to achieve the habits of returning a pointer instantly.