AccelerateHS / AccelerateHS/accelerate
Improved memory management
- Vorherrschende Sprache
- Haskell
- Sterne
- 1k
- Forks
- 135
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
I've created this bug so we can discuss and track work on better memory management in Accelerate.
The status quo:
- The accelerate CUDA backend maintains a table that matches host side arrays to device arrays.
- A second "nursery" table is also maintained. It matches array sizes to device arrays.
- When device arrays are "freed" `cudaFree` is not actually called on the underlying device memory, rather an entry is added to the nursery table.
- When a host side array does not have a corresponding device array, a lookup of the nursery is performed to see if an array of the right size is already available for use.
- Device arrays are only ever actually `cudaFree`d when `cudaMalloc` fails and the nursery is flushed.
The advantage of the current approach is it minimises allocations by use of the nursery. One disadvantage is that we are heavily dependent on GHCs GC to know when arrays are no longer needed. As is evident in #165, this can cause a lot of problems. Additionally, there are cases where an array is still reachable from the perspective of the GC, but we know it's never going to be needed again. This is a problem when using the new streaming interface to process more data than will fit in device memory. Lastly, and perhaps less significantly, we have a large memory footprint, which rather assumes Accelerate is the only thing using the device memory.
A related issue is that a lot of the logic for memory management is the same, or similar, for different backends and it would make sense to move this into the Accelerate base package and provide an interface backends can use.
I know that you had some ideas about this @tmcdonell and @rrnewton, but I've written a simple interface below to show my current thinking. The more I think about this, the more I see that there are many things we need to think about.
``` haskell
class RemoteMemory m where
type Remote m e -- The remote information associated with an array
allocate :: m -> Int -> IO (Maybe (ArrayData e))
free :: m -> ArrayData e -> IO ()
attach :: m -> ArrayData e -> IO () -- Keep track of an existing host-side array in the Remote memory
push :: m -> ArrayData e -> IO ()
pull :: m -> ArrayData e -> IO ()
withRemote :: ArrayData e -> (Remote m e -> IO a) -> IO a
newtype LRU m = LRU m (... some structure used as an array table ...) (... some structure used as a nursery...)
instance RemoteMemory m => RemoteMemory (LRU m) where
type Remote (Lazy m) a = Remote m a
allocate (Lazy m n) sz =
if inNursery n sz then ... use array from nursery ... else
if ... not enough free space in device memory... then ... flush nursery ... else
... remove LRU array from table ...
free (Lazy m n) a = insertNursery n a
... the rest as expected ...
---------------------------------
-- In the CUDA backend.
newtype CudaMem = CudaMem
instance RemoteMemory CudaMem where
type Remote CudaMem a = DevicePtrs a
... All the functions trivially implememnted with just cudaMalloc, cudaFree and cudaMemCpy ...
```
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Bewertung
Dieses Issue wurde noch nicht bewertet.