AccelerateHS / AccelerateHS/accelerate

Improved memory management

未关闭
#240 9 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Haskell
星标
1k
派生
135
PR 合并指标
30 天内没有已合并 PR

描述

I've created this bug so we can discuss and track work on better memory management in Accelerate.

The status quo:
- The accelerate CUDA backend maintains a table that matches host side arrays to device arrays.
- A second "nursery" table is also maintained. It matches array sizes to device arrays.
- When device arrays are "freed" `cudaFree` is not actually called on the underlying device memory, rather an entry is added to the nursery table.
- When a host side array does not have a corresponding device array, a lookup of the nursery is performed to see if an array of the right size is already available for use.
- Device arrays are only ever actually `cudaFree`d when `cudaMalloc` fails and the nursery is flushed.

The advantage of the current approach is it minimises allocations by use of the nursery. One disadvantage is that we are heavily dependent on GHCs GC to know when arrays are no longer needed. As is evident in #165, this can cause a lot of problems. Additionally, there are cases where an array is still reachable from the perspective of the GC, but we know it's never going to be needed again. This is a problem when using the new streaming interface to process more data than will fit in device memory. Lastly, and perhaps less significantly, we have a large memory footprint, which rather assumes Accelerate is the only thing using the device memory.

A related issue is that a lot of the logic for memory management is the same, or similar, for different backends and it would make sense to move this into the Accelerate base package and provide an interface backends can use.

I know that you had some ideas about this @tmcdonell and @rrnewton, but I've written a simple interface below to show my current thinking. The more I think about this, the more I see that there are many things we need to think about.

``` haskell
class RemoteMemory m where
type Remote m e -- The remote information associated with an array

allocate :: m -> Int -> IO (Maybe (ArrayData e))
free :: m -> ArrayData e -> IO ()
attach :: m -> ArrayData e -> IO () -- Keep track of an existing host-side array in the Remote memory
push :: m -> ArrayData e -> IO ()
pull :: m -> ArrayData e -> IO ()
withRemote :: ArrayData e -> (Remote m e -> IO a) -> IO a

newtype LRU m = LRU m (... some structure used as an array table ...) (... some structure used as a nursery...)

instance RemoteMemory m => RemoteMemory (LRU m) where
type Remote (Lazy m) a = Remote m a
allocate (Lazy m n) sz =
if inNursery n sz then ... use array from nursery ... else
if ... not enough free space in device memory... then ... flush nursery ... else
... remove LRU array from table ...
free (Lazy m n) a = insertNursery n a
... the rest as expected ...

---------------------------------
-- In the CUDA backend.
newtype CudaMem = CudaMem

instance RemoteMemory CudaMem where
type Remote CudaMem a = DevicePtrs a
... All the functions trivially implememnted with just cudaMalloc, cudaFree and cudaMemCpy ...
```

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。