haskell / haskell/bytestring

hPutBuf/writeFile fails on OSX for bytestrings over ~2^31 bytes in size

Open
#153 35 comments 0 reactions 0 assignees View on GitHub
bug pr-welcome
Dominant language
Haskell
Stars
301
Forks
144
Avg merge
7d 22h
Merged PRs (30d)
1

Description

Writing more than ~2GB at once to a file appears to fail on OSX. This may be related to a platform issue also encountered in this python ticket: https://bugs.python.org/issue24658

## Repro:
The following code fails on OSX, emitting an exception:

```module Main where

import qualified Data.ByteString as BS
import qualified Data.ByteString.Lazy as LBS

main :: IO ()
main = do
bigBS <- pure $ BS.replicate (2^32) 1
BS.writeFile "testFile" bigBS
```
`*** Exception: testFile: hPutBuf: invalid argument (Invalid argument)`

## Alt approaches
Manually creating a file handle and using hPut directly makes no difference
```module Main where

import System.IO
import qualified Data.ByteString as BS

main :: IO ()
main = do
bigBS <- pure $ BS.replicate (2^32) 1
fileHandle <- openFile "testFile" WriteMode
BS.hPut fileHandle bigBS
```

Writing a lazy bytestring also makes no difference
```
module Main where

import System.IO
import qualified Data.ByteString as BS
import qualified Data.ByteString.Lazy as LBS

main :: IO ()
main = do
bigBS <- pure $ BS.replicate (2^32) 1
fileHandle <- openFile "testFile" WriteMode
LBS.hPut fileHandle (LBS.fromStrict bigBS)
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.