OCFWriter.Append should return number of bytes written
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.1k
- Forks
- 232
- PR merge metrics
- No merged PRs in 30d
Description
Hi! I'm trying to build a data pipeline that writes data to Avro files, and when an individual file reaches a certain size, I want to close it and start a new file. Currently I have to check the file size after every write (records are not fixed-size), but if `OCFWriter.appendDataIntoBlock` returned the number of bytes written through `OCFWriter.Append`, I could just add that number to an accumulator instead of making repeated calls to `os.File.Stat()`.
The exact line where the number of bytes written is thrown away is https://github.com/linkedin/goavro/blob/master/ocf_writer.go#L227
Thank you!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in ocf_writer.go at OCFWriter.Append and the call to appendDataIntoBlock around line 227, where the byte count is discarded. Trace the return values through Append and confirm that callers can receive the number of bytes written; done means Append exposes that count for tracking file size without repeated os.File.Stat() calls.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100