JuliaIO / JuliaIO/StructIO.jl

Performance issues when reading multiple packed structures

Open
#13 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
27
Forks
9
PR merge metrics
No merged PRs in 30d

Description

I've encountered this in replacing manually written out read and write methods with StructIO in visr/LasIO.jl#10. Although using StructIO is elegant, it's also slower.

I've put up a gist here: https://gist.github.com/evetion/2b57d6105cca39b2d3c6ef670a5cc393 with the following results for reading a thousand TwoUInt64s.

➜ julia performance.jl

Using StructIO:
  4.143 ms (2000 allocations: 62.50 KiB)

Using read_generic_array:
  6.138 ms (12000 allocations: 406.25 KiB)

Using read_generic_tuple:
  480.796 μs (8000 allocations: 265.63 KiB)

Using read_written_out:
  29.229 μs (2000 allocations: 31.25 KiB)

Using generated_read:
  29.704 μs (2000 allocations: 31.25 KiB)

The handwritten read version, which also can be generated, is ~200 times faster.

I know this is an unfair comparison, as StructIO has much more functionality than these simple read functions, but it seems it could be faster, especially if you look at the allocations, which are on par.

Let me know if you can't duplicate these results, or if I'm missing a StructIO method for reading multiple packed structures.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the comparison in the referenced performance.jl gist and inspect the StructIO paths behind read_generic_array, read_generic_tuple, read_written_out, and generated_read. Compare their allocation and timing results for multiple packed structures, then document or implement a measured improvement over the current StructIO behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.