[Passing score blocker] `ParquetReader` involves a CPU time in `get_sample` 5x / 4x times longer than the I/O time
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 203
- Forks
- 67
- Avg merge
- 20m
- Merged PRs (30d)
- 8
Description
Testing flux model on multiple CPU architectures and collecting traces with DFTRACER, the time for getting the next sample with ParquetReader is spent for approximately 80% on the deserialization part rather than in I/O.
For the case of UNET3D and NPZ reader, this was not higher than 20-30%
Also, there seems to be a considerable impact of the CPU architecture in the deserialization time.
This may impact the calculation of the Accelerator Utilization % as mainly influenced by elements not strictly related to storage performance.
This could potentially impact other models using ParquetReader
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported ParquetReader timing with the flux model and DFTRACER, comparing deserialization with I/O and with the UNET3D and NPZ reader cases. Done means the deserialization overhead and CPU-architecture impact are measured, with the effect on Accelerator Utilization % assessed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100