microsoft / microsoft/SPTAG

ReadString function is inefficient

Open
#289 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
5k
Forks
622
Avg merge
8h 37m
Merged PRs (30d)
5

Description

Is your feature request related to a problem? Please describe.
When invoking any Helper::DiskIO ReadString function, the function may over estimate the buffer size, by doubling until it fits into the buffer. The function also scans each char for delim and breaks.

Describe the solution you'd like
Instead of having ReadString do resizing and delim parsing. The ReadString should only be responsible for reading into the buffer with the expected size. The expected size should come from the file size, as this is the exact size that must be read. The parsing shouldnt be done at all, instead, in WriteString always write terminating point (the delim always equals \n and then replaced by \0, so why not just let WriteString set the end point?).
Both ReadString / WriteString will basically be boiled down to ReadBinary / WriteBinary.

Additional context
This ReadString is used in reading config (ex. ini) and metadata files (ex. tsv).

By default the read buffer size always starts at 2^16 = 65,536 bytes.

The biggest inefficiency comes from metadata files as they are big. Ex. 100GB file, we know the read is divided up by threads (32), so each thread will eventually resize the buffer to ~4GB (exact size would be ~3GB, so one GB over), which then means we have over estimated the buffer size ~32GB (total over est).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the Helper::DiskIO ReadString and WriteString implementations, then compare them with ReadBinary and WriteBinary. Trace their use for config and metadata files, especially threaded reads, and verify the proposed behavior against the stated buffer-sizing and delimiter assumptions. Done means the string functions avoid repeated resizing and delimiter scanning while preserving correct file contents.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.