microsoft / microsoft/SPTAG

thread local context (#359) causes test issue on Linux: `1: [4] fid:0 channel 2, to submit:64, submitted:Operation not permitted`

Open
#375 21 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
5k
Forks
622
Avg merge
8h 37m
Merged PRs (30d)
5

Description

On Linux, #359 by @PhilipBAdams causes the SSDServingTest TestSearchSSDIndexFloatL2BKTDEFAULTTXT test to continously print this message over and over again:

 1: [4] fid:0 channel 2, to submit:64, submitted:Operation not permitted

This message comes from the BatchReadFileAsync function in AnnService/src/Helper/AsyncFileReader.cpp and is printed when the io_submit syscall fails:

int s = syscall(__NR_io_submit, handler->GetIOCP(channel), iocbs[i].size() - submitted[i], iocbs[i].data() + submitted[i]);
if (s > 0) {
    submitted[i] += s;
    totalSubmitted += s;
}
else {
    SPTAGLIB_LOG(Helper::LogLevel::LL_Error, "fid:%d channel %d, to submit:%d, submitted:%s\n", i, channel, iocbs[i].size() - submitted[i], strerror(-s));
}

According to the io_submit manual page, this indicates incorrectly setting the RT IO priority class:

EPERM  The aio_reqprio field is set with the class IOPRIO_CLASS_RT, but the submitting context does not have the CAP_SYS_ADMIN capability.

However, the code does not set the IO priority class anywhere.

My conclusion is that maybe the iocb structure being passed to io_submit gets corrupted somehow, but I am not sure.

Thoughts and ideas for further debugging of this are welcome.

I also tried commenting out the contents of the Helper::SetThreadAffinity function but that didn't work.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with SSDServingTest, specifically TestSearchSSDIndexFloatL2BKTDEFAULTTXT, and inspect BatchReadFileAsync in AnnService/src/Helper/AsyncFileReader.cpp where io_submit failures are logged. Compare the submitted iocb data and IO priority handling on Linux, including whether SetThreadAffinity is relevant. Done means the test no longer repeatedly reports Operation not permitted.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
operating-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.