IBM-Cloud / IBM-Cloud/ibm-cloud-cli-sdk

CLI is not safe for multi-process use

Open
#239 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
35
Forks
53
Avg merge
1d 15h
Merged PRs (30d)
16

Description

Description:

Running the CLI in parallel frequently causes the config file to be overwritten with empty values. I instrumented the SDK and saw that sometimes when one process was writing, another process would read. Those reads returned 0bytes with no error. They then wrote empty configs back to config.json.

You can see it happening here.

     0B 1615140454645170000-write-beg-34920    <<< PID 34920 write begins
     0B 1615140454645773000-read-beg-34924     <<< PID 34924 read begins
     0B 1615140454646638000-read-end-34924     <<< PID 34924 read ends (0b)
   3.9K 1615140454646771000-write-end-34920    <<< PID 34920 write ends (3.9k)
     0B 1615140454646951000-write-beg-34924    <<< PID 34924 write begins
   945B 1615140454647265000-write-end-34924    <<< PID 34924 write ends (945b)
     0B 1615140454746809000-read-beg-34924     <<< PID 34924 read begins
   945B 1615140454747004000-read-end-34924     <<< PID 34924 read ends (945b)

Here's what happens in order:

  1. proc A starts to write 3.9k bytes
  2. proc B reads 0 bytes
  3. proc A complete write of 3.9k bytes
  4. proc B writes 945 bytes
  5. proc B reads 945 bytes

The 945 byte write is for an empty config struct, effectively logging out all the other processes.

Impact:

The log outs make test automation painful. E.g., they are preventing us from running a large test suite in parallel. I expect users will see more of these errors because IBM Cloud has shortened the TTL of its oauth tokens. That requires plugins to refresh and write oauth tokens more often, which will lead to more conflicts.

This is also a problem when updating plugins.

Repro:

Create a simple plugin that calls RefreshIAMToken on start. Then run the code using a script similar to

#!/bin/bash

set -ex
COUNT=10

for i in `seq 1 $COUNT` ; do
  (
  while ibmcloud myplugin some-command; do
    true
  done
  ) &
done
wait

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the CLI SDK code involved in config.json reads and writes, then trace the RefreshIAMToken path described in the issue. Reproduce the race with the provided shell script and ten parallel plugin processes. Done means concurrent commands no longer overwrite config.json with empty values or log out other processes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.