Flaky MFS sharness test
- Dominant language
- Go
- Stars
- 17.1k
- Forks
- 3.2k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 11
Description
Update 9/13 (@schomatis): **Not much to be done at the moment.** The summary of this flaky test (which we haven't seen failing again nor have we been able to reproduce) is that the source of the fail was `ipfs files write --flush=false` running locally instead of remotely through the daemon. In that scenario the flush flag is virtually ignored as we end up closing the repo after the command which will trigger a full flush from the MFS root anyway.
The reason for the command running locally instead of remotely is unknown but likely one of the following:
* The daemon was in fact not running when it was supposed to (crashed, never started in the first place, etc.). Ideally we would need a more robust mechanism of detecting this throughout the tests (and not just at the beginning when starting the daemon) as proposed in https://github.com/ipfs/go-ipfs/issues/8311, but this is hard to implement and with low return. As a workaround to this we have added an extra check specifically for the failing test to poll the daemon's API and make sure it's running right before calling the `ipfs files` command (https://github.com/ipfs/go-ipfs/pull/8432). That check should be enough to uncover this potential source if it happens again (for this particular test alone).
* The daemon was running but we executed the command locally anyway because `go-ipfs-cmds` failed to detect it (as explained https://github.com/ipfs/go-ipfs/issues/8311#issuecomment-889983795, it has a different detection mechanism). In that case `go-ipfs-cmds` silently calls the command locally and we fail to notice the difference. There is no way to detect this at the moment so https://github.com/ipfs/go-ipfs-cmds/issues/218 was created to add more granularity to the local/remote run choice and force `go-ipfs-cmds` to run the command remotely _or fail_ otherwise in the specific case of the `--flush=false` flag being set.
Update 9/6 (@schomatis). This issue is back on my radar although I'm not sure if there is much more value here other than what has already been noted. We probably want to proceed with https://github.com/ipfs/go-ipfs/issues/8311 instead.
Update 7/14 (@schomatis): My best guess so far is that this was just a race bug in how the daemon is set up in `test_launch_ipfs_daemon` which caused the flush flag to be run in a standalone/direct way without the daemon running and hence trigger the flush even when we didn't intended to.
* Does the daemon do some sort of general MFS flush on stop? Yes, when the daemon ends we close the root and flush everything (`GetNode()` and `sync()` on each child recursively):
https://github.com/ipfs/go-ipfs/blob/ae306994a5254b2d0fc8a7a63ca2e8695c5f745d/core/node/core.go#L159
We need a way to tell in which way we are running the command (standalone or through deamon) as the commands might have a different behavior. The `flush=false` should fail from the start if the write command is run in standalone mode.
Trying to isolate the offending test. There is too much entangling so I need to go deeper on the surrounding tests of the entire `test_files_api` (the next test usually depends on the state left by the previous one).
WIP test branch https://github.com/ipfs/go-ipfs/commits/schomatis/sharness/flaky-mfs-test.
------------------------------------------
Shareness test `1130 - root hash not bubbled up yet (offline, cidv1) - sharnessLinux.t0250-files-api` failed in this CI run https://app.circleci.com/pipelines/github/ipfs/go-ipfs/4611/workflows/e5849e96-e5cf-4df3-bb50-2b4517f43d24/jobs/51165/tests#failed-test-0.
It appears to be a flaky test related to MFS and using `--flush=false`.
Contributor guide
Assessment
This issue has not been assessed yet.