Sanitizer tracking issue

Sanitizer integration in rust is still quite experimental and tends to produce false positives. I went through a bunch of logs and think the issues below could be real bugs. I've added a few points of interest to key points in the codebase after some initial disentangling of the error logs.

AddressSanitizer:

• #187 A lot of leaked memory during cas/store:ref_store_test. Seems to occur in get_test and update_test. POI:
  • The setup_stores call in update_test, in ref_store_owned. Seemingly during some clone operations.
  • The add_store call in setup_stores.
  • The first scope in each of the tests.
  • The ref_store field in the RefStore::new implementation.
  • The unsafe impl Sync for StoreReference {}.
  • The RefStore struct.
  • The name.to_string call in stores.insert in cas/store/lib.rs in add_store.
  • Seems like everything points to the string keys of the hashmap. in the stores.insert call.
  • Related issues:
    • https://users.rust-lang.org/t/understand-memory-leak-with-parkinglot-and-hashmap/92519
    • https://github.com/rust-lang/rust/issues/73307

ThreadSanitizer:

• Data race in cas/worker:running_actions_manager_test in cleanup_happens_on_job_failure. POI:

  • Creation of fast_store in the setup_stores for these tests.
  • The try_join in running_actions_manager.rs in upload_results.

• Data race in cas/worker:local_worker_test in new_local_worker_removes_work_directory_before_start_test. POI:

  • The new_local_worker in the failing test.
  • The fs::canonicalize call in new_local_worker in cas/worker/local_worker.rs.

• Data race in cas/store:filesystem_store_test in oldest_entry_evicted_with_access_times_loaded_from_disk. POI:

  • The match statement at the end of the test on store.get_file_entry_for_digest.
  • The fs::create_dir_all call in the test.
  • The write_file calls in that test.

Start by running the listed AddressSanitizer and ThreadSanitizer tests, especially cas/store:ref_store_test and the three cas/worker or cas/store tests. Trace the points of interest in ref_store_owned, running_actions_manager.rs, local_worker.rs, and filesystem_store tests. Done means determining whether each reported sanitizer finding is a real bug and resolving or documenting the remaining findings.