tensorflow / tensorflow/tensorboard
Clean Bazel test output cache before running CI cron tests
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 7.2k
- Forks
- 1.7k
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 1
Description
The main purpose of our CI cron job is to quickly detect breaking
changes in TensorFlow nightlies. But the Bazel cache that indicates
which tests have passed is preserved across TensorFlow nightly versions,
so in fact our cron job is of very little utility because all* the
tests are already cached due to the post-merge PR hook.
As a result, breaking changes in TensorFlow nightlies are simply
undetected by our cron jobs, causing subsequent pull requests to fail.
This has happened at least twice thrice four times now: #2035,
#2051, #2115, #3260.
A simple solution could be to expunge the Bazel cache when we detect
that the current job is a cron job.
(Ideally, the build cache would actually be keyed off of the TensorFlow
version. The Bazel cache key does include PATH, but we always use
the default virtualenv provided by Travis and simply install our
packages into it, so the actual path does not change.)
* In fact, a select few tests seem to always run. I’m not sure what
makes them special:
//tensorboard:manager_e2e_test
//tensorboard:program_test
//tensorboard/plugins/debugger:tensor_helper_test
//tensorboard/plugins/debugger:tensor_store_test
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the Travis cron configuration and the Bazel cache setup, using the linked cron-job detection documentation as context. Check the listed TensorBoard test targets and confirm that the cron job does not reuse their cached results; done means cron runs expunge or otherwise invalidate the cache before testing.
Written by the indexing model from the issue text.
Assessment
- Domain
- build-system, ci-cd
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100