The test system could also use some :heart:

There is currently a lack of information on how to test the repository. Also it would be nice to have some way of seeing which tests are supposed to pass and which ones are currently expected to fail.

Running oss_scripts/run_tests.sh does not work as expected and is currently not documented. It runs tests in parallel and fails tests at random (roughly 19/36). Note that adding --jobs=1 reduces that number to only these failures on my machine:

//tensorflow_text:whitespace_tokenizer_test                    FAILED TO BUILD
//tensorflow_text:bert_tokenizer_test                          FAILED in 14.7s
//tensorflow_text:gather_with_default_op_test                  FAILED in 5.4s
//tensorflow_text:sentence_breaking_ops_test                   FAILED in 5.9s

Note that the benchmark targets tokenizers_benchmark and ops_benchmark may not be intended to be in there.

Now the problem is that it is unclear whether these tests are expected to fail or the failures come from my setup as there is no official "Hey these tests are expected to fail".
Another issue is e.g. that many more of these test fail when not running with --jobs=1.

The failure for whitespace_tokenizer seems to be expected as some recent commits seem to change that a lot, but I am not sure about the other ones and ill look into it.

To be clear, i do not think it matters much whether these tests fail in general, but it makes it very hard to change things up when there is no information on which tests are currently expected to pass/fail.

For anyone interested in reproducing the following commands should work (same as oss_scripts/run_tests.sh, but singlethreaded and ignoring failures):

# Generate a .bazelrc
./oss_scripts/configure.sh

# Run the tests
bazel test --test_output=errors --keep_going --jobs=1 tensorflow_text:all

Note that for the benchmarks to pass you need tensorflow_datasets in your python toolchain (e.g. via pip install tensorflow_datasets). Also note that running the benchmarks like that may be pointless anyways, but lets just ignore that for now 🎉

Please let me know whether it is safe to add the --jobs=1 flag to the test script. It may break some internal builds or somthing and it will definitely slow testing down by a lot. An alternative would be to find the tests which require singlethreaded execution and tag them with the exclusive keyword, so that only those tests run in serial.

Start with oss_scripts/run_tests.sh and oss_scripts/configure.sh, then run the documented Bazel command with --jobs=1 to reproduce the listed failures. Check which failures are expected and whether benchmark targets belong in the test run. Done means the repository documents how to run tests and identifies expected pass and fail results, including any serial-execution requirements.