bazelbuild / bazelbuild/bazel

[6.2.0] instrumentation_filter doesn't work for bazel coverage with py_test

Open
#18,362 2 comments 0 reactions 0 assignees View on GitHub
coverage P2 team-Rules-Server type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

we run `bazel coverage` over our py_test target `common_tools_test`. Some of our test cases will create temp directory under execution and delete afterwards. Unexpected, this temp directory will be included into the `pylcov.dat` report which causes `no source code` error. Since on macos, pytest tmpdir will create the directory under `/private/var/folders`, I tried to omit this directory with `instrumentation_filter` but it doesn't work.
`Converting coveragepy database to lcov: /private/var/tmp/_bazel_nlou/b43af9baf9d056244183c6c3d6dd8487/sandbox/darwin-sandbox/5/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/_coverage/common_tools_test/test/pylcov.dat `

```
TN:
SF:/private/var/folders/6m/nl3wnqp96hz5b53qpx2y71f40000gp/T/pytest-of-nlou/pytest-375/test_load_config_from_fs0/dev/github.json

```
```
py_test(
name = "common_tools_test",
srcs = glob(["test/**/*.py"]),
data = glob(["test/common_tools/data/*"]) + ["//:config"],
main = "__test__.py",
python_linters = [
],
deps = [
"//:common_tools_lib",
"//confluent/config",
requirement("click"),
requirement("mock"),
requirement("pytest"),
],
)
```
```
% cat .bazelrc
# coverage --combined_report=lcov
coverage --test_env=VERBOSE_COVERAGE=true
coverage --java_runtime_version=remotejdk_11
coverage --action_env=VERBOSE_COVERAGE=true
coverage --sandbox_debug
coverage --test_output=streamed
coverage --verbose_failures
coverage --instrument_test_targets=false
coverage --instrumentation_filter="/confluent[/:],-/private/var/folders"
coverage --combined_report=none
```

Also when I run `coveragepy` tool , I am able to make this work with this configuration in `.coveragerc` .
```
[run]
omit =
.git/*
.tox/*
docs/*
setup.py
test/*
tests/*
bazel-*/*
/private/var/folders/*. <====
```

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

create a test case with pytest which uses `tmpdir` to create a temp directory and files under that directory.

e.g.

```
def test_load_config_from_fs(tmpdir, getpass):
getpass.getuser.return_value = "mzheng"

paths_json = tmpdir.mkdir("dev").join("paths.json")
paths_json.write('{"TMP_PATH": "/tmp/{{ user }}"}')

github_json = tmpdir.join("dev", "github.json")
github_json.write('{"mzheng": "maxzheng"}')

with tmpdir.as_cwd():
paths = load_config(config_name="paths", fs_root=".")

assert paths == {"TMP_PATH": "/tmp/maxzheng"}
```

### Which operating system are you running Bazel on?

darwin-x86_64

### What is the output of `bazel info release`?

bazel info release INFO: Invocation ID: da725c55-1dd1-4435-bc7c-cbe83b030e23 release 6.2.0

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

```text
git@github.com:confluentinc/common-tools.git
```

### Have you found anything relevant by searching the web?

no

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the .bazelrc coverage settings and the py_test target common_tools_test; reproduce with bazel coverage and a pytest tmpdir test, then inspect the generated pylcov.dat. Done means the instrumentation_filter behavior excludes the temporary directory without producing a "no source code" error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.