google / google/testrun

Sporadic results in Services tests

Open
#1,290 4 comments 0 reactions 0 assignees View on GitHub
BE triaged
Dominant language
Python
Stars
56
Forks
20
Avg merge
15h 54m
Merged PRs (30d)
12

Description

**Describe the bug**
In 2.2 I'm still seeing some sporadic results in Services tests. This issue was initially uncovered in #885 and seems to occur on more resource constrained devices, which are prevalent (and constrained _by design_) in the IoT/smart building world.

![Image](https://github.com/user-attachments/assets/2983f80b-a514-4556-9775-85dd2e23fea5)

**Test A**
![Image](https://github.com/user-attachments/assets/2f035405-6b99-4364-b7b3-5d4a5fb50541)

**Test B**
![Image](https://github.com/user-attachments/assets/1da8f66d-436c-4ce4-9878-9c2af5cbf2d3)

**Test C**
![Image](https://github.com/user-attachments/assets/64466248-f246-49e2-998e-e17303696c0a)

**Test D**
![Image](https://github.com/user-attachments/assets/ca224607-f02d-4a79-8b6a-064027e37190)

**Expected behavior**
The same device under the same conditions should yield the same results every time.

**Error logs**
[testA.zip](https://github.com/user-attachments/files/20087767/testA.zip)
[testB.zip](https://github.com/user-attachments/files/20087768/testB.zip) (Ignore the TLS server test errors, I suspect the device lost power during the TLS module tests)
[testC.zip](https://github.com/user-attachments/files/20087770/testC.zip) (Ignore the TLS module timeout, this is due to #[1178](https://github.com/google/testrun/issues/1178#issuecomment-2719142275))
[testD.zip](https://github.com/user-attachments/files/20087776/testD.zip)

**Environment (please provide the following information about your setup):**
- Version 2.2

**Additional context**
There was discussion on #885 previously about utilising nmap's inbuilt rate limiting features to mitigate the false negatives/false positives that can/will occur against devices currently.

The comment [here ](https://github.com/google/testrun/issues/885#issuecomment-2587628036)states that a second HTTP specific scan is carried out in 2.2 that may well resolve false negatives/false positives with the HTTP related tests, but this naturally doesn't cover all the other possible (good/bad?) service/port permutations out there (TCP 502 in the case of the device above, but that's just one device/service in this example).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.