ARMmbed / ARMmbed/ci-test-shield
I2C Test assert not catching real failures
- Dominant language
- C++
- Stars
- 9
- Forks
- 33
- PR merge metrics
- No merged PRs in 30d
Description
The I2C EEPROM tests should fail on K64F for an other issue. Basically, due to this other issue, the K64F isn't able to read from the EEPROM. But, the ci-test-shield I2C EEPROM tests for writing/reading 10 and 100 bytes are showing OK. This is not correct.
Here is the very verbose (--vv) log for one of the tests. It shows that the number of bytes read is 0 and the Read string is null. But the asserts are not catching this.
```
[1481314762.72][CONN][RXD] >>> Running case #5: 'I2C - EEProm WR 10 Bytes'...
[1481314762.77][CONN][RXD]
[1481314762.77][CONN][INF] found KV pair in stream: {{__testcase_start;I2C - EEProm WR 10 Bytes}}, queued...
[1481314762.78][CONN][RXD] ****
[1481314762.82][CONN][RXD] Buffer Len = `10`, String = `DYOQYKFDB`
[1481314762.83][CONN][RXD] ****
[1481314762.83][CONN][RXD]
[1481314762.83][CONN][RXD] ****
[1481314762.86][CONN][RXD] Test String = `DYOQYKFDB`
[1481314762.87][CONN][RXD] ****
[1481314762.90][CONN][RXD] num_read: 0, num_written: 10
[1481314762.90][CONN][RXD]
[1481314762.91][CONN][RXD] ****
[1481314762.93][CONN][RXD] Address = `100`
[1481314762.94][CONN][RXD] Len = `10`
[1481314762.97][CONN][RXD] Num Bytes Written = `10`
[1481314762.99][CONN][RXD] Num Bytes Read = `0`
[1481314763.03][CONN][RXD] Written String = `DYOQYKFDB`
[1481314763.05][CONN][RXD] Read String = ``
[1481314763.05][CONN][RXD] ****
[1481314763.11][CONN][INF] found KV pair in stream: {{__testcase_finish;I2C - EEProm WR 10 Bytes;1;0}}, queued...
[1481314763.16][CONN][RXD] >>> 'I2C - EEProm WR 10 Bytes': 1 passed, 0 failed
```
```
+--------------+---------------+---------------+------------------------------------+--------+--------+--------+--------------------+
| target | platform_name | test suite | test case | passed | failed | result | elapsed_time (sec) |
+--------------+---------------+---------------+------------------------------------+--------+--------+--------+--------------------+
| K64F-GCC_ARM | K64F | tests-api-i2c | I2C - EEProm WR 10 Bytes | 1 | 0 | OK | 0.34 |
| K64F-GCC_ARM | K64F | tests-api-i2c | I2C - EEProm WR 100 Bytes | 1 | 0 | OK | 0.74 |
| K64F-GCC_ARM | K64F | tests-api-i2c | I2C - EEProm WR 2 Bytes | 0 | 1 | FAIL | 0.35 |
| K64F-GCC_ARM | K64F | tests-api-i2c | I2C - EEProm WR Single Byte | 0 | 2 | FAIL | 0.31 |
| K64F-GCC_ARM | K64F | tests-api-i2c | I2C - Instantiation of I2C Object | 1 | 0 | OK | 0.06 |
| K64F-GCC_ARM | K64F | tests-api-i2c | I2C - LM75B Temperature Read | 1 | 0 | OK | 0.12 |
+--------------+---------------+---------------+------------------------------------+--------+--------+--------+--------------------+
```
It looks like one of the assert types is distorting the results from the other asserts.
If these two lines are commented out in void flash_WR(), then the tests result in the expected and correct behavior (fail when it actually fails). However, with these lines not-commented out, the tests result in OK.
```
TEST_ASSERT_MESSAGE(strcmp((char *)test_string,(char *)read_string) == 0,"String Written != String Read");
TEST_ASSERT_MESSAGE(strcmp((char *)read_string,(char *)test_string) == 0,"String Written != String Read");
//TEST_ASSERT_EQUAL_STRING_MESSAGE((char *)test_string,(char *)read_string,"String read does not match the string written");
//TEST_ASSERT_EQUAL_STRING_MESSAGE((char *)read_string,(char *)test_string,"String read does not match the string written");
TEST_ASSERT_EQUAL_MESSAGE(num_written,num_read,"Number of bytes written does not match the number of bytes read");
DEBUG_PRINTF("\r\n****\r\n Address = `%d`\r\n Len = `%d`\r\n Num Bytes Written = `%d` \r\n Num Bytes Read = `%d` \r\n Written String = `%s` \r\n Read String = `%s` \r\n****\r\n",address,size_of_data,num_written,num_read,test_string,read_string);
```
@bridadan mentioned that he thinks this has something to do with the use of STATUS_CONTINUE in greentea_failure_handler
@adbridge
@BlackstoneEngineering
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in void flash_WR() and inspect the TEST_ASSERT and TEST_ASSERT_MESSAGE calls shown in the issue, then read greentea_failure_handler where STATUS_CONTINUE is mentioned. Run the ci-test-shield I2C EEPROM tests on K64F and compare the reported assertions with the verbose log. Done means failed reads are reported as failed tests rather than OK.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- embedded-iot, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100