ARMmbed / ARMmbed/ci-test-shield

I2C Test assert not catching real failures

Open
#16 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
9
Forks
33
PR merge metrics
No merged PRs in 30d

Description

The I2C EEPROM tests should fail on K64F for an other issue. Basically, due to this other issue, the K64F isn't able to read from the EEPROM. But, the ci-test-shield I2C EEPROM tests for writing/reading 10 and 100 bytes are showing OK. This is not correct.

Here is the very verbose (--vv) log for one of the tests. It shows that the number of bytes read is 0 and the Read string is null. But the asserts are not catching this.
```
[1481314762.72][CONN][RXD] >>> Running case #5: 'I2C - EEProm WR 10 Bytes'...
[1481314762.77][CONN][RXD]
[1481314762.77][CONN][INF] found KV pair in stream: {{__testcase_start;I2C - EEProm WR 10 Bytes}}, queued...
[1481314762.78][CONN][RXD] ****
[1481314762.82][CONN][RXD] Buffer Len = `10`, String = `DYOQYKFDB`
[1481314762.83][CONN][RXD] ****
[1481314762.83][CONN][RXD]
[1481314762.83][CONN][RXD] ****
[1481314762.86][CONN][RXD] Test String = `DYOQYKFDB`
[1481314762.87][CONN][RXD] ****
[1481314762.90][CONN][RXD] num_read: 0, num_written: 10
[1481314762.90][CONN][RXD]
[1481314762.91][CONN][RXD] ****
[1481314762.93][CONN][RXD] Address = `100`
[1481314762.94][CONN][RXD] Len = `10`
[1481314762.97][CONN][RXD] Num Bytes Written = `10`
[1481314762.99][CONN][RXD] Num Bytes Read = `0`
[1481314763.03][CONN][RXD] Written String = `DYOQYKFDB`
[1481314763.05][CONN][RXD] Read String = ``
[1481314763.05][CONN][RXD] ****
[1481314763.11][CONN][INF] found KV pair in stream: {{__testcase_finish;I2C - EEProm WR 10 Bytes;1;0}}, queued...
[1481314763.16][CONN][RXD] >>> 'I2C - EEProm WR 10 Bytes': 1 passed, 0 failed
```

```
+--------------+---------------+---------------+------------------------------------+--------+--------+--------+--------------------+
| target | platform_name | test suite | test case | passed | failed | result | elapsed_time (sec) |
+--------------+---------------+---------------+------------------------------------+--------+--------+--------+--------------------+
| K64F-GCC_ARM | K64F | tests-api-i2c | I2C - EEProm WR 10 Bytes | 1 | 0 | OK | 0.34 |
| K64F-GCC_ARM | K64F | tests-api-i2c | I2C - EEProm WR 100 Bytes | 1 | 0 | OK | 0.74 |
| K64F-GCC_ARM | K64F | tests-api-i2c | I2C - EEProm WR 2 Bytes | 0 | 1 | FAIL | 0.35 |
| K64F-GCC_ARM | K64F | tests-api-i2c | I2C - EEProm WR Single Byte | 0 | 2 | FAIL | 0.31 |
| K64F-GCC_ARM | K64F | tests-api-i2c | I2C - Instantiation of I2C Object | 1 | 0 | OK | 0.06 |
| K64F-GCC_ARM | K64F | tests-api-i2c | I2C - LM75B Temperature Read | 1 | 0 | OK | 0.12 |
+--------------+---------------+---------------+------------------------------------+--------+--------+--------+--------------------+
```

It looks like one of the assert types is distorting the results from the other asserts.

If these two lines are commented out in void flash_WR(), then the tests result in the expected and correct behavior (fail when it actually fails). However, with these lines not-commented out, the tests result in OK.

```
TEST_ASSERT_MESSAGE(strcmp((char *)test_string,(char *)read_string) == 0,"String Written != String Read");
TEST_ASSERT_MESSAGE(strcmp((char *)read_string,(char *)test_string) == 0,"String Written != String Read");
//TEST_ASSERT_EQUAL_STRING_MESSAGE((char *)test_string,(char *)read_string,"String read does not match the string written");
//TEST_ASSERT_EQUAL_STRING_MESSAGE((char *)read_string,(char *)test_string,"String read does not match the string written");
TEST_ASSERT_EQUAL_MESSAGE(num_written,num_read,"Number of bytes written does not match the number of bytes read");
DEBUG_PRINTF("\r\n****\r\n Address = `%d`\r\n Len = `%d`\r\n Num Bytes Written = `%d` \r\n Num Bytes Read = `%d` \r\n Written String = `%s` \r\n Read String = `%s` \r\n****\r\n",address,size_of_data,num_written,num_read,test_string,read_string);
```

@bridadan mentioned that he thinks this has something to do with the use of STATUS_CONTINUE in greentea_failure_handler

@adbridge
@BlackstoneEngineering

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in void flash_WR() and inspect the TEST_ASSERT and TEST_ASSERT_MESSAGE calls shown in the issue, then read greentea_failure_handler where STATUS_CONTINUE is mentioned. Run the ci-test-shield I2C EEPROM tests on K64F and compare the reported assertions with the verbose log. Done means failed reads are reported as failed tests rather than OK.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
embedded-iot, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.