arduino / arduino/ArduinoCore-mbed
GalaxyCore GC2145 image corruption on Nicla Vision over serial
- Dominant language
- C
- Stars
- 411
- Forks
- 225
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
We have implemented a library to trasmit the sensors data of the Nicla: https://github.com/ADVRHumanoids/nicla_vision_drivers through WiFi (UDP or TCP)
Recently I am trying to transmit such data over Serial, using the USB cable, which if I am not wrong utilizes UART connection.
By doing so, some images are received (partially) corrupted. This is noticeable since it happens frequently, but most of the time images are fine:
**RGB565, 320x240**
With WiFi connection, the problem happens but very infrequently and hence tolerable.
## Hardware
Hardware-wise, I am trying with different USB cables and different nicla boards and the behaviour is the same. If the problem is not already there, just shaking the nicla (to try to see if cable/port connection is bad) does not make the corruption to occur.
## Software
Software-wise, I am sending data like this:
```c++
Serial.write((uint8_t*)header_buffer, headerLength);
Serial.write((uint8_t*)img_ptr, camPtr->frameSize());
Serial.write(END_BYTE);
```
In receiving the package (on PC, python) where I check for the START_BYTE, END_BYTE. Furthermore I check the packet size, and checksum/CRC-32 (tried both), from bytes inside the header. They are always correct, even when the corruption happen.
Digging out more, I found out that the corruption is indeed just a byte's shift. On the receiver side, when shifting the data of one byte with `data_shifted = data[1:] + data[:1]`, the corruption is "dual":
I am not super sure, but this corruption seems to happen as soon strong light/white areas are captured by the camera:
https://github.com/user-attachments/assets/4fbb36ea-12e4-4101-9246-63add3e82088
### Bayer
For now, I am practically solving trasmitting `CAMERA_BAYER` pixel format instead of `CAMERA_RGB565` (tried CAMERA_R320x240 as well CAMERA_R320x320 this time) . But I am not sure if the problem is solved or only less noticeable because of the Bayer representation. Indeed it seems that images turns from more "blue-ish" to "red-ish" and viceversa:
**BAYER, 320x320**
<\div>
https://github.com/user-attachments/assets/de73398b-6bbd-4399-b224-d3ea21504f6f
I can also see some RGB spurious pixel in the contours but this may related to the Bayer demosacing.
More, arduino code put a warning for BAYER mode but I am unsure about its meaning and if I should handle the data differently:
```cpp
case CAMERA_BAYER:
// There's no BAYER support so it will just look off.
// Make sure odd/even row are switched to work with our bayer conversion.
ret |= regWrite(GC2145_I2C_ADDR,
REG_SYNC_MODE, REG_SYNC_MODE_DEF | REG_SYNC_MODE_ROW_SWITCH);
ret |= regWrite(GC2145_I2C_ADDR,
REG_OUTPUT_FMT, REG_OUTPUT_SET_FMT(reg, REG_OUTPUT_FMT_BAYER));
break;
```
### Bonus
As a bonus issue, here is what happens with bayer and 640 resolution: (`CAMERA_R640x480`)
**BAYER, 640x480**
Colors are always wrong, no matter all the [OpenCV available conversions](https://docs.opencv.org/3.4/d8/d01/group__imgproc__color__conversions.html) for Bayer (with `cv2.COLOR_BayerGBRG2BGR` good for 320 resolution and that should be the right one according to GC2145 documentation).
I can also see the mosaic RGB squared pattern on the image.
---
Anyone has any suggestions?
Thanks
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Serial.write calls shown in the issue and the Python receiver's START_BYTE, END_BYTE, size, and CRC checks. Then inspect the Nicla Vision GC2145 camera-format handling around CAMERA_RGB565 and CAMERA_BAYER, including the cited row-switch and output-format registers. Done means isolating whether corruption comes from capture, serial framing, or reception and documenting a reproducible fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- computer-vision, embedded-iot
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100