[VL] Scan split proto contains duplicate field
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
```json
{
"uriFile": "oss://xxx/tpcds/partitioned/tpcds_sf100/web_sales/ws_sold_date_sk=2451145/part-00086-5f7cfd3c-f467-4d40-82e3-d2033487db84.c000.snappy.parquet",
"partitionIndex": "5",
"length": "5483534",
"parquet": {},
"partitionColumns": [
{
"key": "ws_sold_date_sk",
"value": "2451145"
}
],
"schema": {},
"metadataColumns": [
{
"key": "input_file_name",
"value": "oss://xxx/tpcds/partitioned/tpcds_sf100/web_sales/ws_sold_date_sk=2451145/part-00086-5f7cfd3c-f467-4d40-82e3-d2033487db84.c000.snappy.parquet"
},
{
"key": "input_file_block_length",
"value": "5483534"
},
{
"key": "input_file_block_start",
"value": "0"
}
],
"properties": {
"fileSize": "5483534",
"modificationTime": "1719801698242"
}
}
```
some fields are duplicate, should be merged.
@gaoyangxiaozhu @acvictor
Contributor guide
Research direction
Start with the scan split proto output shown in the issue and trace where its fields are assembled or serialized. No files or tests are named; done means determining the intended merged representation and confirming that duplicate fields no longer appear in the scan split.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100