GoogleCloudPlatform / GoogleCloudPlatform/fluentbit-bigquery-writeapi-sink

[BUG] parseMap() doesn't handle arrays recursively, causing JSON marshal errors

Open
#66 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
14
Forks
6
PR merge metrics
No merged PRs in 30d

Description

## Summary
The `parseMap()` function doesn't recursively process arrays, leaving
`map[interface{}]interface{}`
inside arrays unconverted. This causes JSON marshaling to fail with nested data structures.

## Environment
- Plugin version: latest (main branch)
- Go version: 1.23.0
- Error: `json: unsupported type: map[interface {}]interface {}`

## Problem Description

When processing data with nested arrays containing objects, the plugin fails with a JSON marshal
error.

Example data that causes the error:

```json
{
"request": {
"imp": [{
"banner": {"w": 300, "h": 250}
}]
}
}
```

### Root Cause

The current parseMap() implementation:

```go
func parseMap(m0 map[interface{}]interface{}) map[string]interface{} {
m1 := make(map[string]interface{})
for k, v := range m0 {
switch t := v.(type) {
case []byte:
m1[k.(string)] = string(t)
case map[interface{}]interface{}:
m1[k.(string)] = parseMap(t)
default: // Arrays pass through without processing
m1[k.(string)] = v
}
}
return m1
}

```
The issue: Arrays ([]interface{}) containing map[interface{}]interface{} are not processed,
causing json.Marshal() to fail when it encounters these unprocessed maps.

### Proposed Fix

Add recursive array processing:

```go
func parseMap(m0 map[interface{}]interface{}) map[string]interface{} {
m1 := make(map[string]interface{})
for k, v := range m0 {
switch t := v.(type) {
case []byte:
m1[k.(string)] = string(t)
case map[interface{}]interface{}:
m1[k.(string)] = parseMap(t)
case []interface{}: // Add array handling
m1[k.(string)] = parseArray(t)
default:
m1[k.(string)] = v
}
}
return m1
}

func parseArray(a0 []interface{}) []interface{} {
a1 := make([]interface{}, len(a0))
for i, v := range a0 {
switch t := v.(type) {
case map[interface{}]interface{}:
a1[i] = parseMap(t)
case []interface{}:
a1[i] = parseArray(t)
default:
a1[i] = v
}
}
return a1
}
```

## Impact

- Any data with arrays containing objects fails to write to BigQuery
- Common with OpenRTB bid request data and similar nested structures

I'm happy to submit a PR with this fix if you agree with the approach.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.