ConduitIO / ConduitIO/conduit-commons

paramgen: add support for custom field (un)marshalling

Open
#142 1 comment 0 reactions 0 assignees View on GitHub
feature
Dominant language
Go
Stars
3
Forks
3
Avg merge
1d 24m
Merged PRs (30d)
3

Description

### Feature description

**The problem**: Here's an example from a connector middleware struct:

```go
type SourceWithSchemaExtraction struct {
SchemaType schema.Type `json:"sdk.schema.extract.type" validate:"inclusion=avro" default:"avro"`
}
```
where `schema.Type` is an alias of [schema.Type in conduit-commons](https://github.com/ConduitIO/conduit-commons/blob/main/schema/schema.go#L30).

When parsing `SourceWithSchemaExtraction` with `sdk.Util.ParseConfig`, we get an error saying that there's a type mismatch, because the default value is `"avro"` (a string), and the target value is an `int`. The reason is that in [`Config.Validate()`](https://github.com/ConduitIO/conduit-commons/blob/main/config/config.go#L69) we validate the types.

**Possible solution**:

1. Remove type validation in `Config.Validate()` altogether. In the Connector SDK, it's called only when parsing a config, and after that we decode the map into a struct, where the types are validated too.
2. Use one of `mapstructure`'s hooks ([link](https://github.com/mitchellh/mapstructure/pull/222)), so that developers can implement a `MarshalText`/`UnmarshalText` method and customize parsing.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing sdk.Util.ParseConfig through Config.Validate and the subsequent mapstructure decoding path. Compare removing validation with adding mapstructure hooks for MarshalText/UnmarshalText, using the custom schema.Type example as the acceptance case. Done means custom field unmarshalling handles string defaults without breaking existing config validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.