pathwaycom / pathwaycom/pathway
Add an option to disable or rename special fields in output connectors
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 62.3k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
Is your feature request related to a problem? Please describe.
Currently, output connectors always emit two additional metadata columns alongside user data:
time, which contains the number of computational minibatch in Pathway;diff, which contains either+1or-1, indicating whether the change corresponds to a row insertion or deletion.
In many use cases, this metadata is not required by the user and only adds noise to the output. At the moment, there is no way to disable these fields, even when they are irrelevant for downstream consumers.
Describe the solution you'd like
Introduce an additional configuration field for every output connector, and add an extra optional argument to each emitting method: special_fields_config.
This config should:
- Allow renaming each of the special metadata fields (
time,diff) to custom names; - Allow disabling the emission of each of these fields entirely.
This would give users fine-grained control over what exactly is included in the output.
Describe alternatives you've considered
Keeping the current behavior as-is, with special fields always included.
Additional context
Given the large number of output connectors in Pathway, this feature may be introduced gradually, prioritizing the most commonly used connectors first if necessary.
The change must be fully backward-compatible: the default value of the new entity should reproduce the current behavior exactly (i.e., both fields enabled with their existing names).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by inventorying the output connectors and their emitting methods, then define how special_fields_config is shared across them while preserving current defaults. Done means connectors can independently rename or disable time and diff, with existing behavior unchanged when the option is omitted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100