jbenet / jbenet/transformer

Streams

Open
#3 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
132
Forks
7
PR merge metrics
No merged PRs in 30d

Description

@maxogden

Correctly handling streams is hard, because it's not clear when a codec should be done reading from a stream (i.e. how much xml to parse before the first "object" is found?). Like, if I'm parsing a type with schema:

```
{
"schema": [ {
"dob": "iso-date",
"name": "person-name",
} ]
}
```

It may be able to discern that an array of objects should be parsable object by object.

However, even if that's done correctly, it's not clear that the conversion won't be sync. I may be converting this list of `{name/dob}` objects to a type like:

```
{
"schema": {
"oldest-person": {
"dob": "iso-date",
"name": "person-name",
},
"youngest-person": {
"dob": "iso-date",
"name": "person-name",
}
}
}
```

Which would need to read the whole thing anyway to compute. Hmmmm. this could be a per codec thing-- implement `encodeStream` and `encodeSync` or something.

Also, worth noting most tiny codecs will just be sync -- things like parsing Dates.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the codec and stream-handling design described in the issue, including the proposed encodeStream and encodeSync entry points. Compare the array-of-objects and oldest/youngest-person examples to determine which conversions can be incremental and which require the full input. Done means the stream behavior and codec responsibilities are clearly defined and implemented consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.