apache / apache/arrow-js

[Python][JS] Examples of serializing arrow from python through ipywidgets to JS

Open
#62 4 comments 0 reactions 0 assignees View on GitHub
Type: usage
Dominant language
TypeScript
Stars
112
Forks
23
Avg merge
21h 18m
Merged PRs (30d)
8

Description

### Describe the usage question you have. Please include as many useful details as possible.

I do not know how to use ipywidgets traitlets.Bytes or even base64 encoding to generate an arrow table that will reify in JS.

lets start with Base64

```python
import pyarrow as pa
import base64
df = pd.DataFrame({'a':[10,50], 'b': ['paddy', 'margaret']})

table = pa.Table.from_pandas(df)
batch = table.to_batches()[0]
pabuffer = batch.serialize()
pabuffer

b64_table = base64.b64encode(pabuffer.to_pybytes())
print(base_64_table)
```

given the following JS code, that string doesn't reify properly
```typescript
export function base64ToBytes(base64:string) {
const binString = atob(base64);
//@ts-ignore
return Uint8Array.from(binString, (m) => m.codePointAt(0));
}

const b64bytes = base64ToBytes(base64table)
const t2 = tableFromIPC(b64bytes);
```
I get an error of `TypeError: this.schema is undefined`

I can get an arrow table to reify when I do base64 in JS and use that string.

I have made a sample repo to play with, benchmark, and document dataframe serialization. https://github.com/paddymul/df_cereal

I'm eager to collaborate and help with documentation for using arrow-js. I think it solves a lot of serialization problems for performance and typing vs json. It's a little hard to approach as is, because the docs are written for someone who understands arrow, vs someone who knows less about arrow, but wants to use it for serialization.

### Component(s)

JavaScript, Python

Contributor guide

Open the contributing guide

Research direction

Start with the Python serialization snippet and the TypeScript base64ToBytes/tableFromIPC path, then compare them with the working JavaScript base64 case. Done is a documented, reproducible Python-to-JS Arrow table example covering the ipywidgets or base64 path and explaining the expected result.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, typescript
Domain
documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.