[Python][JS] Examples of serializing arrow from python through ipywidgets to JS
- Dominant language
- TypeScript
- Stars
- 112
- Forks
- 23
- Avg merge
- 21h 18m
- Merged PRs (30d)
- 8
Description
### Describe the usage question you have. Please include as many useful details as possible.
I do not know how to use ipywidgets traitlets.Bytes or even base64 encoding to generate an arrow table that will reify in JS.
lets start with Base64
```python
import pyarrow as pa
import base64
df = pd.DataFrame({'a':[10,50], 'b': ['paddy', 'margaret']})
table = pa.Table.from_pandas(df)
batch = table.to_batches()[0]
pabuffer = batch.serialize()
pabuffer
b64_table = base64.b64encode(pabuffer.to_pybytes())
print(base_64_table)
```
given the following JS code, that string doesn't reify properly
```typescript
export function base64ToBytes(base64:string) {
const binString = atob(base64);
//@ts-ignore
return Uint8Array.from(binString, (m) => m.codePointAt(0));
}
const b64bytes = base64ToBytes(base64table)
const t2 = tableFromIPC(b64bytes);
```
I get an error of `TypeError: this.schema is undefined`
I can get an arrow table to reify when I do base64 in JS and use that string.
I have made a sample repo to play with, benchmark, and document dataframe serialization. https://github.com/paddymul/df_cereal
I'm eager to collaborate and help with documentation for using arrow-js. I think it solves a lot of serialization problems for performance and typing vs json. It's a little hard to approach as is, because the docs are written for someone who understands arrow, vs someone who knows less about arrow, but wants to use it for serialization.
### Component(s)
JavaScript, Python
Contributor guide
Research direction
Start with the Python serialization snippet and the TypeScript base64ToBytes/tableFromIPC path, then compare them with the working JavaScript base64 case. Done is a documented, reproducible Python-to-JS Arrow table example covering the ipywidgets or base64 path and explaining the expected result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, typescript
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100