apache / apache/arrow-js

[JS][Packaging] npm package is bloated with seemingly unnecessary code/deps

Open
#52 2 comments 1 reaction 0 assignees View on GitHub
Type: enhancement
Dominant language
TypeScript
Stars
112
Forks
23
Avg merge
21h 18m
Merged PRs (30d)
8

Description

### Describe the enhancement requested

We're considering using Apache Arrow for a project, but it seems like the dependency is very large. For example, the `@apache-arrow/es5-cjs` package is 2.82MB _compressed_. It seems that the project was already split between target ES version and module loader for size efficiency, but without actually addressing the size problem.

From looking at the folder, I see the following components that could potentially be removed at a glance:

* All `.ts` files, as these are only needed in development when using TypeScript. (this includes definition files and the entire `src` folder)
* All `.dom` and `.map` files since these are only needed in the browser and not in Node.
* The `bin/arrow2csv.js` file as it seems to only be useful for testing.
* All `@swc` dependencies, as these are development dependencies.
* All `@types` dependencies, as those should be installed separately only when using TypeScript.
* The `command-line-args` and `command-line-usage` dependencies as they seem like development dependencies.
* The `tslib` dependency, which again is related to TypeScript and cannot be used by Node.
* The `json-bignum` dependency, as it doesn't seem to make sense if this project is actually an alternative to JSON to begin with.
* For the `flatbuffers` dependency, I'm not entirely sure, but it seems to support generating code from a schema, which may include all or some of the code needed to no longer need `flatbuffers`, although for this one, additional investigation would be needed.
* I'm sure a lot of other files are used to do things like managing schemas and could potentially be removed as well from the production package.

Deleting all the above (except `flatbuffers` as it's unclear if it's needed) just to see the difference results in the uncompressed package size going from 6.4MB to 1.3MB, and it's probably incomplete and could be reduced much further.

I don't know much about the project internals, so my above assumptions might be wrong, but I'd definitely like to open a discussion about the size of this package and how to make it smaller as it seems the current approach to split packages is not quite achieving its goal and it might make more sense to split the project to something closer to a monorepo.

For example, something like this:

```
@apache-arrow/cli
@apache-arrow/types or @types/apache-arrow
@apache-arrow/node (doesn't need to be split by module types as only the entry point needs to change)
@apache-arrow/dom-es2015 (or es5, etc)
etc
```

Or even more granular (since for example we only need the ability to encode and not decode):

```
@apache-arrow/encoder
@apache-arrow/decoder
@apache-arrow/stream-writer
@apache-arrow/stream-reader
etc
```

Since the project already relies on Lerna, it shouldn't be too much work to change what gets released as what package I think, but I'd like to hear thoughts from maintainers before opening a PR with such a large restructuring of the project.

### Component(s)

JavaScript, Packaging

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.