stdlib-js / stdlib-js/stdlib

[RFC]: finish the dimensional ndarray constructors (tensor3d typed packages, tensor4d, tensor5d)

Open
#14,961 1 comment 0 reactions 0 assignees View on GitHub
Feature JavaScript RFC
Dominant language
JavaScript
Stars
6k
Forks
1.3k
Avg merge
1d 3h
Merged PRs (30d)
611

Description

### Description

This RFC proposes the remaining work to provide dedicated dimensional ndarray constructors up to five dimensions. `@stdlib/ndarray/vector/*` and `@stdlib/ndarray/matrix/*` cover one and two dimensions. `@stdlib/ndarray/tensor3d/ctor` covers the three-dimensional case and is in review (see Related Issues). What is left is the typed three-dimensional constructors, the `tensor3d` namespace package, and the same three groups again for four and five dimensions.

Dimensional constructors through 5D cover the large majority of end-user ndarray dimensionality. Arrays of six or more dimensions are uncommon and the general-purpose `@stdlib/ndarray/ctor` and `@stdlib/ndarray/array` APIs already handle them.

### What is already done

`@stdlib/ndarray/tensor3d/ctor` is written and under review.

#### What remains

Forty-one packages, grouped by dimensionality.

| Group | Packages | Count | Approximate size |
| ----- | -------- | ----- | ---------------- |
| `tensor3d` typed constructors | `bool`, `complex64`, `complex128`, `float32`, `float64`, `int8`, `int16`, `int32`, `uint8`, `uint8c`, `uint16`, `uint32` | 12 | 38,600 lines |
| `ndarray/tensor3d` namespace package | re-exports `tensor3d` and the twelve typed constructors | 1 | 4,500 lines |
| `tensor4d` | `ctor` plus the same twelve typed constructors plus the namespace package | 14 | 54,000 lines |
| `tensor5d` | `ctor` plus the same twelve typed constructors plus the namespace package | 14 | 54,000 lines |

The sizes come from measuring the equivalent `vector` and `matrix` packages and the finished `tensor3d/ctor`, which is 10,928 lines. Call it 150,000 lines in total. That number matters, and I return to it below.

#### Data types and constructor names

The typed set matches `vector`, which ships twelve. `matrix` currently ships five, and separate work is under way to bring it up to the full set.

| Data type | Constructor | Description used in `package.json` |
| --------- | ----------- | ---------------------------------- |
| `float64` | `Float64Tensor3D` | double-precision floating-point |
| `float32` | `Float32Tensor3D` | single-precision floating-point |
| `complex128` | `Complex128Tensor3D` | double-precision complex floating-point |
| `complex64` | `Complex64Tensor3D` | single-precision complex floating-point |
| `int32` | `Int32Tensor3D` | signed 32-bit integer |
| `int16` | `Int16Tensor3D` | signed 16-bit integer |
| `int8` | `Int8Tensor3D` | signed 8-bit integer |
| `uint32` | `Uint32Tensor3D` | unsigned 32-bit integer |
| `uint16` | `Uint16Tensor3D` | unsigned 16-bit integer |
| `uint8` | `Uint8Tensor3D` | unsigned 8-bit integer |
| `uint8c` | `Uint8ClampedTensor3D` | clamped unsigned 8-bit integer |
| `bool` | `BooleanTensor3D` | boolean |

Two names do not follow a literal transliteration of the data type, and both follow `vector`: `bool` exports `BooleanTensor3D`, and `uint8c` exports `Uint8ClampedTensor3D`. The 4D and 5D packages should use the same names with the dimensionality changed.

One wording note. Integer descriptions in this repository come in two forms, and the counts are lopsided: 39 packages write "signed 32-bit integer" and 3 write "32-bit signed integer". `matrix/int32` is one of the three. New packages should use the majority form.

#### Prerequisites, all already present

Nothing needs to be built first.

- `@stdlib/array/base/flatten2d` through `flatten5d` all exist. There is no `flatten6d`, so the dependency chain stops exactly where this plan stops.
- `@stdlib/types/ndarray` declares `Shape0D` through `Shape10D`, so `Shape4D` and `Shape5D` are ready for the TypeScript declarations.
- Each typed package is a thin wrapper over `require( '@stdlib/ndarray//ctor' ).factory( dtype )`. The work in a typed package is documentation, TypeScript declarations, tests, and benchmarks, not logic.

#### Porting notes for whoever writes tensor4d and tensor5d

Adding a dimension shifts every `ArrayBuffer` signature form up one position, so the argument-count dispatch is not a find-and-replace. Going from 2D to 3D, seven places needed real thought. The same seven will need it again at 4D and 5D.

1. Maximum arity grows by one. `lib/main.js` gains a branch and the terminal fall-through handles one more argument.
2. A form disappears. `matrix( M, N )` has no 3D counterpart, so two nonnegative integers stopped being a valid shape. At 4D, three integers stop being valid. This branch gets deleted rather than shifted, and it is the easiest change to miss.
3. The lowest shape-specifying arity gains the new full form and loses the `( dims..., dtype )` and `( dims..., options )` forms, which move up one.
4. The default `ArrayBuffer` shape gains a leading `1`. At 3D it is `[ 1, 1, N ]`. At 4D it is `[ 1, 1, 1, N ]`.
5. The collection and iterable paths need `sh.length < N` and `slice( sh, 0, N )` updated, and the `flattenNd` import changed.
6. Shape length assertions move from `x.length === 2` to the new dimensionality at four call sites.
7. Complex data type handling needs no change. The interleaved real and imaginary sniff adjusts the trailing dimension with `sh[ sh.length-1 ] /= 2`, which does not depend on dimensionality. Leave it alone.

Two defects got through the 3D port and were only caught later. Both are worth watching for.

**Two TypeScript overloads kept the old dimension count.** The signatures reading `( buffer, byteOffset, M, N, options )` needed a `K` inserted, and did not get one, while the JSDoc above them already listed K, M, N. Runtime tests cannot catch this because `.d.ts` files never execute. Only the repository's TypeScript lint found it. The same two overloads exist in every typed package, so the mistake multiplied to 26 signatures before anyone noticed.

**Shape literals in final argument position lost the space before the closing paren.** `tensor3d( [ 1, 2, 5 ])` instead of `] )`. The test file ESLint configuration does not enforce `space-in-parens`, so the pre-commit hook passed it. 249 occurrences.

Both argue for the same thing: run the full lint set, including the TypeScript declarations lint, before assuming a mechanical port is finished.

#### Namespace registration

- Each group needs an `ndarray/` namespace package re-exporting the constructor and the typed constructors, matching `ndarray/vector` and `ndarray/matrix`.
- Each group needs an alias in `@stdlib/namespace`, added to `lib/namespace/.js`. The `pkg2alias`, `alias2pkg`, `pkg2related`, `pkg2standalone`, and `standalone2pkg` data files are generated from that source by each subpackage's `scripts/build.js`, so the entry and the regenerated data need to land together.
- There is an existing inconsistency to settle. `ndarray/lib/index.js` registers `vector` in the top-level `@stdlib/ndarray` namespace but does not register `matrix`. Whichever way that gets resolved, `tensor3d`, `tensor4d`, and `tensor5d` should follow the same rule.

#### On writing the dispatch four times

I recommended deferring this question when `tensor3d/ctor` started, on the grounds that the argument-count dispatch resists clean parameterization and that departing from the `vector` and `matrix` precedent would cost reviewers more than it saved. Having now finished the 3D port, I want to revise that.

The case for extracting a shared generator into `@stdlib/ndarray/base` is stronger than I expected:

- The remaining work is roughly 150,000 lines, most of it near-identical. Two typed packages differ by about 3.6% of their lines once you normalize the data type name, the constructor name, and the description.
- Mechanical copying propagates defects. The `matrix/ctor` factory bug described below survived into a released package precisely because nobody re-derived that branch. The two defects listed above came from the same kind of copying.
- The seven adaptation points are the same seven at every dimensionality. That is a pattern, not a series of one-off judgments.

The case against has not changed: error messages are positional ("Third argument must be a nonnegative integer"), so a parameterized version has to generate them; and the JSDoc annotations and TypeScript overloads still need writing by hand per package regardless, which is where most of the line count actually sits.

I lean toward extracting the generator before `tensor4d` begins, and I would rather settle it now than after another 54,000 lines exist. But the line count sits mostly in documentation and tests, which a generator does not remove, so this is a judgment call for maintainers rather than something the evidence decides outright.

### Related Issues

Related issues #14959

### Questions

1. Should the shared dispatch generator be extracted before `tensor4d` starts, or should 4D and 5D follow the same copy-and-adapt path as 3D?
2. Should `matrix` be registered in the top-level `@stdlib/ndarray` namespace alongside `vector`, so the three tensor groups have a consistent rule to follow?
3. Should the twelve typed `tensor3d` constructors land as one pull request or as several smaller ones? They are independent of each other and each is about 3,200 lines.

### Other

#### Duplication of `from_arraybuffer.js`

`lib/from_arraybuffer.js` is dimension-agnostic and gets copied verbatim into each `ctor` package. It carries a `TODO` noting it should move to `@stdlib/ndarray/from-arraybuffer`. There are three copies now. Following this plan without acting on the `TODO` would make five. Separate work on `@stdlib/ndarray/from-arraybuffer` is already under way, which would reclaim all of them.

#### A defect in `matrix/ctor`, found and fixed while porting

The `( M, N, options )` branch of `matrix/ctor`'s `factory` method passed the raw user options object to the main entry point instead of merging it into the validated defaults. Every other branch merged correctly, and so did `vector/ctor`, which made `matrix` the only package with the problem:

```javascript
var f = factory( 'float64', { 'readonly': true } );

isReadOnly( f( 2, 2 ) ); // true
isReadOnly( f( 2, 2, {} ) ); // false, the default was discarded
```

Passing any options to that one signature silently threw away every default set when the factory was created, covering `readonly`, `order`, `mode`, and `submode`. The fix and a regression test are ready. The three-dimensional implementation follows the corrected behavior.

Worth checking the same class of problem elsewhere. Only three packages use this defaults-merging factory pattern: `vector/ctor`, `matrix/ctor`, and `tensor3d/ctor`. Exercising all 43 signature forms across the three with factory-level defaults set confirms nothing else discards them. The other factories that validate options (`array/to-fancy`, `ndarray/to-fancy`, `random/base/beta`, `random/base/gamma`, `random/base/negative-binomial`) return functions that take no options at all, so they cannot hit this.

### Checklist

- [x] I have read and understood the [Code of Conduct](https://github.com/stdlib-js/stdlib/blob/develop/CODE_OF_CONDUCT.md).
- [x] Searched for existing issues and pull requests.
- [x] The issue name begins with `RFC:`.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the existing @stdlib/ndarray/vector and @stdlib/ndarray/matrix packages with ndarray/tensor3d/ctor, especially lib/main.js, typed package declarations, tests, and namespace registration. Before implementation, maintainers need to resolve the generator, namespace, and pull-request-splitting questions. Done means the agreed tensor3d, tensor4d, and tensor5d packages are registered, tested, typed, benchmarked, and pass the full lint set.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, typescript
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.