Point72 / Point72/csp

Remove Numpy1DArray and NumpyNDArray type distinction

Open
#42 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type: enhancement
Dominant language
Python
Stars
441
Forks
90
Avg merge
1d 12m
Merged PRs (30d)
5

Description

Is your feature request related to a problem? Please describe.
The csp.typing Numpy1DArray and NumpyNDArray types are confusing for users and lead to a lot of unexpected typing issues. We should be consistent with Numpy and use the base np.ndarray type for all Numpy time-series objects. With the distinction, you get strange bugs like this:

        inp_arr = np.zeros(shape=(4,))
        @csp.graph
        def arr_1d(x: csp.ts[csp.typing.Numpy1DArray[float]]) -> csp.ts[np.ndarray]:
            return x
        res = csp.run(arr_1d(arr_1d(csp.const(inp_arr))), starttime=st, endtime=datetime(2020, 2, 7, 9, 1)) # should not raise

This raises an error:
csp.impl.types.instantiation_type_resolver.TSArgTypeMismatchError: In function accept_arr_1d: Expected ts[csp.typing.Numpy1DArray[float]] for argument 'x', got ts[csp.typing.NumpyNDArray[float]]

Describe the solution you'd like
For now, set the Numpy1DArray type to be equal to the NDArray type and make associated changes to the parquet reader/writer such that users need to opt-in to using shape information. By default we will treat the Numpy array as 1-dimensional.

Describe alternatives you've considered
Keeping the two types as separate will continue to lead to weird bugs as I showed above.

Additional context
This change has come up as a discussion topic before as the type distinction led to bugs in the past.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the csp.typing Numpy1DArray and NumpyNDArray definitions, then trace csp.impl.types.instantiation_type_resolver with the reproduction in the issue. Inspect the parquet reader and writer paths for their shape handling. Done means the type distinction no longer causes the shown mismatch and shape information is opt-in as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.