twosigma / twosigma/flint

[Python] dataframe summarize and summarizeCycles possible malfunction

Open
#13 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.2k
Forks
199
PR merge metrics
No merged PRs in 30d

Description

I have a dataset which time index is daily, and I want to sum all the values for the same day (same index). I can do this function using the summarize method, but it doesn't work specifying the key 'time' (my flint time index).

Using:
`flintdf.summarize(summarizers.sum('values'), key='time')`
or
`flintdf.summarizeCycles(summarizers.sum('values'), key='time')`

gives me this error:
> Py4JJavaError: An error occurred while calling o449.summarize.
: com.twosigma.flint.timeseries.row.DuplicateColumnsException: Found duplicate columns List(time) in schema...

The point is that I also tried using summarizeCycles without a key, but it sum the time too and gives me the total sum of absolutely everything:

`flintdf.summarizeCycles(summarizers.count())`

returns me something like this:

> |-------------time------------|count|
> |9223372036854775807| 3030|

I think this could be another possible malfunction, because there are many different timestamps in my dataset and as the documentation says,

> "Computes aggregate statistics of rows that share a timestamp." .

And the last thing I tried, was to use another date field, which permits me to use the summarize and summarizeCycles with date as key, but it looks like the summarize method deletes the timeindex, making all the values 0 and the resulting dataframe with the values unsorted.
Using summarizeCycles with key, returns the same dataframe, taking only the first element with a timestamp, the repeated index rows are deleted and as index, it uses the same for every value, that is the sum of all the times, in my case 9223372036854775807

Python version: 3.5

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported calls to flintdf.summarize and summarizeCycles with key='time' and without a key, using the duplicate-column error and the documentation statement about shared timestamps as references. Trace those Python entry points into the Spark implementation and verify that aggregation groups timestamps correctly, avoids duplicate time columns, preserves the time index, and keeps the result ordered.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scala, spark
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.