[Python] dataframe summarize and summarizeCycles possible malfunction
- Dominant language
- Scala
- Stars
- 1.2k
- Forks
- 199
- PR merge metrics
- No merged PRs in 30d
Description
I have a dataset which time index is daily, and I want to sum all the values for the same day (same index). I can do this function using the summarize method, but it doesn't work specifying the key 'time' (my flint time index).
Using:
`flintdf.summarize(summarizers.sum('values'), key='time')`
or
`flintdf.summarizeCycles(summarizers.sum('values'), key='time')`
gives me this error:
> Py4JJavaError: An error occurred while calling o449.summarize.
: com.twosigma.flint.timeseries.row.DuplicateColumnsException: Found duplicate columns List(time) in schema...
The point is that I also tried using summarizeCycles without a key, but it sum the time too and gives me the total sum of absolutely everything:
`flintdf.summarizeCycles(summarizers.count())`
returns me something like this:
> |-------------time------------|count|
> |9223372036854775807| 3030|
I think this could be another possible malfunction, because there are many different timestamps in my dataset and as the documentation says,
> "Computes aggregate statistics of rows that share a timestamp." .
And the last thing I tried, was to use another date field, which permits me to use the summarize and summarizeCycles with date as key, but it looks like the summarize method deletes the timeindex, making all the values 0 and the resulting dataframe with the values unsorted.
Using summarizeCycles with key, returns the same dataframe, taking only the first element with a timestamp, the repeated index rows are deleted and as index, it uses the same for every value, that is the sum of all the times, in my case 9223372036854775807
Python version: 3.5
Contributor guide
Research direction
Start by reproducing the reported calls to flintdf.summarize and summarizeCycles with key='time' and without a key, using the duplicate-column error and the documentation statement about shared timestamps as references. Trace those Python entry points into the Spark implementation and verify that aggregation groups timestamps correctly, avoids duplicate time columns, preserves the time index, and keeps the result ordered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scala, spark
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100