dotnet / dotnet/machinelearning

Q: Faster way to Filter DataView

Open
#6,164 6 comments 1 reaction 0 assignees View on GitHub
area-DataFrame enhancement
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

I filter data from a dataview to get all items within a specific time period.
It seems slow compared to filtering with LINQ from objects in memory. Is there a faster way to do it?

```c#
var boolFilter = df["timestamp"].ElementwiseGreaterThanOrEqual(unixStartTime);
var hourlydata = df.Filter(boolFilter);
var boolFilter2 = hourlydata["timestamp"].ElementwiseLessThan(unixEndTime);
hourlydata = hourlydata.Filter(boolFilter2);
```

In this example, I am creating predictions for a certain time period at a time.
In another example, I may need to filter by exact match. Normally, I might create a dictionary to help, but is there a way to support some type "indices" for DataViews?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.