dotnet / dotnet/machinelearning

Need advanced filtering for downsampling

Open
#3,991 0 comments 0 reactions 0 assignees View on GitHub
area-Core enhancement Priority:2
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

- Version: ML.NET 1.2.1

To filter rows in a performance-friendly way and load the data into an IDataView, I attempted to use:
```
mlContext.Data.FilterRowsByColumn(...)
```
However, this method only supports the ability to filter on the values of a single column. There are cases where you need to do more advanced filtering scenarios - for example, filter based on multiple column values, nest queries, etc. It would be helpful in general to have the ability to provide a linq expression to support a variety of filtering scenarios.

Here's more information on my specific scenario - as I said, it would be helpful to have advanced filtering capabilities provided since this is a useful way to do down-sampling before training on the data.

-----------------------------------------------------------------------
**Scenario:**

My scenario uses a large hotel result dataset for ranking (1,000,000+ records).
Here is simple example of the data:

GroupId | HotelId | Srch_Result_Clicked | Srch_Result_Booked
----------|----------|----------------------|-------------------------|
1 | 12 | 0 | 0 |
1 | 24 | 1 | 0 |
1 | 45 | 1 | 1 |
1 | 55 | 0 | 0 |

Notice that in the above data, the GroupId corresponds to the query or search id. There are multiple hotel results then tied to the GroupId since these are the results corresponding to a given query. Each hotel result may have the following values:
* Srch_Result_Clicked == 1 if the user clicked the hotel search result
* Srch_Result_Booked == 1 if the user both clicked and booked the hotel search result
* Or, the above values are 0 if the user neither clicked nor booked the result

In this scenario, I needed to perform down-sampling so that I only trained on hotel search queries where that had been either clicked or booked. Here's an example of the type of query that I was trying to achieve:

```
//Get those group\query ids that have at least one hotel result that was either clicked or booked
var groupIds = hotelData.Where(h => h.Srch_Result_Clicked == 1 || h.Srch_Result_Booked == 1).Select(h => h.GroupId).Distinct();

//Down sample retrieve all hotel results for a group\query matching the above criteria
IDataView downSampleData = hotelData.Where(h => groupIds.Contains(h.GroupId));

//Train the model
var model = trainingPipeline.Fit(downSampleData);
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.