dotnet / dotnet/machinelearning
Need advanced filtering for downsampling
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
- Version: ML.NET 1.2.1
To filter rows in a performance-friendly way and load the data into an IDataView, I attempted to use:
```
mlContext.Data.FilterRowsByColumn(...)
```
However, this method only supports the ability to filter on the values of a single column. There are cases where you need to do more advanced filtering scenarios - for example, filter based on multiple column values, nest queries, etc. It would be helpful in general to have the ability to provide a linq expression to support a variety of filtering scenarios.
Here's more information on my specific scenario - as I said, it would be helpful to have advanced filtering capabilities provided since this is a useful way to do down-sampling before training on the data.
-----------------------------------------------------------------------
**Scenario:**
My scenario uses a large hotel result dataset for ranking (1,000,000+ records).
Here is simple example of the data:
GroupId | HotelId | Srch_Result_Clicked | Srch_Result_Booked
----------|----------|----------------------|-------------------------|
1 | 12 | 0 | 0 |
1 | 24 | 1 | 0 |
1 | 45 | 1 | 1 |
1 | 55 | 0 | 0 |
Notice that in the above data, the GroupId corresponds to the query or search id. There are multiple hotel results then tied to the GroupId since these are the results corresponding to a given query. Each hotel result may have the following values:
* Srch_Result_Clicked == 1 if the user clicked the hotel search result
* Srch_Result_Booked == 1 if the user both clicked and booked the hotel search result
* Or, the above values are 0 if the user neither clicked nor booked the result
In this scenario, I needed to perform down-sampling so that I only trained on hotel search queries where that had been either clicked or booked. Here's an example of the type of query that I was trying to achieve:
```
//Get those group\query ids that have at least one hotel result that was either clicked or booked
var groupIds = hotelData.Where(h => h.Srch_Result_Clicked == 1 || h.Srch_Result_Booked == 1).Select(h => h.GroupId).Distinct();
//Down sample retrieve all hotel results for a group\query matching the above criteria
IDataView downSampleData = hotelData.Where(h => groupIds.Contains(h.GroupId));
//Train the model
var model = trainingPipeline.Fit(downSampleData);
```
Contributor guide
Assessment
This issue has not been assessed yet.