dotnet / dotnet/machinelearning-samples

Change restaurant dataset in Restaurant Violation Inspections sample

Open
#809 0 comments 0 reactions 0 assignees View on GitHub
bug ModelBuilder
Dominant language
PowerShell
Stars
4.7k
Forks
2.7k
Avg merge
2d 22h
Merged PRs (30d)
1

Description

@luisquintanilla : We should replace the dataset used in [Tutorial: Classify the severity of restaurant health violations with Model Builder](https://github.com/dotnet/docs/blob/3803b263eae3f7575290a1da1142c34d1d4b5b7e/docs/machine-learning/tutorials/health-violation-classification-model-builder.md) in the docs repo, and the [Restaurant Violation Inspections](https://github.com/dotnet/machinelearning-samples/tree/9dc39d7f40c655029dae498e1d0de0406512f66d/samples/modelbuilder/MulticlassClassification_RestaurantViolations) sample in this samples repo.

See discussion: https://github.com/dotnet/docs/issues/17721

> Generally if you're getting 100% accuracy, you're either leaking, have too small of dataset to usefully measure the metrics, or the task is trivial.
>
> In this case the [dataset](https://github.com/luisquintanilla/machinelearning-samples/raw/AB1608219/samples/modelbuilder/MulticlassClassification_RestaurantViolations/RestaurantScores.zip) itself is leaking information or is trivial, depending on how you want to look at it. It has duplicates rows. The `violation_description` column perfectly predicts the label, `risk_category`, as it's the risk category of the given violation.
>
> There are 53,974 rows but only 363 are unique:
>
> ```shell
> $ wc -l RestaurantScores.tsv
> 53974
> $ sort RestaurantScores.tsv | uniq | wc -l
> 363
> ```
>
> See more information about leakage:
> https://en.wikipedia.org/wiki/Leakage_(machine_learning)
>
> # Next steps
> I would recommend replacing the dataset. Perhaps with the original dataset (with all columns).
>
> The dataset seems be [SF Restaurant Scores](https://data.sfgov.org/Health-and-Social-Services/Restaurant-Scores-LIVES-Standard/pyih-qa8i) ([download](Restaurant_Scores_-_LIVES_Standard.csv)). In that case, I would pose the problem either as regression or multi-class classification.
>
> Problem styles:
>
> * Regression -- predict `inspection_score` (**recommended**)
> * Multiclass classification -- predict `violation_description` while ignoring `risk_category` (or the other way around)
>
> For all tasks, I would split the dataset in to train/validate/test based on date (`inspection_date` column). With the oldest data in the train split, newer in validate, and most recent in test.
>
> Why split on time?
> This is to avoid leaking data between the dataset splits as this dataset is time-dependent, as the newer information better predicts the other newer rows than does the older rows. For example the `inspection_score` of a restaurant in June is likely closer to the July score than the January score.
>
> _... (preview of dataset [followed](https://github.com/dotnet/docs/issues/17721))..._

Issue also filed in docs repo to track that change -- https://github.com/dotnet/docs/issues/17962

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.