[Work Around Request] Datasets upload only diffs

Hello Team and Fellow Kagglers,

I maintain an active dataset of image metadata for the [Mars 2020 mission images.](https://www.kaggle.com/sakethramanujam/mars2020imagecatalogue)
At this point there is an update at least once a day or once in two days and the dataset is only going to grow larger.

My issue now is that I have a cron job that handles my update process. I see that the dataset update process uploads all files of the dataset again kind of a replace all situation.

I come here to ask if there is a way to upload only diffs of the dataset i.e., the new items on the dataset and not multiple files.
If not, can we make this a feature request, and I can may be contribute ?

thanks in advance!
Saketh.

The issue does not name files, tests, or a specific command implementation. Start by tracing the CLI's dataset upload flow and determining how it currently decides which files to upload. Done means an update can identify and upload only new or changed dataset items, with tests covering that behavior.

Written by the indexing model from the issue text.