Support for gldv2 and inaturalist datasets
- Dominant language
- Python
- Stars
- 272
- Forks
- 42
- PR merge metrics
- No merged PRs in 30d
Description
I think it would be great to port these datasets from tff to fedjax.
I would be happy to make the effort and contribute to the library, but I need a bit of support from the fedjax team 🙂
By looking at the tff codebase ([gldv2](https://github.com/tensorflow/federated/blob/ea8e3ca2797f3d268f96f94e052cebff9be27705/tensorflow_federated/python/simulation/datasets/gldv2.py), [inaturalist](https://github.com/tensorflow/federated/blob/ea8e3ca2797f3d268f96f94e052cebff9be27705/tensorflow_federated/python/simulation/datasets/inaturalist.py)) it looks that [load_data_from_cache](https://github.com/tensorflow/federated/blob/ea8e3ca2797f3d268f96f94e052cebff9be27705/tensorflow_federated/python/simulation/datasets/vision_datasets_utils.py#L86-L109) function creates a tfrecords file for each client.
The only concrete classes that I see are `SQLiteFederatedData` and `InMemoryFederatedData`, but I don't think they are meant for this use case. What would be the best way to map the clients into a `FederatedDataset`?
We could replicate something like [FilePerUserClientData](https://github.com/tensorflow/federated/blob/ea8e3ca2797f3d268f96f94e052cebff9be27705/tensorflow_federated/python/simulation/datasets/file_per_user_client_data.py#L27).
Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.