google / google/fedjax

Support for gldv2 and inaturalist datasets

Open
#241 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
272
Forks
42
PR merge metrics
No merged PRs in 30d

Description

I think it would be great to port these datasets from tff to fedjax.
I would be happy to make the effort and contribute to the library, but I need a bit of support from the fedjax team 🙂

By looking at the tff codebase ([gldv2](https://github.com/tensorflow/federated/blob/ea8e3ca2797f3d268f96f94e052cebff9be27705/tensorflow_federated/python/simulation/datasets/gldv2.py), [inaturalist](https://github.com/tensorflow/federated/blob/ea8e3ca2797f3d268f96f94e052cebff9be27705/tensorflow_federated/python/simulation/datasets/inaturalist.py)) it looks that [load_data_from_cache](https://github.com/tensorflow/federated/blob/ea8e3ca2797f3d268f96f94e052cebff9be27705/tensorflow_federated/python/simulation/datasets/vision_datasets_utils.py#L86-L109) function creates a tfrecords file for each client.

The only concrete classes that I see are `SQLiteFederatedData` and `InMemoryFederatedData`, but I don't think they are meant for this use case. What would be the best way to map the clients into a `FederatedDataset`?
We could replicate something like [FilePerUserClientData](https://github.com/tensorflow/federated/blob/ea8e3ca2797f3d268f96f94e052cebff9be27705/tensorflow_federated/python/simulation/datasets/file_per_user_client_data.py#L27).

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.