improved packaging

Hi,
As part of our package to easily evaluate clip models, https://github.com/LAION-AI/CLIP_benchmark/issues/1
and my inference lib https://github.com/rom1504/clip-retrieval

I'm interested to have a package like this

however here is what's missing here:
1. pypi packaging
2. much clearer README, the usage of this should be 5 lines of python that can be copy pasted
3. performance evaluation
4. possibly optional dependencies to avoid the dependency list to become too large

I may be interested to contribute all that, but first I'd like to check with you if you're ok with that kind of changes

thanks