Is the 10k files / folder limit a hard limit for a dataset repo?

### Feature request

Can the hard limit of 10k files per folder be extended with acceptable loss in performance?

### Motivation

I'm uploading a lance dataset to huggingface hub and have a folder inside lance internals (`data/*.lance/_transactions`) that has > 20k atomic transaction records and my commits are being rejected.

```
Bad request for commit endpoint:
Your push was rejected because it contains too many files per directory. Each directory in your git repo can only contain up to 10000 files. Offending reference: refs/heads/main Offending directories: /data/expression.lance/_transactions/
```

### Your contribution

Open to suggestions for how to make a PR