Performance regression with v1

Heya. I noticed a significant (~2x) perf regression after upgrading to v1 recently. I also tested against 1.1.0 and didn't see a difference. The regression happens when doing filesystem reads (though it may happen in other situations -- I'm only sharing what my own benchmarks do).

## Repro

1. `git clone https://github.com/zkat/cacache-rs --branch async-std-regression`
2. `cd cacache-rs`
3. `cargo bench`
4. `git checkout head~1`
5. `cargo bench`

## Details

You can see the benchmarks themselves over at https://github.com/zkat/cacache-rs/blob/async-std-regression/benches/benchmarks.rs#L44-L74