Reduce latency sensitivity to small files

Currently, the number of IOPS used to scan files is O(N * M) where N is the number of fragments and M is the number of columns. (This assumes each fragment has one data file, for simplicity.)

We can see this below, in the time to scan 10 columns from 128 rows in a dataset from S3. At 128 fragments, there are is just one row per data file.

For files smaller than the block size, we can reduce this to exactly N IOPS by:

• Storing the data file size in bytes in the manifest, to avoid the initial HEAD request
• Read the whole data file at once, if the file is smaller than the block size.

For files larger than this, we can still make this more O(N) by: