Expanding test coverage

Coverage dimensions:

High priority

• Backwards compatibility - reading and optimizing indices made with old versions
  • We have some basic old test data in https://github.com/lancedb/lance/tree/main/test_data
  • See also: https://github.com/lancedb/lance/issues/4416
• Test utilities to generate different Dataset layouts (index states, fragment sizes, fields in fragments)
  • See: https://github.com/lancedb/lance/issues/4417
• Test Memory - number of allocations, total bytes allocated, no leaked memory, peak memory used
  • We might be able to do this by using some combination of allocator and tracing support.
  • https://github.com/lancedb/lance/issues/4761
  • See also: https://github.com/lancedb/lance/issues/4477

Low priority

• Add line-based code-coverage
• API backwards compat - can we test this?
• Test IO - number of IO requests, bytes scanned, etc.
  • We currently do this with IOTrackingStore: https://github.com/lancedb/lance/blob/662d2c5ec562f61d44f6c3462048de86dcec0d2e/rust/lance/src/utils/test.rs#L306-L310
  • We might also explore a tracing-based solution
• Test concurrency:
  • concurrent reads don't lock each other out
  • concurrent writes don't yield errors or cause data loss
• Disk space usage
  • Test write amplification caused by write commands
  • Test cleanup