Btree index build fail with "No partition metadata files found in index directory"

Copy Discord issue reported:

Ran

updated_dataset = lr.create_scalar_index(
            dataset=dataset_uri,
            column=column,
            index_type=index_type,
            replace=True,
            num_workers=effective_workers,
            ray_remote_args=ray_remote_args,
    )

18:06:11 | INFO     | Ray initialized: {'node:__internal_head__': 1.0, 'memory': 697043922944.0, 'GPU': 4.0, 'accelerator_type:GB200': 1.0, 'CPU': 140.0, 'object_store_memory': 200000000000.0, 'node:xxx': 1.0}
18:06:11 | INFO     | Total fragments: 5000
18:06:11 | INFO     | Ray workers: 16
18:06:11 | INFO     | Creating BTREE index on 'video_uuid' using 16 Ray workers...

18:10:11 | ERROR    | Operation failed: Encountered internal error. Please file a bug report at https://github.com/lance-format/lance/issues. No partition metadata files found in index directory: debug/lance_db/yyy/xxx/_indices/8e4fea4f-92b3-4cc4-affc-911313f95a22 (page_files: 0, lookup_files: 0), /runner/_work/lance/lance/rust/lance-index/src/scalar/btree.rs:1494:23

Some conversations:

I can see that Ray created 16 actors and running but it just fails after some time with no concrete logs/info on what went wrong.
This is a remote Lance table in GCS. I am able to query data from the table (to validate access/operations) at path gs://{bucket_name}/debug/lance_db/yyy/xxx

Start with the create_scalar_index call and the BTREE failure at rust/lance-index/src/scalar/btree.rs:1494. Reproduce the build against the reported remote GCS table with Ray workers, then trace why the index directory contains no partition metadata, page, or lookup files. Done means the failure cause is identified and the BTREE build either completes or reports actionable diagnostics.