cdisc-org / cdisc-org/cdisc-rules-engine
Dask Metadata Bugs
- Dominant language
- Python
- Stars
- 113
- Forks
- 43
- Avg merge
- 14h 51m
- Merged PRs (30d)
- 14
Description
Some rules are skipped ( CORE-000597) -- Dask is having an issue with metadata retrieval that is causing rules to skip.
CORE-000597 using dask: error in log: [ERROR 2025-07-24 08:00:10,165 - console_logger.py:50] - Error occurred during validation.
Error: Failed to generate metadata for Isin(frame=Index(frame=SetIndex(frame=df, _other='USUBJID', options={})), values=Index(frame=SetIndex(frame=ColumnsSetter(frame=ReadParquetFSSpec(e7d4f8b), columns=['STUDYID', 'RDOMAIN', 'USUBJID', 'SUBJID', 'IDVAR', 'IDVARVAL', 'QNAM', 'QLABEL', 'QVAL', 'QORIG', 'QEVAL']), _other='USUBJID', options={}))). This operation may not be supported by the current backend.
Error Type:
Error Message: Failed to generate metadata for Isin(frame=Index(frame=SetIndex(frame=df, _other='USUBJID', options={})), values=Index(frame=SetIndex(frame=ColumnsSetter(frame=ReadParquetFSSpec(e7d4f8b), columns=['STUDYID', 'RDOMAIN', 'USUBJID', 'SUBJID', 'IDVAR', 'IDVARVAL', 'QNAM', 'QLABEL', 'QVAL', 'QORIG', 'QEVAL']), _other='USUBJID', options={}))). This operation may not be supported by the current backend.
Full traceback:
Traceback (most recent call last):
File "dask_expr/_core.py", line 467, in __getattr__
File "functools.py", line 998, in __get__
File "dask_expr/_expr.py", line 1366, in _meta
AttributeError: 'Index' object has no attribute 'iloc'
Additionally, when using Dask, log warnings include: "You did not provide metadata, so Dask is running your function on a small dataset to guess output types. It is possible that Dask will guess incorrectly."
Contributor guide
Research direction
Start with the CORE-000597 Dask validation failure and trace the metadata path through dask_expr/_core.py and dask_expr/_expr.py; console_logger.py:50 is where the error is logged. Reproduce the Isin/Index case described in the traceback, then verify that the rule no longer skips and the missing-metadata warning is addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100