cdisc-org / cdisc-org/cdisc-rules-engine

Dask Metadata Bugs

Open
#1,265 1 comment 0 reactions 0 assignees View on GitHub
dataframe large datasets/performance
Dominant language
Python
Stars
113
Forks
43
Avg merge
14h 51m
Merged PRs (30d)
14

Description

Some rules are skipped ( CORE-000597) -- Dask is having an issue with metadata retrieval that is causing rules to skip.

CORE-000597 using dask: error in log: [ERROR 2025-07-24 08:00:10,165 - console_logger.py:50] - Error occurred during validation.
Error: Failed to generate metadata for Isin(frame=Index(frame=SetIndex(frame=df, _other='USUBJID', options={})), values=Index(frame=SetIndex(frame=ColumnsSetter(frame=ReadParquetFSSpec(e7d4f8b), columns=['STUDYID', 'RDOMAIN', 'USUBJID', 'SUBJID', 'IDVAR', 'IDVARVAL', 'QNAM', 'QLABEL', 'QVAL', 'QORIG', 'QEVAL']), _other='USUBJID', options={}))). This operation may not be supported by the current backend.
Error Type:
Error Message: Failed to generate metadata for Isin(frame=Index(frame=SetIndex(frame=df, _other='USUBJID', options={})), values=Index(frame=SetIndex(frame=ColumnsSetter(frame=ReadParquetFSSpec(e7d4f8b), columns=['STUDYID', 'RDOMAIN', 'USUBJID', 'SUBJID', 'IDVAR', 'IDVARVAL', 'QNAM', 'QLABEL', 'QVAL', 'QORIG', 'QEVAL']), _other='USUBJID', options={}))). This operation may not be supported by the current backend.
Full traceback:
Traceback (most recent call last):
File "dask_expr/_core.py", line 467, in __getattr__
File "functools.py", line 998, in __get__
File "dask_expr/_expr.py", line 1366, in _meta
AttributeError: 'Index' object has no attribute 'iloc'

Additionally, when using Dask, log warnings include: "You did not provide metadata, so Dask is running your function on a small dataset to guess output types. It is possible that Dask will guess incorrectly."

Contributor guide

Open the contributing guide

Research direction

Start with the CORE-000597 Dask validation failure and trace the metadata path through dask_expr/_core.py and dask_expr/_expr.py; console_logger.py:50 is where the error is logged. Reproduce the Isin/Index case described in the traceback, then verify that the rule no longer skips and the missing-metadata warning is addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.