Creating large sample data (>250 patients) for generated Cohort Definition fails on Databricks
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 151
- Forks
- 183
- Avg merge
- 14m
- Merged PRs (30d)
- 2
Description
Expected behavior
Creating large sample data (>250 patients) for generated Cohort Definition on Databricks should work like other dialects does (MS SQL and Azure Synapse).
Actual behavior
example for smaller population sample which works:
No logged errors spotted on WebAPI and no failed jobs on Databricks side.
Steps to reproduce behavior
- WebAPI version 2.12.1
- Databricks Cluster version: 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12)
- Databricks JDBC spark drivers version 2.6.22
- Define Cohort Definition
- Generate Cohort on Databricks
- Create new sample with 500 patients on Databricks
- See popup failure
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or failing test is named. First reproduce the generated Cohort Definition sample flow on Databricks with 500 patients, then compare it with the working MS SQL and Azure Synapse paths and inspect the WebAPI and Databricks job output. Done means large sample creation succeeds on Databricks without a popup failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100