Project-MONAI / Project-MONAI/MONAI

Datasets that cache in some form should be able to handle nested Compose

Open
#7,130 0 comments 2 reactions 1 assignee View on GitHub

@atbenmurray is already working on this.

Since Oct 13, 2023.

Design discussions enhancement
Dominant language
Python
Stars
8.7k
Forks
1.6k
Avg merge
5d 1h
Merged PRs (30d)
20

Description

Is your feature request related to a problem? Please describe.
The issue is as described in the Discussion #6172. Datasets that perform caching in some form can examine a top level Compose object but, if presented with nested Compose objects, treats those as being both randomizable and indivisible.

Describe the solution you'd like
Given the following pipeline

Compose[Compose[A, B], Compose[C, D]]

where A, B and C are deterministic, and D is random
Dataset caching mechanisms should cache the output of C and execute D for each call of the pipeline

This is identical behaviour to the pipeline:

Compose[A, B, C, D]

In order for this to be possible, we either have to:
. flatten the heirarchy of all vanilla Compose objects (although this will cause issues)
. execute datasets on a flattened heirarchy of Compose objects (although this will cause issues)
. make the Caching Dataset iteration behaviour robust to nested Compose objects (my preferred option)

There will also have to be sensible behaviour for OneOf, RandomOrder and SomeOf as these are always effectively random.

Proposed solution

Determining whether composes are acting in a random way
Composes that contain only transforms that we know are not random can be treated as being not random themselves. Subclasses of compose that are always random (Oneof, SomeOf, RandomOrder) should always be treated as random.
We can add is_random() to the base random transform class with a default implementation that always returns True and Compose / subclasses of Compose can override this to perform the necessary logic.

Caching part of a nested compose
I suggest that we implement a compose iterator that is able to step over the transforms in a compose instance and can step into nested compose objects. This iterator can return the index of the first randomizable transform regardless of where it is in the set of nested compose objects, and that index can be subsequently used to fast forward to a given location in the nested pipeline.

This issue is fixed in PR #7140

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.