Performance: ProblemGradeReport reads persisted grades per-learner (no bulk prefetch), plus four resource issues in instructor_task grade reporting

Summary

The asynchronous problem grade report (calculate_problem_grade_report →
ProblemGradeReport)
reads persisted grades one learner at a time, unlike
CourseGradeReport,
which bulk-prefetches the same data for each batch. On a 5,154-learner course this
report takes long enough that pod/worker lifecycle became the binding constraint for us
rather than anything about the report itself.

This is the same defect class as #38911 (which proposes bringing CourseGradeReport's
bulk prefetch to the synchronous CCX report), but in a different code path — the async
instructor task. Filing separately since the fix sites don't overlap; happy to fold them
together if maintainers prefer.

Alongside the N+1 there are four smaller resource issues in the same file that seem worth
recording together, since they all bear on the same report's cost profile.

Measurements

From a production Open edX deployment (Redwood-era, instructor_task unmodified from
upstream — both files referenced here are byte-identical to master at 387ee5d93c134cacff0a7626e28583d701b3ecec).

Successful run, calculate_problem_grade_report, 802 learners over 115 scorable blocks:

Phase (from edx.celery.task logs)	Elapsed
InMemoryReportMixin - 1: Starting grade report	—
InMemoryReportMixin - 2: Compiling grades	0.06 s
InMemoryReportMixin - 3: Uploading grades	660.96 s
InMemoryReportMixin - 4: Completed grades	0.17 s

Total 661.2 s ≈ 0.82 s/learner, essentially all of it in the per-learner compile loop.
Two smaller runs corroborate the rate: 43 learners / 18.4 s, 40 learners / 24.5 s.

Container memory across 30 days of these workers peaked at 2.09 GB
(container_memory_working_set_bytes) against a 4 GiB limit, with zero OOM restarts —
so for our course sizes the wall-clock, not memory, is what hurts. Larger installations
may hit the memory items below first.

Finding 1 — ProblemGradeReport does no bulk prefetch (primary)

CourseGradeReport._rows_for_users
opens with a
_CourseGradeBulkContext,
which calls prefetch_course_and_subsection_grades, bulk_cache_cohorts,
BulkRoleCache.prefetch and BulkCourseTags.prefetch for the whole 100-learner batch.

ProblemGradeReport._rows_for_users
has no equivalent — it goes straight into CourseGradeFactory().iter(...). With nothing
prefetched, PersistentSubsectionGrade.bulk_read_grades misses the per-course
RequestCache and falls through to a per-learner query, and the report then walks
course_grade.problem_scores for every scorable block, so this path touches more
subsection data per learner than the course report does.

Caveat on attribution: I have not profiled query counts, so I can't state that this
accounts for the full 0.82 s/learner. The code asymmetry is clear and #38911 documents
the same pattern costing real time in the CCX view, but the size of the win here is a
hypothesis until someone measures it.

Suggested fix: add a bulk-context step to ProblemGradeReport._rows_for_users
mirroring _CourseGradeBulkContext — at minimum
prefetch_course_and_subsection_grades(course_id, users). Output is unchanged; this
only makes an existing read bulk. As a side benefit it also bounds Finding 3, because the
prefetch cache is keyed per course and replaced each batch rather than accumulating per
learner.

Finding 2 — the collected block structure is deserialized twice

_CourseGradeReportContext.graded_assignments
passes the already-loaded structure in:

grading_cxt = grades_context.grading_context(self.course, self.course_structure)

_ProblemGradeReportContext.graded_scorable_blocks_header
does not:

grading_context = grades_context.grading_context_for_course(self.course)

grading_context_for_course calls get_course_in_cache internally, and
BlockStructureManager.get_collected() deserializes fresh from the cache backend on
every call — there's no request-level memoization — so a large course pays the
deserialization and peak allocation twice.

Suggested fix: use grading_context(self.course, self.course_structure), matching
the course report.

Finding 3 — per-learner RequestCaches are never evicted during the task

RequestCache is flushed on the Celery task_postrun signal, i.e. after the report
finishes. Caches keyed per learner therefore grow monotonically for the life of the task:

• grades.models.VisibleBlocks, keyed (user_id, course_key), holding a dict of
  hash → VisibleBlocks model instance, each carrying the blocks_json TextField
• grades.models.PersistentSubsectionGradeOverride, keyed (user_id, course_key)

Nothing reads those entries again once a learner's row is written. The hook to evict them
already exists —
GradeReportBase._clear_caches
is called per batch — but the base is a no-op, CourseGradeReport never overrides it,
and
ProblemGradeReport._clear_caches
only clears two enrollment caches.

This is the term that scales with enrollment, and it is unaffected by the
instructor_task.use_on_disk_grade_reporting toggle.

Suggested fix: extend _clear_caches to drop the VisibleBlocks and override
namespaces, and implement it for CourseGradeReport too.

Finding 4 — the on-disk report path still buffers the whole file to upload

TemporaryFileReportMixin
(behind instructor_task.use_on_disk_grade_reporting) correctly streams rows to a
TemporaryFile instead of materializing them all, avoiding
InMemoryReportMixin._compile's
zip(*batched_rows) + list(chain(...)).

But the upload then undoes much of it —
DjangoStorageReportStore.store:

buff_contents = buff.read()                       # whole file into RAM
if not isinstance(buff_contents, bytes):
    buff_contents = buff_contents.encode('utf-8') # second copy
buff = ContentFile(buff_contents)                 # third
self.storage.save(path, buff)

So the on-disk path still peaks at roughly 3× the CSV text size. The
ReportStore docstring
anticipates this ("Should probably refactor later to create a ReportFile object that can
simply be appended to for the sake of memory efficiency").

Suggested fix: wrap the file object in django.core.files.File and hand it to
storage.save() directly so S3Boto3Storage can stream it. That requires the temp
files to be binary — TemporaryFile('r+') at
grades.py#L354 would
become a binary temp file wrapped in io.TextIOWrapper(..., newline='', encoding='utf-8')
for csv.writer.

Finding 5 — USER_BATCH_SIZE is dead code

CourseGradeReport.USER_BATCH_SIZE = 100
is never referenced. The effective batch size is the hardcoded default on the grouper
closure inside
_batch_users, so
anyone tuning that constant gets no effect.

Suggested fix: pass it through to grouper, or delete it.

Backward compatibility

All five are internal to report generation. CSV contents and format are unchanged by any
of them; no schema or data migration is involved. Findings 1–3 reuse helpers the codebase
already relies on.

Additional context

We hit this on a 5,154-learner course (128 scorable blocks, 261 CSV columns). The
immediate failure was operational rather than in this code — the report was sharing a
Celery deployment scaled on an unrelated queue's depth, so it was repeatedly killed
mid-run with a 30s termination grace period and left no terminal state, showing in the
instructor dashboard as permanently in progress. We fixed that on our side with a
dedicated worker and a grace period longer than the job.

That makes the report reliable for us but not fast, which is why these are worth
raising upstream: at 0.82 s/learner the runtime is what forces the operational
workaround in the first place. Happy to open a PR for any subset — Findings 2 and 5 are
one-liners, and 1 is a small, well-precedented change.

Start in lms/djangoapps/instructor_task/tasks_helper/grades.py with ProblemGradeReport._rows_for_users, the report context methods, _clear_caches, and _batch_users; compare them with CourseGradeReport. Then inspect instructor_task/models.py, especially DjangoStorageReportStore.store and the ReportStore contract. Done means the report keeps the same CSV output while avoiding per-learner reads, repeated deserialization, unbounded caches, buffered uploads, and ineffective batch configuration.