Node-free variant grid export (CSV) for an arbitrary list of variants

🤖 Written by Claude

## Goal

Make it possible to export a **VariantGrid analysis-style grid** (CSV, with all the configured annotation columns and identical formatting) for an **arbitrary list/queryset of variants**, *without* creating an `Analysis` or `AnalysisNode`.

Today the only way to produce a fully-formatted variant grid export is through the analysis grid machinery, which is hard-bound to an `AnalysisNode`. To export "a dozen variants" outside an analysis (e.g. from search results, a variant-tag list, a gene page) you currently have to spin up a throwaway `Analysis` + `AllVariantsNode`, restrict its queryset, run the export, then delete it. That works but has real costs: DB writes (`Analysis` + `AnalysisNode` + `NodeVersion` rows per export), a required user, audit-log noise, and fragility (it relies on `AllVariantsNode` being queryable while unloaded).

We want a clean, node-free path that produces output **byte-identical** to the existing analysis CSV export for the same set of columns.

## Background — why this is feasible

The coupling to `Analysis`/`AnalysisNode` is almost entirely *incidental*. The actual export pipeline has no intrinsic need for a node — `VariantGrid` just happens to read all its config off `node.analysis`.

The core pieces are already node-free:

- **Columns + overrides**: `snpdb/grid_columns/custom_columns.py::get_custom_column_fields_override_and_sample_position(ccc, annotation_version, analysis_tags=...)` — pure function of `(CustomColumnsCollection, AnnotationVersion)`.
- **Base queryset**: `annotation/annotation_version_querysets.py::get_variant_queryset_for_annotation_version(annotation_version)` — installs the SQL partition transformer so `variantannotation__*` joins resolve to the correct partition. (This is *not* an ordinary FK filter — you MUST start from this queryset or annotation columns will not join correctly.)
- **Extra annotations**: `snpdb/grid_columns/custom_columns.py::get_variantgrid_extra_annotate(user)` — needs a `user`, not a node (provides `tags_global`, `internally_classified`, etc.).
- **colmodels / server-side formatters**: built by the jqgrid base class from `self.fields` + `self._overrides` — driven by data the grid already holds, not by the node.
- **CSV writing**: `library/django_utils/jqgrid_view.py::grid_export_csv(colmodels, items)` — already node-free.
- **Export-only row fixups**: `analysis/grid_export.py::format_items_iterator(sample_ids, items, variant_tags_dict)` — only needs `sample_ids` (which is `[]` when there are no samples), plus each row's `tags_global` and `id`.

## Current coupling to untangle

In `analysis/grids.py`:

- `VariantGrid.__init__(self, user, node, extra_filters=None, af_show_in_percent=None)` (line ~52) reads from `node.analysis`: `genome_build`, `custom_columns_collection`, `annotation_version`, `default_sort_by_column`, `grid_sample_label_template`, plus `node.get_cohorts_and_sample_visibility()`, `node.get_extra_columns()`, `node.get_extra_grid_config()`, `NodeCount.load_for_node(...)`.
- `VariantGrid._get_base_queryset()` (line ~83) returns `self.node.get_queryset()`.
- `VariantGrid._get_fields_and_overrides(node, af_show_in_percent)` (line ~154) — already mostly delegates to `get_custom_column_fields_override_and_sample_position(...)`; the node-specific parts are the extra columns and sample/genotype columns.
- `VariantGrid.get_colmodels(...)` (line ~101) injects `{"analysisNode": {"visible": node.visible}}` into every colmodel — analysis-only.
- `VariantGrid._get_q()` (line ~109) — `NodeCount`/extra-filters (diff-to-parent) logic — analysis-only.
- `ExportVariantGrid` (line ~306) — paginates contig-at-a-time via `node.analysis.genome_build`; otherwise generic.

In `analysis/grid_export.py`:

- `node_grid_get_export_iterator(request, node, export_type, ...)` (line ~23) hardcodes `ExportVariantGrid(request.user, node, ...)` and pulls `node.get_sample_ids()` / `node.analysis.genome_build`.

### Classification of each coupling

**Incidental (drop / default for the node-free path):**
- `get_colmodels` `analysisNode` injection
- `_get_q` / `NodeCount` extra-filters
- `default_sort_by_column` / `grid_sample_label_template` (these originate from `UserSettings` *via* the analysis — read them from `UserSettings.get_for_user(user)` directly)
- `get_extra_columns()` / `get_extra_grid_config()` (default empty)
- `node.get_queryset()` DAG filtering (replaced by an explicit base queryset / `pk__in`)

**Genuinely context-bound — but to a cohort/sample, NOT an analysis:**
- Sample/genotype columns (zygosity, AD/AF/DP/GQ/PL/FT, `CohortGenotype` packed-data columns + per-sample visibility), built in `VariantGrid.get_grid_genotype_columns_and_overrides(...)`. This needs a cohort context, which a variant-level export does not have.

## Proposed implementation

### Scope (first cut)

**Variant-level columns only.** No sample/genotype columns in this issue — that's the one genuinely context-bound piece and untangling it is out of scope here. Get the node-free path shipped for the clean 90%; sample support can be added later by passing cohorts into the context explicitly.

### Approach

1. Introduce a small context object (e.g. `VariantGridExportConfig`) carrying everything the grid needs, independent of a node:
- `genome_build: GenomeBuild`
- `annotation_version: AnnotationVersion` (the umbrella `AnnotationVersion`, which has `.variant_annotation_version` — NOT a bare `VariantAnnotationVersion`)
- `custom_columns_collection: CustomColumnsCollection`
- `user: User`
- `base_queryset: QuerySet[Variant]` (already built via `get_variant_queryset_for_annotation_version`, optionally pre-filtered)
- `af_show_in_percent: bool` (default from `settings.VARIANT_ALLELE_FREQUENCY_CLIENT_SIDE_PERCENT`)
- `order_by: Optional[list[str]]`
- (later) optional `cohorts` / `visibility` for sample columns

2. Refactor a node-free base (e.g. `BaseVariantGrid`) that depends on the context object instead of `node`. Move the node-free logic (`_get_fields_and_overrides` minus extra/sample columns, base queryset, extra annotate kwargs, colmodels) onto it.

3. Make the existing `VariantGrid` a **thin adapter** that builds the context from a node and adds the node-specific bits (extra columns, sample/genotype columns, `analysisNode` colmodel injection, `_get_q`). **The existing analysis export must be unchanged.**

4. Provide a node-free entry point, e.g.:
`analysis/grid_export.py::variant_list_grid_export(variants_or_qs, annotation_version, ccc, user, export_type='csv') -> tuple[str, Iterator[str]]`
mirroring `node_grid_get_export_iterator` but driven by the context object. `user` should default to `library.guardian_utils.admin_bot()` (note: tag/classification columns are permission-filtered per user).

5. Parameterize `node_grid_get_export_iterator` / `format_items_iterator` off the context where they currently reach for `node` (e.g. `sample_ids` becomes `[]` for the variant-list case).

### Reference prototype

`export_analysis.py` - issue comment - already contains a working **fake-analysis** implementation (`get_annotated_variants_csv`, `get_annotated_variant_rows`) that produces correctly-formatted output by creating a throwaway `Analysis` + `AllVariantsNode`. Use it as the behavioural reference / oracle for what correct output looks like — the new node-free path should produce identical output. It is NOT the desired end state (the whole point is to remove the fake analysis); treat it as a spec, then delete it once the real path lands.

## Acceptance criteria

- A public function exists that takes a list (or queryset) of `Variant`, an `AnnotationVersion`, a `CustomColumnsCollection`, and a user, and returns formatted CSV lines (header first) — with **no** `Analysis`/`AnalysisNode`/`NodeVersion` rows created or deleted, and no `FakeRequest`/grid-handler request needed.
- For a shared variant-level column set, the output is **byte-identical** to the existing analysis grid CSV export over the same variants (header row + data rows + server-side formatting + the `-1 → "."` and tag-summarising fixups from `format_items_iterator`).
- The existing analysis grid export (the node path) is unchanged — verified by existing tests still passing.
- A new test asserts the node-free export matches the analysis export byte-for-byte on a shared column set, using `annotation/tests/test_data_fake_genes.py` / `analysis/tests/utils.py::AnalysisSetupMixin` style fixtures. **This test is required** — the two paths will silently drift without it.

## Out of scope (note as follow-ups)

- Sample/genotype columns in the node-free path (needs cohort context passed explicitly).
- VCF export via the node-free path (CSV first; VCF can follow the same context pattern afterward).

## Key files

- `analysis/grids.py` — `VariantGrid`, `ExportVariantGrid` (the refactor target)
- `analysis/grid_export.py` — `node_grid_get_export_iterator`, `format_items_iterator`, `grid_export_csv` usage
- `snpdb/grids.py` — `AbstractVariantGrid` (base `get_queryset`/field-name logic, ~line 499)
- `snpdb/grid_columns/custom_columns.py` — `get_custom_column_fields_override_and_sample_position`, `get_variantgrid_extra_annotate`
- `annotation/annotation_version_querysets.py` — `get_variant_queryset_for_annotation_version` (partition transformer)
- `library/django_utils/jqgrid_view.py` — `grid_export_csv`
- `library/jqgrid/jqgrid.py` — base `get_items` / `iter_format_items` / `get_colmodels`