aws / aws/aws-fpga

F2 small shell: place_design (Vivado 2025.2) drops a physical-only VCC tie on two locked static slices at ~1.3M-cell CL scale -> HDPRVerify-41, AFI ingestion rejected (deterministic, scale-triggered)

Open
#799 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
SystemVerilog
Stars
1.7k
Forks
538
PR merge metrics
No merged PRs in 30d

Description

## Summary

Building a partial-reconfiguration CL against the **F2 small shell** (HDK kit **v2.3.3**, shell version **0x10212415**, **Vivado 2025.2** as shipped on FPGA Developer AMI 1.19.2), `place_design` deterministically alters the site programming of two **LOCKED static-region slices** — `SLICE_X117Y357` and `SLICE_X117Y358`, which hold the static shell's clock-A MMCM DRP state machine — by **dropping a physical-only VCC tie** (site pin `D6` VCC physical-only net + arc `D6 -> D6LUT[A6]`) that the shell reference checkpoint (`cl_bb_routed.small_shell.dcp`, sha256 matches the kit pin) carries.

`pr_verify -full_check` therefore fails with `HDPRVerify-41` on those two sites, and `create-fpga-image` rejects the AFI at ingestion.

The trigger is **design scale**, and the failure is **deterministic**: 4/4 builds of the CL at ~1.3M-cell scale fail identically (including a minimal one-engine, place-only configuration), while 3/3 builds of the same CL at a smaller scale pass the identical check and ingested fine. Four candidate mechanisms were each exonerated by direct A/B experiment (details below). No large configuration of this CL can ship until there is a workaround or an ingestion-side answer.

## AWS-side symptom (verbatim, from the failed ingestion)

```
ERROR: [Constraints 18-13341] HDPRVerify-41: Static site SLICE_X117Y358 has different
site programming in design checkpoint in-memory-design and ../checkpoints/SH_CL_BB_routed.dcp.
ERROR: [Constraints 18-13341] HDPRVerify-41: Static site SLICE_X117Y357 ... (same)
ERROR: [Common 17-39] 'pr_verify' failed due to earlier errors. (ingest.tcl line 82)
```

- Failing AFI: **`afi-04b0d37f1789bb2f6`** (us-west-2, submitted with `--logs-storage-location`; ingestion logs preserved).
- Passing controls, same CL / same flow / same kit at smaller scale: `afi-03232169ea962036d` and `afi-036d83cafa00d26ea` (the latter is currently in active use).

## The byte-level delta

`pr_verify`'s own site-diff dumps show **exactly one difference per site**. The shell reference (BB) checkpoint **has**, and our post-place checkpoints **lack**:

```
Site pin 'D6' VCC Physical-only net
arc D6 -> D6LUT[A6]
```

This is a fractured-LUT unused-input VCC tie, **physical-only** — invisible to logical Tcl queries, which is why every queryable comparison matched first: INIT/INIT_A–D, per-pin inversions, BEL bindings, LOCK_PINS, used site PIPs, and pin→BEL maps are identical **139/139** between our checkpoint and the BB at both sites. The two slices hold the static shell's `WRAPPER/IO_SHIM/RL_CLKS/MMCM_CLK_MAIN_A/.../mmcm_drp_inst` state machine and its LUTRAM recipe ROM (`ram_reg[24][35]`, `ram_do_reg[32]`, …), identically placed in both checkpoints.

## Stage bisection

Running `pr_verify -full_check -in_memory -additional ` on each saved stage checkpoint of the same build:

| Checkpoint | pr_verify |
|---|---|
| `post_link` | **CLEAN** |
| `post_opt` | **CLEAN** (and the affected site pin's `IS_USED` property stays `1` through opt) |
| `post_place` | **DAMAGED** (both sites) — every later stage inherits it |

So the damage occurs inside **`place_design` proper**; `opt_design` is exonerated.

Possibly related: AMD knowledge-base articles 000040413 / 000040409 / 000035406 describe a family of Vivado defects around dropped unused-LUT-input tie programming — but that documented family fingerprints in `opt_design`, with `IS_USED` visibly changing. Our failure preserves everything through opt and dies in the placer, with the resulting delta **below the property-queryable layer** (the post-place checkpoint still looks correct to every Tcl query we ran; only pr_verify's byte-level site comparison sees the missing tie). The association with those KB articles is our reading, not a vendor confirmation. If it is the same defect class, it has a placer-phase manifestation the published articles do not describe, and their opt_design-targeted workarounds cannot be assumed to apply.

## Variables exonerated by direct A/B experiment

| # | Variable | Experiment | Result |
|---|---|---|---|
| 0 | Kit drift (BB checkpoint) | sha256 of the kit's `cl_bb_routed.small_shell.dcp` vs the kit pin | **No drift** — hash matches |
| 1 | Clock-recipe TCL (`aws_clock_properties.tcl` `set_property` calls on the static shell MMCM) | Guarded out the three `proc_set_mmcm $RL_A0_MMCM` calls, rebuilt the same design | **Fails identically — exonerated.** (Corroboration: an earlier smaller build ran the same recipe on an unpatched kit and ingested fine) |
| 2 | Placement adjacency (CL cells packed at `SLICE_X116Y358`, sharing the interconnect column with damaged static `X117Y358`) | Site PROHIBITs on the CL-facing apron (`SLICE_X116Y355..360` + `X117Y359..360`), verified to hold at post_place | **Fails identically with the apron empty — exonerated** |
| 3 | Placer directive | Place-only run under `AltSpreadLogic_high` | **Fails identically — exonerated** |
| 4 | Engine count (4 → 1) | Minimal one-engine configuration, place-only + in-memory pr_verify | **Fails identically — exonerated** (so CL utilization is not the lever) |
| 5 | Design scale | Small-scale controls 3/3 green + ingested; large-scale builds 4/4 red | **CORRELATED — the trigger.** Threshold not further localized |
| 6 | Vivado version | **Not varied** — every build and every pr_verify ran 2025.2 | Open variable |

Checker validation: our local `pr_verify` procedure agrees with AWS's ingestion verdict **4-for-4** across the checkpoints where both exist.

## Reproduction

1. FPGA Developer AMI (Ubuntu) 1.19.2 (`ami-07a164f1a402ab274`), `m6a.4xlarge` or larger (a 30 GiB host OOMs during routing at this scale). Vivado 2025.2 as shipped.
2. aws-fpga branch `f2` at v2.3.3; `source hdk_setup.sh`; BB checkpoint sha256 confirmed against the kit pin.
3. CL: proprietary design (~1.3M CL cells), CL_TEMPLATE-based, built with `aws_build_dcp_from_cl.py -c --aws_clk_gen --clock_recipe_a A2`. The failing routed build meets timing (worst slack +0.711 ns) — this is not a timing/congestion failure.
4. Local check, no AWS submission needed: open the post-place (or any later) checkpoint in Vivado 2025.2 and run
```
pr_verify -full_check -in_memory -additional /…/from_aws/cl_bb_routed.small_shell.dcp
```
Expected: `HDPRVerify-41` on `SLICE_X117Y357` and `SLICE_X117Y358`. The same command on the same design's `post_link` / `post_opt` checkpoint reports clean.

We can provide **everything** to AWS/AMD on request through a private channel: the failing routed tarball (`2026_07_27-195354.Developer_CL.tar`, 191,272,960 bytes), every stage checkpoint (post_link / post_opt / post_place / routed) of failing and control builds, full build and pr_verify logs, and the two site-diff dumps.

## What we are asking for

1. **A supported workaround** — a placer parameter / directive / patch for Vivado 2025.2 that preserves static-region physical tie programming at this design scale (`AltSpreadLogic_high` is already exonerated above).
2. **Or an ingestion-policy answer** — can `create-fpga-image`'s pr_verify tolerate deltas that are *provably physical-only VCC ties on static sites* (functionally a constant; every logical/queryable property matches)? If yes, under what attestation?
3. **A tracking id** — for the tool defect (AMD) and/or the ingestion case (AWS), so future kit/tool updates can be tested against a named issue.

## Environment

| Item | Value |
|---|---|
| HDK kit | aws-fpga branch `f2`, v2.3.3 (BB checkpoint sha256 matches kit pin) |
| Shell | small shell, AFI manifest shell version 0x10212415 |
| Vivado | 2025.2 (only version used — see matrix row 6) |
| Build AMI | FPGA Developer AMI (Ubuntu) 1.19.2, `ami-07a164f1a402ab274` |
| Build instance | m6a.4xlarge (61 GiB; a 30 GiB host OOMs during route at this scale) |
| Runtime instance | f2.6xlarge (VU47P), us-west-2 |
| Failing AFI | `afi-04b0d37f1789bb2f6` (ingestion logs preserved via `--logs-storage-location`) |
| Passing AFIs (same CL, smaller scale) | `afi-03232169ea962036d`, `afi-036d83cafa00d26ea` |
| Failing sites | `SLICE_X117Y357` / `SLICE_X117Y358` (static shell `MMCM_CLK_MAIN_A` DRP block) |

Contributor guide

Open the contributing guide

Research direction

Start with the saved post_opt and post_place checkpoints and run pr_verify -full_check -in_memory -additional against cl_bb_routed.small_shell.dcp in Vivado 2025.2. Compare the clean post_opt result with the HDPRVerify-41 failures at SLICE_X117Y357 and SLICE_X117Y358, using the stage logs and site-diff dumps. Done means identifying a supported workaround, an ingestion-policy answer, or a tracking ID from AWS or AMD.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, ubuntu
Domain
cloud, embedded-iot, tooling
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.