feat: quantized runtime input sidecars for Gemma 4 E-models

Detailed description of the requested feature

Gemma 4 E-models externalize two large runtime input tables as FP16:

• token embedding [262144, 2560]: 1.25 GiB
• per-layer embedding (PLE) [262144, 10752]: 5.25 GiB

For google/gemma-4-E4B-it with W4A16 AWQ, those tables make the exported weight payload at least 10.64 GiB before TensorRT contexts, KV cache, activations, and image preprocessing. This prevents a documented supported model from fitting responsibly on a 16 GB Jetson Orin NX.

I would like to contribute opt-in symmetric INT8 runtime sidecars for these tables:

• per-row scales for the token embedding
• grouped per-row scales for PLE, preserving one independent scale per PLE layer
• gather/dequantize only requested rows into the existing FP16 runtime outputs
• preserve FP16 sidecar compatibility and Gemma multimodal-placeholder zero filling
• exporter/runtime validation, CUDA unit tests, Python tests, and documentation

A tested implementation is available at:
https://github.com/mihai-chiorean/TensorRT-Edge-LLM/tree/feat/gemma4-int8-sidecars

Measured on the real E4B artifacts, INT8 sidecars save 3.21 GiB versus FP16. Sampled reconstruction cosine similarity is 0.999958 for token embeddings and 0.999969 for PLE. On an Orin NX 16 GB, the resulting runtime completes a 12-image Gemma 4 workload at 8.99 GB peak unified memory. With 546 visual tokens per image it matches the existing runtime's 12/12 task labels at 3.26 s p50 and 3.31 s p95 end-to-end latency.

The submission branch is now feature-only and DCO-signed. The separate Jetson startup-allocation work remains on the combined validation branch and will be proposed independently after a broader allocation-order design and its own approved issue.

Per CONTRIBUTING.md, I am opening this issue for approval before requesting code review. Would this contribution be welcome, and would maintainers prefer the INT8 sidecars and Jetson allocation fixes as two PRs?

Timeline

Blocker for running the already-supported Gemma 4 E4B VLM on Jetson Orin NX 16 GB. There is a slower llama.cpp fallback, so no release deadline.

Describe alternatives you've considered

• FP16 sidecars: numerically ideal, but consume 6.50 GiB by themselves.
• Output-vocabulary reduction: reduces lm_head, but intentionally leaves both input tables at the original 262k vocabulary.
• Packed INT4 sidecars: saves more memory, but adds unpacking complexity and greater accuracy risk. INT8 is the conservative first format for SM87.
• FP8/NVFP4: unavailable on Orin SM87.

Target hardware/use case

• Jetson Orin NX 16 GB, SM87
• JetPack 6.2.x / CUDA 12.6 / TensorRT 10.3
• google/gemma-4-E4B-it, image-input VLM, W4A16 AWQ
• Batch 1 edge inference under a constrained unified-memory budget

Start with CONTRIBUTING.md and review the linked feature branch to understand the proposed exporter and runtime changes. Check the existing validation, CUDA unit-test, Python-test, and documentation entry points before assessing scope. Done means preserving FP16 behavior and placeholder handling while adding validated INT8 sidecars across exporter, runtime, tests, and documentation.