convrot is just a wrapper around QuaRot Stage 1c, ported from LLM linear layers to DiT/FLUX linear layers
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Feature Idea
ConvRot is just a wrapper of QuaRot Stage 1c, ported from linear layers from LLM to DiT/FLUX. ConvRot should not be described as a new method. It is just QuaRot Stage 1c repackaged for DiT/FLUX linear layers. **Applying an existing method into a new architecture is a useful work. But presenting that application as a fundamentally new method is not acceptable.** The work should be framed honestly as a DiT/FLUX wrapper or adaptation of QuaRot Stage 1c, with clear attribution.
## 1. Both apply "group-wise" operator
Both ConvRot and QuaRot Stage 1c in essence apply the same block-wise Hadamard operator, though in different written forms.
QuaRot Stage 1c writes the rotation as one Kronecker-structured multiplication:
$$
R = I_G \otimes H_{N_0},
$$
which means applying the same Hadamard transform independently inside each channel group.
ConvRot write in a chunked form but in fact same operation as stage 1c in QuoRot.
$$
Y =
\sum_{i=1}^{\lceil K/N_0\rceil}
\mathrm{RHT}(X_i)\mathrm{RHT}(W_i)^{\top}.
$$
If concatenate the chunks as
$$
X = [X_1,\ldots,X_G],
\qquad
W = [W_1,\ldots,W_G],
$$
then applying RHT to every chunk is equivalent to multiplying the concatenated tensor by a block-diagonal Hadamard matrix. Assuming \(K = G N_0\), or after padding the last block if necessary, this block rotation is
$$
R=I_G \otimes H_{N_0}=
\begin{bmatrix}
H_{N_0} & 0 & \cdots & 0 \\
0 & H_{N_0} & \cdots & 0 \\
\vdots & \vdots & \ddots & \vdots \\
0 & 0 & \cdots & H_{N_0}
\end{bmatrix}.
$$
Therefore,
$$
X\cdot R=[X_1,\ldots,X_G]
\begin{bmatrix}
H_{N_0} & 0 & \cdots & 0 \\
0 & H_{N_0} & \cdots & 0 \\
\vdots & \vdots & \ddots & \vdots \\
0 & 0 & \cdots & H_{N_0}
\end{bmatrix}= [X_1H_{N_0},\ldots,X_GH_{N_0}]= \sum_{i=1}^{\lceil K/N_0\rceil}
\mathrm{RHT}(X_i)\mathrm{RHT}(W_i)^{\top}.
$$
So ConvRot’s chunk-wise RHT formulation is the same as QuaRot Stage 1c’s Kronecker formulation. The difference is that QuaRot Stage 1c applies this to the attention projections, usually with group size $d_h$, while ConvRot seems to apply the grouped operation to DIT , i.e.
$$
\text{QuaRot Stage 1c:}
W_{\mathrm{attn}}(I_G \otimes H_{N_0}),
$$
$$
\text{ConvRot: }
W_{\mathrm{linear}}(I_G \otimes H_{256}).
$$
## 2. The equivalent Hadamard matrix
I’m also trying to understand whether ConvRot’s choice of a **regular** Hadamard matrix, rather than QuaRot Stage 1c’s Hadamard matrix, changes the operator in an essential way.
For $$N_0 = 4^k$$, my understanding is that the two Hadamard matrices are equivalent up to input/output coordinate reorderings and sign flips:
$$
H^{\mathrm{regular}} = P H^{\mathrm{Sylv}} Q.
$$
where $P$ and $Q$ are signed permutation matrices. In other words, the regular Hadamard does not seem to introduce a different kind of mixing from the Quarot stage 1c's Hadamard . It seems to use the same Hadamard mixing, but with a different ordering and sign convention for the input and output coordinates.
##
To conclude, ConvRot is not a new method, but an engineering wrapper around QuaRot Stage 1c for DiT/FLUX model: adapt QuaRot stage 1C from LLM linear layers to DiT/FLUX linear layers.
Contributor guide
Research direction
No files, tests, or entry points are named. Start by locating the ConvRot and QuaRot Stage 1c implementations and their DiT/FLUX integration, then compare the operators and attribution. Done means the adaptation is framed as a QuaRot Stage 1c wrapper with clear attribution, rather than as a new method.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100