Comfy-Org / Comfy-Org/ComfyUI

convrot is just a wrapper around QuaRot Stage 1c, ported from LLM linear layers to DiT/FLUX linear layers

Open
#14,778 2 comments 1 reaction 0 assignees View on GitHub
Feature
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

### Feature Idea

ConvRot is just a wrapper of QuaRot Stage 1c, ported from linear layers from LLM to DiT/FLUX. ConvRot should not be described as a new method. It is just QuaRot Stage 1c repackaged for DiT/FLUX linear layers. **Applying an existing method into a new architecture is a useful work. But presenting that application as a fundamentally new method is not acceptable.** The work should be framed honestly as a DiT/FLUX wrapper or adaptation of QuaRot Stage 1c, with clear attribution.

## 1. Both apply "group-wise" operator
Both ConvRot and QuaRot Stage 1c in essence apply the same block-wise Hadamard operator, though in different written forms.
QuaRot Stage 1c writes the rotation as one Kronecker-structured multiplication:

$$
R = I_G \otimes H_{N_0},
$$

which means applying the same Hadamard transform independently inside each channel group.

ConvRot write in a chunked form but in fact same operation as stage 1c in QuoRot.

$$
Y =
\sum_{i=1}^{\lceil K/N_0\rceil}
\mathrm{RHT}(X_i)\mathrm{RHT}(W_i)^{\top}.
$$

If concatenate the chunks as

$$
X = [X_1,\ldots,X_G],
\qquad
W = [W_1,\ldots,W_G],
$$

then applying RHT to every chunk is equivalent to multiplying the concatenated tensor by a block-diagonal Hadamard matrix. Assuming \(K = G N_0\), or after padding the last block if necessary, this block rotation is

$$
R=I_G \otimes H_{N_0}=
\begin{bmatrix}
H_{N_0} & 0 & \cdots & 0 \\
0 & H_{N_0} & \cdots & 0 \\
\vdots & \vdots & \ddots & \vdots \\
0 & 0 & \cdots & H_{N_0}
\end{bmatrix}.
$$

Therefore,

$$
X\cdot R=[X_1,\ldots,X_G]
\begin{bmatrix}
H_{N_0} & 0 & \cdots & 0 \\
0 & H_{N_0} & \cdots & 0 \\
\vdots & \vdots & \ddots & \vdots \\
0 & 0 & \cdots & H_{N_0}
\end{bmatrix}= [X_1H_{N_0},\ldots,X_GH_{N_0}]= \sum_{i=1}^{\lceil K/N_0\rceil}
\mathrm{RHT}(X_i)\mathrm{RHT}(W_i)^{\top}.
$$

So ConvRot’s chunk-wise RHT formulation is the same as QuaRot Stage 1c’s Kronecker formulation. The difference is that QuaRot Stage 1c applies this to the attention projections, usually with group size $d_h$, while ConvRot seems to apply the grouped operation to DIT , i.e.

$$
\text{QuaRot Stage 1c:}
W_{\mathrm{attn}}(I_G \otimes H_{N_0}),
$$

$$
\text{ConvRot: }
W_{\mathrm{linear}}(I_G \otimes H_{256}).
$$

## 2. The equivalent Hadamard matrix
I’m also trying to understand whether ConvRot’s choice of a **regular** Hadamard matrix, rather than QuaRot Stage 1c’s Hadamard matrix, changes the operator in an essential way.

For $$N_0 = 4^k$$, my understanding is that the two Hadamard matrices are equivalent up to input/output coordinate reorderings and sign flips:

$$
H^{\mathrm{regular}} = P H^{\mathrm{Sylv}} Q.
$$

where $P$ and $Q$ are signed permutation matrices. In other words, the regular Hadamard does not seem to introduce a different kind of mixing from the Quarot stage 1c's Hadamard . It seems to use the same Hadamard mixing, but with a different ordering and sign convention for the input and output coordinates.

##
To conclude, ConvRot is not a new method, but an engineering wrapper around QuaRot Stage 1c for DiT/FLUX model: adapt QuaRot stage 1C from LLM linear layers to DiT/FLUX linear layers.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by locating the ConvRot and QuaRot Stage 1c implementations and their DiT/FLUX integration, then compare the operators and attribution. Done means the adaptation is framed as a QuaRot Stage 1c wrapper with clear attribution, rather than as a new method.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.