openvinotoolkit / openvinotoolkit/nncf
Concat scales not being grouped
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 305
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 27
Description
I am trying to quantize a pytorch model using NNCF.
The output of my model is a concatenation of two tensors.
To quantize my outputs I set:
advanced_parameters = AdvancedQuantizationParameters(quantize_outputs=True)
When I quantize the model I get a separate quantizer for each input:
ModuleDict(
(/nncf_model_input_0|OUTPUT): AsymmetricQuantizer(bit=8, ch=False)
(/nncf_model_input_1|OUTPUT): AsymmetricQuantizer(bit=8, ch=False)
)
Based on what I saw in NNCF I would expect to get something like this.
ModuleDict(
(/nncf_model_input_0|OUTPUT;/nncf_model_input_1|OUTPUT): AsymmetricQuantizer(bit=8, ch=False)
)
I am guessing it's an edge case which comes up due to AdvancedQuantizationParameters.
NNCF version: 2.6.0
Run the following to reproduce:
import nncf
import torch
import numpy as np
from nncf.quantization.advanced_parameters import AdvancedQuantizationParameters
class DummyDataset(torch.utils.data.Dataset):
""" Loads images from a folder """
def __init__(self, input_shapes, input_names):
self.input_shapes = input_shapes
self.input_names = input_names
def __len__(self):
return 1
def __getitem__(self, index):
return { self.input_names[i]: np.random.rand(*input_shape) for i, input_shape in enumerate(self.input_shapes) }
class DummyModel(torch.nn.Module):
def __init__(self):
super().__init__()
self.conv = torch.nn.Conv2d(6, 6, 3, 1, 1)
def forward(self, x, y):
x_cat_y = torch.cat((x,y), dim=1)
return x_cat_y
# return self.conv(x_cat_y) # use this to verify that quantizers get grouped if concat isn't the output
def quantize_model(model, input_shapes, input_names):
def transform_fn(data):
data_dict = data
return tuple(data_dict[key][0].to(torch.float32) for key in data_dict)
dummy_dl = torch.utils.data.DataLoader(DummyDataset(input_shapes, input_names))
calibration_dataset = nncf.Dataset(dummy_dl, transform_fn)
advanced_parameters = AdvancedQuantizationParameters(quantize_outputs=True)
return nncf.quantize(model, calibration_dataset, subset_size=1, preset=nncf.QuantizationPreset.MIXED, advanced_parameters=advanced_parameters)
def main():
quantized_model = quantize_model(DummyModel(), [(1, 3, 256, 256), (1, 3, 256, 256)], ["x", "y"])
print()
print(quantized_model._nncf.external_quantizers)
if __name__ == "__main__":
main()
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.