tensorflow / tensorflow/model-optimization

How to user quantize to imporve inference performance on tensorflow-serving?

Open
#450 3 comments 2 reactions 1 assignee View on GitHub

@nutsiepully is already working on this.

Since Aug 7, 2020.

bug technique:qat
Dominant language
Python
Stars
1.6k
Forks
349
Avg merge
3d 2h
Merged PRs (30d)
1

Description

Prior to filing: check that this should be a bug instead of a feature request. Everything supported, including the compatible versions of TensorFlow, is listed in the overview page of each technique. For example, the overview page of quantization-aware training is here. An issue for anything not supported should be a feature request.

Describe the bug
I use the "Quantization aware training in Keras example"
and save 4 models: model(model without quantization aware training), q_aware_model(model with quantization aware training), quantized_tflite_model(TFLite model with quantization aware training ), float_tflite_model(TFLite model without quantization), and depoly the models with tensorflow-serving,but I got the same performance. I dont konw the reason. How can I use the quantized model to imporve inference performance on tensorflow-serving?
System information

TensorFlow version (installed from source or binary): nightly

TensorFlow Model Optimization version (installed from source or binary): binary

Python version: python3.8

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.