NVIDIA / NVIDIA/TensorRT-Edge-LLM

LLM Quantization Cannot Load `text_dataset` (Default and Custom Values Fail)

Open
#150 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
563
Forks
135
Avg merge
14h 13m
Merged PRs (30d)
1

Description

Describe the bug

I cannot reproduce the quantization step from the Quick Start Guide because the text_dataset argument consistently fails.

  • When set to cnn_dailymail, it returns:
Repository id must be 'namespace/name', got 'cnn_dailymail'.
  • When set to abisee/cnn_dailymail, it returns:
Registered text datasets: cnn_dailymail, wikitext.
Steps/Code to reproduce bug

Installation method:
As described in TensorRT Edge-LLM on Jetson, I used an NVIDIA Jetson Thor as both the host and target device. I installed TensorRT Edge-LLM v0.9.1 by following the Installation Guide. The installation completed successfully without any issues.

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Jul_16_07:31:19_PM_PDT_2025
Cuda compilation tools, release 13.0, V13.0.48
Build cuda_13.0.r13.0/compiler.36260728_0

$ nvidia-smi
Mon Jul 27 08:52:49 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.00                 Driver Version: 580.00         CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA Thor                    Off |   00000000:01:00.0 Off |                  N/A |
| N/A   36C  N/A               3W /  N/A  | Not Supported          |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A           15507      G   /usr/lib/xorg/Xorg                       92MiB |
|    0   N/A  N/A           15707      G   /usr/bin/gnome-shell                     55MiB |
+-----------------------------------------------------------------------------------------+

$ python3 --version
Python 3.12.3

$ cat /etc/nv_tegra_release
# R38 (release), REVISION: 4.0, GCID: 43443517, BOARD: generic, EABI: aarch64, DATE: Wed Dec 31 00:15:19 UTC 2025
# KERNEL_VARIANT: oot
TARGET_USERSPACE_LIB_DIR=nvidia
TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia
INSTALL_TYPE=

Export command used:

  • When text_dataset is set to cnn_dailymail (also default value):
$ tensorrt-edgellm-quantize llm --model_dir Qwen/Qwen3-0.6B --output_dir Qwen3-0.6B-My-fp8-fp16 --quantization fp8 --text_dataset cnn_dailymail

/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/modelopt/torch/__init__.py:53: UserWarning: transformers>=5.0 support is experimental. Unified Hugging Face checkpoint export for quantized checkpoints may not work for some models yet.
_warnings.warn(
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 311/311 [00:00<00:00, 3993.24it/s]
Text calibration dataset: cnn_dailymail
Traceback (most recent call last):
File "/workspace/TensorRT-Edge-LLM/host_venv/bin/tensorrt-edgellm-quantize", line 8, in <module>
    sys.exit(main())
            ^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/tensorrt_edgellm/scripts/quantize.py", line 171, in main
    quantize_and_export(
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/tensorrt_edgellm/quantization/quantize.py", line 805, in quantize_and_export
    loader = _text_calib_dataloader(tokenizer,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/tensorrt_edgellm/quantization/quantize.py", line 62, in _text_calib_dataloader
    texts = list(islice(text_dataset(), num_samples))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/tensorrt_edgellm/quantization/datasets/builtin.py", line 27, in cnn_dailymail
    ds = load_hf_split("cnn_dailymail",
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/tensorrt_edgellm/quantization/datasets/__init__.py", line 238, in load_hf_split
    return load_dataset(hf_id, config, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/datasets/load.py", line 1688, in load_dataset
    builder_instance = load_dataset_builder(
                    ^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/datasets/load.py", line 1315, in load_dataset_builder
    dataset_module = dataset_module_factory(
                    ^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/datasets/load.py", line 1207, in dataset_module_factory
    raise e1 from None
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/datasets/load.py", line 1182, in dataset_module_factory
    ).get_module()
    ^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/datasets/load.py", line 598, in get_module
    standalone_yaml_path = cached_path(
                        ^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/datasets/utils/file_utils.py", line 180, in cached_path
    ).resolve_path(url_or_filename)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/huggingface_hub/hf_file_system.py", line 305, in resolve_path
    parsed = parse_hf_uri(f"{constants.HF_PROTOCOL}{path}")
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/huggingface_hub/utils/_hf_uris.py", line 319, in parse_hf_uri
    return _parse_repo_body(location, type_, raw=raw)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/huggingface_hub/utils/_hf_uris.py", line 617, in _parse_repo_body
    raise HfUriError(uri=raw, msg=f"Repository id must be 'namespace/name', got '{repo_id}'.")
huggingface_hub.errors.HfUriError: Invalid HF URI 'hf://datasets/cnn_dailymail@96df5e686bee6baa90b8bee7c28b81fa3fa6223d/.huggingface.yaml'. Repository id must be 'namespace/name', got 'cnn_dailymail'.
  • When text_dataset is set to abisee/cnn_dailymail:
$ tensorrt-edgellm-quantize llm --model_dir Qwen/Qwen3-0.6B --output_dir Qwen3-0.6B-My-fp8-fp16 --quantization fp8 --text_dataset abisee/cnn_dailymail

/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/modelopt/torch/__init__.py:53: UserWarning: transformers>=5.0 support is experimental. Unified Hugging Face checkpoint export for quantized checkpoints may not work for some models yet.
_warnings.warn(
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 311/311 [00:00<00:00, 3993.24it/s]
Text calibration dataset: cnn_dailymail
Traceback (most recent call last):
File "/workspace/TensorRT-Edge-LLM/host_venv/bin/tensorrt-edgellm-quantize", line 8, in <module>
    sys.exit(main())
            ^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/tensorrt_edgellm/scripts/quantize.py", line 171, in main
    quantize_and_export(
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/tensorrt_edgellm/quantization/quantize.py", line 805, in quantize_and_export
    loader = _text_calib_dataloader(tokenizer,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/tensorrt_edgellm/quantization/quantize.py", line 62, in _text_calib_dataloader
    texts = list(islice(text_dataset(), num_samples))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/tensorrt_edgellm/quantization/datasets/builtin.py", line 27, in cnn_dailymail
    ds = load_hf_split("cnn_dailymail",
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/tensorrt_edgellm/quantization/datasets/__init__.py", line 238, in load_hf_split
    return load_dataset(hf_id, config, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/datasets/load.py", line 1688, in load_dataset
    builder_instance = load_dataset_builder(
                    ^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/datasets/load.py", line 1315, in load_dataset_builder
    dataset_module = dataset_module_factory(
                    ^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/datasets/load.py", line 1207, in dataset_module_factory
    raise e1 from None
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/datasets/load.py", line 1182, in dataset_module_factory
    ).get_module()
    ^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/datasets/load.py", line 598, in get_module
    standalone_yaml_path = cached_path(
                        ^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/datasets/utils/file_utils.py", line 180, in cached_path
    ).resolve_path(url_or_filename)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/huggingface_hub/hf_file_system.py", line 305, in resolve_path
    parsed = parse_hf_uri(f"{constants.HF_PROTOCOL}{path}")
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/huggingface_hub/utils/_hf_uris.py", line 319, in parse_hf_uri
    return _parse_repo_body(location, type_, raw=raw)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/huggingface_hub/utils/_hf_uris.py", line 617, in _parse_repo_body
    raise HfUriError(uri=raw, msg=f"Repository id must be 'namespace/name', got '{repo_id}'.")
huggingface_hub.errors.HfUriError: Invalid HF URI 'hf://datasets/cnn_dailymail@96df5e686bee6baa90b8bee7c28b81fa3fa6223d/.huggingface.yaml'. Repository id must be 'namespace/name', got 'cnn_dailymail'.
(host_venv) root@d1cdc56c1c93:/workspace/tensorrt-edgellm-workspace/LLM# tensorrt-edgellm-quantize llm --model_dir Qwen/Qwen3-0.6B --output_dir Qwen3-0.6B-My-fp8-fp16 --quantization fp8 --text_dataset abisee/cnn_dailymail
/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/modelopt/torch/__init__.py:53: UserWarning: transformers>=5.0 support is experimental. Unified Hugging Face checkpoint export for quantized checkpoints may not work for some models yet.
_warnings.warn(
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 311/311 [00:00<00:00, 3688.66it/s]
Traceback (most recent call last):
File "/workspace/TensorRT-Edge-LLM/host_venv/bin/tensorrt-edgellm-quantize", line 8, in <module>
    sys.exit(main())
            ^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/tensorrt_edgellm/scripts/quantize.py", line 171, in main
    quantize_and_export(
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/tensorrt_edgellm/quantization/quantize.py", line 802, in quantize_and_export
    text_ds = resolve_dataset(text_dataset, "text")
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/tensorrt_edgellm/quantization/datasets/__init__.py", line 152, in resolve_dataset
    return get_dataset(modality, value)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/TensorRT-Edge-LLM/host_venv/lib/python3.12/site-packages/tensorrt_edgellm/quantization/datasets/__init__.py", line 133, in get_dataset
    raise UnknownCalibrationDataset(modality, name,
tensorrt_edgellm.quantization.datasets.UnknownCalibrationDataset: Unknown text calibration dataset 'abisee/cnn_dailymail'. Registered text datasets: cnn_dailymail, wikitext. To calibrate on a custom dataset, write a text generator and register it with @register('text', ...). See the customization guide (docs/source/developer_guide/customization/calibration-datasets.md) and copy the template at tensorrt_edgellm/quantization/datasets/custom_dataset_template.py.

System information (Nvidia Jetson Thor)

  • Container used: nvcr.io/nvidia/pytorch:25.12-py3
  • OS: Ubuntu 24.04.3 LTS
  • CPU architecture: aarch64
  • GPU name: Nvidia Jetson T5000
  • GPU memory size: 128 GB
  • Number of GPUs: 1
  • Library versions:
    • Python: 3.12.3
    • TensorRT Edge-LLM version or commit hash: 0.9.1
    • CUDA: 13.1
    • PyTorch: 2.12.0+cu130
    • Transformers: 5.9.0
    • ModelOpt: 0.44.0
    • ONNX: 1.19.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tensorrt_edgellm/quantization/datasets/builtin.py and datasets/init.py, then trace the call from quantization/quantize.py:805. Re-run the documented tensorrt-edgellm-quantize command with both cnn_dailymail and abisee/cnn_dailymail. Done means the documented dataset values load successfully for text calibration without the shown repository or registered-dataset errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.