deepspeedai / deepspeedai/DeepSpeed

[BUG] Multiple missing zero-guards cause ZeroDivisionError / non-finite values across DeepSpeed (4 locations)

Open
#7,838 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

Describe the bug
I found 4 related zero-handling issues where divisors/scales are not validated before use.
Three can raise ZeroDivisionError; one can silently propagate inf/nan.

Affected locations

  1. deepspeed/utils/groups.py
    _ensure_divisibility(numerator, denominator) uses numerator % denominator without guarding denominator == 0.

  2. deepspeed/utils/timer.py
    ThroughputTimer._is_report_boundary() checks None but not 0 before:
    self.global_step_count % self.steps_per_output.

  3. deepspeed/inference/v2/inference_utils.py
    ceil_div(a, b) returns -(-a // b) without guarding b == 0.

  4. op_builder/hpu/fp_quantizer.py
    FPQuantizer.dequantize() computes (1.0 / scale) without guarding zero elements in scale, which can produce non-finite values and corrupt outputs silently.

To Reproduce

  1. _ensure_divisibility(8, 0) -> ZeroDivisionError
  2. _is_report_boundary() with steps_per_output=0 -> ZeroDivisionError
  3. ceil_div(10, 0) -> ZeroDivisionError
  4. 1.0 / torch.tensor([0.0, 1.0]) -> tensor([inf, 1.]) (same pattern used in HPU dequantize path)

Expected behavior

  • Explicit validation for invalid zero values.
  • Clear user-facing error messages (or safe clamping where appropriate).
  • No raw modulo/division-by-zero exceptions.
  • No silent non-finite propagation in dequantization.

Suggested fixes

  1. _ensure_divisibility: add guard before modulo (denominator != 0).
  2. _is_report_boundary: treat 0 as invalid/disabled (if not self.steps_per_output: or explicit <= 0 validation).
  3. ceil_div: reject b == 0 with clear error.
  4. FPQuantizer.dequantize: clamp scale to torch.finfo(scale.dtype).tiny (or validate and fail clearly) before inversion.

System info

  • Can provide full ds_report output if needed.
  • Repros above are minimal and mostly pure-Python, except item 4 which is on the HPU backend path.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the four affected functions in deepspeed/utils/groups.py, deepspeed/utils/timer.py, deepspeed/inference/v2/inference_utils.py, and op_builder/hpu/fp_quantizer.py. Run the four minimal reproductions from the issue, then determine the intended validation or safe handling for each case. Done means zero inputs produce clear behavior without raw division errors or silent non-finite dequantized values.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.