microsoft / microsoft/ark

[Bug]Ark0.4.1 multi_gou_tutorial.py run error

Open
#207 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
133
Forks
21
PR merge metrics
No merged PRs in 30d

Description

Describe the bug
ark0.4.1: run multi_gou_tutorial.py fail in sched_default.cc
line393 in configure_gpu_buf, tensor.cc line246 in update_pads, errors as follow:
invalid padding detected. This is likely caused because one GPU buffer is used by multiple operators that require different padding. A possible workaround is to let each operator use a different buffer by creating a new tensor rather than overwriting an existing tensor op name:send.

To Reproduce
run multi_gou_tutorial.py in ark0.4.1

Expected behavior

  1. explain why has the error;
  2. what relationship "ldims, type_bytes, tile" between ref_tensor and this_tensor satisfy in updae_pads?

System (please complete the following information):

  • ark0.4.1
  • OS: [e.g. Ubuntu18.04]
  • GPU [A100]
  • Networking Environment [Single-node, Multi-gpu]

Additional context
Add any other context about the problem here.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run multi_gou_tutorial.py on the reported Ark 0.4.1 setup and inspect the failure in sched_default.cc line 393 and tensor.cc line 246. Trace configure_gpu_buf and update_pads to determine why the padding is invalid and document the required relationship among ldims, type_bytes, and tile for ref_tensor and this_tensor.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
ai-infra-agents
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.