awslabs / awslabs/ratex

torch.arange free var issue

Open
#44 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
23
Forks
13
PR merge metrics
No merged PRs in 30d

Description

I noticed while compiling model that I was getting RAF free var. Using the IR_DUMP Python frames, we were able to trace the free var to be caused be torch.arange op.
In my bert_layer.py when I create position_ids for embedding like this:
```
position_ids = torch.arange(
seq_length, dtype=torch.long, device=
input_ids.device)
```
I get free var issue. However, this is simply resolved by doing this instead:
```
position_ids = torch.arange(
seq_length, dtype=torch.long
).to(input_ids.device)

```
I think the solution to this might be to implemented arange op & provide lowering to raf.arange in ratex. But I did not confirm as arange op does not show in LTC IR. Regardless, in case anyone sees similar issue, this would be a good reference.

Contributor guide

Open the contributing guide

Research direction

Reproduce the BERT position_ids case from the issue, comparing torch.arange with device placement against the workaround that calls .to(input_ids.device). Use the IR_DUMP Python frames and inspect the LTC IR to determine how arange is represented; done means the direct torch.arange form compiles without the RAF free-variable issue and has the appropriate ratex lowering.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
compilers, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.