Robbyant / Robbyant/lingbot-map

compile problem

Open
#57 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
17.1k
Forks
1.9k
No merged PRs in 30d

Description

During the acceleration process, everything worked fine when I ran python gct_profile.py --backend flashinfer --dtype bf16 --compile, but I encountered an error when I ran python demo.py --compile .... I look forward to your reply.

(lingbot-map) gabriel@ubuntu-2:~/code/lingbot-map$ python gct_profile.py --backend flashinfer --dtype bf16 --compile
torchtitan not available for ulysses cp
========================================================================
GCTStream FPS profiling  |  378×504  |  500 frames  |  sw=64  |  kf_int=1
  backends=['flashinfer']  dtypes=['bf16']
========================================================================
Generating 500 synthetic images on CUDA...

========================================================================
  Run: flashinfer_bf16
========================================================================
pretrained_path: 
Failed to load pretrained weights: [Errno 2] No such file or directory: ''
  Warmup eager (scale + 10 streaming)...
  Compiling hot modules...
  Warmup compiled (3× dress rehearsal)...
  Phase 1: 294.8 ms for 8 scale frames

  [flashinfer_bf16]  (500 total frames: 8 scale + 492 streaming)
    ── Global FPS ─────────────────────────────────────
      total time: 49.74 s  (294.8 ms phase1 + 49446.0 ms phase2)
      per frame :  99.48 ms  →   10.05 FPS
    ── Windowed FPS (±30 streaming frames) ────────────
      frame    57 (10%):  86.84 ms  →   11.52 FPS
      frame   254 (50%): 103.71 ms  →    9.64 FPS
      frame   451 (90%): 105.92 ms  →    9.44 FPS
    ── FPS trace (every 100 global frames) ────────────
      frame   100:   7.44 FPS  (134.45 ms)
      frame   200:   9.71 FPS  (103.01 ms)
      frame   300:   9.58 FPS  (104.38 ms)
      frame   400:   9.48 FPS  (105.49 ms)

  Saved to /tmp/profile_results_378x504_500f_bf16.json

(lingbot-map) gabriel@ubuntu-2:~/code/lingbot-map$ python demo.py  --compile   --model_path /data/gabriel/checkpoints/lingbot/lingbot-map.pt     --image_folder /home/gabriel/code/examples/007
Loading 160 images...
Loading images: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
  File "/home/gabriel/code/lingbot-map/demo.py", line 490, in main
    _warm_streaming(model, images, scale_for_warm, warm_stream_n, dtype, passes=3)
  File "/home/gabriel/code/lingbot-map/demo.py", line 194, in _warm_streaming
    model.forward(
  File "/home/gabriel/code/lingbot-map/lingbot_map/models/gct_base.py", line 322, in forward
    aggregated_tokens_list, patch_start_idx = self._aggregate_features(
  File "/home/gabriel/code/lingbot-map/lingbot_map/models/gct_stream.py", line 275, in _aggregate_features
    aggregated_tokens_list, patch_start_idx = self.aggregator(
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/gabriel/code/lingbot-map/lingbot_map/aggregator/base.py", line 589, in forward
    tokens, global_idx, global_intermediates = self._process_global_attention(
  File "/home/gabriel/code/lingbot-map/lingbot_map/aggregator/stream.py", line 409, in _process_global_attention
    return self._process_causal_stream(
  File "/home/gabriel/code/lingbot-map/lingbot_map/aggregator/stream.py", line 509, in _process_causal_stream
    tokens = self.global_blocks[global_idx](
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/gabriel/code/lingbot-map/lingbot_map/layers/block.py", line 289, in forward
    x = x + self.ls1(self.attn(
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/gabriel/code/lingbot-map/lingbot_map/layers/attention.py", line 618, in forward
    x = self.proj(x)
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 414, in call
    return super().call(*args, **kwargs)
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper
    return fn(*args, **kwargs)
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/_dynamo/external_utils.py", line 66, in inner
    @functools.wraps(fn)
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1044, in _fn
    return fn(*args, **kwargs)
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1130, in forward
    return compiled_fn(full_args)
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 353, in runtime_wrapper
    all_outs = call_func_at_runtime_with_args(
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 129, in call_func_at_runtime_with_args
    out = normalize_as_list(f(args))
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 526, in wrapper
    return compiled_fn(runtime_args)
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/_inductor/output_code.py", line 613, in call
    return self.current_callable(inputs)
  File "/tmp/torchinductor_gabriel/cq/ccqupl7majfqt53lshu3iig3cyhpxege376fcogeko3uigbdtvwh.py", line 70, in call
    (buf0,) = self.partitions[0](partition0_args)
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1772, in run
    return compiled_fn(new_inputs)  # type: ignore[arg-type]
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 388, in deferred_cudagraphify
    return fn(inputs)
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/_inductor/utils.py", line 3017, in run
    out = model(new_inputs)
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 2012, in run
    out = self._run(new_inputs, function_id)
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 2179, in _run
    self.apply_checkpoint_execution_state_in_allocator()
  File "/home/gabriel/anaconda3/envs/lingbot-map/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 2549, in apply_checkpoint_execution_state_in_allocator
    torch._C._cuda_setCheckpointPoolState(
RuntimeError: Expected curr_block->next == nullptr to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
[W429 03:15:30.865757981 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator())

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the demo.py --compile command and inspect _warm_streaming in demo.py, then follow the traceback through gct_base.py, aggregator/stream.py, and layers/attention.py. Compare this path with the successful gct_profile.py compile run and determine why the compiled projection reaches the CUDA checkpoint-pool error. Done means the demo command completes with --compile without the reported RuntimeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.