microsoft / microsoft/TRELLIS.2

shapes not match

Open
#64 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
11.3k
Forks
1.4k
No merged PRs in 30d

Description

when i generate, it reports like this:
Traceback (most recent call last):
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/gradio/queueing.py", line 763, in process_events
response = await route_utils.call_process_api(
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/gradio/route_utils.py", line 354, in call_process_api
output = await app.get_blocks().process_api(
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/gradio/blocks.py", line 2106, in process_api
result = await self.call_function(
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/gradio/blocks.py", line 1588, in call_function
prediction = await anyio.to_thread.run_sync( # type: ignore
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/anyio/to_thread.py", line 61, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2525, in run_sync_in_worker_thread
return await future
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 986, in run
result = context.run(func, *args)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/gradio/utils.py", line 1048, in wrapper
response = f(*args, **kwargs)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/gradio/utils.py", line 1048, in wrapper
response = f(*args, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/app.py", line 423, in image_to_3d
outputs, latents = pipeline.run(
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/pipelines/trellis2_image_to_3d.py", line 542, in run
coords = self.sample_sparse_structure(
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/pipelines/trellis2_image_to_3d.py", line 212, in sample_sparse_structure
z_s = self.sparse_structure_sampler.sample(
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/pipelines/samplers/flow_euler.py", line 208, in sample
return super().sample(model, noise, cond, steps, rescale_t, verbose, neg_cond=neg_cond, guidance_strength=guidance_strength, guidance_interval=guidance_interval, **kwargs)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/pipelines/samplers/flow_euler.py", line 121, in sample
out = self.sample_once(model, sample, t, t_prev, cond, **kwargs)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/pipelines/samplers/flow_euler.py", line 79, in sample_once
pred_x_0, pred_eps, pred_v = self._get_model_prediction(model, x_t, t, cond, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/pipelines/samplers/flow_euler.py", line 49, in _get_model_prediction
pred_v = self._inference_model(model, x_t, t, cond, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/pipelines/samplers/guidance_interval_mixin.py", line 11, in _inference_model
return super()._inference_model(model, x_t, t, cond, guidance_strength=guidance_strength, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/pipelines/samplers/classifier_free_guidance_mixin.py", line 15, in _inference_model
pred_pos = super()._inference_model(model, x_t, t, cond, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/pipelines/samplers/flow_euler.py", line 46, in _inference_model
return model(x_t, t, cond, **kwargs)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/models/sparse_structure_flow.py", line 240, in forward
h = block(h, t_emb, cond, self.rope_phases)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/modules/transformer/modulated.py", line 164, in forward
return self._forward(x, mod, context, phases)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/modules/transformer/modulated.py", line 151, in _forward
h = self.cross_attn(h, context)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/modules/attention/modules.py", line 90, in forward
kv = self.to_kv(context)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 125, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1029x768 and 1024x3072)
Traceback (most recent call last):
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/gradio/queueing.py", line 763, in process_events
response = await route_utils.call_process_api(
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/gradio/route_utils.py", line 354, in call_process_api
output = await app.get_blocks().process_api(
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/gradio/blocks.py", line 2106, in process_api
result = await self.call_function(
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/gradio/blocks.py", line 1588, in call_function
prediction = await anyio.to_thread.run_sync( # type: ignore
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/anyio/to_thread.py", line 61, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2525, in run_sync_in_worker_thread
return await future
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 986, in run
result = context.run(func, *args)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/gradio/utils.py", line 1048, in wrapper
response = f(*args, **kwargs)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/gradio/utils.py", line 1048, in wrapper
response = f(*args, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/app.py", line 423, in image_to_3d
outputs, latents = pipeline.run(
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/pipelines/trellis2_image_to_3d.py", line 542, in run
coords = self.sample_sparse_structure(
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/pipelines/trellis2_image_to_3d.py", line 212, in sample_sparse_structure
z_s = self.sparse_structure_sampler.sample(
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/pipelines/samplers/flow_euler.py", line 208, in sample
return super().sample(model, noise, cond, steps, rescale_t, verbose, neg_cond=neg_cond, guidance_strength=guidance_strength, guidance_interval=guidance_interval, **kwargs)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/pipelines/samplers/flow_euler.py", line 121, in sample
out = self.sample_once(model, sample, t, t_prev, cond, **kwargs)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/pipelines/samplers/flow_euler.py", line 79, in sample_once
pred_x_0, pred_eps, pred_v = self._get_model_prediction(model, x_t, t, cond, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/pipelines/samplers/flow_euler.py", line 49, in _get_model_prediction
pred_v = self._inference_model(model, x_t, t, cond, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/pipelines/samplers/guidance_interval_mixin.py", line 11, in _inference_model
return super()._inference_model(model, x_t, t, cond, guidance_strength=guidance_strength, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/pipelines/samplers/classifier_free_guidance_mixin.py", line 15, in _inference_model
pred_pos = super()._inference_model(model, x_t, t, cond, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/pipelines/samplers/flow_euler.py", line 46, in _inference_model
return model(x_t, t, cond, **kwargs)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/models/sparse_structure_flow.py", line 240, in forward
h = block(h, t_emb, cond, self.rope_phases)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/modules/transformer/modulated.py", line 164, in forward
return self._forward(x, mod, context, phases)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/modules/transformer/modulated.py", line 151, in _forward
h = self.cross_attn(h, context)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/AIGC_Group/3d/TRELLIS.2-main/trellis2/modules/attention/modules.py", line 90, in forward
kv = self.to_kv(context)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/AIGC_Group/miniconda3/envs/trellis2/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 125, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1029x768 and 1024x3072)

Is this error caused by the fact that I am using dinov3-base, which has a hidden layer size of 768 instead of 3072? Do I need to switch to the dinov3-large model? The AI suggested I download dinov3-vitl14, but I haven't been able to find that model. I am currently using the dinov3-vitb16-pretrain-lvd1689m model.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at app.py:image_to_3d and trace the call into trellis2/pipelines/trellis2_image_to_3d.py. Inspect the conditioning dimensions through models/sparse_structure_flow.py, modules/transformer/modulated.py, and modules/attention/modules.py, then reproduce the generation failure. Done means image generation completes without the reported matrix-shape error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.