GCG evaluation error
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 55
- PR merge metrics
- No merged PRs in 30d
Description
When running the following command:
NPROC_PER_NODE=4 xtuner gcg_omg_seg_llava \
${PATH_TO_CONFIG} \
${PATH_TO_PTH} \
--output-name gcg_pred
I encountered this error:
Traceback (most recent call last):
File "/root/OMG-Seg/omg_llava/omg_llava/tools/gcg_omg_seg_llava.py", line 484, in <module>
main()
File "/root/OMG-Seg/omg_llava/omg_llava/tools/gcg_omg_seg_llava.py", line 295, in main
predict, seg_hidden_states = forward_model(
File "/root/OMG-Seg/omg_llava/omg_llava/tools/gcg_omg_seg_llava.py", line 370, in forward_model
generate_output = llm.generate(
File "/root/miniconda3/envs/omg-llava/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/root/miniconda3/envs/omg-llava/lib/python3.10/site-packages/transformers/generation/utils.py", line 1718, in generate
return self.greedy_search(
File "/root/miniconda3/envs/omg-llava/lib/python3.10/site-packages/transformers/generation/utils.py", line 2579, in greedy_search
outputs = self(
File "/root/miniconda3/envs/omg-llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/envs/omg-llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/omg-llava/lib/python3.10/site-packages/accelerate/hooks.py", line 169, in new_forward
output = module._old_forward(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/internlm2-chat-7b/modeling_internlm2.py", line 1060, in forward
logits = self.output(hidden_states)
File "/root/miniconda3/envs/omg-llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/envs/omg-llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/omg-llava/lib/python3.10/site-packages/accelerate/hooks.py", line 169, in new_forward
output = module._old_forward(*args, **kwargs)
File "/root/miniconda3/envs/omg-llava/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::Half
My conda environment:
# packages in environment at /root/miniconda3/envs/omg-llava:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
_openmp_mutex 5.1 1_gnu https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
accelerate 0.33.0 pypi_0 pypi
addict 2.4.0 pypi_0 pypi
aiofiles 23.2.1 pypi_0 pypi
aiohappyeyeballs 2.4.0 pypi_0 pypi
aiohttp 3.10.5 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
altair 5.4.0 pypi_0 pypi
annotated-types 0.7.0 pypi_0 pypi
anyio 4.4.0 pypi_0 pypi
argon2-cffi 23.1.0 pypi_0 pypi
argon2-cffi-bindings 21.2.0 pypi_0 pypi
arrow 1.3.0 pypi_0 pypi
arxiv 2.1.3 pypi_0 pypi
asttokens 2.4.1 pypi_0 pypi
async-lru 2.0.4 pypi_0 pypi
async-timeout 4.0.3 pypi_0 pypi
attrs 24.2.0 pypi_0 pypi
babel 2.16.0 pypi_0 pypi
beautifulsoup4 4.12.3 pypi_0 pypi
bitsandbytes 0.43.3 pypi_0 pypi
bleach 6.1.0 pypi_0 pypi
blinker 1.8.2 pypi_0 pypi
brotli 1.1.0 pypi_0 pypi
bzip2 1.0.8 h5eee18b_6 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ca-certificates 2024.7.2 h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
cachetools 5.5.0 pypi_0 pypi
certifi 2022.12.7 pypi_0 pypi
cffi 1.17.0 pypi_0 pypi
charset-normalizer 2.1.1 pypi_0 pypi
click 8.1.7 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
comm 0.2.2 pypi_0 pypi
contourpy 1.2.1 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
datasets 2.21.0 pypi_0 pypi
debugpy 1.8.5 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
deepspeed 0.15.0 pypi_0 pypi
defusedxml 0.7.1 pypi_0 pypi
dill 0.3.8 pypi_0 pypi
distro 1.9.0 pypi_0 pypi
duckduckgo-search 5.3.1b1 pypi_0 pypi
einops 0.8.0 pypi_0 pypi
et-xmlfile 1.1.0 pypi_0 pypi
exceptiongroup 1.2.2 pypi_0 pypi
executing 2.0.1 pypi_0 pypi
fastapi 0.112.1 pypi_0 pypi
fastjsonschema 2.20.0 pypi_0 pypi
feedparser 6.0.11 pypi_0 pypi
ffmpy 0.4.0 pypi_0 pypi
filelock 3.13.1 pypi_0 pypi
fonttools 4.53.1 pypi_0 pypi
fqdn 1.5.1 pypi_0 pypi
frozenlist 1.4.1 pypi_0 pypi
fsspec 2024.2.0 pypi_0 pypi
ftfy 6.2.3 pypi_0 pypi
func-timeout 4.3.5 pypi_0 pypi
gitdb 4.0.11 pypi_0 pypi
gitpython 3.1.43 pypi_0 pypi
gradio 4.37.2 pypi_0 pypi
gradio-client 1.0.2 pypi_0 pypi
gradio-image-prompter 0.1.0 pypi_0 pypi
griffe 1.1.1 pypi_0 pypi
h11 0.14.0 pypi_0 pypi
h2 4.1.0 pypi_0 pypi
hjson 3.1.0 pypi_0 pypi
hpack 4.0.0 pypi_0 pypi
httpcore 1.0.5 pypi_0 pypi
httpx 0.27.0 pypi_0 pypi
huggingface-hub 0.24.6 pypi_0 pypi
hyperframe 6.0.1 pypi_0 pypi
idna 3.4 pypi_0 pypi
imageio 2.35.1 pypi_0 pypi
importlib-metadata 8.4.0 pypi_0 pypi
importlib-resources 6.4.4 pypi_0 pypi
ipykernel 6.29.5 pypi_0 pypi
ipython 8.26.0 pypi_0 pypi
ipywidgets 8.1.5 pypi_0 pypi
isoduration 20.11.0 pypi_0 pypi
jedi 0.19.1 pypi_0 pypi
jinja2 3.1.3 pypi_0 pypi
json5 0.9.25 pypi_0 pypi
jsonpointer 3.0.0 pypi_0 pypi
jsonschema 4.23.0 pypi_0 pypi
jsonschema-specifications 2023.12.1 pypi_0 pypi
jupyter 1.0.0 pypi_0 pypi
jupyter-client 8.6.2 pypi_0 pypi
jupyter-console 6.6.3 pypi_0 pypi
jupyter-core 5.7.2 pypi_0 pypi
jupyter-events 0.10.0 pypi_0 pypi
jupyter-lsp 2.2.5 pypi_0 pypi
jupyter-server 2.14.2 pypi_0 pypi
jupyter-server-terminals 0.5.3 pypi_0 pypi
jupyterlab 4.2.4 pypi_0 pypi
jupyterlab-pygments 0.3.0 pypi_0 pypi
jupyterlab-server 2.27.3 pypi_0 pypi
jupyterlab-widgets 3.0.13 pypi_0 pypi
kiwisolver 1.4.5 pypi_0 pypi
kornia 0.7.3 pypi_0 pypi
kornia-rs 0.1.5 pypi_0 pypi
lagent 0.2.3 pypi_0 pypi
lazy-loader 0.4 pypi_0 pypi
ld_impl_linux-64 2.38 h1181459_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libffi 3.4.4 h6a678d5_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libgcc-ng 11.2.0 h1234567_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libgomp 11.2.0 h1234567_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libstdcxx-ng 11.2.0 h1234567_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libuuid 1.41.5 h5eee18b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
markdown 3.7 pypi_0 pypi
markdown-it-py 3.0.0 pypi_0 pypi
markupsafe 2.1.5 pypi_0 pypi
mat4py 0.6.0 pypi_0 pypi
matplotlib 3.9.2 pypi_0 pypi
matplotlib-inline 0.1.7 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
mistune 3.0.2 pypi_0 pypi
mmcv 2.0.1 pypi_0 pypi
mmdet 3.1.0 pypi_0 pypi
mmengine 0.10.4 pypi_0 pypi
mmpretrain 1.0.1 pypi_0 pypi
mmsegmentation 1.1.1 pypi_0 pypi
model-index 0.1.11 pypi_0 pypi
modelindex 0.0.2 pypi_0 pypi
modelscope 1.17.1 pypi_0 pypi
mpi4py-mpich 3.1.5 pypi_0 pypi
mpmath 1.3.0 pypi_0 pypi
multidict 6.0.5 pypi_0 pypi
multiprocess 0.70.16 pypi_0 pypi
narwhals 1.5.3 pypi_0 pypi
nbclient 0.10.0 pypi_0 pypi
nbconvert 7.16.4 pypi_0 pypi
nbformat 5.10.4 pypi_0 pypi
ncurses 6.4 h6a678d5_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
nest-asyncio 1.6.0 pypi_0 pypi
networkx 3.2.1 pypi_0 pypi
ninja 1.11.1.1 pypi_0 pypi
notebook 7.2.1 pypi_0 pypi
notebook-shim 0.2.4 pypi_0 pypi
numpy 1.26.3 pypi_0 pypi
nvidia-ml-py 12.560.30 pypi_0 pypi
opencv-python 4.10.0.84 pypi_0 pypi
openpyxl 3.1.5 pypi_0 pypi
openssl 3.0.14 h5eee18b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ordered-set 4.1.0 pypi_0 pypi
orjson 3.10.7 pypi_0 pypi
overrides 7.7.0 pypi_0 pypi
packaging 24.1 pypi_0 pypi
pandas 2.2.2 pypi_0 pypi
pandocfilters 1.5.1 pypi_0 pypi
parso 0.8.4 pypi_0 pypi
peft 0.12.0 pypi_0 pypi
pexpect 4.9.0 pypi_0 pypi
phx-class-registry 4.1.0 pypi_0 pypi
pillow 10.2.0 pypi_0 pypi
pip 24.2 py310h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
platformdirs 4.2.2 pypi_0 pypi
prettytable 3.11.0 pypi_0 pypi
prometheus-client 0.20.0 pypi_0 pypi
prompt-toolkit 3.0.47 pypi_0 pypi
protobuf 5.27.3 pypi_0 pypi
psutil 6.0.0 pypi_0 pypi
ptyprocess 0.7.0 pypi_0 pypi
pure-eval 0.2.3 pypi_0 pypi
py-cpuinfo 9.0.0 pypi_0 pypi
pyarrow 17.0.0 pypi_0 pypi
pycocotools 2.0.8 pypi_0 pypi
pycparser 2.22 pypi_0 pypi
pydantic 2.8.2 pypi_0 pypi
pydantic-core 2.20.1 pypi_0 pypi
pydeck 0.9.1 pypi_0 pypi
pydub 0.25.1 pypi_0 pypi
pygments 2.18.0 pypi_0 pypi
pyparsing 3.1.2 pypi_0 pypi
python 3.10.14 h955ad1f_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
python-dateutil 2.9.0.post0 pypi_0 pypi
python-json-logger 2.0.7 pypi_0 pypi
python-multipart 0.0.9 pypi_0 pypi
pytz 2024.1 pypi_0 pypi
pyyaml 6.0.2 pypi_0 pypi
pyzmq 26.2.0 pypi_0 pypi
qtconsole 5.5.2 pypi_0 pypi
qtpy 2.4.1 pypi_0 pypi
readline 8.2 h5eee18b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
referencing 0.35.1 pypi_0 pypi
regex 2024.7.24 pypi_0 pypi
requests 2.32.3 pypi_0 pypi
rfc3339-validator 0.1.4 pypi_0 pypi
rfc3986-validator 0.1.1 pypi_0 pypi
rich 13.7.1 pypi_0 pypi
rpds-py 0.20.0 pypi_0 pypi
ruff 0.6.2 pypi_0 pypi
safetensors 0.4.4 pypi_0 pypi
scikit-image 0.24.0 pypi_0 pypi
scipy 1.14.1 pypi_0 pypi
semantic-version 2.10.0 pypi_0 pypi
send2trash 1.8.3 pypi_0 pypi
sentencepiece 0.2.0 pypi_0 pypi
setuptools 72.1.0 py310h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
sgmllib3k 1.0.0 pypi_0 pypi
shapely 2.0.6 pypi_0 pypi
shellingham 1.5.4 pypi_0 pypi
six 1.16.0 pypi_0 pypi
smmap 5.0.1 pypi_0 pypi
sniffio 1.3.1 pypi_0 pypi
socksio 1.0.0 pypi_0 pypi
soupsieve 2.6 pypi_0 pypi
sqlite 3.45.3 h5eee18b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
stack-data 0.6.3 pypi_0 pypi
starlette 0.38.2 pypi_0 pypi
streamlit 1.37.1 pypi_0 pypi
sympy 1.12 pypi_0 pypi
tenacity 8.5.0 pypi_0 pypi
termcolor 2.4.0 pypi_0 pypi
terminado 0.18.1 pypi_0 pypi
terminaltables 3.1.10 pypi_0 pypi
tifffile 2024.8.10 pypi_0 pypi
tiktoken 0.7.0 pypi_0 pypi
timeout-decorator 0.5.0 pypi_0 pypi
timm 1.0.8 pypi_0 pypi
tinycss2 1.3.0 pypi_0 pypi
tk 8.6.14 h39e8969_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
tokenizers 0.15.2 pypi_0 pypi
toml 0.10.2 pypi_0 pypi
tomli 2.0.1 pypi_0 pypi
tomlkit 0.12.0 pypi_0 pypi
torch 2.1.2+cu118 pypi_0 pypi
torchaudio 2.1.2+cu118 pypi_0 pypi
torchvision 0.16.2+cu118 pypi_0 pypi
tornado 6.4.1 pypi_0 pypi
tqdm 4.66.5 pypi_0 pypi
traitlets 5.14.3 pypi_0 pypi
transformers 4.36.0 pypi_0 pypi
transformers-stream-generator 0.0.5 pypi_0 pypi
triton 2.1.0 pypi_0 pypi
typer 0.12.4 pypi_0 pypi
types-python-dateutil 2.9.0.20240821 pypi_0 pypi
typing-extensions 4.12.2 pypi_0 pypi
tzdata 2024.1 pypi_0 pypi
uri-template 1.3.0 pypi_0 pypi
urllib3 2.2.2 pypi_0 pypi
uvicorn 0.30.6 pypi_0 pypi
watchdog 4.0.2 pypi_0 pypi
wcwidth 0.2.13 pypi_0 pypi
webcolors 24.8.0 pypi_0 pypi
webencodings 0.5.1 pypi_0 pypi
websocket-client 1.8.0 pypi_0 pypi
websockets 11.0.3 pypi_0 pypi
wheel 0.43.0 py310h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
widgetsnbextension 4.0.13 pypi_0 pypi
xtuner 0.1.21 dev_0 <develop>
xxhash 3.5.0 pypi_0 pypi
xz 5.4.6 h5eee18b_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
yapf 0.40.2 pypi_0 pypi
yarl 1.9.4 pypi_0 pypi
zipp 3.20.0 pypi_0 pypi
zlib 1.2.13 h5eee18b_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with omg_llava/tools/gcg_omg_seg_llava.py, especially main() around line 295 and forward_model() around line 370, then reproduce the documented xtuner gcg_omg_seg_llava command. Trace the tensor and model dtypes reaching llm.generate and the output layer. Done means GCG evaluation completes without the reported dtype mismatch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100