danhper / danhper/suplearn-clone-detection

InvalidArgumentError while training the model

Open
#8 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
18
Forks
7
PR merge metrics
No merged PRs in 30d

Description

Hi @danhper

I am trying to reproduce the project, and this is what I have done so far
1)Generated java and python vocab using below commands (from bigtool)
I have used the ast files given [here](https://www.csg.ci.i.u-tokyo.ac.jp/projects/clone/)
`docker-bigcode bigcode-ast-tools generate-vocabulary -s 10000 --include-types workspace/java-asts.json -o workspace/java-vocab.tsv`
`docker-bigcode bigcode-ast-tools generate-vocabulary -s 10000 --include-types workspace/python-asts.json -o workspace/python-vocab.tsv`

2)Modified the config file in super-clone-detection to point to the generated data files
3)Generated the dataset
`./bin/suplearn-clone generate-dataset -c config.yml
`
4) Train the model
`./bin/suplearn-clone train -c /path/to/config.yml
`
But after reaching 202 steps out of 4467 in the first epoch, the model throws the error as below:

InvalidArgumentError (see above for traceback): indices[112,180] = 9998 is not in [0, 9994)
[[node encoder_java_1/embedding_java_1/GatherV2 (defined at /opt/anaconda3/envs/msr_project/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:1193) ]]

I have attached my config and error log file( line no 121 to 163)
[config.txt](https://github.com/danhper/suplearn-clone-detection/files/5883528/config.txt)
[training_error.txt](https://github.com/danhper/suplearn-clone-detection/files/5883529/training_error.txt)

Contributor guide

No contributing guide indexed for this repository

Research direction

Review config.txt alongside training_error.txt lines 121–163, then reproduce the failure with ./bin/suplearn-clone train -c /path/to/config.yml after generating the dataset with ./bin/suplearn-clone generate-dataset -c config.yml. Trace the reported embedding lookup during the first epoch and consider the issue done when training completes without the InvalidArgumentError.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python, tensorflow
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.