danhper / danhper/suplearn-clone-detection
InvalidArgumentError while training the model
- Dominant language
- Python
- Stars
- 18
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
Hi @danhper
I am trying to reproduce the project, and this is what I have done so far
1)Generated java and python vocab using below commands (from bigtool)
I have used the ast files given [here](https://www.csg.ci.i.u-tokyo.ac.jp/projects/clone/)
`docker-bigcode bigcode-ast-tools generate-vocabulary -s 10000 --include-types workspace/java-asts.json -o workspace/java-vocab.tsv`
`docker-bigcode bigcode-ast-tools generate-vocabulary -s 10000 --include-types workspace/python-asts.json -o workspace/python-vocab.tsv`
2)Modified the config file in super-clone-detection to point to the generated data files
3)Generated the dataset
`./bin/suplearn-clone generate-dataset -c config.yml
`
4) Train the model
`./bin/suplearn-clone train -c /path/to/config.yml
`
But after reaching 202 steps out of 4467 in the first epoch, the model throws the error as below:
InvalidArgumentError (see above for traceback): indices[112,180] = 9998 is not in [0, 9994)
[[node encoder_java_1/embedding_java_1/GatherV2 (defined at /opt/anaconda3/envs/msr_project/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:1193) ]]
I have attached my config and error log file( line no 121 to 163)
[config.txt](https://github.com/danhper/suplearn-clone-detection/files/5883528/config.txt)
[training_error.txt](https://github.com/danhper/suplearn-clone-detection/files/5883529/training_error.txt)
Contributor guide
No contributing guide indexed for this repository
Research direction
Review config.txt alongside training_error.txt lines 121–163, then reproduce the failure with ./bin/suplearn-clone train -c /path/to/config.yml after generating the dataset with ./bin/suplearn-clone generate-dataset -c config.yml. Trace the reported embedding lookup during the first epoch and consider the issue done when training completes without the InvalidArgumentError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python, tensorflow
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100