deepseek-ai / deepseek-ai/DeepSeek-Coder

使用fim后human eval分数很低?

Open
#84 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
24.3k
Forks
2.9k
PR merge metrics
No merged PRs in 30d

Description

不使用fim,跑出来的分数跟官方给的比较接近,但是使用fim后分数很低,低了8个点,以下是我使用的fim格式,请问有啥问题吗?

```
def get_full_prompt(model_name, prompt_prefix, prompt, suffix="\n"):
"""
get_full_prompt 通过匹配模型名,使用不同的FIM格式来准备prompt
"""
output = prompt
if "star-coder" in model_name.lower():
...
elif "deep-seek" in model_name.lower() or "deepseek" in model_name.lower():
template = Template("<|fim▁begin|>${preprefix}${prefix}<|fim▁hole|>${suffix}<|fim▁end|>")
output = template.substitute(preprefix=prompt_prefix, prefix=prompt, suffix=suffix)
else:
if prompt_prefix:
output = prompt_prefix + output
return output
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.