Azure / Azure/Azure-Sentinel-Notebooks

Mitremap-notebook: Error - Gpt2tokenizer object has no attribute _added_tokens_decoder

Open
#242 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
628
Forks
236
PR merge metrics
No merged PRs in 30d

Description

When I run the below mentioned code I get the error "Gpt2tokenizer object has no attribute _added_tokens_decoder"

configs, inference_df, iocs_df = main.go(
config_widgets
)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the error with the shown main.go(config_widgets) call in the Mitremap notebook and inspect how the Gpt2tokenizer object is created. Trace the tokenizer setup to identify the compatibility failure, then verify that the notebook runs past this call without the _added_tokens_decoder error.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.