sillsdev / sillsdev/serval

Warning! phrase-based model vocabularies are now different from lexical model vocabularies.

Open
#503 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
3
Forks
2
Avg merge
3d 1h
Merged PRs (30d)
3

Description

It appears to happen when there are a lot of get word graphs and a few trains:

2024-10-01 23:43:30	
{"log":"Warning! phrase-based model vocabularies are now different from lexical model vocabularies.\n","stream":"stderr","time":"2024-10-02T03:43:30.677122153Z"}
2024-10-01 23:43:30	
{"log":"Warning! phrase-based model vocabularies are now different from lexical model vocabularies.\n","stream":"stderr","time":"2024-10-02T03:43:30.677113643Z"}
2024-10-01 23:43:30	
{"log":"Warning! phrase-based model vocabularies are now different from lexical model vocabularies.\n","stream":"stderr","time":"2024-10-02T03:43:30.677064168Z"}
2024-10-01 23:42:48	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:42:46.065731712Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:42:46	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:42:26.782587856Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:42:27	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:42:08.251130903Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:42:09	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:42:07.784591941Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:42:08	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:42:04.68842351Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:42:05	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:42:04.441732675Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:42:05	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:41:48.257841234Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:41:49	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:41:40.446429763Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:41:41	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:41:30.164171281Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:41:30	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7\n","stream":"stdout","time":"2024-10-02T03:40:32.77032131Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:40:33	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:39:39.581977747Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:39:40	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7\n","stream":"stdout","time":"2024-10-02T03:38:57.585376561Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:38:58	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7\n","stream":"stdout","time":"2024-10-02T03:38:46.285474743Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:38:47	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Trained segment pair for engine 66bcbbd6ec21e6eb9f9ae9e7\n","stream":"stdout","time":"2024-10-02T03:38:45.443995883Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:38:46	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7\n","stream":"stdout","time":"2024-10-02T03:38:23.597854677Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:38:45	
{"log":"Warning! phrase-based model vocabularies are now different from lexical model vocabularies.\n","stream":"stderr","time":"2024-10-02T03:38:22.645678887Z"}
2024-10-01 23:38:24	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Trained segment pair for engine 66bcbbd6ec21e6eb9f9ae9e7\n","stream":"stdout","time":"2024-10-02T03:38:22.673247098Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:38:23	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:38:10.071637977Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:38:23	
{"log":"info: Serval.Machine.Shared.Services.ClearMLAuthenticationService[0]\n      ClearML Authentication Token Refresh Successful.\n","stream":"stdout","time":"2024-10-02T03:38:01.354824092Z","type":"info","source":"Serval.Machine.Shared.Services.ClearMLAuthenticationService"}
2024-10-01 23:38:22	
{"log":"Warning! phrase-based model vocabularies are now different from lexical model vocabularies.\n","stream":"stderr","time":"2024-10-02T03:38:22.645633643Z"}
2024-10-01 23:38:10	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:37:56.767955145Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:38:01	
{"log":"Warning! phrase-based model vocabularies are now different from lexical model vocabularies.\n","stream":"stderr","time":"2024-10-02T03:37:06.37522718Z"}
2024-10-01 23:37:57	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:37:54.862996674Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:37:55	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7\n","stream":"stdout","time":"2024-10-02T03:37:50.677922355Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:37:51	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7\n","stream":"stdout","time":"2024-10-02T03:37:42.061922543Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:37:42	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7\n","stream":"stdout","time":"2024-10-02T03:37:32.483666598Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the Serval.Translation.Controllers.TranslationEnginesController entries for “Got word graph” and “Trained segment pair,” using the supplied logs to reproduce the warning during repeated word-graph requests and training. Trace why phrase-based and lexical model vocabularies diverge; done means the warning no longer occurs in this workflow without breaking those operations.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.