Lookahead model only recognizes words from word list

I was able to get the lookahead model working, but it seems to recognize only words from the wordlist, for example, lets say I pass "one two three" and the person speaks my name is alpha and I can count to one two three, it produces wrong results because the words prior to one two three are also recognized as one two three. I wanted to implement a phrase hint system for example, perform normal asr but put more weight on words passed through a list.