Define tokenizer strategy

## Summary
Define how conventional commit messages will be tokenized for both n-gram lookup and neural fallback.

## Success Criteria
- [ ] Tokenization strategy chosen (BPE, WordPiece, character, or hybrid)
- [ ] Special tokens defined (, , , etc.)
- [ ] Vocabulary size determined
- [ ] Tokenizer can be trained on our dataset
- [ ] Roundtrip test: tokenize → detokenize preserves meaning

## Context
The tokenizer must work well for both:
1. **N-gram lookup**: Needs discrete, interpretable tokens
2. **Neural fallback**: Needs subword handling for rare terms

## Considerations
- Conventional commits have structured grammar (type(scope): subject)
- Technical terms may need subword tokenization
- Should we use different tokenizers for different components?