baidu / baidu/lac

包含正则表达式匹配的简单实现 Greedy implementation of lac with regex

Open
#187 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
4k
Forks
592
PR merge metrics
No merged PRs in 30d

Description

I use your tools on my project. Since I need regex to assist with my tokenization, I come up with a way for solving this.
Noticing that it might still need some time since "后续还将开放以通配符配置词典的模式", I think I could help people with similar needs like me.

Pseudocode:

import re
def get_regex_lac(sent, lac, regex_patterns, tags),:
    """
    :param sent:  input sentence
    :param lac: LAC object
    :param lac: A list of regex patterns
    :param tags: A list of corresponding tags
    :return: just as lac.run(sent)
    """
    for regex_pattern, tag in zip(regex_patterns, tags):
        sent_remained = sent                                            # process remained sentence from left to right
        search_result = re.search(regex_pattern, sent_remained)

        while search_result:
            l_index, r_index = search_result.span()
            new_word = sent_remained[l_index:r_index]

            if new_word not in lac.custom.dictitem:
                lac.custom.dictitem[new_word] = ([tag], [len(new_word)])     # add new word to lac inner data structure
                lac.custom.ac.add_word(new_word)

            sent_remained  = sent_remained[r_index:]
            search_result = re.search(regex_pattern, sent_remained)

    return lac.run(sent)

I use it in my project. And it works fine. If you think this implementation is OK and would like to assign me as a committer, I could assist to insert the function into this project.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no repository files or tests. Start by reviewing the existing lac.run entry point and the lac.custom.dictitem and lac.custom.ac structures referenced in the pseudocode; clarify how regex patterns and tags should be exposed before defining what an integrated implementation and its tests must cover.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.