Unstructured-IO / Unstructured-IO/unstructured

bug/parse chinese document ( tranditional chinese ) with attachment

Open
#4,119 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
HTML
Stars
15.5k
Forks
1.3k
Avg merge
4d 2h
Merged PRs (30d)
13

Description

govform02.pdf

result_from_official_unstructured.json

result_from_testing_server_installed_unstructured.json

  1. first element , official give type : Title, testing server , give UncategorizedText
  2. testing server , language , said : kor, it should be chinese , lang detect problem

additional info : about the testing server config

self.extraction_config = {
            'strategy'                      : 'hi_res',                 # 高精度策略
            'model_name'                    : "detectron2_onnx",        # 佈局檢測模型:yolox, detectron2_onnx
            
            'chunking_strategy'             : 'by_title',
            
            
            # 添加 OCR 支援(如果 PDF 是掃描件)
            'ocr_languages'                 : 'chi_tra',
            'languages'                     : ['chi_tra', 'chi_sim', 'eng'],
            
            # 提取圖片配置
            'extract_images_in_pdf'         : True,
            'extract_image_block_types'     : ["Image", "Table"],
            
            # 重要:添加這些參數來改善標題識別
            'pdf_infer_table_structure'     : True,
            'infer_table_structure'         : True,         # 推斷表格結構
            'include_page_breaks'           : False,
            'include_metadata'              : True,
            
            # 調整分塊參數
            'max_characters'                : 4000,
            'new_after_n_chars'             : 3800,
            'combine_text_under_n_chars'    : 500,
            
            #'pdf_image_dpi'                 : 300
            
            # 其它
            #'extract_image_block_to_payload'    : False,  # 不將圖片存入 payload(節省空間)
        }

** file source, from public online , google , just for testing

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing govform02.pdf with result_from_official_unstructured.json and result_from_testing_server_installed_unstructured.json, then review the supplied extraction_config, especially the hi_res strategy, detectron2_onnx model, OCR languages, and language list. Done means the testing-server output agrees on the first element's Title type and identifies the document as Chinese.

Written by the indexing model from the issue text.

Assessment

Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.