Unstructured-IO / Unstructured-IO/unstructured

bug/Unstructured always gets stuck

Open
#4,053 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
HTML
Stars
15.5k
Forks
1.3k
Avg merge
4d 2h
Merged PRs (30d)
13

Description

Describe the bug
Unstructured on Debian, no matter what type of file is loaded, it will get stuck when calling the load method. Sometimes there may be an error message 'File is not a zip file'

To Reproduce
`
def load_document_from_file_path(file_path: str, file_type: str) -> List[Document]:
"""根据文件路径和类型加载文档内容"""
try:
if file_type == "docx":
# 步骤1: 首先校验是否是有效的docx格式
is_valid_docx = _is_valid_docx_format(file_path)

        if is_valid_docx:
            # 步骤4: 如果是有效的docx,使用UnstructuredWordDocumentLoader
            try:
                loader = UnstructuredWordDocumentLoader(file_path)
                logger.info(f"使用UnstructuredWordDocumentLoader加载文件: {file_path}")
                documents = loader.load()
                logger.info(f"成功使用UnstructuredWordDocumentLoader加载文件 {file_path}")
                return documents
            except Exception as e:
                logger.warning(f"UnstructuredWordDocumentLoader加载失败: {e},尝试使用python-docx备用方案")
                # 如果UnstructuredWordDocumentLoader失败,降级到python-docx
                return _parse_docx_with_python_docx(file_path)
        else:
            # 步骤2: 如果不是有效格式,使用python-docx解析
            logger.info(f"文件 {file_path} 不是有效的docx格式,尝试使用python-docx解析")
            try:
                return _parse_docx_with_python_docx(file_path)
            except Exception as e:
                # 步骤3: 如果python-docx也无法解析,丢弃文件并发出warning
                warning_msg = f"文件 {file_path} 无法被任何docx解析器处理,跳过该文件。错误: {e}"
                raise DocxLoadError(warning_msg)
            
    elif file_type == "md":
        loader = UnstructuredMarkdownLoader(file_path)
    elif file_type == "txt":
        # 尝试多种编码方式加载txt文件
        encodings = ['utf-8', 'utf-8-sig', 'gbk', 'gb2312', 'big5', 'latin-1']
        last_error = None

        for encoding in encodings:
            try:
                loader = TextLoader(file_path, encoding=encoding)
                # 直接返回加载结果,不重复调用load()
                documents = loader.load()
                logger.info(f"成功使用编码 {encoding} 加载文件 {file_path}")
                return documents
            except Exception as e:
                logger.debug(f"使用编码 {encoding} 加载文件失败: {str(e)}")
                last_error = e
                continue

        # 如果所有编码都失败,抛出最后一个错误
        raise ValueError(f"无法使用任何编码方式加载文件 {file_path},最后错误: {str(last_error)}")
    elif file_type == "html":
        loader = UnstructuredHTMLLoader(file_path)
    elif file_type == "pdf":
        loader = PyPDFLoader(file_path)
    else:
        raise ValueError(f"不支持的文件类型: {file_type}")
        
    # 对于非docx文件类型,正常加载
    tmp = loader.load()
    return tmp
except Exception as e:
    logger.error(f"加载文档失败: {str(e)}")
    raise

`

Expected behavior
After calling the load method in an unstructured manner, it should be able to continue running.

Environment Info
aiofiles==24.1.0
aiohappyeyeballs==2.6.1
aiohttp==3.12.12
aiosignal==1.3.2
alembic==1.14.0
annotated-types==0.7.0
anyio==4.9.0
argon2-cffi==25.1.0
argon2-cffi-bindings==21.2.0
attrs==25.3.0
azure-core==1.35.0
azure-storage-blob==12.25.1
backoff==2.2.1
beautifulsoup4==4.13.4
black==24.10.0
certifi==2025.4.26
cffi==1.17.1
chardet==5.2.0
charset-normalizer==3.4.2
click==8.2.1
colorama==0.4.6
cryptography==42.0.8
dataclasses-json==0.6.7
distro==1.9.0
emoji==2.14.1
environs==9.5.0
fastapi==0.115.12
filetype==1.2.0
flake8==7.1.1
frozenlist==1.7.0
greenlet==3.2.3
grpcio==1.60.0
h11==0.16.0
html5lib==1.1
httpcore==1.0.9
httpx==0.28.1
httpx-sse==0.4.0
idna==3.10
iniconfig==2.1.0
isodate==0.7.2
isort==5.13.2
jiter==0.10.0
joblib==1.5.1
jsonpatch==1.33
jsonpointer==3.0.0
langchain==0.3.25
langchain-community==0.3.25
langchain-core==0.3.65
langchain-text-splitters==0.3.8
langdetect==1.0.9
langsmith==0.3.45
loguru==0.7.2
lxml==6.0.0
Mako==1.3.10
Markdown==3.8
MarkupSafe==3.0.2
marshmallow==3.26.1
mccabe==0.7.0
minio==7.2.15
multidict==6.4.4
mypy_extensions==1.1.0
nacos-sdk-python==1.0.0
nest-asyncio==1.6.0
nltk==3.9.1
numpy==2.3.0
olefile==0.47
openai==1.88.0
orjson==3.10.18
packaging==24.2
pandas==2.2.3
pathspec==0.12.1
platformdirs==4.3.8
pluggy==1.6.0
propcache==0.3.2
protobuf==6.31.1
psutil==7.0.0
pyarrow==20.0.0
pycodestyle==2.12.1
pycparser==2.22
pycryptodome==3.23.0
pydantic==2.11.5
pydantic-settings==2.9.1
pydantic_core==2.33.2
pyflakes==3.2.0
pymilvus==2.4.0
PyMySQL==1.1.0
pypdf==5.7.0
pytest==8.3.4
pytest-asyncio==0.24.0
python-dateutil==2.9.0.post0
python-docx==1.2.0
python-dotenv==1.1.0
python-iso639==2025.2.18
python-magic==0.4.27
python-multipart==0.0.20
python-oxmsg==0.0.2
pytz==2025.2
PyYAML==6.0.2
RapidFuzz==3.13.0
redis==5.2.1
regex==2024.11.6
requests==2.32.4
requests-toolbelt==1.0.0
scipy==1.16.0
six==1.17.0
sniffio==1.3.1
soupsieve==2.7
SQLAlchemy==2.0.41
starlette==0.46.2
tenacity==9.1.2
tiktoken==0.9.0
tqdm==4.67.1
typing-inspect==0.9.0
typing-inspection==0.4.1
typing_extensions==4.14.0
tzdata==2025.2
ujson==5.10.0
unstructured==0.17.2
unstructured-client==0.37.4
urllib3==2.4.0
uvicorn==0.34.3
webencodings==0.5.1
wrapt==1.17.2
yarl==1.20.1
zstandard==0.23.0

Additional context
Add any other context about the problem here.
I am very strange because on CentOS 8 in the testing environment, Unstructured can run successfully:
=== Package Info on localhost.localdomain ===
gcc-8.5.0-4.el8_5.x86_64
Python level package not installed
The software package libXML2 develop is not installed
The software package libxlt develop is not installed
File level package not installed
file-libs-5.33-16.el8_3.1.x86_64
poppler-utils-20.11.0-2.el8.x86_64
The software package Tesseract is not installed
The software package teseract langpack eng is not installed
The software package libreoffice is not installed
The software package Pandoc is not installed
The software package libjpeg level is not installed
zlib-devel-1.2.11-17.el8.x86_64
libxml2-2.9.7-9.el8.x86_64
libxslt-1.1.32-6.el8.x86_64
poppler-utils-20.11.0-2.el8.x86_64
The poppler develop package is not installed
The poppler cpp develop package is not installed

And I checked the dify backend container (debian system) that also uses the unstructured library, and it can still run normally without these libraries:

=== Package Info on ebce3a5d76b7 ===
dpkg-query: no packages found matching gcc
dpkg-query: no packages found matching python3-dev
dpkg-query: no packages found matching libxml2-dev
dpkg-query: no packages found matching libxslt-dev
dpkg-query: no packages found matching libmagic-dev
file unknown ok not-installed
dpkg-query: no packages found matching poppler-utils
dpkg-query: no packages found matching tesseract-ocr
dpkg-query: no packages found matching tesseract-ocr-eng
dpkg-query: no packages found matching libreoffice
dpkg-query: no packages found matching pandoc
dpkg-query: no packages found matching libjpeg-dev
dpkg-query: no packages found matching zlib1g-dev
libxml2 unknown ok not-installed
dpkg-query: no packages found matching libxslt1-dev
dpkg-query: no packages found matching poppler-utils
dpkg-query: no packages found matching libpoppler-dev
dpkg-query: no packages found matching libpoppler-cpp-dev

My production environment's debian container doesn't work:
=== Package Info on 8947af66fce1 ===
dpkg-query: no packages found matching gcc
dpkg-query: no packages found matching python3-dev
dpkg-query: no packages found matching libxml2-dev
dpkg-query: no packages found matching libxslt-dev
dpkg-query: no packages found matching libmagic-dev
dpkg-query: no packages found matching file
dpkg-query: no packages found matching poppler-utils
dpkg-query: no packages found matching tesseract-ocr
dpkg-query: no packages found matching tesseract-ocr-eng
dpkg-query: no packages found matching libreoffice
dpkg-query: no packages found matching pandoc
dpkg-query: no packages found matching libjpeg-dev
dpkg-query: no packages found matching zlib1g-dev
libxml2 unknown ok not-installed
dpkg-query: no packages found matching libxslt1-dev
dpkg-query: no packages found matching poppler-utils
dpkg-query: no packages found matching libpoppler-dev
dpkg-query: no packages found matching libpoppler-cpp-dev

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the load_document_from_file_path function and its UnstructuredWordDocumentLoader, UnstructuredMarkdownLoader, and UnstructuredHTMLLoader calls. Compare the reported Debian production environment with the working containers and reproduce the hang or “File is not a zip file” error; done means the load call completes for the affected document types.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.