apache / apache/hudi

Utilities bundle ships tika-core only, so unstructured ingestion extracts no text from any file

Open
#19,676 0 comments 0 reactions 1 assignee Claimed by @rahil-c View on GitHub
area:ingest priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Only `tika-core` ships in the utilities bundles, and `tika-core` registers no parser at all, not even for `text/plain`. Running the bundle as documented therefore produces `parse_status=EMPTY` for every file, empty `extracted_text`, no chunks, and a null vector on every row, with the job exiting 0 and logging nothing.

`EMPTY` is also the normal outcome for an image, so nothing distinguishes "this file has no text" from "this deployment cannot extract text at all".

Measured on a 2 GB corpus of 616 text and 616 binary files:

| classpath | parse_status | chunks |
| --- | --- | --- |
| bundle as shipped | EMPTY = 1232 | 0 |
| plus `tika-parsers-standard-package` | SUCCESS 554, TRUNCATED 62, EMPTY 616 | 115,873 |

The 616 EMPTY in the second row are exactly the binary files, which is correct. In the first row every text file was empty too.

The documented remedy, supplying the parsers at runtime with `spark-submit --packages`, does not work reliably. Ivy failed to resolve three of Tika's transitive dependencies. Passing the jars with `--jars` instead gets further and then throws `NoSuchMethodError: GzipUtils.getUncompressedFileName`, because Spark ships commons-compress 1.23.0 and `--jars` does not take precedence over Spark's own jars.

That last failure is reached in practice rather than in theory. Tika detects format by magic number, and arbitrary binary content occasionally matches a compressed signature. A 1,232-file corpus never hit it; a 64,487-file corpus did. At document-lake scale it is a certainty.

The same applies to metadata, which is easy to miss: a JPEG through `TikaDocumentParser` yields 25 metadata fields when the parser modules are present, and nothing at all without them.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.