apache / apache/doris-streamloader

Why is the Parquet file format not supported?

Offen
#34 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Go
Sterne
36
Forks
24
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

The operation succeeds when using streamload curl
**curl -v -L --location-trusted -u "${USER}:${PASSW}" \
-H "format: parquet" \
-H "label: ${LABEL}" \
-H "timeout: ${TIMEOUT}" \
-H "Expect: 100-continue" \
-T "${SOURCE_FILE}" \
"http://${DORIS_HOST}:${DORIS_PORT}/api/${DB_NAME}/${TABLE_NAME}/_stream_load" \
| tee -a "${LOG_FILE}"**

yet an error occurs with streamloader after adding the header format:parquet,
**${DORIS_LOADER} \
--source_file="${SOURCE_FILE}" \
--url="${DORIS_URL}" \
--header="format:parquet" \
--db="${DB_NAME}" \
--table="${TABLE_NAME}" \
--u="${USER}" \
--p="${PASSW}" \
--workers=${WORKERS} \
--check_utf8=false \
2>&1 | tee -a "${LOG_FILE}"**\

as shown below:

regent@dorisFEServer01:~/testdata$ bash streamload_parquet.sh
[2026-01-27 09:00:05] Start Doris Stream Load
time=2026-01-27 09:00:05 level=error msg=Read file failed, error message: EOF, before retrying, we suggest:
1.Check the input data files and fix if there is any problem.
2.Do select count(*) to check whether data is partially loaded.
3.If the data is partially loaded and duplication is unacceptable, consider dropping the table (with caution that all data in the table will be lost) and retry.
4.Otherwise, just retry.
line=reader/reader.go:139
time=2026-01-27 09:00:05 level=error msg=5.When using a specified line delimiter, the file must end with that delimiter. line=reader/reader.go:141

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Start with reader/reader.go around lines 139-141 and reproduce the streamloader command using the supplied Parquet header and source-file options. Compare its behavior with the successful curl stream load, then establish whether the loader should support this input and define done as resolving the EOF or clearly reporting the format limitation.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
go
Bereich
cli, database
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.