Doris Sink Error Log in verbose mode

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

I am struggle with pushing data into doris. when it failed ,vector have few infomation, and I have to guess what is wrong.

the doris steam log response is mostly like :

{
    "TxnId": 2288,
    "Label": "aabd7e19-2831-45cb-aab0-67227df68e5c",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Success",
    "Message": "OK",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 1,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 5516,
    "LoadTimeMs": 40,
    "BeginTxnTimeMs": 1,
    "StreamLoadPutTimeMs": 3,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 12,
    "ReceiveDataTimeMs": 7,
    "CommitAndPublishTimeMs": 22
}

but when the status is not success, it will return ErrorLog , it is very helpful for debug the data flow

{
    "TxnId": 2289,
    "Label": "628c4ac3-5a20-4a32-bc15-7c7f58a1ccbe",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[DATA_QUALITY_ERROR]too many filtered rows",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 0,
    "NumberFilteredRows": 1,
    "NumberUnselectedRows": 0,
    "LoadBytes": 5516,
    "LoadTimeMs": 9,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 3,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 5,
    "ReceiveDataTimeMs": 6,
    "CommitAndPublishTimeMs": 0,
    "ErrorURL": "http://xxxxxxx/api/_load_error_log?file=__shard_711/error_log_insert_stmt_c646ac01567619fe-6a399d092673181_c646ac01567619fe_6a399d092673181"
}

Use Cases

I want vector to expose the errorlog in verbose mode ,so i can debug the data remap directly.

Attempted Solutions

doris response is just receive status and message in sinks/doris/retry.rs

struct DorisStreamLoadResponse {
    #[serde(rename = "Status")]
    status: String,

    #[serde(rename = "Message")]
    message: Option<String>,

}

we can add ErrorLog and expose it in debug message

struct DorisStreamLoadResponse {
    #[serde(rename = "Status")]
    status: String,

    #[serde(rename = "Message")]
    message: Option<String>,

    #[serde(rename = "ErrorURL")]
    error_url: Option<String>,
}

Proposal

No response

References

No response

Version

No response

Start in sinks/doris/retry.rs and inspect DorisStreamLoadResponse plus the path that handles its status and message. Trace how failed responses reach verbose or debug output. Done means the Doris ErrorURL is captured and exposed in verbose mode so data-flow failures can be diagnosed.