[Connector] Capture log files using streaming mode, but the process stops after capturing a file.
- Dominant language
- Java
- Stars
- 9.7k
- Forks
- 2.4k
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 204
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
Collect log files using streaming mode, but the process stops after collecting one file.
Even if data is appended to the file, it still doesn't work.
### SeaTunnel Version
2.3.12
### SeaTunnel Config
```conf
env {
job.mode = "STREAMING"
}
source {
LocalFile {
path = "/data/app/message_log/message-service-center.log"
file_format_type = "text"
schema {
fields {
raw_line = "string"
}
}
file_read_mode = "rolling"
monitoring_interval_ms = 10000
poll_interval = 10000
plugin_output = "original_log_line"
}
}
transform {
Split {
plugin_input = "original_log_line"
separator = "发送消息成功:"
split_field = "raw_line"
output_fields = [first_name, second_name]
plugin_output = "split_log_parts"
}
Filter {
plugin_input = "split_log_parts"
plugin_output = "filter_log_parts"
include_fields = [second_name]
}
JsonPath {
plugin_input = "filter_log_parts"
row_error_handle_way = FAIL
columns = [
{
"src_field" = "second_name"
"path" = "$.id"
"dest_field" = "id"
"column_error_handle_way" = "SKIP_ROW"
},
{
"src_field" = "second_name"
"path" = "$.receiver"
"dest_field" = "receiver"
"column_error_handle_way" = "SKIP_ROW"
}
]
plugin_output = "json_parsed_data"
}
FieldMapper {
plugin_input = "json_parsed_data"
plugin_output = "final_parsed_data"
field_mapper = {
id = id
receiver = receiver
}
}
}
```
### Running Command
```shell
nohup bin/seatunnel.sh --config ../job/test_file_to_mysql.conf -m local >seatunnel.log 2>&1 &
```
### Error Exception
```log
don't report an error, but it stops.
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the supplied LocalFile connector configuration in STREAMING mode, using the nohup command and the stated 10-second polling intervals. Check why processing stops after one file or ignores appended data; done means the connector continues collecting subsequent and newly appended log content without an error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100