apache / apache/pulsar

File Connector repeats reading the same file if not deleted

Open
#13,301 4 comments 0 reactions 0 assignees View on GitHub
help wanted lifecycle/stale Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**

When using the file connector, if specifying a directory, a file name and setting `keepFile = true` the connector simply re-reads the file once it is complete, this is regardless of whether it was modified or not. If setting `keepFile = false` then it does not, as after the first read the connector deletes the file.

**To Reproduce**
Steps to reproduce the behavior:
1. First run pulsar standalone, and create a local file reader connector, using the following config (any.yaml) read the file. Note any.csv can be any file

```yaml
configs:
inputDirectory: "/tmp/"
recurse: false
keepFile: true
fileFilter: "any.csv"
```
2. Run the connector via the following command

```bash
pulsar-admin sources localrun \
--archive connectors/pulsar-io-file-2.8.1.nar \
--name spout \
--destination-topic-name file-raw \
--source-config-file any.yaml
```

3. Check the stats via the stats msgIn from the topic stats

```
./pulsar-admin topics stats lotr-file-raw | grep msgIn
```

You will see that the msgIn exceeds the number of lines in the csv file, if you now repeat this experiment but change the yaml to `keepFile: false` you will see that the msgIn will not exceed the number of lines in the csv file.

**Expected behavior**

The file connector should not repeatedly read the file, it should read the file once.
If the file has changed, then it should either read and continue where it left off. But this should be a config set.

**Desktop (please complete the following information):**
- OS: Mac OSX latest, latest pulsar 2.8.1.
This issue is still in the latest build of pulsar because the source code for the file connector has not changed in years.

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with the file connector archive, the provided any.yaml settings, and the localrun and topic-stats commands. Then inspect the file connector implementation used by connectors/pulsar-io-file-2.8.1.nar to trace how keepFile affects subsequent reads. Done means an unchanged retained file is read only once, with changed-file continuation behavior handled according to an explicit configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.