[Feature] Support periodically refresh existing dataFileMetas in CompactManager

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

It is very usual user will use a streaming job and a batch job to update different columns of the same paimon table. User can choose to configure the batch job as write-only or streaming job as write-only to avoid conflict.

In either way, the write and compact job will not receive the new files generated by write-only job. This will lead to insufficient compaction both for MergeTreeCompactManager and BucketedAppendCompactManager.

This issue could be addressed by supporting periodically refresh dataFileMetas in CompactManager.

By doing this, there is no need to introduce an extra dedicated compaction job.

### Solution

Same solution like the refreshFiles in LocalTableQuery

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Start by reading CompactManager and the refreshFiles approach in LocalTableQuery. Trace how MergeTreeCompactManager and BucketedAppendCompactManager obtain dataFileMetas, then determine where periodic refreshing belongs. Done means both managers can discover files produced by a separate write-only job and perform sufficient compaction without a dedicated compaction job.

Written by the indexing model from the issue text.