Design a system which consumes columnar file notifications from HDFS/cloud storage and generates delta, hudi and iceberg metadata

### Feature Request / Improvement

Sub-task of main feature request https://github.com/apache/incubator-xtable/issues/550

Look at **Using cloud notifications queue** in the approach mentioned below. It's only a high level approach, with many open ended items. We can do a design doc and finalize the approach through a discussion in dev mailing list.
https://github.com/apache/incubator-xtable/issues/550#issuecomment-2386526797

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Start with the main feature request in issue #550 and its “Using cloud notifications queue” approach. The issue calls for a design document covering how notifications from HDFS or cloud storage would produce Delta, Hudi, and Iceberg metadata; resolve the open questions through discussion on the dev mailing list.

Written by the indexing model from the issue text.