Replace expensive file existance check (in object store) with spark options

The incremental loading from S3 and GCS performs a existence check for an object. This is expensive. This happens [here|[http://example.com|https://github.com/apache/hudi/blob/130498708bb1cd5da1d0e725971b3d721eeef231/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/CloudObjectsSelectorCommon.java#L161]]
 
This can be replaced with spark provided options:
spark.sql.files.ignoreMissingFiles
spark.sql.files.ignoreCorruptFiles
 
Ref for these options: [https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#ignore-missing-files]
 
 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-7504
- Type: Improvement

---

## Comments

21/Mar/24 02:47;vinay.bhat;Support for these configs (as part of DataSourceOptions) in spark was added only in 3.4.0. Will hold on to the PR for now.;;;

Start in hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/CloudObjectsSelectorCommon.java around line 161, then read Spark's generic file-source options for ignoreMissingFiles and ignoreCorruptFiles. Check the project's supported Spark versions, since these DataSourceOptions were added in Spark 3.4. Done means the expensive object existence check is replaced where supported without breaking incremental S3 or GCS loading.

Written by the indexing model from the issue text.