Credential Rotation

Hello,

In my company we are building an internal platform that runs many ETL processes over Spark 4.1 with PySpark and spark-connect.
We are using an AWS RDS Postgres DB as well as custom Postgres deployments for usage with AGE etc., all read/write via Spark.
We would like to have an option to change the credentials for connections where the pipeline configuration is coming from config files we have that are dynamically translated into Spark code.
For writers this is not a problem as we can add our own retry mechanism that recreates the writer with the updated credentials.
However, For readers, this is a problem because at the point of materialization, the reader is already baked into the pipeline with the original credentials and it does not have any way to change the credentials without restarting the entire ETL process, which could be running for several hours before the plan reaches the point where it needs to use the JDBC connection.

I'm wondering if it would be possible to add some way of reconfiguring just the reader/writer credentials from PySpark code without having to write dedicated Java code, compile into a JAR and load when running Spark?

Thanks!

Start by tracing the PySpark reader and writer paths for JDBC connections and how credentials are captured before materialization. Determine whether credentials can be refreshed from PySpark without restarting the ETL process, and define completion as supporting rotation for readers and writers without dedicated Java code or a separately loaded JAR.

Written by the indexing model from the issue text.