Ignition Timeout logic is confusing, inconsistent

Bug

Ignition Version

v0.29.0

Environment

Any

Expected Behavior

Ignition has reasonable, easy to understand timeout behavior

Actual Behavior

Ignition's timeout handling is confusing, only configurable in some cases

Other Information

Ignition currently has too many different types of timeouts. Some only apply to specific URL schemes. It's not well documented which apply where.

Here is the current handling for the URL schemes Ignition supports:

• HTTP(S): Ignition tries to fetch the resource, times out after ignition.timeouts.httpheaders if no connection is established. Times out after ignition.timeouts.httptotal if the body is not fetched.
• TFTP: Uses it's tftp library's default timeout of 5s for round trip time.
• S3: Uses the HTTP client so presumably behaves the same as HTTP(S)
• DATA: N/A
• OEM (not user facing): N/A

Proposal for both 2.x and 3.x configs:

• Remove exponential backoff code. It's dwarfed by the timeouts in the HTTP or TFTP connections and only complicates the codebase making it harder to reason about how long Ignition will try for.
• Verify and document that s3 behaves likes HTTP

Prospoal for 3.x only:

• change ignition.timeouts.httpheaders to ignition.timeouts.connection. Specify that it is the timeout for initializing a connection. This would be the same behavior for HTTP(S) and S3, but also set the round trip timeout for TFTP. Should more protocols be added, it should set the closest analogous timeout for that protocol. Alternatively could move this the URL section to allow per-source behavior
• change ignition.timeouts.httptotal to ignition.timeouts.transfer and have it be the timeout for fetching any resource regardless of protocol. Alternatively this could move to the URL section to allow per-source behavior.
• Add a ignition.timeouts.retries which applies to all protocols (except data).

Related bugs + PRs:
https://github.com/coreos/bugs/issues/2527
https://github.com/coreos/bugs/issues/2532
https://github.com/coreos/ignition/pull/679

Start by tracing the current timeout handling for HTTP(S), TFTP, and S3, then read related issues #2527 and #2532 and pull request #679. The proposal lists several alternative configuration designs; work is complete only after the intended timeout and retry behavior is chosen, implemented consistently, and documented for each supported URL scheme.