crate / crate/cratedb-toolkit

Cluster API: Improve DNS propagation handling after cluster deployment

Open
#422 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10
Forks
5
Avg merge
7d 5h
Merged PRs (30d)
7

Description

## Overview

Currently, the code in `cratedb_toolkit/api/main.py` (line ~249) uses a fixed sleep time of 3.25 seconds to wait for DNS propagation after cluster deployment:

```python
# Wait a bit to let the deployment settle, mostly to work around DNS propagation issues.
time.sleep(3.25)
```

## Problem

A fixed sleep time is brittle and unreliable for DNS propagation:
- It might be unnecessarily long in some cases (slowing down operations)
- It might be too short in other cases (causing operational failures)
- There's no guarantee that DNS will have propagated after waiting for a fixed duration

Additionally, immediate polling after deployment could result in negative DNS caching, where a 'not found' response gets cached by DNS resolvers for some period (often 5-30 seconds depending on TTL settings).

## Recommendation

Implement a hybrid approach:
1. Keep a shorter initial sleep (maybe 1-2 seconds) to allow initial DNS propagation
2. Then implement polling with exponential backoff
3. Add a DNS cache-busting mechanism if necessary

This would maintain the reliability benefit of waiting for DNS propagation while avoiding the brittleness of a fixed sleep duration.

## References
- Referenced in PR: #81
- Referenced in comment: https://github.com/crate/cratedb-toolkit/pull/81#discussion_r2061393865

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.