[abctl] fills up /tmp directory causing "no space left on device" errors
- Lingua principale
- Python
- Stelle
- 22.1k
- Fork
- 5.3k
- Metriche di merge delle PR
- Metriche PR in attesa
Descrizione
## Issue Type
- [x] Bug Report
- [ ] Feature Request
- [ ] Documentation Issue
## Description
When running `abctl local install`, the installation process can fail with the error:
```
OCI runtime exec failed: write /tmp/runc-process: no space left on device
```
This occurs because abctl (via Docker/kind) creates numerous temporary files in `/tmp` during cluster creation and installation. When the installation fails or is interrupted, these temporary files are not cleaned up, causing `/tmp` to fill up over multiple installation attempts.
## Environment
- **OS**: Ubuntu 22.04 (or similar Linux distribution)
- **abctl version**: v0.30.3
- **Docker version**: 29.1.3
- **kind version**: (as installed by abctl)
- **/tmp filesystem**: tmpfs (RAM-based) with 2GB limit
## Steps to Reproduce
1. Install abctl:
```bash
curl -LsfS https://get.airbyte.com | bash -
```
2. Run `abctl local install` multiple times (especially if previous attempts failed):
```bash
abctl local install
```
3. Check `/tmp` disk usage:
```bash
df -h /tmp
```
4. Observe that `/tmp` fills up (100% usage) after multiple failed/successful installations
5. Subsequent `abctl local install` attempts fail with:
```
ERROR Cluster 'airbyte-abctl' could not be created
ERROR unable to create kind cluster: failed to generate kubeadm config content: failed to get kubernetes version from node: failed to get file: command "docker exec --privileged airbyte-abctl-control-plane cat /kind/version" failed with error: exit status 128: OCI runtime exec failed: write /tmp/runc-process: no space left on device
```
## Expected Behavior
1. abctl should check `/tmp` disk space before starting installation
2. abctl should warn the user if `/tmp` is getting full (>80% usage)
3. abctl should clean up temporary files after failed installations
4. abctl should provide guidance on how to free up space if `/tmp` is full
## Actual Behavior
1. abctl does not check `/tmp` disk space before installation
2. abctl does not warn when `/tmp` is getting full
3. Temporary files created during installation (especially on failure) are not cleaned up
4. Multiple installation attempts accumulate temporary files in `/tmp`
5. Eventually `/tmp` fills up completely, causing all subsequent installations to fail
## Root Cause Analysis
### Why /tmp Gets Full
1. **Docker/kind creates many temp files**:
- Container runtime operations create `runc-process*` files in `/tmp`
- Image pulling and extraction creates temporary files
- Cluster initialization creates temporary configuration files
- Container creation and execution creates temporary mounts
2. **Failed operations don't clean up**:
- When `abctl local install` fails partway through, temporary files are not automatically removed
- Docker/kind processes may leave orphaned temporary files
- Multiple failed attempts accumulate files over time
3. **/tmp is limited**:
- `/tmp` is typically a tmpfs (RAM-based filesystem) with limited size (often 2GB)
- This can fill up quickly with Docker/kind operations
- The limit is system-dependent and may be too small for abctl operations
### Files Accumulating in /tmp
Common temporary files found in `/tmp` after failed abctl installations:
- `runc-process*` files (Docker container runtime)
- Docker temporary files
- kind cluster initialization files
- Kubernetes temporary files
## Impact
- **Severity**: Medium
- **Frequency**: High (occurs after multiple installation attempts)
- **User Impact**:
- Users cannot install Airbyte via abctl
- Users must manually clean `/tmp` or restart the instance
- No clear error message explaining the root cause
- Poor user experience
## Workarounds
### Immediate Fix
1. **Restart the instance** (clears /tmp completely):
```bash
sudo reboot
```
2. **Manual cleanup**:
```bash
# Remove old files
find /tmp -type f -mtime +7 ! -path "*/systemd-private/*" ! -path "*/snap-private/*" -delete
# Remove Docker/kind temp files
find /tmp -name "*runc*" -type f ! -path "*/systemd-private/*" -delete
find /tmp -name "*docker*" -type f ! -path "*/systemd-private/*" -delete
find /tmp -name "*kind*" -type f ! -path "*/systemd-private/*" -delete
```
### Prevention
1. **Check /tmp before installation**:
```bash
df -h /tmp
# If > 80% full, clean up or restart
```
2. **Regular cleanup** (cron job):
```bash
0 0 * * 0 find /tmp -type f -mtime +7 -delete
```
## Proposed Solutions
### 1. Pre-flight Checks (Recommended)
Add disk space checks before installation:
```bash
# Check /tmp space before starting
TMP_USAGE=$(df /tmp | tail -1 | awk '{print $5}' | sed 's/%//')
if [ "$TMP_USAGE" -gt 80 ]; then
echo "WARNING: /tmp is ${TMP_USAGE}% full"
echo "Please free up space or restart the instance"
exit 1
fi
```
### 2. Automatic Cleanup
Clean up temporary files after failed installations:
```bash
# Clean up on failure
cleanup_temp_files() {
find /tmp -name "*runc*" -type f -mtime +1 -delete 2>/dev/null || true
find /tmp -name "*docker*" -type f -mtime +1 -delete 2>/dev/null || true
find /tmp -name "*kind*" -type f -mtime +1 -delete 2>/dev/null || true
}
trap cleanup_temp_files EXIT
```
### 3. Better Error Messages
Provide clear error messages when `/tmp` is full:
```
ERROR: /tmp directory is full (100% used)
This prevents Docker/kind from creating temporary files.
Solutions:
1. Restart the instance: sudo reboot
2. Clean up /tmp: find /tmp -type f -mtime +7 -delete
3. Increase /tmp size (system configuration required)
```
### 4. Use Alternative Temp Directory
Allow users to configure a different temporary directory:
```bash
# Environment variable
export DOCKER_TMPDIR=/var/tmp
abctl local install
```
### 5. Post-Installation Cleanup
Clean up temporary files after successful installation:
```bash
# After successful installation
cleanup_installation_temp_files() {
# Remove abctl/kind specific temp files
find /tmp -name "*abctl*" -type f -delete 2>/dev/null || true
find /tmp -name "*kind*" -type f -mtime +1 -delete 2>/dev/null || true
}
```
## Additional Context
- This issue affects users who:
- Run multiple installation attempts
- Have limited `/tmp` space (tmpfs with small size)
- Experience installation failures (which leave temp files behind)
- Related issues:
- Docker/kind also create temp files that may not be cleaned up
- This is a known limitation of Docker/kind, but abctl should handle it better
## References
- [Docker tmp directory usage](https://docs.docker.com/config/containers/logging/)
- [kind cluster creation](https://kind.sigs.k8s.io/)
- Similar issues in other projects (e.g., Apache Flink blobstore files in /tmp)
## Checklist
- [x] I have searched existing issues to ensure this is not a duplicate
- [x] I have provided clear steps to reproduce
- [x] I have provided expected vs actual behavior
- [x] I have provided workarounds
- [x] I have proposed solutions
- [x] I have included environment details
---
**Labels**: `bug`, `installation`, `docker`, `kind`, `tmp`, `disk-space`
### What did you expect to happen?
1) abctl should check /tmp disk space before starting installation
2) abctl should warn the user if /tmp is getting full (>80% usage)
3) abctl should clean up temporary files after failed installations
4) abctl should provide guidance on how to free up space if /tmp is full
### Abctl Version
```console
$ abctl version
[v0.30.3]
```
### Docker Version
```console
$ docker version
Found Docker installation: version 29.1.3
```
### OS Version
```console
# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.