airbytehq / airbytehq/airbyte

[abctl] fills up /tmp directory causing "no space left on device" errors

Aperta
#71,134 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
area/abctl needs-triage team/triage type/bug
Lingua principale
Python
Stelle
22.1k
Fork
5.3k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

## Issue Type
- [x] Bug Report
- [ ] Feature Request
- [ ] Documentation Issue

## Description

When running `abctl local install`, the installation process can fail with the error:
```
OCI runtime exec failed: write /tmp/runc-process: no space left on device
```

This occurs because abctl (via Docker/kind) creates numerous temporary files in `/tmp` during cluster creation and installation. When the installation fails or is interrupted, these temporary files are not cleaned up, causing `/tmp` to fill up over multiple installation attempts.

## Environment

- **OS**: Ubuntu 22.04 (or similar Linux distribution)
- **abctl version**: v0.30.3
- **Docker version**: 29.1.3
- **kind version**: (as installed by abctl)
- **/tmp filesystem**: tmpfs (RAM-based) with 2GB limit

## Steps to Reproduce

1. Install abctl:
```bash
curl -LsfS https://get.airbyte.com | bash -
```

2. Run `abctl local install` multiple times (especially if previous attempts failed):
```bash
abctl local install
```

3. Check `/tmp` disk usage:
```bash
df -h /tmp
```

4. Observe that `/tmp` fills up (100% usage) after multiple failed/successful installations

5. Subsequent `abctl local install` attempts fail with:
```
ERROR Cluster 'airbyte-abctl' could not be created
ERROR unable to create kind cluster: failed to generate kubeadm config content: failed to get kubernetes version from node: failed to get file: command "docker exec --privileged airbyte-abctl-control-plane cat /kind/version" failed with error: exit status 128: OCI runtime exec failed: write /tmp/runc-process: no space left on device
```

## Expected Behavior

1. abctl should check `/tmp` disk space before starting installation
2. abctl should warn the user if `/tmp` is getting full (>80% usage)
3. abctl should clean up temporary files after failed installations
4. abctl should provide guidance on how to free up space if `/tmp` is full

## Actual Behavior

1. abctl does not check `/tmp` disk space before installation
2. abctl does not warn when `/tmp` is getting full
3. Temporary files created during installation (especially on failure) are not cleaned up
4. Multiple installation attempts accumulate temporary files in `/tmp`
5. Eventually `/tmp` fills up completely, causing all subsequent installations to fail

## Root Cause Analysis

### Why /tmp Gets Full

1. **Docker/kind creates many temp files**:
- Container runtime operations create `runc-process*` files in `/tmp`
- Image pulling and extraction creates temporary files
- Cluster initialization creates temporary configuration files
- Container creation and execution creates temporary mounts

2. **Failed operations don't clean up**:
- When `abctl local install` fails partway through, temporary files are not automatically removed
- Docker/kind processes may leave orphaned temporary files
- Multiple failed attempts accumulate files over time

3. **/tmp is limited**:
- `/tmp` is typically a tmpfs (RAM-based filesystem) with limited size (often 2GB)
- This can fill up quickly with Docker/kind operations
- The limit is system-dependent and may be too small for abctl operations

### Files Accumulating in /tmp

Common temporary files found in `/tmp` after failed abctl installations:
- `runc-process*` files (Docker container runtime)
- Docker temporary files
- kind cluster initialization files
- Kubernetes temporary files

## Impact

- **Severity**: Medium
- **Frequency**: High (occurs after multiple installation attempts)
- **User Impact**:
- Users cannot install Airbyte via abctl
- Users must manually clean `/tmp` or restart the instance
- No clear error message explaining the root cause
- Poor user experience

## Workarounds

### Immediate Fix

1. **Restart the instance** (clears /tmp completely):
```bash
sudo reboot
```

2. **Manual cleanup**:
```bash
# Remove old files
find /tmp -type f -mtime +7 ! -path "*/systemd-private/*" ! -path "*/snap-private/*" -delete

# Remove Docker/kind temp files
find /tmp -name "*runc*" -type f ! -path "*/systemd-private/*" -delete
find /tmp -name "*docker*" -type f ! -path "*/systemd-private/*" -delete
find /tmp -name "*kind*" -type f ! -path "*/systemd-private/*" -delete
```

### Prevention

1. **Check /tmp before installation**:
```bash
df -h /tmp
# If > 80% full, clean up or restart
```

2. **Regular cleanup** (cron job):
```bash
0 0 * * 0 find /tmp -type f -mtime +7 -delete
```

## Proposed Solutions

### 1. Pre-flight Checks (Recommended)

Add disk space checks before installation:

```bash
# Check /tmp space before starting
TMP_USAGE=$(df /tmp | tail -1 | awk '{print $5}' | sed 's/%//')
if [ "$TMP_USAGE" -gt 80 ]; then
echo "WARNING: /tmp is ${TMP_USAGE}% full"
echo "Please free up space or restart the instance"
exit 1
fi
```

### 2. Automatic Cleanup

Clean up temporary files after failed installations:

```bash
# Clean up on failure
cleanup_temp_files() {
find /tmp -name "*runc*" -type f -mtime +1 -delete 2>/dev/null || true
find /tmp -name "*docker*" -type f -mtime +1 -delete 2>/dev/null || true
find /tmp -name "*kind*" -type f -mtime +1 -delete 2>/dev/null || true
}
trap cleanup_temp_files EXIT
```

### 3. Better Error Messages

Provide clear error messages when `/tmp` is full:

```
ERROR: /tmp directory is full (100% used)
This prevents Docker/kind from creating temporary files.

Solutions:
1. Restart the instance: sudo reboot
2. Clean up /tmp: find /tmp -type f -mtime +7 -delete
3. Increase /tmp size (system configuration required)
```

### 4. Use Alternative Temp Directory

Allow users to configure a different temporary directory:

```bash
# Environment variable
export DOCKER_TMPDIR=/var/tmp
abctl local install
```

### 5. Post-Installation Cleanup

Clean up temporary files after successful installation:

```bash
# After successful installation
cleanup_installation_temp_files() {
# Remove abctl/kind specific temp files
find /tmp -name "*abctl*" -type f -delete 2>/dev/null || true
find /tmp -name "*kind*" -type f -mtime +1 -delete 2>/dev/null || true
}
```

## Additional Context

- This issue affects users who:
- Run multiple installation attempts
- Have limited `/tmp` space (tmpfs with small size)
- Experience installation failures (which leave temp files behind)

- Related issues:
- Docker/kind also create temp files that may not be cleaned up
- This is a known limitation of Docker/kind, but abctl should handle it better

## References

- [Docker tmp directory usage](https://docs.docker.com/config/containers/logging/)
- [kind cluster creation](https://kind.sigs.k8s.io/)
- Similar issues in other projects (e.g., Apache Flink blobstore files in /tmp)

## Checklist

- [x] I have searched existing issues to ensure this is not a duplicate
- [x] I have provided clear steps to reproduce
- [x] I have provided expected vs actual behavior
- [x] I have provided workarounds
- [x] I have proposed solutions
- [x] I have included environment details

---

**Labels**: `bug`, `installation`, `docker`, `kind`, `tmp`, `disk-space`

### What did you expect to happen?

1) abctl should check /tmp disk space before starting installation
2) abctl should warn the user if /tmp is getting full (>80% usage)
3) abctl should clean up temporary files after failed installations
4) abctl should provide guidance on how to free up space if /tmp is full

### Abctl Version

```console
$ abctl version
[v0.30.3]
```

### Docker Version

```console
$ docker version
Found Docker installation: version 29.1.3
```

### OS Version

```console
# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.