Bug Report: Distributed AI Inference Template Issues & Architectural Fixes

## 🐛 Bug Report & Solutions

**Candidate:** Sujeet Kumar (sk0990670)
**Project:** Distributed AI Inference on AWS
**Solution Repository:** [Alchemyst-ai-Devops-Assignment](https://github.com/sk0990670/Alchemyst-ai-Devops-Assignment)

During the deployment of the DevOps assignment, I identified four critical bugs/flaws in the provided `quickstart` template code and default architectural assumptions. Below is the technical breakdown of the root causes and the engineering solutions I implemented to achieve a production-ready, stable deployment.

---

### 1. Engine Configuration Crash (`unknown field 'engine'`)

**Symptom:**
The `iii-engine` service crashed immediately on startup with the error:
`Failed to parse config file: unknown field 'engine'`.

**Root Cause:**
The provided `quickstart/config.yaml` included an `engine:` block (e.g., attempting to bind to `host: 0.0.0.0`). In the `0.12.0` release of the `iii` framework, `engine` is no longer a valid top-level key. The engine now only expects a `workers:` list in the configuration file.

**Solution:**
I removed the invalid `engine:` block from `config.yaml`. Network binding is now managed entirely by the framework's default behavior and exposed securely via AWS VPC Security Groups.

---

### 2. Multi-Engine Architecture Flaw (`function_not_found`)

**Symptom:**
When the `caller-worker` attempted to trigger the inference worker via an API request, it resulted in an HTTP 500 error:
`function_not_found: inference::run_inference`.

**Root Cause:**
The original design implied that the Caller VM and Inference VM would each run their own isolated `iii-engine`. However, the `iii` RPC framework requires functions to be registered with the *same* engine instance to be discoverable. Because the `inference-worker` was registered to its own local engine, the Caller VM's engine was entirely unaware of the `inference::run_inference` function.

**Solution: Single-Engine Architecture**
1. Configured the `iii-engine` to run **only** on the Caller VM.
2. Configured the `inference-worker.py` on the private VM to connect remotely to the Caller VM's engine via the private network by setting `III_URL=ws://:49134`.
3. Updated the Caller VM's AWS Security Group to allow inbound TCP traffic on port `49134` from the private subnet (`10.0.2.0/24`).

---

### 3. SDK Version Incompatibility

**Symptom:**
Workers would silently disconnect or fail to register functions correctly with the engine, despite the network being reachable.

**Root Cause:**
The `package.json` (Node.js) and `requirements.txt` (Python) in the provided templates were hardcoded to older versions of the `iii-sdk` (v0.11.x or v0.10.x). The infrastructure bootstrap scripts installed the latest `iii` CLI (v0.12.0). The version mismatch between the engine and the SDK caused silent failures during RPC negotiation.

**Solution:**
I upgraded the dependencies across both worker projects to ensure strict compatibility with the v0.12.0 engine:
- `caller-worker/package.json`: Updated to `"iii-sdk": "^0.12.0"`
- `inference-worker/requirements.txt`: Updated to `iii-sdk>=0.12.0`

---

### 4. Systemd `EXEC` Failures in Bootstrap

**Symptom:**
The systemd services (`caller-worker` and `inference-worker`) failed to start, logging a `status=203/EXEC` error in `journalctl`.

**Root Cause:**
The systemd service files assumed the `iii` binary would be located at a specific legacy path (e.g., `~/.iii/bin`). The newer installation script places these binaries in `~/.local/bin`. Because systemd runs as root or a specific service user with a restricted `$PATH`, it could not locate the binaries to execute the workers.

**Solution:**
I added a standardizing step to the Terraform `user_data` bootstrap scripts. After installation, the binaries are explicitly moved to the global `/usr/local/bin/` directory, ensuring reliable execution by systemd regardless of the user environment:
```bash
mv /root/.local/bin/iii* /usr/local/bin/