allenai / allenai/molmoweb

Enhancements for macOS (Apple Silicon) Compatibility and Playwright Stability

Đang mở
#11 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
592
Fork
81
Merge trung bình
9 ngày 22 giờ
Pull request đã merge (30 ngày)
1

Mô tả

# Enhancements for macOS (Apple Silicon) Compatibility and Playwright Stability

### **Summary**
When running MolmoWeb locally on macOS (M-series), users encounter a "float64 on MPS" runtime error and indefinite hangs during Playwright screenshots. I have implemented a set of surgical fixes to address these, along with minor bug fixes for missing imports and client-side validation.

---

### **System Environment**
- **Hardware:** Mac Studio M4 Max (128GB Unified Memory)
- **OS:** macOS Tahoe 26.4
- **Python:** 3.12.x
- **Models Tested:** MolmoWeb-4B and MolmoWeb-8B (HuggingFace versions)

---

### **Issue 1: MPS `float64` Crash**
The standard `torch` backend on macOS (MPS) does not support double precision (`float64`), but the current model/server implementation defaults to `cuda` or `cpu` and occasionally introduces `float64` tensors during inference.

#### **How to Reproduce**
1. Install MolmoWeb on an M-series Mac.
2. Start the server: `bash scripts/start_server.sh ./checkpoints/MolmoWeb-8B-HF 8001` (with `PREDICTOR_TYPE=hf`).
3. Run an inference query via `test_server.py`.

#### **Observed Error**
```text
Predictor error: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.
```

---

### **Issue 2: Playwright Screenshot Hangs**
In headless mode on Apple Silicon, `page.screenshot()` frequently hangs indefinitely. This appears to be a known conflict between Chromium’s GPU acceleration and macOS frame synchronization.

#### **How to Reproduce**
1. Run any agent task using `MolmoWeb.run()` in local/headless mode.
2. Observe the process hanging at the first step (taking a screenshot).

#### **Observed Logs**
```text
playwright._impl._errors.TimeoutError: Page.screenshot: Timeout 30000ms exceeded.
Call log: taking page screenshot ... waiting for fonts to load...
```

---

### **Issue 3: Minor Bugs & UX**
1. **Missing Import:** `agent/model_backends.py` is missing `import os`, causing a `NameError`.
2. **Cryptic Error:** Initializing `MolmoWeb` without an endpoint results in:
`AttributeError: 'NoneType' object has no attribute 'reset'`

---

### **Proposed Fixes**
I have verified a set of fixes locally that resolve these issues:
1. **MPS Support:** Added `mps` device auto-detection and forced `float32` dtypes.
2. **Stability Flags:** Added `--disable-gpu` and `--force-cpu-draw` flags to `SimpleEnv`.
3. **Screenshot Robustness:** Set `PW_TEST_SCREENSHOT_NO_FONTS_READY=1` and implemented a Chrome DevTools Protocol (CDP) fallback for screenshots (which bypasses standard API hangs related to frame synchronization).

I have these fixes verified and running in a local branch. Let me know if I can help out by filing a PR or providing the diffs in the comments.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.