google / google/artemis

Fix analyzer clip framerate

Open Beginner friendly
#52 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.9k
Forks
516
Avg merge
22m
Merged PRs (30d)
5

Description

## Summary

`render_timeline_clip()` in `artemis/utils/video.py` renders every analyzer clip at ~25fps instead of the requested `fps` (default 15), silently duplicating frames. Since the whole point of this function is to keep a frame index mappable back to a real recording timestamp, the mismatch breaks that guarantee on every analyzer clip the agent renders.

## Root cause

Each input segment's ffmpeg filter chain sets an explicit `fps={fps}`, but the final encode command never sets an explicit output frame rate (`-r`). Without one, ffmpeg has no framerate to encode the muxed output with and falls back to a default of 25fps, inserting duplicate frames (via PTS) to preserve the real wall-clock duration:

```
frame= 77 fps=0.0 q=28.0 Lsize= 4KiB time=00:00:03.00 bitrate=10.6kbits/s dup=31 drop=0
```

A 3.0s window that should produce ~45 frames at 15fps instead produces 77 (~40% duplicate frames). Duration is correct; encoded framerate is not.

## Impact

`UnifiedMobileController.render_timeline_clip` (`artemis/controllers/unified_controller.py:408`) is the only caller and runs on the production path — every analyzer clip the agent renders during a task goes through this. Any downstream logic assuming `frame_index / fps ≈ recording_time` (which is exactly what this function's docstring promises: *"so `start_time + frame_offset` still names the true recording time"*) is working against a clip whose actual framerate silently diverges from `fps`.

## Reproduction

This is caught by an existing, currently-failing test on `main`:

```
tests/unit/test_unified_controller_video.py::test_analyzer_clip_keeps_timeline_time_across_restart_gap
FAILED - assert 43 <= len(frames) <= 47
AssertionError: assert 77 <= 47
```

Reproduces the same way outside pytest by calling `render_timeline_clip()` directly and reading frame count with `cv2.VideoCapture`.

## Fix

Pass `-r {fps}` to the output stage of the ffmpeg command so the encoder gets an explicit framerate instead of guessing:

```diff
"-filter_complex", ";".join(filter_parts),
"-map", "[outv]",
+ "-r", str(fps),
"-c:v", "libx264",
```

Verified: frame count drops from 77 → 46 (within the test's expected 43–47 range); the previously-failing test now passes, and the full `tests/unit/test_unified_controller_video.py` file (22 tests) passes. `ruff format --check`, `ruff check`, and `pyright` are clean on the changed file.

Contributor guide

Open the contributing guide

Research direction

Start in artemis/utils/video.py at render_timeline_clip() and inspect the final ffmpeg output stage, then read the caller at artemis/controllers/unified_controller.py:408. Run tests/unit/test_unified_controller_video.py::test_analyzer_clip_keeps_timeline_time_across_restart_gap and the full test file; done means the expected frame count passes and the stated format, lint, and type checks remain clean.

Written by the indexing model from the issue text.

Assessment

Tech stack
opencv, python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
95/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.