ml-opensource / ml-opensource/reality-check

Describe simulator scene with `gpt-4-vision`

Open
#12 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Swift
Stars
73
Forks
1
PR merge metrics
No merged PRs in 30d

Description

// Define a function to capture the screen of the Xcode simulator app window
func captureSimulatorWindow() {
    let targetAppName = "Simulator"  // The name of the Xcode simulator app
    
    if let windowList = CGWindowListCopyWindowInfo(.optionOnScreenOnly, kCGNullWindowID) as? [[String: AnyObject]] {
        for window in windowList {
            if let windowOwnerName = window[kCGWindowOwnerName] as? String, windowOwnerName.contains(targetAppName) {
                if let windowID = window[kCGWindowNumber] as? CGWindowID {
                    let screenCaptureSession = ScreenCaptureSession(windowID: windowID)
                    
                    // Start the screen capture session for the Xcode simulator window
                    screenCaptureSession.startCapture { image, error in
                        if let image = image {
                            // Process the captured image
                            // For example, you can save the image to a file
                            saveImageToFile(image: image)
                        } else if let error = error {
                            // Handle the error
                            print("Error capturing the simulator window: \(error.localizedDescription)")
                        }
                    }
                }
            }
        }
    }
}

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the proposed Swift captureSimulatorWindow entry point and review ScreenCaptureKit and CGWindowListCopyWindowInfo for restricting capture to the Simulator window. Determine how the captured image should be passed to gpt-4-vision and what scene description is expected; done means the Simulator scene is captured and described reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, swift
Domain
ai, desktop, devtools
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.