ml-opensource / ml-opensource/reality-check
Describe simulator scene with `gpt-4-vision`
Open
Nobody has claimed this yet.
- Dominant language
- Swift
- Stars
- 73
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
- Consider using ScreenCaptureKit
- CGWindowListCopyWindowInfo(::) can help to restrict captures to simulator
// Define a function to capture the screen of the Xcode simulator app window
func captureSimulatorWindow() {
let targetAppName = "Simulator" // The name of the Xcode simulator app
if let windowList = CGWindowListCopyWindowInfo(.optionOnScreenOnly, kCGNullWindowID) as? [[String: AnyObject]] {
for window in windowList {
if let windowOwnerName = window[kCGWindowOwnerName] as? String, windowOwnerName.contains(targetAppName) {
if let windowID = window[kCGWindowNumber] as? CGWindowID {
let screenCaptureSession = ScreenCaptureSession(windowID: windowID)
// Start the screen capture session for the Xcode simulator window
screenCaptureSession.startCapture { image, error in
if let image = image {
// Process the captured image
// For example, you can save the image to a file
saveImageToFile(image: image)
} else if let error = error {
// Handle the error
print("Error capturing the simulator window: \(error.localizedDescription)")
}
}
}
}
}
}
}
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the proposed Swift captureSimulatorWindow entry point and review ScreenCaptureKit and CGWindowListCopyWindowInfo for restricting capture to the Simulator window. Determine how the captured image should be passed to gpt-4-vision and what scene description is expected; done means the Simulator scene is captured and described reliably.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- macos, swift
- Domain
- ai, desktop, devtools
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100