firebase / firebase/quickstart-ios

[FirebaseAI] FR: Extend GenerativeAIMultimodalExample to support analysis of multiple media types

Open
#1,729 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
3k
Forks
1.5k
Avg merge
2d 5h
Merged PRs (30d)
3

Description

# Overview

The current `GenerativeAIMultimodalExample` sample project only supports image analysis, with a single input method—using `PhotosPicker` to select images, and only processes image (`UIImage`) formats.

However, according to the Swift version of the Firebase AI API, it can actually support analysis of more media types, including video, audio, and PDF.

## Firebase AI API Analysis

According to the official documentation: https://firebase.google.com/docs/ai-logic, all media types use the same `generateContent` format, which makes things much simple.

```swift
// Image analysis (implemented)
let response = try await model.generateContent(image, prompt)

// Video analysis (to be extended)
let video = InlineDataPart(data: videoData, mimeType: "video/mp4")
let response = try await model.generateContent(video, prompt)

// Audio analysis (to be extended)
let audio = InlineDataPart(data: audioData, mimeType: "audio/mpeg")
let response = try await model.generateContent(audio, prompt)

// PDF document analysis (to be extended)
let pdf = InlineDataPart(data: pdfData, mimeType: "application/pdf")
let response = try await model.generateContent(pdf, prompt)
```

## Proposed Design Plan

### 1. UI Enhancement

#### Tab Navigation Design
Add 4 media type selection tabs at the top of the interface:

- **📷 Image**
- **🎥 Video**
- **🎵 Audio**
- **📄 Document**

#### Input Component Upgrade
Expand the `MultimodalInputField` component:

1. **Dynamic Button**: Show the corresponding file picker based on the selected tab. Not all the pickers for every tab.
2. **Type Indicator**: Clearly display the currently selected media type
3. **Preview Optimization**: Provide appropriate preview for different media types

### 2. Data Processing Extension

#### File Handler
Implement file processing logic for each media type, support DocumentPicker selection, convert to `InlineDataPart`.

#### ViewModel Refactor
Extend `PhotoReasoningViewModel` to support state management for multiple media types.

### 3. User Experience Optimization

Smart Prompt: Provide corresponding prompt to the `MultimodalInputField` for preview based on the selected media type, for example:

- **Image**: "Describe the content of this image"
- **Video**: "Summarize the main content of this video"
- **Audio**: "Transcribe and analyze this audio"
- **Document**: "Extract key information from the document"

## Conclusion

This feature extension will fully showcase the complete multimedia analysis capabilities of Firebase AI, providing iOS developers with more comprehensive learning and reference resources. @peterfriese

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.