google-deepmind / google-deepmind/gemma

Not able to pass audio to `gemma3n:e4b` using genai SDK?

Open
#371 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.7k
Forks
1k
Avg merge
10h 33m
Merged PRs (30d)
2

Description

Hey guys I have gemma3n deployed on cloud run which I can interact with just fine when I try text but I still haven't figured out a way to pass audio to the model. In the official documentation I see the use of huggingface transformers library. But I am not using it. Can someone please tell me how it is done using python-sdk.

```python

load_dotenv()
CLOUD_RUN_URL = os.environ.get("CLOUD_RUN_URL")
API_KEY = os.environ.get("API_KEY")
MODEL_NAME = os.environ.get("MODEL_NAME")

# --------------------------------------------------------------------
# FastAPI + CORS
# --------------------------------------------------------------------
app = FastAPI()
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_methods=["POST", "OPTIONS"],
allow_headers=["*"],
)

try:
client = genai.Client(
api_key=API_KEY,
http_options=HttpOptions(base_url=CLOUD_RUN_URL)
)
except Exception as e:
print(f"Error initializing GenAI Client: {e}")
client = None

@app.post("/text")
async def ask_text(payload: TextPayload):
"""Handles a text-based prompt and returns a generated text response."""
if not client:
raise HTTPException(503, "GenAI client not initialized. Check configuration.")

try:
response = client.models.generate_content(
model=MODEL_NAME, # type: ignore
contents=[payload.prompt] # Use the prompt from the incoming payload
)

return {"text": response.text}

except Exception as exc:
raise HTTPException(500, f"An error occurred: {exc}")

# doesn't work for this
@app.post("/audio")
async def ask_audio(file: UploadFile = File(...)):
"""
Accepts a direct WAV file upload, transcribes it, and returns the text.
"""
if not client:
raise HTTPException(503, "GenAI client not initialized. Check configuration.")

if file.content_type not in ["audio/wav", "audio/x-wav"]:
raise HTTPException(400, "Invalid file type. Please upload a WAV file.")

wav_path = None
try:
audio_bytes = await file.read()

audio_part = types.Part(
inline_data=types.Blob(
mime_type=file.content_type,
data=audio_bytes
)
)
response = client.models.generate_content(
model=MODEL_NAME, # type: ignore
contents = [
types.Part.from_text(text='Transcribe this audio file:'),
audio_part,
]
)

return {"text": response.text}

except Exception as exc:
raise HTTPException(500, f"An error occurred: {exc}") from exc
finally:
if wav_path and os.path.exists(wav_path):
os.remove(wav_path)
```

Any help is appreciated

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.