google / google/skills

Feature Request: Cloud Pub/Sub Skill

Open
#107 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
20k
Forks
1.6k
Avg merge
3d 13h
Merged PRs (30d)
1

Description

# Feature Request: Cloud Pub/Sub Skill

## Summary

Cloud Pub/Sub is the backbone of event-driven and asynchronous architectures on Google Cloud. Existing skills cover individual services (BigQuery, Cloud Run, GKE) but do not explain how to connect them via Pub/Sub — leaving agents unable to design reliable messaging pipelines.

## The Gap

A typical agent workflow today:

User asks 'How do I trigger a Cloud Run job when a file lands in a bucket?' → Agent may answer with Cloud Functions or direct Eventarc, but misses Pub/Sub as the preferred decoupled alternative for high-throughput or multi-consumer scenarios.

There is no skill covering:

- Topic vs subscription architecture (fan-out, fan-in, dead-letter queues)
- Push vs Pull delivery semantics and retry policies
- Ordering keys, exactly-once delivery, and message deduplication
- BigQuery subscription for direct ingestion without intermediate compute
- Schema validation (Avro / Protobuf) for strongly typed message contracts
- Flow control, batching, and throughput tuning
- Security: Pub/Sub IAM, VPC-SC, CMEK encryption

## Proposed Skill

A `pubsub-basics` skill that agents load when users mention: Pub/Sub, event-driven, message queue, topic, subscription, async pipeline, streaming ingestion, dead letter queue, or BigQuery subscription.

### Suggested SKILL.md frontmatter

```yaml
---
name: pubsub-basics
description: >
Use when the user asks about event-driven messaging, asynchronous pipelines,
decoupling services, or streaming data ingestion on Google Cloud. Covers Cloud Pub/Sub
topics, subscriptions (push/pull/BigQuery), message ordering, exactly-once delivery,
dead-letter queues, schema validation, IAM, and integration with Cloud Run, GKE,
Dataflow, and BigQuery. WHEN: create topic, create subscription, event-driven,
message queue, async job, dead letter, BigQuery subscription, Pub/Sub schema.
compatibility: Requires pubsub.editor IAM role and Cloud Pub/Sub API enabled.
---
```

### Key reference topics

1. **Core Concepts**
- Publisher → Topic → Subscription(s) → Subscriber
- Fan-out (multiple subscriptions per topic) vs fan-in (single subscription per topic)
- Message retention: default 7 days, configurable up to 31 days
2. **Subscription Types**
- **Pull**: Manual ack, batch processing, GKE / compute workloads
- **Push**: HTTP POST to endpoint, Cloud Run / Cloud Functions
- **BigQuery**: Direct streaming write without code, schema mapping
- **Cloud Storage**: Export messages to GCS (integration with analytics)
3. **Reliability Patterns**
- Dead-letter topics (DLQ) and max delivery attempts
- Retry policies: immediate vs exponential backoff
- Ordering keys: when to use, throughput limitations
- Exactly-once delivery: semantics, prerequisites, idempotency
4. **Schema & Governance**
- Avro / Protobuf / JSON schema definitions
- Schema evolution and backward compatibility
- Constraining publishers to enforce contracts
5. **Performance & Cost**
- Flow control (max outstanding messages / bytes)
- Batching and compression for high throughput
- Pricing model: ingestion + egress + storage
6. **Security**
- IAM roles: publisher, subscriber, viewer
- VPC Service Controls for private-only access
- CMEK encryption at rest

## Why Now?

- Pub/Sub is implicitly required by at least 4 existing skills (Cloud Run event triggers, BigQuery streaming, GKE async jobs, Dataflow ingestion) but never taught as a first-class skill.
- BigQuery Subscription is a powerful zero-code pattern that is under-documented across the ecosystem.
- The rise of agentic workflows increases demand for reliable async handoff between tasks; Pub/Sub is the canonical GCP answer.

## Reference Implementation

Google Cloud official docs:
- https://cloud.google.com/pubsub/docs
- https://cloud.google.com/pubsub/docs/bigquery
- https://cloud.google.com/pubsub/docs/handling-failures

Happy to contribute a SKILL.md draft if this direction is accepted.

Contributor guide

Open the contributing guide

Research direction

Start with the proposed pubsub-basics SKILL.md frontmatter and the linked Google Cloud Pub/Sub, BigQuery subscription, and failure-handling references. Compare the requested concepts and integrations with the repository’s existing service skills, then define the accepted scope; done should be a documented skill covering the agreed Pub/Sub patterns, triggers, reliability, governance, performance, and security guidance.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud
Domain
cloud, documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.