Feature Request: Cloud Pub/Sub Skill
- Dominant language
- Python
- Stars
- 20k
- Forks
- 1.6k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 1
Description
# Feature Request: Cloud Pub/Sub Skill
## Summary
Cloud Pub/Sub is the backbone of event-driven and asynchronous architectures on Google Cloud. Existing skills cover individual services (BigQuery, Cloud Run, GKE) but do not explain how to connect them via Pub/Sub — leaving agents unable to design reliable messaging pipelines.
## The Gap
A typical agent workflow today:
User asks 'How do I trigger a Cloud Run job when a file lands in a bucket?' → Agent may answer with Cloud Functions or direct Eventarc, but misses Pub/Sub as the preferred decoupled alternative for high-throughput or multi-consumer scenarios.
There is no skill covering:
- Topic vs subscription architecture (fan-out, fan-in, dead-letter queues)
- Push vs Pull delivery semantics and retry policies
- Ordering keys, exactly-once delivery, and message deduplication
- BigQuery subscription for direct ingestion without intermediate compute
- Schema validation (Avro / Protobuf) for strongly typed message contracts
- Flow control, batching, and throughput tuning
- Security: Pub/Sub IAM, VPC-SC, CMEK encryption
## Proposed Skill
A `pubsub-basics` skill that agents load when users mention: Pub/Sub, event-driven, message queue, topic, subscription, async pipeline, streaming ingestion, dead letter queue, or BigQuery subscription.
### Suggested SKILL.md frontmatter
```yaml
---
name: pubsub-basics
description: >
Use when the user asks about event-driven messaging, asynchronous pipelines,
decoupling services, or streaming data ingestion on Google Cloud. Covers Cloud Pub/Sub
topics, subscriptions (push/pull/BigQuery), message ordering, exactly-once delivery,
dead-letter queues, schema validation, IAM, and integration with Cloud Run, GKE,
Dataflow, and BigQuery. WHEN: create topic, create subscription, event-driven,
message queue, async job, dead letter, BigQuery subscription, Pub/Sub schema.
compatibility: Requires pubsub.editor IAM role and Cloud Pub/Sub API enabled.
---
```
### Key reference topics
1. **Core Concepts**
- Publisher → Topic → Subscription(s) → Subscriber
- Fan-out (multiple subscriptions per topic) vs fan-in (single subscription per topic)
- Message retention: default 7 days, configurable up to 31 days
2. **Subscription Types**
- **Pull**: Manual ack, batch processing, GKE / compute workloads
- **Push**: HTTP POST to endpoint, Cloud Run / Cloud Functions
- **BigQuery**: Direct streaming write without code, schema mapping
- **Cloud Storage**: Export messages to GCS (integration with analytics)
3. **Reliability Patterns**
- Dead-letter topics (DLQ) and max delivery attempts
- Retry policies: immediate vs exponential backoff
- Ordering keys: when to use, throughput limitations
- Exactly-once delivery: semantics, prerequisites, idempotency
4. **Schema & Governance**
- Avro / Protobuf / JSON schema definitions
- Schema evolution and backward compatibility
- Constraining publishers to enforce contracts
5. **Performance & Cost**
- Flow control (max outstanding messages / bytes)
- Batching and compression for high throughput
- Pricing model: ingestion + egress + storage
6. **Security**
- IAM roles: publisher, subscriber, viewer
- VPC Service Controls for private-only access
- CMEK encryption at rest
## Why Now?
- Pub/Sub is implicitly required by at least 4 existing skills (Cloud Run event triggers, BigQuery streaming, GKE async jobs, Dataflow ingestion) but never taught as a first-class skill.
- BigQuery Subscription is a powerful zero-code pattern that is under-documented across the ecosystem.
- The rise of agentic workflows increases demand for reliable async handoff between tasks; Pub/Sub is the canonical GCP answer.
## Reference Implementation
Google Cloud official docs:
- https://cloud.google.com/pubsub/docs
- https://cloud.google.com/pubsub/docs/bigquery
- https://cloud.google.com/pubsub/docs/handling-failures
Happy to contribute a SKILL.md draft if this direction is accepted.
Contributor guide
Research direction
Start with the proposed pubsub-basics SKILL.md frontmatter and the linked Google Cloud Pub/Sub, BigQuery subscription, and failure-handling references. Compare the requested concepts and integrations with the repository’s existing service skills, then define the accepted scope; done should be a documented skill covering the agreed Pub/Sub patterns, triggers, reliability, governance, performance, and security guidance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud
- Domain
- cloud, documentation
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100