Azure / Azure/azure-functions-nodejs-extensions

[Feature] Add E2E tests with Docker Kafka broker for @azure/functions-extensions-kafka

Open
#105 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
2
Forks
8
PR merge metrics
No merged PRs in 30d

Description

## Summary

Add Docker-based E2E tests for the `@azure/functions-extensions-kafka` package, following the existing pattern used by Blob and ServiceBus extensions.

## Background

PR #104 adds the `@azure/functions-extensions-kafka` package with unit tests, CI build/lint integration, and a sample app. However, E2E testing (Docker Compose with a real Kafka broker) is not yet included because it requires additional infrastructure setup.

## Required Changes

### 1. Docker Compose service: Kafka broker

Add a Kafka service to `e2e/docker-compose.yml` (e.g., `bitnami/kafka` or `confluentinc/cp-kafka`):

```yaml
kafka:
image: bitnami/kafka:latest
ports:
- "9092:9092"
environment:
KAFKA_CFG_NODE_ID: 0
KAFKA_CFG_PROCESS_ROLES: broker,controller
KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 0@kafka:9093
KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
healthcheck:
test: ["CMD", "kafka-topics.sh", "--bootstrap-server", "localhost:9092", "--list"]
interval: 10s
timeout: 5s
retries: 5
```

### 2. Kafka function app

Create `e2e/kafka-app/` following the `e2e/servicebus-app/` pattern:

- `Dockerfile` — Multi-stage build: builds base + kafka extensions, compiles TypeScript app
- `src/functions/kafkaTrigger.ts` — KafkaTrigger with `KafkaRecord` binding, stores processed records in global array
- `src/functions/getProcessedRecords.ts` — HTTP GET to retrieve processed records
- `src/functions/health.ts` — Health check endpoint
- `host.json` — Runtime config with Kafka extension bundle
- `package.json` — Depends on `@azure/functions-extensions-kafka`

### 3. E2E test file

Create `e2e/tests/kafka.e2e.test.ts`:

- **Setup**: Wait for Kafka app healthy, produce a test message to a topic using `kafkajs`
- **Test 1**: Verify single message processing — produce message, poll `/api/processed-records`, verify metadata (topic, partition, offset, headers, timestamp)
- **Test 2**: Verify batch processing — produce multiple messages, verify all processed

### 4. Build script update

Add `'azure-functions-nodejs-extensions-kafka'` to `e2e/scripts/build-extensions.mjs` extensions list (already done in PR #104).

### 5. E2E pipeline integration

Add Kafka app to `eng/templates/jobs/e2e-tests.yml` — Docker Compose up with Kafka service, wait for health, run vitest.

## Dependencies

- `kafkajs` npm package (for producing test messages in E2E tests)
- Kafka broker Docker image (`bitnami/kafka` or similar)
- Kafka Extension NuGet (`Microsoft.Azure.WebJobs.Extensions.Kafka` 4.3.1) in extension bundle

## Complexity

This is more complex than Blob/ServiceBus E2E because:
- Kafka broker requires KRaft mode configuration (no ZooKeeper in new versions)
- The Kafka Extension host-side requires the extension bundle to include `Microsoft.Azure.WebJobs.Extensions.Kafka` 4.3.1+
- Topic auto-creation and consumer group coordination need careful setup

## Related

- Parent package: PR #104
- Parent issue: #103
- Kafka Extension: [Azure/azure-functions-kafka-extension#612](https://github.com/Azure/azure-functions-kafka-extension/issues/612)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.