DefangLabs / DefangLabs/defang

`defang tail` stops streaming events mid-deployment on GCP provider

Open
#2,001 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
166
Forks
24
Avg merge
10h 8m
Merged PRs (30d)
33

Description

Description

defang tail stops delivering log events partway through a GCP deployment while the deployment continues to make progress. The stream appears to hang silently. No error, no disconnect, the process stays alive. But no new events are delivered.

Steps to Reproduce

  1. Deploy a multi-service stack (4 services: api, frontend, worker + managed db/redis) to GCP using defang compose up --detach
  2. Run defang tail --stack=beta to follow the deployment
  3. Observe that log streaming stops mid-deployment

Expected Behavior

defang tail should continue streaming events for the entire deployment lifecycle, including:

  • Cloud Build completion for all services
  • Cloud Run service updates and revision readiness
  • Pulumi resource creation (NEG, backend service, URL map, HTTPS proxy, forwarding rule)
  • Worker instance group updates
  • Final "Update succeeded/failed" message

Actual Behavior

defang tail streams initial events (Pulumi update start, Cloud Build progress, early apt-get output from Docker builds) but then stops around the 1-2 minute mark. The last events received were Cloud Build Docker layer pulls and apt-get package resolution for the API image build.

Meanwhile, the deployment continued successfully for another ~6+ minutes in GCP:

  • All three Cloud Build jobs completed
  • API service ran Django migrations and started gunicorn successfully
  • Both Cloud Run services (api, frontend) reached CONDITION_SUCCEEDED
  • Load balancer resources (NEG, backend service, URL map, HTTPS proxy, forwarding rule) were all created
  • Worker instance group was updated

None of these events appeared in defang tail. The process remained alive with no error output — just silent.

I confirmed the deployment was progressing by querying the GCP APIs directly (Cloud Run services API, Compute Engine instance group manager API), which showed healthy services while defang tail showed nothing.

Environment

  • Defang CLI: v3.3.2
  • Defang Fabric: v0.6.0-1519-g598a4a1d
  • Provider: GCP (DEFANG_PROVIDER=gcp)
  • GCP Project: ecgpt-a9bf8
  • Region: us-central1
  • Stack: beta
  • Deployment ID: ang5tnyb63sp
  • OS: Linux

Additional Context

  • defang compose up (non-detached, with inline log tailing) appeared to have the same issue in a prior deployment attempt: it showed early build logs but stopped delivering events before the deployment completed or failed.
  • The defang tail process never exited or printed an error; it just stopped receiving events while still appearing connected.
  • defang ls also returned "No active deployments found" during the deployment, which may be a related issue with deployment state visibility.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the deployment with defang compose up --detach, then follow it with defang tail --stack=beta and compare the output with GCP's Cloud Run and Compute Engine APIs. Also check whether defang ls reports the deployment during this period. Done means defang tail remains connected and reports events through the final success or failure message.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, go, google-cloud
Domain
cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.