Endpoint plugin backwards invocation fails with "session not found" when HTTP callback times out

Endpoint plugin backwards invocation fails with "session not found" when HTTP callback times out

Summary

When an endpoint plugin calls self.session.app.chat.invoke(response_mode="blocking") and the workflow execution time exceeds the HTTP callback timeout (e.g., WeChat Work's 5-second limit), the plugin daemon returns ErrSessionNotFound ("session not found"). This is because the HTTP request context cancellation triggers a cascade that removes the session from sessionToInstanceMap before the backwards invocation goroutine can write the response back.

This affects all webhook-based endpoint plugins (WeChat Work, DingTalk, Feishu, etc.) that use backwards invocations with long-running workflows.

Environment

• Plugin Daemon: langgenius/dify-plugin-daemon:0.6.10-local
• Dify API: langgenius/dify-api:1.17.0
• Plugin: anspire/wecom-bot:0.0.2 (WeChat Work bot)
• Runtime: Docker Compose, local runtime mode
• Workflow: advanced-chat app with agent node (execution time ~14-113 seconds)

Reproduction

1. Install an endpoint plugin that receives webhooks from WeChat Work (or any service with a callback timeout)
2. Configure the plugin to call self.session.app.chat.invoke(response_mode="blocking") in the endpoint handler
3. Send a message from WeChat Work
4. The workflow takes >5 seconds to complete
5. WeChat Work closes the connection after 5 seconds (callback timeout)
6. The plugin daemon logs: backwards_invocation.write_error "session not found"
7. The plugin returns an error to the user

Root Cause Analysis

The cascade of events:

HTTP callback timeout (5s)
    → HTTP request context cancelled
    → context.AfterFunc(ctx, response.Close) fires
    → response stream closed
    → listener.OnClose callback fires
    → sessionToInstanceMap.Delete(sessionId)
    → backwards invocation goroutine tries to write response
    → LocalPluginRuntime.Write() looks up sessionId in sessionToInstanceMap
    → NOT FOUND → ErrSessionNotFound
    → WriteError/WriteResponse fails
    → "session not found" logged

Key code locations:

1. Session registration — internal/core/local_runtime/io.go:

func (r *LocalPluginRuntime) Listen(sessionId string) (...) {
    instance, err := r.pickLowestLoadInstance()
    r.sessionToInstanceMap.Store(sessionId, instance)  // ← session registered here

    listener.OnClose(func() {
        instance.removeStdioHandlerListener(sessionId)
        r.sessionToInstanceMap.Delete(sessionId)        // ← session removed here
    })
}

2. Context cancellation triggers cleanup — internal/core/io_tunnel/generic.go:

stopCloseOnCancel := context.AfterFunc(ctx, response.Close)  // ← fires on timeout

3. Write fails after cleanup — internal/core/local_runtime/io.go:

func (r *LocalPluginRuntime) Write(sessionId string, ...) error {
    instance, ok := r.sessionToInstanceMap.Load(sessionId)
    if !ok {
        return ErrSessionNotFound  // ← "session not found"
    }
    // ...
}

4. Error logged but result lost — internal/core/io_tunnel/backwards_invocation/request.go:

func (bi *BackwardsInvocation) WriteError(err error) {
    err1 := bi.writer.Write(...)  // ← fails with "session not found"
    if err1 != nil {
        log.WarnContext(context.Background(), "backwards_invocation.write_error", err1)
    }
}

The fundamental issue:

The backwards invocation is dispatched as an async goroutine (routine.Submit in task.go:InvokeDify), but the session lifecycle is tied to the HTTP request context. When the HTTP context is cancelled (callback timeout), the session is cleaned up, but the goroutine is still running. The goroutine's attempt to write the response back fails because the session no longer exists in the map.

This creates a race condition: the backwards invocation goroutine and the session cleanup are not coordinated.

Expected Behavior

The plugin daemon should either:

1. Keep the session alive for the duration of the backwards invocation (even after the HTTP callback returns), so the goroutine can write the response back.
2. Provide an alternative mechanism for endpoint plugins to make long-running invocations that aren't tied to the HTTP request lifecycle.
3. At minimum, provide a clear error message that explains the timeout, rather than the confusing "session not found".

Actual Behavior

• Error: backwards_invocation.write_error "session not found"
• Error: backwards_invocation.write_response error "session not found"
• HTTP response status: 100 (Continue) — the default status code, indicating no response was sent
• The workflow continues running and succeeds in the database, but the result is lost
• WeChat Work shows no response or an error message

Workaround

We implemented a workaround by bypassing the backwards invocation mechanism entirely:

1. The endpoint plugin returns finish=False immediately (within the callback timeout)
2. A background thread makes a direct HTTP call to the Dify API's internal endpoint (/inner/api/invoke/app) using urllib.request, which is not tied to the session lifecycle
3. Results are cached to a file (/tmp/qw_bot_cache/)
4. Subsequent stream callbacks check the cache and return the result when ready

This workaround requires:

• DIFY_INNER_API_URL and DIFY_INNER_API_KEY environment variables (already available in the plugin process)
• DIFY_TENANT_ID and DIFY_USER_ID environment variables (manually added)
• Parsing the length-prefixed binary protocol used by the internal endpoint (14-byte header + JSON payload)

Log Evidence

22:30:42.541  session info cached  session_id=ac1c8430-...  plugin=anspire/wecom-bot:0.0.2
22:30:47.502  backwards_invocation.write_error  "session not found"
22:30:47.502  backwards_invocation.write_response error  "session not found"
22:30:47.502  session info deleted from cache  session_id=ac1c8430-...
22:30:47.502  HTTP request  status=100  latency_ms=4972  (5 second timeout)

The workflow itself succeeds (verified in workflow_runs table with succeeded status), but the result is never delivered to the plugin.

Suggested Fix Directions

1. Deferred session cleanup: Delay sessionToInstanceMap.Delete(sessionId) until all pending backwards invocations for that session have completed. A reference counter or sync.WaitGroup could track pending invocations.

2. Separate invocation sessions: Create a separate session for the backwards invocation that's not tied to the HTTP request context. The invocation session would persist until the invocation completes.

3. Graceful degradation: When the HTTP context is cancelled but backwards invocations are still pending, log a warning and let the goroutine complete. The result may not be delivered to the HTTP client, but at least the workflow isn't left in an inconsistent state.

Read internal/core/local_runtime/io.go, internal/core/io_tunnel/generic.go, internal/core/io_tunnel/backwards_invocation/request.go, and task.go:InvokeDify, then reproduce the callback-timeout sequence described in the issue. Trace session cleanup against the pending backwards invocation; done means the timeout no longer produces a lost result with a misleading "session not found" error, or the resulting failure is explicitly handled.