iterative / iterative/cml

BitBucket implementation + webhook events / integration with Studio

Open
#817 8 comments 2 reactions 0 assignees View on GitHub
ci-bitbucket cml-comment documentation flaky
Dominant language
JavaScript
Stars
4.2k
Forks
345
PR merge metrics
No merged PRs in 30d

Description

Hey CML team! As you've probably noticed, I'm working on the integration between Studio + CML PR comments on BitBucket.

I'm running into a limitation with BitBucket webhooks and wanted to check if you are open to changing the CML BitBucket implementation to make the integration work more reliably.

## Problem: `pullrequest:comment_updated` is very unreliable

BitBucket does not send webhook events for `pullrequest:comment_updated` in a lot of scenario's.

From the [BB docs](https://support.atlassian.com/bitbucket-cloud/docs/event-payloads/#Comment-updated):

> If a user updates the same comment with not much time in between, Bitbucket only sends the event request the first time the comment is updated. However, if some time passes and that user updates the comment again, Bitbucket sends the event request a second time.

From my experimentation, "not much time in between" can mean "not even after 20 minutes" (!). I have as-of-yet not been able to make BitBucket send me a single comment updated event[^wrong]. (Not when editing a PR comment with CML, but also not when editing manually).

Update: I've edited a comment I've posted after 3 days, and BitBucket still did not send any update payload. Seems like this is just plain broken.

[^wrong]: I triple checked that I'm not doing anything weird. The webhook I have is configured to send updates when PR comments are edited. The created+deleted events work fine.

## How CML performs edits

When `cml send-comment` is invoked with the `--pr` and `--update` flags, CML checks if a PR comment exists and updates it if this is the case. The update contains the new contents of the report.

This update happens by sending a PUT request to BitBucket:

https://github.com/iterative/cml/blob/0a674fef92b0ff210c95fb83bd710e972199d630/src/drivers/bitbucket_cloud.js#L154-L166

## Why Studio needs to know about updates

When CML updates a PR comment, the new comment may contain a new SHA. In these cases, Studio needs to associate the CML report with a different commit and display it accordingly. We are therefore interested in knowing when things change.

## Options to resolve this

1. In theory, Studio could periodically poll BitBucket to see if any CML reports have been updated. I'd really like to avoid this though, as providing these updates to users in a timely manner would take up quite a sizeable portion of the API limits.
1. Make CML delete old BitBucket reports when invoked with `--pr` and `--update`.
1. Provide a different flag (e.g. `--delete` / `--delete-old`) and tell users to use this if they want smooth integration with Studio.
1. Don't do anything special / tell users not to use `--update` if they want to use CML with Studio

Out of all these options, I'm inclined to lean towards `--delete` / `--delete-old` (name up for discussion ofc.):

- The flag could be useful to other users as well, if they want notifications every time a new CML report is available, but don't care about the old ones any longer.
- Some CML users may use BitBucket and not use Studio and appreciate the current behavior of `--edit`.

What do you think about the options I posted? Anything I missed?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.