Traditional spider is missing URLs in v2.16.0

Describe the bug:

I have a very simple Flask app that I use as part of a Gitlab test pipeline to confirm our ZAP configuration is working correctly.

After recently upgrading to v2.16.0 some of the tests have started failing intermittently for the traditional spider with missing URLs reported by zap.spider.all_urls and /or zap.core.urls, even though I can see an alert was raised referencing the URLs.

Steps to reproduce the behavior:

1. Configure a form-based authentication for an application.
2. Kick off the traditional spider against the application.
3. Check if login URL is present in zap.core.urls.

Expected behavior:

URLs, especially those present in alerts, should be returned in zap.core.urls and / or zap.spider.all_urls.

Software Versions:

This is occurring for ZAP v2.16.0 using Docker image tag 20250304 and the Python zap-api-python library v0.4.0.

Screenshots:

Here are some screenshots of the problem (the URL in question below is http://<flask-app-id>/login):

zap.core.urls

zap.core.alerts

This mainly fails for the missing /login URL, although there is also a /blog?title=foo endpoint missing from the results in the screenshot as well, which happens less frequently.

Errors from the zap.log file:

No response

Additional context:

This happens using of a form-based authentication and the presence of the URL /profile in the list of URLs indicates that the login was successful because it is behind authentication, and I have a separate test that confirms the reverse, i.e. /profile is not present when unauthenticated, which passes.

The tests are isolated and a fresh Flask app instance is spun up for each test with no caching or anything between test runs. It happens when running tests locally and on the pipeline so seems unrelated to environment. One thing to note is the failure is not consistent - I'd say it happens roughly 70% of the time, whereas this never failed prior to v2.16.0 and has been working for ~6-8 months.

Finally, I tried switching the spider to use a thread count of 1 but it had no impact.

Would you like to help fix this issue?

• Yes

Start by reproducing the traditional spider against the form-authenticated Flask app and inspect the zap.spider.all_urls and zap.core.urls entry points on ZAP v2.16.0. Compare the returned URLs with zap.core.alerts, focusing on /login and /blog?title=foo and the intermittent behavior. Done means URLs referenced by alerts are consistently returned by the relevant URL APIs.