Idea: better soft404-handling

Not sure this is the right place, but here goes.
The combination of soft404 (responding with 200 when the page isn't found) and relative links (or javascriptextraction) results in tons of 'fake' pages being crawled.
The crawler don't know to stop, without 404.

I am wondering if we could do something about it?

Here is an example of soft404: https://budgets.dk/_ignition/horizon/api/jobs/_ignition/salt-edge/get-consent

One idea I have (but I lack the skill to implement) is this approach:

As one of the first things crawling a domain (right after the seeds maybe?), request a bogus file with a random name (could be seed + heritrix_soft404check_[random string]

If the response code from this absolute bogusurl is 200, then continue as usual. But handle all files from this domain with the exact same filesize/content as the bogus-file as 404s. (except the first one, because the frontpage is often used as redirect for soft404)

Logfile example with created 'wrong' urls:

`2026-04-30T08:19:29.329Z 200 39155 https://budgets.dk/heritrix_soft404check_qaegqygqegbdbq7938 - - text/html #004 20260430081900342+212 sha1:RXAU2UXVNF5EHYGEWMD3XM5HZWNNWEJC https://budgets.dk/heritrix_soft404check_qaegqygqegbdbq7938 content-size:40542

2026-04-30T08:18:59.867Z 200 39155 https://budgets.dk/_ignition/horizon/api/jobs2/_ignition/salt-edge/get-consent - - text/html #039 20260430081830401+223 sha1:VUKPYVCE4LKFF53LT2LKVNTZFWFWLKVY https://budgets.dk/_ignition/horizon/api/jobs2/_ignition/salt-edge/get-consent content-size:40542

2026-04-30T08:18:29.431Z 200 39155 https://budgets.dk/_ignition/horizon/api/jobs/_ignition/salt-edge/get-consent3 - - text/html #005 20260430081801987+519 sha1:D36K2L4E3GSGHNNVHM5FGUUPRTF7MY5R https://budgets.dk/_ignition/horizon/api/jobs/_ignition/salt-edge/get-consent3 content-size:40542

2026-04-30T08:12:28.746Z 200 39155 https://budgets.dk/_ignition/horizon/api2/jobs/_ignition/salt-edge/get-consent - - text/html #006 20260430081200329+264 sha1:2OBDMMEUJDHQZC7DFO2ABNFYFUDM5JGB https://budgets.dk/_ignition/horizon/api2/jobs/_ignition/salt-edge/get-consent content-size:40542

2026-04-30T08:11:59.825Z 200 39155 https://budgets.dk/_ignition/horizon/api/jobs/_ignition/salt-edge/get-consent2 - - text/html #012 20260430081137453+228 sha1:ZBDDLKXA2ZJUXKQNNVTBQNUCDKUWVYVB https://budgets.dk/_ignition/horizon/api/jobs/_ignition/salt-edge/get-consent2 content-size:40542

2026-04-30T08:11:37.068Z 200 39155 https://budgets.dk/_ignition/horizon/api/jobs/_ignition/salt-edge/get-consent6 - - text/html #004 20260430081114501+228 sha1:XHSCMJQJBPKKOGCSGJW47HJEK5QMM6KR https://budgets.dk/_ignition/horizon/api/jobs/_ignition/salt-edge/get-consent6 content-size:40542

2026-04-30T08:11:14.030Z 200 39155 https://budgets.dk/_ignition/horizon/api/jobs/_ignition/salt-edge/get-consent5 - - text/html #010 20260430081057524+254 sha1:PRBLRJR4Q6U34BW5WRFLHWJNC7KVIP7C https://budgets.dk/_ignition/horizon/api/jobs/_ignition/salt-edge/get-consent5 content-size:40542

2026-04-30T08:10:57.003Z 200 39155 https://budgets.dk/_igni3tion/horizon/api/jobs/_ignition/salt-edge/get-consent - - text/html #002 20260430081036423+315 sha1:HKLOAEKH3CT3SYNZBXOBWCOUVHRGUBLB https://budgets.dk/_igni3tion/horizon/api/jobs/_ignition/salt-edge/get-consent content-size:40542,3t

2026-04-30T08:10:35.875Z 200 24 https://budgets.dk/robots.txt P https://budgets.dk/_igni3tion/horizon/api/jobs/_ignition/salt-edge/get-consent text/plain #003 20260430081035286+448 sha1:YR6M6GSJYJGMLBBEGCVHLRZO6SISSJAS https://budgets.dk/_igni3tion/horizon/api/jobs/_ignition/salt-edge/get-consent content-size:359

2026-04-30T08:10:34.566Z 1 53 dns:budgets.dk P https://budgets.dk/_igni3tion/horizon/api/jobs/_ignition/salt-edge/get-consent text/dns #001 20260430081033889+153 sha1:W27X2YRE7Y37QHMWNWP57ZGNBS3QXNFD https://budgets.dk/_igni3tion/horizon/api/jobs/_ignition/salt-edge/get-consent content-size:53`