filecoin-project / filecoin-project/boost

Boost provider not providing CIDs and timing out with no error in response to ad entries

Open
#2,029 1 comment 0 reactions 0 assignees View on GitHub
hint/needs-decision
Dominant language
Go
Stars
121
Forks
79
Avg merge
10d 14h
Merged PRs (30d)
1

Description

### Checklist

- [x] This is **not** a question or a support request. If you have any boost related questions, please ask in the [discussion forum](https://github.com/filecoin-project/boost/discussions).
- [x] This is **not** a new feature request. If it is, please file a [feature request](https://github.com/filecoin-project/boost/issues/new?assignees=&labels=need%2Ftriage%2Ckind%2Ffeature&template=feature_request.yml) instead.
- [x] This is **not** an enhancement request. If it is, please file a [improvement suggestion](https://github.com/filecoin-project/boost/issues/new?assignees=&labels=need%2Ftriage%2Ckind%2Fenhancement&template=enhancement.yml) instead.
- [x] I **have** searched on the [issue tracker](https://github.com/filecoin-project/boost/issues) and the [discussion forum](https://github.com/filecoin-project/boost/discussions), and there is no existing related issue or discussion.
- [x] I am running the [`Latest release`](https://github.com/filecoin-project/boost/releases), or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
- [x] I did not make any code changes to boost.

### Boost component

- [x] boost daemon - storage providers
- [ ] boost client
- [ ] boost UI
- [ ] boost data-transfer
- [x] boost index-provider
- [ ] Other

### Boost Version

```text
(2 separate boost nodes)
```

### Describe the Bug

Filecoin users are reporting some instances of being unable to publish to IPNI. In both cases, IPNI is making a request for CIDs and the server is not returning after 15 minutes in 6 successive tries. @willscott believes this is a bug in the boost provider.

If this long delay is intentional and working as intended, it is incompatible with IPNI design and needs to be jointly resolved somehow. If the server returns 404 or some error, IPNI indexer can progress with ingest one way or another. But getting no response combined with linear ingestion of ads means that the indexer cannot progress until that URL resolves (result or error). Otherwise we end up with gap in index.

We have received 2 reports in the #ipni channel on [Filecoin Slack](https://filecoinproject.slack.com/archives/C02T827T9N0/p1759884828259359) in the past week. It looks like there are ~15-20 providers that are in this same state per cid.contact/providers.

### First report - 02 October 2025

([thread in #ipni channel](https://filecoinproject.slack.com/archives/C02T827T9N0/p1759443301204649))

@tippyflitsUK is seeing mixed results over the last few days/months with network indexing from Boost node f08403.

> Deals over the last week do not appear to be indexing, or are doing so very slowly (these are Lighthouse deals) and I still have a ton of deals from months ago that remain in indexing state (these are mostly Solana deals). I have added a recent history covering the last few months to the [F08403 - IPNI Status](https://docs.google.com/spreadsheets/d/1avuICbQEIzgr_pufOG1LELsQ3E-V7e-CPkW80q6JOHc/edit?gid=1179090631#gid=1179090631) Google sheet with open access.

IPNI is seeing repeated timeouts when fetching ads from that endpoint.

```
2025-10-03T07:49:16.907Z ERROR indexer/ingest ingest/ingest.go:1152 Error while ingesting ad. Bailing early, not ingesting later ads. {"publisher": "12D3KooWMdvJ2PT8Rph2UYUneXwGKFu5DfUNbmkZBziJweXV6Nws", "worker": 39, "adCid": "baguqeeral2z6tmsgw7lgtw6z3d4kmx5lukl2sqcwmwixtan5mmk2g4d4xqza", "err": "syncEntriesErr: failed to sync first entry while checking entries type: sync handler failed: failed to traverse requested dag: failed to load node for root cid baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q: failed to fetch block for cid baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q: fetch request failed: Get \"http://yablufc.ddns.net:3104/ipni/v1/ad/baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q\": GET http://yablufc.ddns.net:3104/ipni/v1/ad/baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q giving up after 6 attempt(s): Get \"http://yablufc.ddns.net:3104/ipni/v1/ad/baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "adsLeftToProcess": 9840}
2025-10-03T07:49:22.668Z ERROR indexer/admin admin/handler.go:369 Cannot sync with peer {"peerID": "12D3KooWMdvJ2PT8Rph2UYUneXwGKFu5DfUNbmkZBziJweXV6Nws", "err": "syncEntriesErr: failed to sync first entry while checking entries type: sync handler failed: failed to traverse requested dag: failed to load node for root cid baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q: failed to fetch block for cid baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q: fetch request failed: Get \"http://yablufc.ddns.net:3104/ipni/v1/ad/baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q\": GET http://yablufc.ddns.net:3104/ipni/v1/ad/baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q giving up after 6 attempt(s): Get \"http://yablufc.ddns.net:3104/ipni/v1/ad/baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
```

### Second report - 08 October 2025

([thread in #ipni channel](https://filecoinproject.slack.com/archives/C02T827T9N0/p1759884828259359))
User Edward creates an online deal on Filecoin. That deal was supposed to advertise payload CID on IPNI but the publish did not happen.

Original Filecoin deal: https://filfox.info/en/deal/129424823
CID: https://cid.contact/ipfs/bafybeibsszq36h6fv4k6r4a4rnrz4tguubsgyfdp4y2qzk3rprfwzcenra

IPNI sees this response:
`'error while ingesting ad baguqeeravhwxgdnvcvjxb34qn4jhwohqtibycsbwi347r6p7at54aufjx7pa: syncEntriesErr: failed to sync first entry while checking entries type: sync handler failed: failed to traverse requested dag: failed to load node for root cid baguqeera6e6lo53vmew6v5bxmls77mk55s3r34q4qgtd2rbqqemzvrbrsmna: failed to fetch block for cid baguqeera6e6lo53vmew6v5bxmls77mk55s3r34q4qgtd2rbqqemzvrbrsmna: fetch request failed: Get "http://79.104.211.73:3402/ipni/v1/ad/baguqeera6e6lo53vmew6v5bxmls77mk55s3r34q4qgtd2rbqqemzvrbrsmna": GET http://79.104.211.73:3402/ipni/v1/ad/baguqeera6e6lo53vmew6v5bxmls77mk55s3r34q4qgtd2rbqqemzvrbrsmna giving up after 6 attempt(s): Get "http://79.104.211.73:3402/ipni/v1/ad/baguqeera6e6lo53vmew6v5bxmls77mk55s3r34q4qgtd2rbqqemzvrbrsmna": net/http: request canceled (Client.Timeout exceeded while awaiting headers)'`

### Logging Information

```text
See error logs inline
```

### Repo Steps

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the failures reported at ingest/ingest.go:1152 and admin/handler.go:369, then inspect the Boost index-provider ad endpoint involved in the logged /ipni/v1/ad requests. Reproduce or trace the timeout behavior using the examples in the issue. Done means IPNI receives CIDs or a timely error instead of repeated requests that wait for 15 minutes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.