filecoin-project / filecoin-project/boost
Add cli command to download offline deal data
- Dominant language
- Go
- Stars
- 121
- Forks
- 79
- Avg merge
- 10d 14h
- Merged PRs (30d)
- 1
Description
### Background
To download the data for an offline deal, a Storage Provider typically
- downloads the data manually (eg using wget)
- calls `boostd import-data `
In practice Storage Providers may want to take advantage of the Boost UI to manage the download. The advantages are
- Boost will keep track of how much space the download takes and prevent the SP from downloading more than will fit into their download staging area
For example if the SP has a 500GB download staging area, and is downloading enough data to fit into 480GB, boost will prevent the SP from starting a download for a 32GB file.
- Manage downloads in the Boost UI
Boost shows the progress of downloads, automatically resumes downloads when boost is restarted, and allows the SP to manually cancel a download
### Proposal
Add a `boostd download-data` command:
```
$ boostd download-data --help
NAME:
boostd download-data - Download data for an offline deal made with Boost
USAGE:
boostd download-data [command options] or
```
The command should output an error if downloading this data would exceed the available space in the staging area (including ongoing downloads). For example if
- the staging area is 500GB
- there are 400GB of completed downloads (whose deals have not been added to a sector)
- there are in-progress downloads for deals whose total data size is 80GB
- ie total "tagged" space in the download area is 400GB + 80GB = 480GB
- the command should return an error if the user attempts to download a 32GB file
Note that if the size of the data to download was not specified as part of the deal, boostd should do an HTTP HEAD request to get the size from the client. If it's not possible to get the size, boostd should download the data anyway.
The command will process the deal in the same way as if it were an online deal:
- download the data from the client
- perform commp on the data
- publish the deal
- add the deal data to a sector
- index and announce the deal
The advantage of using `boostd download-data` instead of doing on online deal is that the SP can control when to download the data (online deals start downloading immediately).
### Related commands
SPs will likely want to use this command in scripts, so it would also be useful for us to document how to check the remaining space in the download staging area:
```
$ curl 'http://localhost:8080/graphql/query' \
--data-raw '{"query":"query { storage { Staged Transferred Pending Free MountPoint }}"}' | jq
{
"data": {
"storage": {
"Staged": {
"__typename": "BigInt",
"n": "0"
},
"Transferred": {
"__typename": "BigInt",
"n": "25123893565"
},
"Pending": {
"__typename": "BigInt",
"n": "7966307422"
},
"Free": {
"__typename": "BigInt",
"n": "611154893413"
},
"MountPoint": "/home/nonsense/.boost/incoming"
}
}
}
```
Contributor guide
Research direction
Start by locating the existing boostd import-data command and the online-deal processing path described in the issue. Compare their handling of download, commp, publishing, sector addition, indexing, and announcing; done means boostd download-data accepts a proposal CID or deal UUID and enforces the stated staging-space behavior. No file or test path is named, so repository search is required.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100