filecoin-project / filecoin-project/boost

Add cli command to download offline deal data

Open
#1,435 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
121
Forks
79
Avg merge
10d 14h
Merged PRs (30d)
1

Description

### Background

To download the data for an offline deal, a Storage Provider typically
- downloads the data manually (eg using wget)
- calls `boostd import-data `

In practice Storage Providers may want to take advantage of the Boost UI to manage the download. The advantages are
- Boost will keep track of how much space the download takes and prevent the SP from downloading more than will fit into their download staging area
For example if the SP has a 500GB download staging area, and is downloading enough data to fit into 480GB, boost will prevent the SP from starting a download for a 32GB file.
- Manage downloads in the Boost UI
Boost shows the progress of downloads, automatically resumes downloads when boost is restarted, and allows the SP to manually cancel a download

### Proposal

Add a `boostd download-data` command:
```
$ boostd download-data --help
NAME:
boostd download-data - Download data for an offline deal made with Boost

USAGE:
boostd download-data [command options] or
```

The command should output an error if downloading this data would exceed the available space in the staging area (including ongoing downloads). For example if
- the staging area is 500GB
- there are 400GB of completed downloads (whose deals have not been added to a sector)
- there are in-progress downloads for deals whose total data size is 80GB
- ie total "tagged" space in the download area is 400GB + 80GB = 480GB
- the command should return an error if the user attempts to download a 32GB file

Note that if the size of the data to download was not specified as part of the deal, boostd should do an HTTP HEAD request to get the size from the client. If it's not possible to get the size, boostd should download the data anyway.

The command will process the deal in the same way as if it were an online deal:
- download the data from the client
- perform commp on the data
- publish the deal
- add the deal data to a sector
- index and announce the deal

The advantage of using `boostd download-data` instead of doing on online deal is that the SP can control when to download the data (online deals start downloading immediately).

### Related commands

SPs will likely want to use this command in scripts, so it would also be useful for us to document how to check the remaining space in the download staging area:
```
$ curl 'http://localhost:8080/graphql/query' \
--data-raw '{"query":"query { storage { Staged Transferred Pending Free MountPoint }}"}' | jq

{
"data": {
"storage": {
"Staged": {
"__typename": "BigInt",
"n": "0"
},
"Transferred": {
"__typename": "BigInt",
"n": "25123893565"
},
"Pending": {
"__typename": "BigInt",
"n": "7966307422"
},
"Free": {
"__typename": "BigInt",
"n": "611154893413"
},
"MountPoint": "/home/nonsense/.boost/incoming"
}
}
}

```

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing boostd import-data command and the online-deal processing path described in the issue. Compare their handling of download, commp, publishing, sector addition, indexing, and announcing; done means boostd download-data accepts a proposal CID or deal UUID and enforces the stated staging-space behavior. No file or test path is named, so repository search is required.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.