Joystream SDK: RFC / draft

In this issue, I'll outline the proposed tech stack, requirements and functionality to be supported by the initial version of the Joystream SDK, based on the description here: https://github.com/Joystream/joystream/issues/4596

**I encourage everyone interested in building on top of Joystream to leave their comments and suggestions here. Any feedback at this stage is invaluable and will help shape the SDK for the maximum benefit of everyone.**

# Repository, Package Management, and Tooling

- **TypeScript Monorepo**
- **Environment-agnostic** (works in both Node.js and browser contexts)
- **High test coverage** (ideally 100%)
- Proper **semantic versioning** and **`CHANGELOG`s** for each package
- Regular **publishing to npm registry** under **`@joystream/sdk-*`**, where `*` is the SDK package name (see: [Library structure](#library-structure-packages))
- Extensive, versioned, interactive **[documentation](#documentation)**
- Striving to **limit the number and size of dependencies** (excluding `devDependencies`) to absolute minimum
- **Code examples** included in the repository
## Tooling
- **Package manager:** `yarn` (`2+`)
- **Versioning&publishing:** Changesets / Lerna
- **Code formatting & standards:** ESLint & Prettier
- **Tests:** Jest
- **CI/CD:** Github workflows
- **Documentation:** TypeDoc, Docosaurus, LLMs
- **Logging:** [`debug`](https://www.npmjs.com/package/debug) by default, but allows passing custom logger instance (e.g. [winston](https://github.com/winstonjs/winston))

## Repository structure:
```
/docs
/packages
/core
/query
/keys
...
/storage
/content
/examples
...
```

## Documentation
### Features
- Includes both a lower-level API documentation (TypeDoc) and a higher-level overview of the SDK modules with examples (Docosaurus).
- Consider using LLMs to facilitate the production of higher-level docs.
- Includes links to useful existing dev resources / guides (Joystream playground setup, Atlas and Orion guides etc.)
- Includes interactive Joystream network architecture diagram
- Some partial examples:
- https://blog.joystream.org/content/images/2022/12/joystream_stack.png
- https://github.com/Joystream/atlas/blob/master/docs/architecture.md
### Good examples:
- **Subsquid documentation:** https://docs.sqd.ai/

# Library structure (packages)

**Disclaimer:** The package and module descriptions below are **not** a final or exhaustive list of features. They are a potential way of categorizing SDK packages, along with illustrative examples of commonly repeated code patterns in existing Joystream tools (CLI, Atlas, Pioneer, Orion, YouTube Sync, Network Tests, etc.), plus a few of my own ideas.

- [`@joystream/sdk-core`](#joystreamsdk-core):
- [`@joystream/sdk-core/query`](#joystreamsdk-corequery)
- [`@joystream/sdk-core/keys`](#joystreamsdk-corekeys)
- [`@joystream/sdk-core/chain`](#joystreamsdk-corechain)
- [`@joystream/sdk-core/tx`](#joystreamsdk-coretx)
- [`@joystream/sdk-core/assets`](#joystreamsdk-coreassets)
- [`@joystream/sdk-core/roles`](#joystreamsdk-coreroles)
- **(?)** [`@joystream/sdk-core/metaprotocol`](#joystreamsdk-coremetaprotocol)
- [`@joystream/sdk-storage`](#joystreamsdk-storage)
- [`@joystream/sdk-content`](#joystreamsdk-content)

**Out of scope:**
- **(?)** `@joystream/sdk-react`
- `@joystream/sdk-governance`
- `@joystream/sdk-forum`
- ...

## `@joystream/sdk-core`

This package would include a set of core features and utilities that can be used in a wide variety of contexts, beyond just a single Joystream runtime module (like _Content_ or _Elections_). It would be the main building block of other SDK packages targeting more specific use cases.

### `@joystream/sdk-core/query`

This module would consist of utilities related to interacting with GraphQL servers provided by tools such as Hydra or Subsquid. The three main data sources supported will be:

- [Joystream Query Node](https://github.com/Joystream/joystream/tree/master/query-node) (e.g. https://query.joystream.org/graphql)
- [Orion](https://github.com/Joystream/orion) GraphQL server (e.g. https://orion.gleev.xyz/graphql)
- [Storage Squid](https://github.com/Joystream/storage-squid) GraphQL server

#### Features:
- Includes the current GraphQL schema of GraphQL servers listed above
- Supports pagination and chunking of GraphQL queries based on predefined limits
- Provides a simple, type-safe interface for executing GraphQL queries (without the need to run `graphql-codegen`, etc.)
- This should be possible with tools such as [`@genql/cli`](https://genql.dev/docs)
- Works with `ApolloClient` (for compatibility with React-based apps)

#### Existing code references:

Show references

- Multiple Joystream tools currently use a wrapper class over `ApolloClient` to implement features listed above:
- **CLI:** [`QueryNodeApi`](https://github.com/Joystream/joystream/blob/master/cli/src/QueryNodeApi.ts)
- **Network tests:** [`QueryNodeApi`](https://github.com/Joystream/joystream/blob/master/tests/network-tests/src/QueryNodeApi.ts)
- **Colossus:** [`QueryNodeApi`](https://github.com/Joystream/joystream/blob/master/storage-node/src/services/queryNode/api.ts)
- **Argus:** [`QueryNodeApi`](https://github.com/Joystream/joystream/blob/master/distributor-node/src/services/networking/query-node/api.ts)

### `@joystream/sdk-core/keys`

This module would be responsible for managing keys, signatures, and integrations with external signers.

#### Features:
- Simplifies integration with external signers and wallets:
- [WalletConnect](https://walletconnect.network/): Enables integration with SubWallet, Tangem and other mobile/hardware wallets that support the WalletConnect protocol.
- [Talisman connect](https://github.com/TalismanSociety/talisman-connect): Enables integration with multiple wallets and browser extensions like Talisman, Nova Wallet, SubWallet, Polkadot.js etc.
- Provides a wrapper over [`@polkadot/keyring`](https://www.npmjs.com/package/@polkadot/keyring) with Joystream network defaults
- Injecting dev accounts (`//Alice`, `//Bob` and possibly Joystream-specific ones like `//testing//worker//Storage/0` etc.)
- **(?)** Offline transaction signing utilities

#### Similar / related projects:
- [@polkadot-onboard](https://github.com/paritytech/polkadot-onboard/tree/main) - provides a universal interface for working with different type of wallets (e.g. Injected-wallets, Wallet-Connect, hardware-wallets)
- [@polkadot/extension-dapp](https://polkadot.js.org/docs/extension/) - similar to Talisman connect, an earlier library of this type developed by Parity.
- [injectWeb3-connect](https://github.com/WRadoslaw/injectWeb3-connect) - @WRadoslaw's fork of Talisman connect (used by Pioneer)

#### Related issues
- https://github.com/Joystream/joystream/issues/5038

#### Existing code references:

Show references

- Wallets integration:
- **Atlas:** [Wallet provider](https://github.com/Joystream/gleev/blob/main/packages/atlas/src/providers/wallet)
- **Pioneer:** [Accounts provider](https://github.com/Joystream/pioneer/tree/dev/packages/ui/src/accounts/providers/accounts)
- Offline transaction signing:
- https://github.com/Joystream/joystream/pull/5197
- **CLI:** [`SignUnsignedTxCommand`](https://github.com/Joystream/joystream/blob/b08de4fb515cdd738c014b7f84bc66d1095c908e/cli/src/commands/sign-offline/signUnsignedTx.ts)

### `@joystream/sdk-core/chain`

This module would provide a set of utilities related to retrieving, decoding and processing data from the Joystream blockchain (chain state, blocks, events, extrinsics etc.).
It would rely on `ApiPromise.derive`, `ApiPromise.query` and `ApiPromise.rpc` from `@polkadot/api` to communicate with Joystream nodes through a Websocket RPC API.

#### Features
- Calculating average blocktime / retrieving blocks in given time range
- Conversion utilities:
- between `blockHeight` and date/time
- between number of blocks and duration (in various units of time)
- Blockchain state querying utilities (e.g. processing map entries)
- Retrieving and decoding events and extrinsics from a given block
- **(?)** **Migration of `@joystream/types`:** Includes current Joystream runtime metadata and helpers for constructing, encoding and decoding Joystream runtime types

#### Existing code references

Show references

- Conversion utilities / average blocktime:
- **Pioneer:** `inBlocksDate`, `formatBlocksToDuration`, `blocksToTime`
- **Atlas:** [`useBlockTimeEstimation`](https://github.com/Joystream/atlas/blob/4416fddcf6001a6270dbfc201c1fefe619647cad/packages/atlas/src/hooks/useBlockTimeEstimation.ts)
- **Network tests** ([`Api.ts`](https://github.com/Joystream/joystream/blob/master/tests/network-tests/src/Api.ts)): `durationInMsFromBlocks`
- https://joyutils.org/
- Blockchain state querying utilities:
- **CLI:** `entriesByIds`
- Retrieving and decoding events and extrinsics from a given block:
- https://github.com/Joystream/joystream/pull/5197/files#diff-f3d5eff469075007aabef06a6213ceb8dcc5d5e1acd988395ce83f3b044b90cb
- https://github.com/Joystream/joystream/pull/5197/files#diff-8f242fe7cfa37562f96e85eea794b907cf60a199cf0f08bfcf7d8d9a8c5c2c9f
- **Atlas:** [`joystream-lib/lib`](https://github.com/Joystream/atlas/blob/1f086a392211bdca36ab687f6475b6a190443f00/packages/atlas/src/joystream-lib/lib.ts): `getChainMetadata`, `getChainConstants`, `subscribeAccountBalance`, `subscribeCurrentBlock` (...)
- [`@joystream/types`](https://github.com/Joystream/joystream/tree/master/types)

### `@joystream/sdk-core/tx`

This module would provide a set of utilities for preparing, sending and tracking transactions.

#### Features
- Extrinsic status tracking & errors:
- Allows tracking transaction/extrinsic status through all of the stages:
- Client-side validation (may include validation based on other features provided by the SDK, e.g. balance / permissions validation)
- Signer events (e.g. signing cancelled)
- RPC node validation (see: [Transaction validity](https://github.com/paritytech/substrate/blob/2951d340841523f6897e55e7a613b0fd0e1c7380/primitives/runtime/src/transaction_validity.rs#L100))
- Runtime dispatch errors and events
- Query node / Orion processing status (metaprotocol message parsing errors etc.)
- Transaction batching and parallel execution utilities
- Extracting status of individual calls in a `utility.batch` / `utility.batchAll` / `utility.forceBatch` transaction,
- Nonce caching
- **(?)** Auto-batching of transactions
- Calculating transaction costs (fees, bloat bonds, storage fee etc.)

#### Existing code references

Show references

- Extrinsic status and error handling:
- **Network tests** ([`Api.ts`](https://github.com/Joystream/joystream/blob/master/tests/network-tests/src/Api.ts)): `sendExtrinsicsAndGetResults`, `getEventDetails`, `findEvents`, `getEvent`, `getErrorNameFromExtrinsicFailedRecord` (...)
- **Atlas** [`joystream-lib/extrinsics`](https://github.com/Joystream/atlas/blob/master/packages/atlas/src/joystream-lib/extrinsics.ts), [`joystream-lib/helpers`](https://github.com/Joystream/atlas/blob/641ec805a2b47ba20a7218dc099b78e77948d157/packages/atlas/src/joystream-lib/helpers.ts)(`sendExtrinsic`, `sendExtrinsicAndParseEvents`, `parseExtrinsicEvents`, `extractExtrinsicErrorMsg`, ...),[`joystream-lib/errors`](https://github.com/Joystream/atlas/blob/1f086a392211bdca36ab687f6475b6a190443f00/packages/atlas/src/joystream-lib/errors.ts)
- **CLI** ([`ApiCommandBase`](https://github.com/Joystream/joystream/blob/master/cli/src/base/ApiCommandBase.ts)): `sendExtrinsic`, `sendAndFollowTx`, `findEvent`, `getEvent`, `getEventDetails` (...)
- **Pioneer**: [`useProcessTransaction`](https://github.com/Joystream/pioneer/blob/dev/packages/ui/src/common/hooks/useProcessTransaction.ts), [`useQueryNodeTransactionStatus`](https://github.com/Joystream/pioneer/blob/dev/packages/ui/src/common/hooks/useQueryNodeTransactionStatus.ts) hooks, [`model/JoystreamNode`](https://github.com/Joystream/pioneer/blob/dev/packages/ui/src/common/model/JoystreamNode) (`errorEvents`, `getDataFromEvent`) (...)
- Transaction costs (fees, bloat bonds etc.)
- **CLI:** [`FeeProfile`](https://github.com/Joystream/joystream/tree/master/cli/src/commands/fee-profile) commands, [`prepareAssetsForExtrinsic`](https://github.com/Joystream/joystream/blob/master/cli/src/base/UploadCommandBase.ts#L350) (storage fee / bloat bonds)
- **Atlas:** [`useFee`](https://github.com/Joystream/atlas/blob/master/packages/atlas/src/providers/joystream/joystream.hooks.ts) hook
- **Network tests** ([`Api.ts`](https://github.com/Joystream/joystream/blob/349606fee9febf60a600644403b857727b339a06/tests/network-tests/src/Api.ts)): `prepareAccountsForFeeExpenses`, `estimateTxFee`
- Nonce caching / parallel transactions:
- **Network tests:** [`Sender`](https://github.com/Joystream/joystream/blob/master/tests/network-tests/src/sender.ts) class
- Sending batch transactions and extracting results:
- **Youtube Sync:** `sendBatchExtrinsic`
- **Colossus** (`extrinsics.ts`): `updateStorageBucketsForBags`, `acceptPendingDataObjectsBatch`

### `@joystream/sdk-core/assets`

This module would provide a set of utilities for managing balances, vesting, locks and stakes.

#### Features
- Simplifies checking balances available for different purposes ([reference](https://gist.github.com/Lezek123/88b85b6af866feaa4f6b5064ce528a93))
- HAPI <=> JOY <=> USD conversion (possibly with reference to a specific point in time)
- Balance formatting utilities
- Managing stakes & locks:
- checking if locks are conflicting
- retrieving all stakes by key(s) / membership(s)
- retrieving membership's staking accounts
- checking status of a stake (unbonding / recoverable etc.)
- setting up staking accounts (addStakingAccount + confirmStakingAccount)
- **(?)** recovering stakes
- Vesting utilities:
- constructing vesting scheme based on amount, cliff percentage and period
- calculating vested amount at specific date

**Out of scope:**
- **(?)** Utilities related to managing other assets owned on Joystream:
- CRTs
- NFTs

#### Existing code references:

Show references

- Parsing balances:
- **Atlas** ([`joystream-lib/utils`](https://github.com/Joystream/atlas/blob/master/packages/atlas/src/joystream-lib/utils.ts)): `parseAccountBalance` (...)
- **Pioneer**: [`toBalances`](https://github.com/Joystream/pioneer/blob/b7f95c1a2d80c261c44378dd9c7f70b714f05000/packages/ui/src/accounts/model/toBalances.ts)
- Balance formatting:
- **CLI** ( [`Api.ts`](https://github.com/Joystream/joystream/blob/72ccc84104cfebb6d9b69137a05cc8cd6bf4638d/cli/src/Api.ts)): `formatBalance` config
- **Argus:** [`createApi`](https://github.com/Joystream/joystream/blob/f6afceccf863ca85f8b8d91f23273e887d9b1fbf/storage-node/src/services/runtime/api.ts) (`formatBalance` config)
- **Pioneer** ([`formatters`](https://github.com/Joystream/pioneer/blob/def29998e6e8aeda9c1afcaacfe8aa75e326d909/packages/ui/src/common/model/formatters.ts)): `formatJoyValue`, `formatTokenValue`
- Setting up staking account(s):
- **CLI** ([`AccountsCommandBase`](https://github.com/Joystream/joystream/blob/1455c24d24dd838a6609b95356e0239a2fc2c89c/cli/src/base/AccountsCommandBase.ts)): `setupStakingAccount`
-
- HAPI <=> JOY <=> USD conversion:
- **Atlas** ([`joystream-lib/utils`](https://github.com/Joystream/atlas/blob/master/packages/atlas/src/joystream-lib/utils.ts)): `hapiBnToTokenNumber`, `tokenNumberToHapiBn`, `HAPI_TO_JOY_RATE_BN` (...)
- **CLI:** [`FeeProfileCommandBase`](https://github.com/Joystream/joystream/blob/f6afceccf863ca85f8b8d91f23273e887d9b1fbf/cli/src/base/FeeProfileCommandBase.ts): `asJoy`, `asUsd`
- **Network tests** ([`Utils`](https://github.com/Joystream/joystream/blob/349606fee9febf60a600644403b857727b339a06/tests/network-tests/src/utils.ts)) `joy`
- https://joyutils.org/
- Stakes and locks
- **Pioneer:** [lockTypes](https://github.com/Joystream/pioneer/blob/dev/packages/ui/src/accounts/model/lockTypes.ts) (types of locks / stakes, conflicting stakes etc.), [`hooks`](https://github.com/Joystream/pioneer/blob/be37baa6a8708bd97531bed3b8d27c2630a567ff/packages/ui/src/accounts/hooks): `useGroupLocks`, `useHasRequiredStake`, `useStakingAccountsLocks`, `useStakingAccountStatus` (...)
- **CLI** ([`Api.ts`](https://github.com/Joystream/joystream/blob/72ccc84104cfebb6d9b69137a05cc8cd6bf4638d/cli/src/Api.ts)): `fetchStake`, `nonRivalrousLocks`, `isLockRivalrous`, `allStakingLedgers` (...)

### `@joystream/sdk-core/roles`

This module would provide a set of utilities related to roles, permissions, authorization and access control on the Joystream blockchain.

#### Features

- Filter keys/memberships by role(s)
- Get roles by key(s) / membership(s)
- Get available contexts based on action (example: channel creation / deletion)
- Get permissions based on role/roles
- Roles and permissions tree (see [Data model](#data-model))
- Basic validation if user has sufficient permissions for a given action

#### Data model
- A **Role** can have a *parent* role (for example: *MemberController* is a parent role of *ChannelCollaborator*)
- Some roles may have an associated entity / set of entities (for example *MemberController* role is associated with *Member* entity)
- Each role can have an associated set of permissions:
- Some roles, like _MemberController_, have a static set of permissions
- Other roles, like _ChannelCollaborator_ or _CuratorGroupMember_ can have fine-grained permissions assigned individually

Expand to show some examples

```
MemberController {
memberId
permissions {
MemberRemark
CreateChannel
CreateProposal
...
}
}

MemberRoot {
memberId
permissions {
UpdateControllerAccount
UpdateRootAccount
}
}

Worker {
workerId
workingGroup
permissions {
UpdateRewardAccount
...
}
}

CuratorGroupMember {
parent: Worker
permissions {
...
}
}

ChannelOwner {
parent: MemberController
channelId
permissions {
AddVideo
UpdateChannelMetadata
UpdateChannelAssets
...
}
}
```

#### Existing code references:

Show references

- **Network tests** ([`Api.ts`](https://github.com/Joystream/joystream/blob/349606fee9febf60a600644403b857727b339a06/tests/network-tests/src/Api.ts)): `getLeadRoleKey`, `getLeaderStakingKey`, `getMemberSigners`, `getWorkerRoleAccounts`
- **CLI:** `getRequiredMemberContext`, `getRequiredWorkerContext`, `getRequiredLeadContext`, `getCuratorContext`, `getForumModeratorContext`, `isModeratorWithRequiredPermission`, `getChannelManagementActor`, `getModerationActionActor`, `getCategoryManagementActor`, `getContentActor`, `getChannelOwner` (...)
- **Pioneer:** `useWorker`, `useMyMemberships`, `useRoleAccount`, `useMyRoleIds`, `useMyWorkers`, `useAllMemberRoles` (...)

### `@joystream/sdk-core/metaprotocol`

This module may provide Joystream metaprotocol (protobuf) related utilities migrated from [`@joystream/metadata-protobuf`](https://github.com/Joystream/joystream/tree/master/metadata-protobuf) (ie. constructing, validating, encoding and decoding metaprotocol messages)

## `@joystream/sdk-storage`

This package would simplify interactions with Joystream network storage & distribution infrastructure.
### Features
- Retrieving a list of storage / distributor nodes (endpoints, location metadata, ping (optionally), supported buckets, status etc.)
- Retrieving an url to an assets by id (with the possibility of choosing the most optimal one according to ping / location)
- Downloading data objects by id:
- Argus / Colossus endpoints can either be derived or provided explicitly,
- retry mechanism,
- allows handling events like individual Argus/Colossus node errors, switching to a different endpoint etc.
- multiple download targets possible: into file, into stream, into buffer etc.
- Uploading data objects by id (assumes existence on chain):
- Argus / Colossus endpoints can either be derived or provided explicitly,
- retry mechanism,
- multiple sources supported: from file, from stream, from buffer
- Calculating multihash of a file
- Storage bags and buckets utilities:
- Parsing BagId/BucketId to/from string
- ...
### Existing code references

Show references

- Retrieving a list of Argus / Colossus operators:
- **Atlas:** [`OperatorsContextProvider`](https://github.com/Joystream/atlas/blob/fc1e3a3c97873b5fc3db98bf1692ca03a6d23a0b/packages/atlas/src/providers/assets/assets.provider.tsx)
- **CLI:** [`QueryNodeApi`](https://github.com/Joystream/joystream/blob/e36fd92b87a4cbbb73dda7e88253e00ff33af25f/cli/src/QueryNodeApi.ts) `storageNodesInfoByBagId`, `storageBucketsForNewChannel`, `distributionBucketsForNewChannel`
- **Colossus:** [`getAllBuckets`](https://github.com/Joystream/joystream/blob/6fcd6a739fd3982b6538bb086107a8a949b5a08c/storage-node/src/services/sync/storageObligations.ts)
- **Argus:** [`NetworkingService`](https://github.com/Joystream/joystream/blob/d83b4e93c1726564665af53d863e5147aead1846/distributor-node/src/services/networking/NetworkingService.ts): `checkActiveStorageNodeEndpoints`, `getDataObjectActiveDistributorsSet`, `prepareStorageNodeEndpoints`, [`QueryNodeApi`](https://github.com/Joystream/joystream/blob/c7b8d46e18fb1185decde7867cc7370a40dbb7bf/distributor-node/src/services/networking/query-node/api.ts): `getDistributionBucketsWithObjectsByIds`, `getDistributionBucketsWithObjectsByWorkerId`, `getActiveStorageBucketOperatorsData`
- **Orion** ([`AssetsResolver`](https://github.com/Joystream/orion/blob/05fb302f6e0560dee10e52a77b071603109ff07c/src/server-extension/resolvers/AssetsResolver/index.ts)): `DistributionBucketsCache`
- Downloading data objects:
- **Colossus:** [`DownloadFileTask`](https://github.com/Joystream/joystream/blob/7b7bf268c07280639ac14c4434508d5df83751a7/storage-node/src/services/sync/tasks.ts)
- **Argus** ([`NetworkingService`](https://github.com/Joystream/joystream/blob/d83b4e93c1726564665af53d863e5147aead1846/distributor-node/src/services/networking/NetworkingService.ts)): `downloadJob`
- **Network tests:** [`ColossusApi`](https://github.com/Joystream/joystream/blob/ec6a2788eb731901a48deb665ff084d8420404fc/tests/network-tests/ColossusApi.ts): `fetchAssetAsBuffer`
- Resolving asset url(s):
- **Atlas:** [`useGetAssetUrl`](https://github.com/Joystream/atlas/blob/8a105a330b6186f804a35253847fea2875584768/packages/atlas/src/hooks/useGetAssetUrl.ts)
- **Orion:** [`getAssetUrls`](https://github.com/Joystream/orion/blob/05fb302f6e0560dee10e52a77b071603109ff07c/src/server-extension/resolvers/AssetsResolver/utils.ts)
- [https://assets.joyutils.org](https://assets.joyutils.org/video/123566)
- Uploading data objects:
- **CLI** ([`UploadCommandBase`](https://github.com/Joystream/joystream/blob/b32d4d0bb53dfb5d4ea80804a85b3b3b58a65776/cli/src/base/UploadCommandBase.ts)): `uploadAsset`, `uploadAssets`
- **Atlas**: [`useStartFileUpload`](https://github.com/Joystream/atlas/blob/0434c346afe6b07b6ad6b894fadbcaa89d5e0bab/packages/atlas/src/providers/uploads/uploads.hooks.ts) hook
- **Youtube sync** ([`StorageNodeApi`](https://github.com/Joystream/youtube-synch/blob/f478c48b86a485646475fed37f1c4e68212c5f8c/src/services/storage-node/api.ts)): `upload` / `uploadVideo`
- Calculating file multihash:
- **CLI** ([`UploadCommandBase`](https://github.com/Joystream/joystream/blob/b32d4d0bb53dfb5d4ea80804a85b3b3b58a65776/cli/src/base/UploadCommandBase.ts)): `calculateFileHash`
- **Argus:** [`ContentHash`](https://github.com/Joystream/joystream/blob/d42f98bc029808c0d62c02dc6880a4e338a1aa96/distributor-node/src/services/crypto/ContentHash.ts)
- **Colossus:** [`hashFile`](https://github.com/Joystream/joystream/blob/948a8883adcbe7a11d760da68823115676cd5a94/storage-node/src/services/helpers/hashing.ts) (used by `util:multihash` command)
- **Youtube sync:** [`computeFileHashAndSize`](https://github.com/Joystream/youtube-synch/blob/c8f310df1cd02f3fc1282880fc6f47a99640dd75/src/utils/hasher.ts)
- **Atlas:** [`computeFileHash`](https://github.com/Joystream/atlas/blob/d8fa95bee2457846b0529466adfc6b9c18349fa9/packages/atlas/src/utils/hashing/worker.ts)
- Storage bags and buckets utils:
- **Argus:** [`BagIdParserService`](https://github.com/Joystream/joystream/blob/5786a19467f72bc60c182754341032ff2285fa4e/distributor-node/src/services/parsers/BagIdParserService.ts), [`BucketIdParserService`](https://github.com/Joystream/joystream/blob/3d4f00b4290492a30486fd1aaeda1732f44cd563/distributor-node/src/services/parsers/BucketIdParserService.ts)
- **Colossus:** [`BagIdParser`](https://github.com/Joystream/joystream/blob/5786a19467f72bc60c182754341032ff2285fa4e/storage-node/src/services/helpers/bagTypes.ts)

#### Related links
- https://joystream.notion.site/SDK-Storage-e98130394a88494080509d585646600d

### `@joystream/sdk-content`

This package would simplify common tasks involving Joystream's content module (channels, videos etc.).

### Features
- Simplifies creation / updating of channels and videos:
- Provides an interface similar to CLI's (without the need to run CLI programatically):
- Provides a single abstraction over a complex, multi-step process (extrinsic data & channel/video metadata preparation, sending & processing extrinsic, uploading assets etc.)
- **(?)** Allows auto-derivation of some metadata fields (like video length, codecs etc.)
- Supports batching of channel / video creation operations
- Allows handling events and errors related to transaction processing, uploading etc.
- Supports `AppAction` wrapping
- Supports issuing video NFTs
- Retrieval of video / channel assets
- Video / channel removal (with all associated assets)
- Channel payout utils:
- constructing and verifying merkle proofs
- claiming rewards
- **(?)** Channel reward account from channel id (mirroring runtime encoding impl.)
- Channel payment history (nft revenue, crt sales, tips, channel payments, direct payments etc.)
- Direct channel payments
- Known licenses
- **(?)** Orion authentication cryptographic utilities

**Out of scope:**
- **(?)** YPP API features
- **(?)** Content moderation features
- **(?)** NFT utils
- **(?)** CRT utils

### Existing code references:

Show references

- Creating / updating channels and videos:
- **CLI:** [`createChannel`](https://github.com/Joystream/joystream/blob/8f29d433816b84e468a9251a3ba99a037d228ef8/cli/src/commands/content/createChannel.ts), [`createVideo`](https://github.com/Joystream/joystream/blob/8f29d433816b84e468a9251a3ba99a037d228ef8/cli/src/commands/content/createVideo.ts), [`updateChannel`](https://github.com/Joystream/joystream/blob/8f29d433816b84e468a9251a3ba99a037d228ef8/cli/src/commands/content/updateChannel.ts), [`updateVideo`](https://github.com/Joystream/joystream/blob/627caa325e90a6b98f51d0a0905aa7682db9dfb5/cli/src/commands/content/updateVideo.ts) commands
- **Network tests** [`Api.ts`](https://github.com/Joystream/joystream/blob/349606fee9febf60a600644403b857727b339a06/tests/network-tests/src/Api.ts): `createMockChannel`, `createMockVideo`, [`CreateChannelsAndVideosFixture`](https://github.com/Joystream/joystream/blob/f859d392f5c49814b3a5cf9a714da9100a32c8f8/tests/network-tests/src/fixtures/content/createChannelsAndVideos.ts)
- **Atlas:** [`useChannelForm`](https://github.com/Joystream/atlas/blob/e457a9102f0b48e750801b770039c12bcdaff689/packages/atlas/src/hooks/useChannelForm.ts), [`useChannelFormSubmit`](https://github.com/Joystream/atlas/blob/bdfd8ae1ee820de7e0c1768d34547e42fd084fdc/packages/atlas/src/hooks/useChannelFormSubmit.ts), [`useVideoForm`](https://github.com/Joystream/atlas/blob/63e2fb7980e684177e8cd6eccb70d4f989ed35b4/packages/atlas/src/hooks/useVideoForm.ts), [`useHandleVideoWorkspaceSubmit`](https://github.com/Joystream/atlas/blob/ec76f5399d20747b617c2bb8b0076d937c0ade80/packages/atlas/src/views/studio/VideoWorkspace/VideoWorkspace.hooks.ts), [joystream-lib/extrinsics](https://github.com/Joystream/atlas/blob/8a105a330b6186f804a35253847fea2875584768/packages/atlas/src/joystream-lib/extrinsics.ts): `createVideo`, `createChannel` etc.
- **Youtube sync:** [`JoystreamClient`](https://github.com/Joystream/youtube-synch/blob/f478c48b86a485646475fed37f1c4e68212c5f8c/src/services/runtime/client.ts): `createVideoTx` (...)
- Deleting videos / channels and their assets
- **CLI:** [`deleteChannel`](https://github.com/Joystream/joystream/blob/5786a19467f72bc60c182754341032ff2285fa4e/cli/src/commands/content/deleteChannel.ts), [`deleteVideo`](https://github.com/Joystream/joystream/blob/5786a19467f72bc60c182754341032ff2285fa4e/cli/src/commands/content/deleteVideo.ts), [`removeChannelAssets`](https://github.com/Joystream/joystream/blob/20e5d597c290bbf83f789019207c3e5d45240732/cli/src/commands/content/removeChannelAssets.ts) etc.
- **Atlas:** [`useDeleteVideo`](https://github.com/Joystream/atlas/blob/641ec805a2b47ba20a7218dc099b78e77948d157/packages/atlas/src/hooks/useDeleteVideo.ts),
- Channel payouts:
- **JoystreamJS lib:** [`channelPayouts`](https://github.com/Joystream/joystream/blob/01783329dfefdcf4098c040aadf5d1dad8e7fcc3/joystreamjs/src/content/channelPayouts.ts)
- Detecting file metadata:
- **CLI:** `getVideoFileMetadata`
- **Argus:** ([`ContentService`](https://github.com/Joystream/joystream/blob/48b1b9c51db5b32b4b7cb5375856df352eee8f86/distributor-node/src/services/content/ContentService.ts)): `detectMimeType`
- **Colossus:** [`getFileInfo`](https://github.com/Joystream/joystream/blob/d5f021a0661aad1cb4459862f9f120ebd04583b9/storage-node/src/services/helpers/fileInfo.ts)
- Channel payments history:
- **Atlas:** [`useChannelPaymentsHistory`](https://github.com/Joystream/atlas/blob/8a105a330b6186f804a35253847fea2875584768/packages/atlas/src/views/studio/MyPaymentsView/PaymentsTransactions/PaymentTransactions.hooks.ts)
- Direct channel payments:
- **CLI:** [`directChannelPayment`](https://github.com/Joystream/joystream/blob/e36fd92b87a4cbbb73dda7e88253e00ff33af25f/cli/src/commands/content/directChannelPayment.ts)
- Known licenses:
- **Atlas:** [`knownLicenses.json`](https://github.com/Joystream/atlas/blob/5685f996f388f258cc12baaaed07e8f051ae49a2/packages/atlas/src/data/knownLicenses.json)
- **`@joystream/metadata-protobuf`**: [`KnownLicenses.json`](https://github.com/Joystream/joystream/blob/bdda285859fae414f2cfd62bfae8a63ef16d60bf/metadata-protobuf/src/KnownLicenses.json)
- Orion authentication cryptographic utilities:
- **Atlas:** [`auth.helpers.ts`](https://github.com/Joystream/atlas/blob/37e9652e438b3e75629378d237d687c3be603caf/packages/atlas/src/providers/auth/auth.helpers.ts)
- **Orion:** [`auth-server/tests/common.ts`](https://github.com/Joystream/orion/blob/ce06d4b179da0ea598cb696a2bc7fdfd6d92c99e/src/auth-server/tests/common.ts): `prepareEncryptionArtifacts`, `decryptSeed`, `signedAction` (...)