Data Point and API specification

Introduction

This issue is the first step of the process for the implementation of the dashboard-api, specified here: https://github.com/Joystream/dashboard-api/issues/1

The scope and necessary APIs have been extracted and inferred from the original issue but also, more details has been given to the APIs specified in the designs and comments that can be found here: https://github.com/Joystream/joystream-org/issues/650

Based on the previous information, I have arrived at the following scope for the initial version of the dashboard-api:

- Project scope:
  => Community:
    - [DYNAMIC] Twitter
      => Number of followers (+ over time)
      => Featured followers

    - [DYNAMIC] Discord
      => Number of users in server (+ over time)
      => DAU and MAU
      => Daily messages (+ over time)
      => Current events

    - [DYNAMIC] Telegram
      => Number of users in chat (+ over time)
      => DAU and MAU
      => Daily messages (+ over time)

    - [DYNAMIC] Tweetscout
      => Tweetscout score
      => Top followers

  => GitHub:
    - [AGGREGATED ACROSS ALL REPOS]
    - [DYNAMIC] Number of stars.
    - [DYNAMIC] Number of commits (+ over time).
    - [DYNAMIC] Number of issues. (EXPLANATION: Issues are both issues and PRs in the GitHub API)
    - [DYNAMIC] Number of open PRs.
    - [DYNAMIC] Number of repos.
    - [DYNAMIC] Number of followers.
    - [DYNAMIC] Number of contributors.

  => Token metrics:
    - [STATIC] Markets list:
      => [STATIC] Exchange itself + name
      => [DYNAMIC] price, volume
    - [DYNAMIC] Price (+ over time)
    - [DYNAMIC] Supply (circulating, locked, total)
    - [DYNAMIC] Volume (+ over time)
    - [STATIC] Token allocation
    - [STATIC] Release schedule
    - [DYNAMIC] Marketcap
    - [DYNAMIC] FDV
    - [DYNAMIC] Annual Inflation
    - [DYNAMIC] Minting (different types of minting, reward amount for each and USD equivalent)
    - [DYNAMIC] % of Supply in account
    - [DYNAMIC] % of Supply staked for validation
    - [DYNAMIC] APR on staking
    - [DYNAMIC] ROI (+ over time)
    - [DYNAMIC] Supply distribution (list of supply numbers as it relates to specific address milestones)

  => Traction:
    - [DYNAMIC] # of Content Creators (+ over time)
    - [DYNAMIC] # of members (+ over time)
    - [DYNAMIC] # of Videos uploaded (+ over time)
    - [DYNAMIC] # of Comments + reactions (+ over time)
    - [DYNAMIC] # of Video NFTs (+ over time)
    - [DYNAMIC] value of sold NFTs (+ over time)
    - [DYNAMIC] # of NFT sales (+ over time)
    - [DYNAMIC] # of followers across all creators in total (+ over time)
    - [DYNAMIC] Top creators list
    - [DYNAMIC] # of Daily Active Accounts (+ over time)
    - [DYNAMIC] Chain metrics (# of transactions, # of holders, block number, average block time)

  => Project comparison:
    - [STATIC] General data
    - [STATIC] FDV's for all projects

  => Team:
    - [STATIC] What is a council?
    - [DYNAMIC?] Council plan.
    - [DYNAMIC] Council members data (handle, connected socials, times served, total amount earned)
    - [DYNAMIC] Council Budget (+ past budgets)

    - [STATIC] What are working groups?
    - [DYNAMIC?] WG Plan
    - [DYNAMIC] Workers data (WG -> # of workers, budget, openings)
    - [DYNAMIC] Total Budget of all WGs (+past budgets(
    - [DYNAMIC] WG Leads data (handle, connected socials, times served, total amount earned, name of wg)

    - [STATIC] Who is JSG?
    - [STATIC] JSG members.

  => Project roadmap:
    - [STATIC] List of roadmap items (static cause they will be accessible from the website repo).

In front of all of the potential data points, I've gone ahead and classified them into dynamic and static data points. The dynamic data points are ones that will find themselves in the API and the static ones will be statically integrated into the final dashboard page.

Code

Community

Twitter

import { TwitterApi, TwitterApiV2Settings } from "twitter-api-v2";

const twitterClient = new TwitterApi(
  {
    appKey: "",
    appSecret: "",
    accessToken: "",
    accessSecret: "",
  }
);
const readonlyClient = twitterClient.readOnly;

const user = await readonlyClient.v2.me({
  "user.fields": [
    "created_at",
    "description",
    "entities",
    "id",
    "location",
    "name",
    "most_recent_tweet_id" as any,
    "pinned_tweet_id",
    "profile_image_url",
    "protected",
    "public_metrics",
    "url",
    "username",
    "verified",
    "verified_type",
    "withheld",
  ],
  expansions: ["pinned_tweet_id"],
});

console.log(`Number of followers: ${user.data.public_metrics?.followers_count}`);

Discord

import { REST, Routes } from "discord.js";

const BOT_TOKEN = "";
const TEST_SERVER_GUILD_ID = "";
const TEST_SERVER_GENERAL_CHANNEL_ID = "";

const dateToSnowflake = (date: Date) => {
  const discordEpoch = 1420070400000n; // Discord's epoch time
  const timestamp = BigInt(date.getTime()) - discordEpoch; // Subtract Discord's epoch time
  const snowflake = (timestamp << 22n) | 0n; // Left-shift and ensure it's an unsigned 64-bit integer
  return snowflake.toString(); // Convert to string (Snowflakes are typically represented as strings)
};

const rest = new REST({ version: "10" }).setToken(BOT_TOKEN);

// Users in a server

const data = (await rest.get(Routes.guildPreview(TEST_SERVER_GUILD_ID), {
  query: new URLSearchParams([["with_counts", "true"]]),
})) as any;

console.log(`Approximate member count: ${data.approximate_member_count}`);
console.log(`Approximate presence count: ${data.approximate_presence_count}`);

// DAU and MAU

const pruneData = (await rest.get(Routes.guildPrune(TEST_SERVER_GUILD_ID), {
  query: new URLSearchParams([
    ["days", "1"],
    // ["include_roles", "comma-separated list of roles (as snowflakes)"],
  ]),
})) as any;

console.log(`Number of inactive users in the last day: ${pruneData.pruned}`);
console.log(
  `Number of active users in the last day: ${data.approximate_member_count - pruneData.pruned}`
);

// Daily messages in a channel

const messages = (await rest.get(Routes.channelMessages(TEST_SERVER_GENERAL_CHANNEL_ID), {
  query: new URLSearchParams([
    ["limit", "100"],
    ["after", dateToSnowflake(new Date(Date.now() - 1000 * 60 * 60 * 24))],
  ]),
})) as any[];

console.log(`Number of messages in the general channel since yesterday: ${messages.length}`);

// Scheduled events

const events = (await rest.get(Routes.guildScheduledEvents(TEST_SERVER_GUILD_ID))) as any[];

for (const event of events) {
  console.log(`Event ${event.id} with name ${event.name} starts at ${event.scheduled_start_time}`);
}

Telegram

import { Api, TelegramClient } from "telegram";
import { StringSession } from "telegram/sessions";
import input from "input";

const stringSession = new StringSession(
  ""
);

const client = new TelegramClient(stringSession,0, "", {
  connectionRetries: 5,
});
await client.start({
  phoneNumber: async () => await input.text("Please enter your number: "),
  password: async () => await input.text("Please enter your password: "),
  phoneCode: async () => await input.text("Please enter the code you received: "),
  onError: (err) => console.log(err),
});

const generalChannelData = await client.invoke(
  new Api.channels.GetFullChannel({
    channel: "JoystreamOfficial",
  })
);

console.log(`Number of users in channel ${generalChannelData.fullChat.participantsCount}`);

const result = await client.invoke(
  new Api.messages.GetHistory({
    peer: "JoystreamOfficial",
    offsetDate: 1696846213,
    limit: 100,
  })
);

console.log(`Number of messages in channel since inception: ${result.count}`);

let lastMessageId = result.messages[99].id;

let messages: any[] = result.messages;

let multipleOfMessages = 1;

while (multipleOfMessages <= 2) {
  const result = await client.invoke(
    new Api.messages.GetHistory({
      peer: "JoystreamOfficial",
      offsetId: lastMessageId,
      limit: 100,
    })
  );

  messages = [...messages, ...result.messages];

  multipleOfMessages++;
  lastMessageId = result.messages[99].id;
}

const timestamp_24h = Date.now() - 1000 * 60 * 60 * 24;

const messages_24h = messages.filter((message) => message.date * 1000 > timestamp_24h);

console.log(`Number of messages in channel in the last day: ${messages_24h.length}`);

// DAU and MAU:

const activeUsers = messages_24h.map((message) => message.fromId.userId.value.toString());
const uniqueActiveUsers = [...new Set(activeUsers)];

console.log(`Number of active users in channel in the last day: ${uniqueActiveUsers.length}`);

Tweetscout

import axios from "axios";

const API_KEY = "";
const BASE_URL = "https://api.tweetscout.io/api";

const {
  data: { score: tweetScoutScore },
} = await axios({
  method: "get",
  url: `${BASE_URL}/score/joystreamdao`,
  headers: {
    ApiKey: API_KEY,
  },
});

const { data: topFollowers } = await axios({
  method: "get",
  url: `${BASE_URL}/top-followers/joystreamdao`,
  headers: {
    ApiKey: API_KEY,
  },
});

console.log(`Tweetscout score is: ${tweetScoutScore}`);
console.log(`Top @JoystreamDao followers are:`);

for (const follower of topFollowers.sort((a: any, b: any) => b.followersCount - a.followersCount)) {
  console.log(`  - ${follower.name} with ${follower.followersCount} followers`);
}

GitHub

import { Octokit } from "octokit";

const JOYSTREAM_ORG_NAME = "Joystream";

const getNumberOfItemsFromPageNumbers = (linkString: string | undefined) => {
  const result = linkString
    ?.split(",")[1]
    ?.match(/&page=(\d+)/g)?.[0]
    .replace(/&page=(\d+)/g, "$1");

  return result ? parseInt(result) : 0;
};

const octokit = new Octokit({
  auth: "",
});

const {
  data: { public_repos, followers },
  headers,
} = await octokit.request("GET /orgs/{org}", {
  org: JOYSTREAM_ORG_NAME,
});

const { data: repoData } = await octokit.request("GET /orgs/{org}/repos", {
  org: JOYSTREAM_ORG_NAME,
  per_page: 1000,
});

const repos: string[] = repoData.map((repo: any) => repo.name);
const finalRepoInformation: any[] = [];
const allContributors: any = {};

for (const repo of repos) {
  const { data } = await octokit.request("GET /repos/{username}/{repo}", {
    username: JOYSTREAM_ORG_NAME,
    repo,
  });
  const { headers: pullRequestHeaders } = await octokit.request("GET /repos/{owner}/{repo}/pulls", {
    owner: JOYSTREAM_ORG_NAME,
    repo,
    per_page: 1,
    page: 1,
  });
  const { headers: commitHeaders } = await octokit.request("GET /repos/{username}/{repo}/commits", {
    username: JOYSTREAM_ORG_NAME,
    repo,
    per_page: 1,
    page: 1,
  });
  const { data: contributorData } = await octokit.request(
    "GET /repos/{owner}/{repo}/contributors",
    {
      owner: JOYSTREAM_ORG_NAME,
      repo,
      per_page: 5000,
    }
  );

  const numberOfPullRequests = getNumberOfItemsFromPageNumbers(pullRequestHeaders.link);

  finalRepoInformation.push({
    name: repo,
    numberOfStars: data.stargazers_count,
    numberOfCommits: getNumberOfItemsFromPageNumbers(commitHeaders.link),
    numberOfOpenIssues: data.open_issues_count - numberOfPullRequests,
    numberOfPullRequests,
    numberOfContributors: contributorData.length,
  });

  contributorData.forEach((contributor) => {
    if (contributor.login) allContributors[contributor.login] = contributor;
  });
}

console.log(`Number of public repos: ${public_repos}`);
console.log(`Number of followers: ${followers}`);
console.log(`Number of contributors: ${Object.keys(allContributors).length}`);
console.log(
  `Remaining repo data (stars, commit, issue, PR, contributor info): ${JSON.stringify(
    finalRepoInformation
  )}`
);

Token

import axios from "axios";
import { GraphQLClient, gql } from "graphql-request";

const CMC_API_KEY = "";
const JOYSTREAM_CMC_ID = "6827";
const URLs = [
  "https://pro-api.coinmarketcap.com/v2/cryptocurrency/info?slug=joystream",
  `https://pro-api.coinmarketcap.com/v1/cryptocurrency/quotes/latest?id=${JOYSTREAM_CMC_ID}&convert=USD`,
  `https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest?convert=USD`,
  // For all of the exchanges that have joystream:
  `https://pro-api.coinmarketcap.com/v1/exchange/map?crypto_id=${JOYSTREAM_CMC_ID}`,
];

// *==================================================*
// TOKEN METRICS
// *==================================================*

// MEXC (fetching price and volume information)

const {
  data: { data },
} = await axios.get(`https://www.mexc.com/open/api/v2/market/ticker?symbol=joystream_usdt`, {
  headers: {
    "Content-Type": "application/json",
    "api-key": "",
    "api-secret": "",
  },
});

console.log(`Price: ${data[0].last}`);
console.log(`Volume (in JOY): ${data[0].volume}`);
console.log(`Volume (in USD): ${data[0].amount}`);

// BITGET (fetching price and volume information)

const { data: bitgetData } = await axios.get(
  `https://api.bitget.com/api/v2/spot/market/tickers?symbol=JOYUSDT`,
  {
    headers: {
      "Content-Type": "application/json",
    },
  }
);

const { lastPr, baseVolume, usdtVolume } = bitgetData.data[0];

console.log(`Price: ${lastPr}`);
console.log(`Volume (in JOY): ${baseVolume}`);
console.log(`Volume (in USD): ${usdtVolume}`);

// Rest of the info

const {
  data: { data: generalBlockchainData },
} = await axios.post("https://joystream.api.subscan.io/api/scan/metadata", {
  Headers: {
    "Content-Type": "application/json",
    "X-API-Key": "",
  },
});
const {
  data: { data: currencyData },
} = await axios.post(`https://joystream.api.subscan.io/api/scan/token`, {
  Headers: {
    "Content-Type": "application/json",
    "X-API-Key": "",
  },
});

const {
  detail: { JOY },
} = currencyData;
console.log(`Price: ${JOY.price}`);
console.log(`Circulating supply: ${JOY.available_balance}`);
console.log(`Locked supply: ${JOY.locked_balance}`);
console.log(`Total supply: ${JOY.total_issuance}`);
console.log(`Inflation: ${JOY.inflation}`);
console.log(`Value staked for validation: ${JOY.validator_bonded}`);
console.log(`Value staked for nomination: ${JOY.nominator_bonded}`);

const fetchCMCInformation = async (url: string) => {
  return await axios.get(url, {
    headers: {
      Accepts: "application/json",
      "X-CMC_PRO_API_KEY": CMC_API_KEY,
    },
  });
};

const {
  data: { data: priceAndVolumeData },
} = await fetchCMCInformation(URLs[1]);

console.log(priceAndVolumeData);
console.log(`Price: ${priceAndVolumeData[JOYSTREAM_CMC_ID].quote.USD.price}`);
console.log(`Volume (in USD): ${priceAndVolumeData[JOYSTREAM_CMC_ID].quote.USD.volume_24h}`);
console.log(`Market cap: ${priceAndVolumeData[JOYSTREAM_CMC_ID].self_reported_market_cap}`);
console.log(`FDV: ${priceAndVolumeData[JOYSTREAM_CMC_ID].quote.USD.fully_diluted_market_cap}`);
console.log(`ROI (1h): ${priceAndVolumeData[JOYSTREAM_CMC_ID].quote.USD.percent_change_1h}`);
console.log(`ROI (24h): ${priceAndVolumeData[JOYSTREAM_CMC_ID].quote.USD.percent_change_24h}`);
console.log(`ROI (7d): ${priceAndVolumeData[JOYSTREAM_CMC_ID].quote.USD.percent_change_7d}`);
console.log(`ROI (30d): ${priceAndVolumeData[JOYSTREAM_CMC_ID].quote.USD.percent_change_30d}`);
console.log(`ROI (90d): ${priceAndVolumeData[JOYSTREAM_CMC_ID].quote.USD.percent_change_90d}`);

// APR Calculation (using chain data):
const apr =
  rewardHistory.length && !stakingInfo.total.toBn().isZero()
    ? last(rewardHistory)
        .eraReward.toBn()
        .muln(ERAS_PER_YEAR)
        .mul(validatorInfo.commission.toBn())
        .div(stakingInfo.total.toBn())
        .divn(10 ** 7) // Convert from Perbill to Percent
        .toNumber()
    : 0;

const {
  data: { data: exchangeList },
} = await fetchCMCInformation(URLs[3]);

console.log(exchangeList);

const {
  data: { data: accountData },
} = await axios({
  method: "post",
  url: "https://joystream.api.subscan.io/api/v2/scan/accounts",
  headers: {
    "Content-Type": "application/json",
    "X-API-Key": "",
  },
  data: JSON.stringify({ order_field: "balance", order: "desc", page: 0, row: 100, filter: "" }),
});

const numberOfAddresses = accountData.count;
const onePercentOfAddressesCount = numberOfAddresses * 0.01;
const numberOfPagesToFetch = Math.ceil(onePercentOfAddressesCount / 100) - 1;
let currentPageCount = 1;

let addresses: any[] = accountData.list;

while (currentPageCount <= numberOfPagesToFetch) {
  const {
    data: { data: accountData },
  } = await axios({
    method: "post",
    url: "https://joystream.api.subscan.io/api/v2/scan/accounts",
    headers: {
      "Content-Type": "application/json",
      "X-API-Key": "",
    },
    data: JSON.stringify({
      order_field: "balance",
      order: "desc",
      page: currentPageCount,
      row: 100,
      filter: "",
    }),
  });

  addresses = [...addresses, ...accountData.list];
  currentPageCount++;
}

const top100Addresses = addresses.slice(0, 100);
const top1PercentAddresses = addresses.slice(0, onePercentOfAddressesCount);

// We can also use the min_balance filter as shown in the next query to fetch
// the necessary address values for the dashboard (e.g., >$100 or >1M JOY).
// An optimized way to do it would be to fetch by the lowest value and
// do filtering via code.

const res = await axios({
  method: "post",
  url: "https://joystream.api.subscan.io/api/v2/scan/accounts",
  headers: {
    "Content-Type": "application/json",
    "X-API-Key": "",
  },
  data: JSON.stringify({
    order_field: "balance",
    order: "desc",
    page: 0,
    row: 100,
    filter: "",
    // The min_balance value here is in HAPI
    min_balance: `${100_000_000 * 10_000_000_000}`,
  }),
});

console.log(res.data.data.list);

// Minting

const client = new GraphQLClient("https://query.joystream.org/graphql");
const workersData = await client.request(`
{
  workers (orderBy:createdAt_DESC) {
    group {
      name
    }
    membership {
      handle
    }
    payouts {
      amount
      createdAt
    }
  }
  proposals (orderBy:createdAt_DESC, where: { details_json: {isTypeOf_eq: "FundingRequestProposalDetails"}, status_json: { isTypeOf_eq: "ProposalStatusExecuted"}}) {
    title
    id
    details {
      __typename
      ... on FundingRequestProposalDetails {
        destinationsList {
          destinations {
            amount
            account
          }
        }
      }
    }
  }
}
`);

// Workers budget data
console.log(workersData.workers);

// Funding/spending proposals data
console.log(workersData.proposals);

// Validator rewards data
console.log(api.query.staking.activeEra())
console.log(api.query.staking.erasValidatorReward(activeEra))

Traction

import axios from "axios";

const NO_LIMIT_NUMBER = "1000000000";

// Number of content creators
const CHANNELS_QUERY = `
channels(limit:${NO_LIMIT_NUMBER},orderBy:createdAt_DESC,where:{totalVideosCreated_gt:0}){
  id
}
`;

// Number of members
const MEMBERS_QUERY = `
memberships(limit: ${NO_LIMIT_NUMBER}) {
  handle
}
`;

// Number of videos uploaded
const VIDEOS_QUERY = `
videos(limit:${NO_LIMIT_NUMBER},orderBy:createdAt_DESC){
  id
}
`;

// Number of comments
const COMMENTS_QUERY = `
comments(limit:${NO_LIMIT_NUMBER}){
  id
}
`;

// Number of reactions
const REACTIONS_QUERY = `
videoReactions(limit:${NO_LIMIT_NUMBER}){
  id   
}
commentReactions(limit:${NO_LIMIT_NUMBER}){
  id
}
`;

// Number of video nfts
const NUMBER_OF_NFTS_QUERY = `
nftIssuedEvents(limit:${NO_LIMIT_NUMBER}){
  id
}
`;

// NFT sales value and volume
const NUMBER_OF_NFT_SALES_QUERY = `
nftBoughtEvents(limit:${NO_LIMIT_NUMBER}){
  id
  price
}
auctions(limit:${NO_LIMIT_NUMBER},where:{isCompleted_eq:true}){
    id
    topBid{
      amount
    }
}
`;

const CHANNEL_PAYMENT_EVENTS = `
channelPaymentMadeEvents(limit: ${NO_LIMIT_NUMBER}, orderBy: createdAt_DESC) {
  createdAt
  amount
}
`;

const data = await axios({
  url: "https://query.joystream.org/graphql",
  method: "post",
  data: {
    query: `
      query MyQuery {
        ${NUMBER_OF_NFTS_QUERY}
      }
    `,
  },
});

console.log(data.data.data.nftIssuedEvents.length);

// *==================================================*
// Chain Metrics (TRACTION)
// *==================================================*

const {
  data: { data: generalBlockchainData },
} = await axios.post("https://joystream.api.subscan.io/api/scan/metadata", {
  Headers: {
    "Content-Type": "application/json",
    "X-API-Key": "",
  },
});


// Number of transactions

console.log(`Number of transactions: ${generalBlockchainData.count_signed_extrinsic}`);

// Number of holders

console.log(`Number of holders: ${generalBlockchainData.count_account}`);

// Block number

console.log(`Block number: ${generalBlockchainData.finalized_blockNum}`);

// Average block time

console.log(`Average block time: ${generalBlockchainData.avgBlockTime}`);

Team

import axios from "axios";
import { GraphQLClient, gql } from "graphql-request";

const NO_LIMIT_NUMBER = "1000000000";

const hapiToJoy = (hapi: number) => {
  return hapi / 10_000_000_000;
};

const COUNCIL_QUERY = `
councilMembers(limit: 3, orderBy: updatedAt_DESC) {
  member {
    handle
    metadata {
      externalResources {
        type
        value
      }
    }
    councilCandidacies {
      id
    }
    councilMembers {
      id
      accumulatedReward
    }
  }
}
`;

const WORKERS_QUERY = `
workingGroups {
  id
  budget
  leader {
    entry {
      createdAt
    }
    membership {
      id
      handle
      externalResources {
        type
        value
      }
    }
  }
  openings {
    id
    status {
      __typename
    }
  }
  workers {
    isActive
    entry {
      createdAt
    }
    membership {
      id
      handle
    }
    payouts {
      amount
    }
  }
}
`;

// *=================================================================*
// COUNCIL MEMBERS
// *=================================================================*

const { data: councilMemberData } = await axios({
  url: "https://query.joystream.org/graphql",
  method: "post",
  data: {
    query: `
      query MyQuery {
        ${COUNCIL_QUERY}
      }
    `,
  },
});

const {
  data: { councilMembers },
} = councilMemberData;

const currentCouncilMembers = councilMembers.map((cm: any) => ({
  handle: cm.member.handle,
  councilCandidacies: cm.member.councilCandidacies.length,
  timesServed: cm.member.councilMembers.length,
  accumulatedRewardInJOY: cm.member.councilMembers.reduce(
    (acc: number, cm: any) => acc + hapiToJoy(Number(cm.accumulatedReward)),
    0
  ),
  socials: cm.member.metadata.externalResources,
}));
const currentCouncilMemberHandles = councilMembers.map((cm: any) => cm.member.handle);

// Council members data:

console.log(`Current council members: ${currentCouncilMemberHandles.join(", ")}`);
console.log(currentCouncilMembers);

// Council budget (from rpc):
console.log(api.query.council.budget())

// *=================================================================*
// WORKING GROUPS
// *=================================================================*

const query = gql`{
  ${WORKERS_QUERY}
}`;

const client = new GraphQLClient("https://query.joystream.org/graphql");
const { workingGroups } = await client.request(query);

const workingGroupsData = workingGroups.reduce((acc: any, wg: any) => {
  acc[wg.id] = {
    currentWorkers: wg.workers.filter((w: any) => w.isActive).length,
    openings: wg.openings.filter((o: any) => o.status.__typename === "OpeningStatusOpen").length,
    budget: hapiToJoy(Number(wg.budget)),
  };

  return acc;
}, {});

console.log(workingGroupsData);
console.log(
  `Total WG budget: ${Object.values(workingGroupsData).reduce(
    (acc: number, wg: any) => acc + wg.budget,
    0
  )}`
);

const workingGroupsLeadsData = workingGroups.reduce((acc: any, wg: any) => {
  const servedData = wg.workers.filter((w: any) => wg.leader.membership.id === w.membership.id);

  acc[wg.id] = {
    memberHandle: wg.leader.membership.handle,
    socials: wg.leader.membership.externalResources,
    timesServed: servedData.length,
    amountEarned: servedData.reduce((totalEarnedAmount: number, newServedDataInstance: any) => {
      const servedInstanceEarnedAmount = newServedDataInstance.payouts.reduce(
        (acc: number, p: any) => acc + hapiToJoy(Number(p.amount)),
        0
      );

      return totalEarnedAmount + servedInstanceEarnedAmount;
    }, 0),
  };

  return acc;
}, {});

console.log(workingGroupsLeadsData);

API Rate Limiting

The following list explains the amount of rate limiting for each of the APIs above. Some have rate limiting for specific endpoints and some are global.

The list:

• Twitter
  • GET /2/users/me - 25req/24hours
• Discord: 50req/sec
• Telegram: 30req/sec
  • Not clearly specified in their documentation but information online seems to indicate it's up to 30req/sec.
  • Sources #1 and #2
• Tweetscout: 10,000req
• GitHub: 5000req/hour
  • Note: The query we run for GitHub is intense and considerably eats away at this limit. Around ~270 requests per query per latest testing.
• CMC: 30req/minute but there is a hard monthly limit of 10,000req.
  • For this we want to use a separate API key from the one used on the status server.
• MEXC: 20req/sec
• Bitget: 20req/sec
• SubScan: 5req/sec
• Joystream Graphql API: No limits afaik.

Questions

These are the questions that arose during research which I wasn't sure about how to resolve:

• Community:
  • Twitter:
    • What are "featured" followers?
    • Note: Twitter/X heavily neutered their API where in the free tier you can only access super basic information about your own profile. We should carefully weigh what we want to take from them as any of the paid access levels are very expensive (>$100).
    • Edit: We can get top 20 followers from Tweetscout now. It might be a way to step around the twitter API limitation on this front.
  • What other socials do we want to track?
    • Currently it's Twitter, Discord, Telegram. Tweetscout was mentioned in the issue, anything else?
• Token:
  • What is % of supply in account? (mentioned in issue that it doesn't mean anything but still in design..)
  • % of Supply staked for validation -> is this just validators or also nominators?
  • ROI data is only available up to 90 days on CMC free plan, is that enough?
    • Potential q: I took ROI to mean the change in token price over "x" time, so ROI for buying and holding JOY. Is that what was meant?
  • Minting: It is my understanding that we want to fetch creator payouts by the council here? My reasoning is that those that are being paid by jsg are paid out of pocket from already minted (pre-reserved) tokens. If that is the case, there was some discussion on discord and we came to a conclusion that it is not possible to fetch this information yet. For that, tomato created the following issue: https://github.com/Joystream/joystream/issues/4929
• Traction:
  • When talking about number of followers, are we saying number of followers on gleev?
  • For the top creators list, it seems we're taking top creators based on the number of followers. Again, followers on gleev or somewhere else?
  • What constitutes a daily active account?
• Team:
  • Council/WG plan: We can dynamically find and link to a plan but parsing the text and displaying it in the UI I don't think is realistic. The data/content is way too irregular in my opinion. Thoughts?

Notes

• Tweetscout private API is in the process of being arranged as they currently do not offer a free public variant of it.

The issue names no repository files or tests; start by reviewing dashboard-api issue #1 and joystream-org issue #650, then compare their requirements with the listed data points and TypeScript API examples. Done means an agreed, implementable initial dashboard-api scope covering the specified dynamic metrics and integrations.