Giza integration tests: Distributor node: Cache eviction

Preconditions:

• A running Joystream node
• A running query-node
• A running storage node which accepts channel assets and has sufficient limits set to handle the scenario
• A running distributor node which accepts channel assets and has sufficient limits set to handle the scenario

Flow parameters:

• lg - lowest (best) data object initial LRU-SP cache group
• hg - highest (worst) data object initial LRU-SP cache group

Other variables:

• ts - total created data objects size, always equal to [2^lg + 2^(lg+1) + ... + 2^hg]
• gc - CACHE_GROUPS_COUNT distributor node constant
• sl - distributor node's storage limit

Constraints:

• hg >= lg + 2
• lg >= 2
• sl >= ts

Part 1: Movement between cache groups

1. Check the cache limit in the distributor node using /status endpoint. Populate the distributor node cache with data objects of total size equal to sl - ts.
2. For each group i between lg and hg upload 3 data objects (let's call them {i}-0, {i}-1 and {i}-2) of size 2^i to storage
3. Let t1 be the timestamp taken at this setp (in seconds)
4. Request all created objects from distributor node (GET request)
5. Wait at least one minute or until all objects are confirmed to be fully cached by the distributor node (whichever takes longer)
6. For each group i between lg and hg (from lowest to highest):
   1. Send 1 request to retrieve object {i}-1
   2. Send 3 requests to retrieve object {i}-2
   3. Wait at least 1 minute or until all responses are recieved (whichever takes longer)
7. Let t2 be the timestamp at this step (in seconds)
8. Wait at least ceil( (t2-t1)/60 ) minutes. This will make sure the elapsed parameter of LRU cache items (e) won't affect the expected result, because the ratio of highest e to lowest e will become < 2
9. We expect the cache structure to be the following:

• Group lg-2:

  • Each item has (s/p) = 2^(lg-2)
    LRU set:
    • object {lg}-2

• Group lg-1:

  • Each item has: (s/p) = 2^(lg-1)
    LRU set:
    • object {lg+1}-2, accessed at least 1 minute before:
    • object {lg}-1

• Group {lg}:

  • Each item has: (s/p) = 2^(lg)
    LRU set:
    • object {lg+2}-2, accessed at least 1 minute before:
    • object {lg+1}-1, accessed at least 1 minute before:
    • object {lg}-0
      ...

• Group {hg-2}:

  • Each item has: (s/p) = 2^(hg-2)
    LRU set:
    • object {hg}-2, accessed at least 1 minute before:
    • object {hg-1}-1, accessed at least 1 minute before:
    • object {hg-2}-0

• Group {hg-1}:

  • Each item has: (s/p) = 2^(hg-1)
    LRU set:
    • object {hg}-1, accessed at least 1 minute before:
    • object {hg-1}-0

• Group {hg}:

  • Each item has: (s/p) = 2^(hg)
    LRU set:
    • object {hg}-0

  Note that each time cache eviction is triggered:

  1. The last item from the LRU set in each group is chosen (least recently accessed item)
  2. We calculate the cost of each chosen item (where the cost is: s/p*e)
  3. We evict item with highest cost

  Because in step 9. we mitigated the effect of e for the purpose of this test, we can assume the cost of each item is just s/p.

  Given items {a}-{b} and {c}-{d} we know that {a}-{b} will be dropped before {c}-{d}, if:

  • a-b > c-d or,
  • a-b == c-d and b < d

  We sort all items accordingly to the rules above (from first-dropping to last-dropping).

11. For each item {i} in the set created in step 10.:
    1. Make sure item {i} is cached by distributor node (use HEAD request to verify)
    2. Create another data object with size equal to the size of item {i} and upload it to storage
    3. Request the created object from distributor node
    4. Make sure the request succeeds and item {i} is dropped from the cache (use HEAD request to verify)

Part 2: Dropping order depending on elapsed parameter

1. Upload 1 data object of size 2^(lg+1) (let's call it A)
2. Upload 2 data objects of size 2^(lg) (let's call them B and C)
3. Request all objects (A-C) from distributor node using GET requests
4. Wait until the objects are fully cached
5. Request object C from distributor node using GET request
6. Wait 4 minutes
7. Request objects A and B from distributor node using GET requests (can be requested simultaneously)
8. We expect the cache structure to be the following:

• Group lg-1:

  • Each item has: (s/p) = 2^(lg-1)
    LRU set:
    • object B, elapsed = ~1 minute
    • object C, elapsed = ~5 minutes

• Group lg:

  • Each item has: (s/p) = 2^(lg)
    LRU set:
    • object A, elapsed = ~1 minute

  Therefore the items should be dropped in the following order:

  1. C (because 5*2^(lg-1) > 1*2^(lg))
  2. A (because 1*2^(lg) > 1*2^(lg-1))
  3. B

9. For each item {i} in [C, A, B] set (starting from C and ending with B):

• Make sure item {i} is cached by distributor node (use HEAD request to verify)
• Create another data object with size equal to the size of item {i} and upload it to storage
• Request the created object from distributor node
• Make sure the request succeeds and item {i} is dropped from the cache (use HEAD request to verify)

Start by locating the Giza integration-test entry point for the distributor node and the /status, GET, and HEAD request helpers; no file paths are named in the issue. Run the existing distributor integration tests with a running Joystream node, query-node, storage node, and distributor node, then verify both described eviction orders and cache-presence checks pass.