progress-fd issue tracker

Followup to https://github.com/containers/bootc/pull/921 which we merged, and I was hoping to do some post-merge followup on. This issue will track some of that followup.

- [ ] docs https://github.com/containers/bootc/pull/961
- [x] integration tests https://github.com/containers/bootc/pull/1017
- [ ] size concerns

## IPC relationship concerns

This also relates to https://github.com/bootc-dev/bootc/issues/522 in that ideally the progress notifications can go seamlessly into an IPC protocol

## size concerns

In the PR we kept going back and forth on whether the progress data should require the client to keep track of state or not. @antheas was calling this "normalization". There's clear simplicity in having the protocol be stateless.

However...looking at the layer progress (one of the most important phases) in a local run here during a download, we keep accumulating and emitting more data (completed subtasks) for each layer. In this run the final `ProgressBytes` I got was 12 KiB which...is a bit more than I was expecting.

Click for (prettified) example:

{
"type": "ProgressBytes",
"apiVersion": "org.containers.bootc.progress/v1",
"task": "pulling",
"description": "Pulling Image: sha256:127c32e1418aebdd9a6cbeb6517605e11c1aa4004689e4a396307babe801f309",
"id": "sha256:127c32e1418aebdd9a6cbeb6517605e11c1aa4004689e4a396307babe801f309",
"bytesCached": 0,
"bytes": 1212857687,
"bytesTotal": 1212857687,
"stepsCached": 0,
"steps": 74,
"stepsTotal": 74,
"subtasks": [
{
"subtask": "ostree chunk",
"description": "ostree chunk: bca923230b7dde927d4f2",
"id": "bca923230b7dde927d4f2",
"bytesCached": 0,
"bytes": 434743772,
"bytesTotal": 434743772
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 7787da09b798ff4c245e5",
"id": "7787da09b798ff4c245e5",
"bytesCached": 0,
"bytes": 23673835,
"bytesTotal": 23673835
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 46b215c7346346e749145",
"id": "46b215c7346346e749145",
"bytesCached": 0,
"bytes": 35716920,
"bytesTotal": 35716920
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 5ce7d0f46f7ce0107eaab",
"id": "5ce7d0f46f7ce0107eaab",
"bytesCached": 0,
"bytes": 14600588,
"bytesTotal": 14600588
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 1ef5317b04cbde5294f5f",
"id": "1ef5317b04cbde5294f5f",
"bytesCached": 0,
"bytes": 14419888,
"bytesTotal": 14419888
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: bfb6ab72f6e960dc7c91f",
"id": "bfb6ab72f6e960dc7c91f",
"bytesCached": 0,
"bytes": 29870153,
"bytesTotal": 29870153
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 41b301e3bb75ca998bf86",
"id": "41b301e3bb75ca998bf86",
"bytesCached": 0,
"bytes": 12703409,
"bytesTotal": 12703409
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 43c7299a13755a59c3570",
"id": "43c7299a13755a59c3570",
"bytesCached": 0,
"bytes": 12076932,
"bytesTotal": 12076932
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 82fe717d3cd099e4576e7",
"id": "82fe717d3cd099e4576e7",
"bytesCached": 0,
"bytes": 16657939,
"bytesTotal": 16657939
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 46c7b3da4902f903056a0",
"id": "46c7b3da4902f903056a0",
"bytesCached": 0,
"bytes": 8510610,
"bytesTotal": 8510610
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 4e60e3792abde5311f750",
"id": "4e60e3792abde5311f750",
"bytesCached": 0,
"bytes": 8003859,
"bytesTotal": 8003859
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 0b796653effe66b6b6f4a",
"id": "0b796653effe66b6b6f4a",
"bytesCached": 0,
"bytes": 6904178,
"bytesTotal": 6904178
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: decebbcc09fcb2d10260b",
"id": "decebbcc09fcb2d10260b",
"bytesCached": 0,
"bytes": 6778508,
"bytesTotal": 6778508
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 743d8170de20edd185b8d",
"id": "743d8170de20edd185b8d",
"bytesCached": 0,
"bytes": 6637766,
"bytesTotal": 6637766
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 08daf3e7216c98939f930",
"id": "08daf3e7216c98939f930",
"bytesCached": 0,
"bytes": 5206174,
"bytesTotal": 5206174
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 760cadc63f561de092591",
"id": "760cadc63f561de092591",
"bytesCached": 0,
"bytes": 5662725,
"bytesTotal": 5662725
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 820a6b1995bf16e7c42e4",
"id": "820a6b1995bf16e7c42e4",
"bytesCached": 0,
"bytes": 11252209,
"bytesTotal": 11252209
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 15f10449a2c985e5d097e",
"id": "15f10449a2c985e5d097e",
"bytesCached": 0,
"bytes": 4403265,
"bytesTotal": 4403265
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 0473f5256cda332e113a3",
"id": "0473f5256cda332e113a3",
"bytesCached": 0,
"bytes": 2792039,
"bytesTotal": 2792039
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 854f47b2e8b312a25526c",
"id": "854f47b2e8b312a25526c",
"bytesCached": 0,
"bytes": 4506580,
"bytesTotal": 4506580
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 0f0753883ec53774d8cbf",
"id": "0f0753883ec53774d8cbf",
"bytesCached": 0,
"bytes": 4409259,
"bytesTotal": 4409259
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: b66bae9ab05760c2cae7c",
"id": "b66bae9ab05760c2cae7c",
"bytesCached": 0,
"bytes": 3817407,
"bytesTotal": 3817407
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 9168a98efc0f20fe7222c",
"id": "9168a98efc0f20fe7222c",
"bytesCached": 0,
"bytes": 5073270,
"bytesTotal": 5073270
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 635fa553888bf7fb02804",
"id": "635fa553888bf7fb02804",
"bytesCached": 0,
"bytes": 4202725,
"bytesTotal": 4202725
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 17bb2ef0dfb2941a561e6",
"id": "17bb2ef0dfb2941a561e6",
"bytesCached": 0,
"bytes": 3628297,
"bytesTotal": 3628297
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: dc34b802800e0af2bf1b4",
"id": "dc34b802800e0af2bf1b4",
"bytesCached": 0,
"bytes": 3487384,
"bytesTotal": 3487384
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 263e158a4a0ca3e3e0312",
"id": "263e158a4a0ca3e3e0312",
"bytesCached": 0,
"bytes": 3492706,
"bytesTotal": 3492706
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: a82c571fabd4910a1d1bb",
"id": "a82c571fabd4910a1d1bb",
"bytesCached": 0,
"bytes": 906047,
"bytesTotal": 906047
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 492c5bcbae66c24743149",
"id": "492c5bcbae66c24743149",
"bytesCached": 0,
"bytes": 3402775,
"bytesTotal": 3402775
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: a0230ba465f3959c97432",
"id": "a0230ba465f3959c97432",
"bytesCached": 0,
"bytes": 3148819,
"bytesTotal": 3148819
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: dbd1e7e16e61ac1780670",
"id": "dbd1e7e16e61ac1780670",
"bytesCached": 0,
"bytes": 3176990,
"bytesTotal": 3176990
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 8cd5a12c083744402472d",
"id": "8cd5a12c083744402472d",
"bytesCached": 0,
"bytes": 2998918,
"bytesTotal": 2998918
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: dfc1365a2d046d0ecf8f9",
"id": "dfc1365a2d046d0ecf8f9",
"bytesCached": 0,
"bytes": 1894124,
"bytesTotal": 1894124
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: b8732c5b9580f3710b2f4",
"id": "b8732c5b9580f3710b2f4",
"bytesCached": 0,
"bytes": 2799520,
"bytesTotal": 2799520
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 64eea8d70bf3b3b31cdb5",
"id": "64eea8d70bf3b3b31cdb5",
"bytesCached": 0,
"bytes": 2553266,
"bytesTotal": 2553266
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 5ccf9e04fb9dcaaa00514",
"id": "5ccf9e04fb9dcaaa00514",
"bytesCached": 0,
"bytes": 2080498,
"bytesTotal": 2080498
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 27f93ce479a2a4f2ed9cf",
"id": "27f93ce479a2a4f2ed9cf",
"bytesCached": 0,
"bytes": 3732082,
"bytesTotal": 3732082
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: b0ed1c30c1810ec35555c",
"id": "b0ed1c30c1810ec35555c",
"bytesCached": 0,
"bytes": 2654549,
"bytesTotal": 2654549
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: bf39286872071851063ce",
"id": "bf39286872071851063ce",
"bytesCached": 0,
"bytes": 410836,
"bytesTotal": 410836
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 20220a8b5841d3190b23d",
"id": "20220a8b5841d3190b23d",
"bytesCached": 0,
"bytes": 3383311,
"bytesTotal": 3383311
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 0c322000c68c88a8a6e19",
"id": "0c322000c68c88a8a6e19",
"bytesCached": 0,
"bytes": 8447384,
"bytesTotal": 8447384
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 9b374af64e2b534ba7997",
"id": "9b374af64e2b534ba7997",
"bytesCached": 0,
"bytes": 1242438,
"bytesTotal": 1242438
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 17880fd3670dc1d3ad3d7",
"id": "17880fd3670dc1d3ad3d7",
"bytesCached": 0,
"bytes": 1205548,
"bytesTotal": 1205548
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: d514efcf47822aff479d9",
"id": "d514efcf47822aff479d9",
"bytesCached": 0,
"bytes": 5192130,
"bytesTotal": 5192130
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 678dee538c6f43289e3fe",
"id": "678dee538c6f43289e3fe",
"bytesCached": 0,
"bytes": 8546705,
"bytesTotal": 8546705
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 36a6412eee2f43f80129e",
"id": "36a6412eee2f43f80129e",
"bytesCached": 0,
"bytes": 5906596,
"bytesTotal": 5906596
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 569b5c232ced65ff58fec",
"id": "569b5c232ced65ff58fec",
"bytesCached": 0,
"bytes": 16706185,
"bytesTotal": 16706185
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: f04aba303f2d539dfe8ff",
"id": "f04aba303f2d539dfe8ff",
"bytesCached": 0,
"bytes": 1426478,
"bytesTotal": 1426478
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: c94d1855b699eaa99335c",
"id": "c94d1855b699eaa99335c",
"bytesCached": 0,
"bytes": 1853709,
"bytesTotal": 1853709
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: f72316d3eff3bea7dfe73",
"id": "f72316d3eff3bea7dfe73",
"bytesCached": 0,
"bytes": 1363165,
"bytesTotal": 1363165
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: aeabcb9a5095245f0a88b",
"id": "aeabcb9a5095245f0a88b",
"bytesCached": 0,
"bytes": 990031,
"bytesTotal": 990031
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: c85e204532a1b23fb9be4",
"id": "c85e204532a1b23fb9be4",
"bytesCached": 0,
"bytes": 753533,
"bytesTotal": 753533
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: f47ae488c3046092ace03",
"id": "f47ae488c3046092ace03",
"bytesCached": 0,
"bytes": 1354135,
"bytesTotal": 1354135
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 7c6dd2bb4648b78a02b9d",
"id": "7c6dd2bb4648b78a02b9d",
"bytesCached": 0,
"bytes": 1006770,
"bytesTotal": 1006770
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 5a5652b1d7ce060341943",
"id": "5a5652b1d7ce060341943",
"bytesCached": 0,
"bytes": 1431015,
"bytesTotal": 1431015
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 3371a52ab2ae279c9c097",
"id": "3371a52ab2ae279c9c097",
"bytesCached": 0,
"bytes": 1214839,
"bytesTotal": 1214839
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 49e518db5f56651bb7f5c",
"id": "49e518db5f56651bb7f5c",
"bytesCached": 0,
"bytes": 1145057,
"bytesTotal": 1145057
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 007142b4c9013f6ced073",
"id": "007142b4c9013f6ced073",
"bytesCached": 0,
"bytes": 8810138,
"bytesTotal": 8810138
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 60511678ad429b0f823db",
"id": "60511678ad429b0f823db",
"bytesCached": 0,
"bytes": 8981116,
"bytesTotal": 8981116
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 61291d26047700443893e",
"id": "61291d26047700443893e",
"bytesCached": 0,
"bytes": 12053670,
"bytesTotal": 12053670
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 951bf96f06ffdb2f01dc4",
"id": "951bf96f06ffdb2f01dc4",
"bytesCached": 0,
"bytes": 9501139,
"bytesTotal": 9501139
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 6ef4a9ab822ece2f9001e",
"id": "6ef4a9ab822ece2f9001e",
"bytesCached": 0,
"bytes": 13734035,
"bytesTotal": 13734035
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: a61f018731661d738ea3a",
"id": "a61f018731661d738ea3a",
"bytesCached": 0,
"bytes": 113384098,
"bytesTotal": 113384098
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 8997f32f8b56ede6bfdee",
"id": "8997f32f8b56ede6bfdee",
"bytesCached": 0,
"bytes": 3190,
"bytesTotal": 3190
},
{
"subtask": "ostree chunk",
"description": "ostree chunk: 0be1d11f3c17d9670acc9",
"id": "0be1d11f3c17d9670acc9",
"bytesCached": 0,
"bytes": 1948304,
"bytesTotal": 1948304
},
{
"subtask": "layer",
"description": "layer: 90cc6a627d6af3292093d",
"id": "90cc6a627d6af3292093d",
"bytesCached": 0,
"bytes": 10263,
"bytesTotal": 10263
},
{
"subtask": "layer",
"description": "layer: b4c762a4bb09d379184a5",
"id": "b4c762a4bb09d379184a5",
"bytesCached": 0,
"bytes": 10263,
"bytesTotal": 10263
},
{
"subtask": "layer",
"description": "layer: cd0caee131ccee6fce0aa",
"id": "cd0caee131ccee6fce0aa",
"bytesCached": 0,
"bytes": 89152818,
"bytesTotal": 89152818
},
{
"subtask": "layer",
"description": "layer: e853d0b60a65fdcdb01f5",
"id": "e853d0b60a65fdcdb01f5",
"bytesCached": 0,
"bytes": 10263,
"bytesTotal": 10263
},
{
"subtask": "layer",
"description": "layer: 2b38f58e8f4ab5cb12812",
"id": "2b38f58e8f4ab5cb12812",
"bytesCached": 0,
"bytes": 10263,
"bytesTotal": 10263
},
{
"subtask": "layer",
"description": "layer: 08faeb75bbf7617c9a6fd",
"id": "08faeb75bbf7617c9a6fd",
"bytesCached": 0,
"bytes": 10263,
"bytesTotal": 10263
},
{
"subtask": "layer",
"description": "layer: dc91651243b6a48f8a96d",
"id": "dc91651243b6a48f8a96d",
"bytesCached": 0,
"bytes": 169057488,
"bytesTotal": 169057488
},
{
"subtask": "layer",
"description": "layer: 90cc6a627d6af3292093d",
"id": "90cc6a627d6af3292093d",
"bytesCached": 0,
"bytes": 10263,
"bytesTotal": 10263
},
{
"subtask": "layer",
"description": "layer: d2ebaa84aa5f16a1351d1",
"id": "d2ebaa84aa5f16a1351d1",
"bytesCached": 0,
"bytes": 10263,
"bytesTotal": 10263
}
]
}

Does that size actually matter? Well...obviously we're talking about local IPC so 12k isn't large exactly, but we are also trying to emit this data at a relatively high frequency. Note that a default Linux pipe buffer is 64k, so if the reader slows down a bit that's only a maximum of 5 messages queued before we fill the pipe.

The size of this progress data will grow with the number of layers...let's say in the future we start using 100 layers (not at all an unreasonable thing!) that'd double to 25k of JSON etc.

This is data we're serializing and having the caller to deserialize ~5 times a second, competing with actual real work (like decompression, sha256 etc.).

OK well, benchmarking this obviously CPUs are super fast and python parses it in `44.4 usec` here...but...still...it wouldn't be a huge burden at all to have a stateful protocol either.