Improve image build speed (Discussion,Ideas)

This is a discussion for a concept I'm working on; it is also preliminary work on hopefully support for arm64.
I'm working on a concept to improve the build speed of the image.

The concept I'm using within my work and in my private repositories has a base image automatically updated with a CI cron job at least once a week. This image is written to the registry with the tag base. The next stage of the build is simply using this image tag.

Furthermore, I want several time-consuming build stages of this image to be extracted to speed up the building process.
For example, ruby 3.0.6 is used throughout several versions. However, building this every time is time-consuming.
Therefor, I'm looking towards having this specific ruby bing built by a CI process, which then writes it to the repository as an asset as a *.deb file stored in the repository under git-lfs. The install stage has to checkout the repository, perform git-lfs checkout, start the build context, and then only install the *.deb package. This can also be done with the other compilation parts.

This is a rough draft; I'm looking forward to input and comments.

Please let me know what you think.

Disclaimer: Due to my profession, I'm a hefty GitLab user; I can quickly build all of this in GitLab; however, building it within GitHub actions will be a challenge and probably require help and input from others.

The issue names Docker image build stages, a CI cron job, GitHub Actions, Git LFS, and Ruby 3.0.6, but no files or tests. Start by mapping the repository's existing image build and CI entry points, then clarify the proposed caching and arm64 design with maintainers; done would require an agreed approach and evidence of faster builds.