docker/build-push-action@v4 with cache-from/to type gha keep adding entries into github cache

Hi, I have GitHub workflow in which I use docker/build-push-action@v4. it correctly caches the dependencies built the problem is that it keeps filling my cache with these entries even after it has cached the result correctly:

index-armx-1-17e638f6#7 and buildkit-blob-1-sha256:9fd03ee654113a4b91adf098f734052358584f641606799c45069ccc1bb1c300 they are super small but it still clouds the cache and I was wondering if there might be an option to avoid this. I've tried using scope, mode=min, and mode=max. even try to check the source code but I couldn't find anything.

This is my current action:

testarm64:
    name: ARM64 Test
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3
      - name: Setup qemu
        uses: docker/setup-qemu-action@v2
        with:
          platforms: arm64
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      - name: Build arm64 image
        uses: docker/build-push-action@v4
        with:
          file: test/extra/arm64_dockerfile
          platforms: linux/arm64
          load: true
          push: false
          tags: arm64
          # Create a cache entry for later use in github actions
          cache-from: type=gha,scope=armx
          cache-to: type=gha,mode=min,scope=armx
      - name: Test Dtypes
        run: docker run -e ENABLE_METHOD_CACHE=1 -e CI=1 -e ARM64=1 -e CLANG=1 arm64 python3 -m pytest -n=auto test/test_dtype.py

Start with the shown workflow, especially the type=gha cache-from/cache-to settings and test/extra/arm64_dockerfile, then trace the BuildKit GitHub Actions cache handling. Reproduce the arm64 build and inspect the reported index-armx and buildkit-blob entries. Done means determining whether these entries are expected and whether a supported configuration avoids them.