docker builder prune should produce error when using "label" filter

When I attempt to clear select build cache files using the --filter option of docker builder prune, the filter option is seemingly ignored. For example if I use docker builder prune --filter "label=asdfasdfasdf", it removes all build cache files.

Steps to reproduce the issue:

1. Dockerfile:

FROM ubuntu:18.04
RUN echo "Hello World"

2. sudo DOCKER_BUILDKIT=1 docker build .
3. sudo docker builder prune --filter "label=asdfasdfasdf"

Describe the results you received:

WARNING! This will remove all dangling build cache. Are you sure you want to continue? [y/N] y
Deleted build cache objects:
ccsi71vdr28pnus85bldmoonf
hk1yxod8uv58i2tpui5it3rqn

Total reclaimed space: 42B

All cache files were deleted.

Describe the results you expected:

In the case of a non-existent label like described above (for example asdfasdfasdf), it should not delete anything.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 01:25:46 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b7f0
  Built:            Wed Mar 11 01:24:19 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 5
  Running: 0
  Paused: 0
  Stopped: 5
 Images: 48
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Operating System: Ubuntu 18.04.4 LTS
 OSType: linux
 Architecture: x86_64
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

Start by reproducing the issue with the documented docker builder prune --filter "label=asdfasdfasdf" command and inspect the CLI entry point for builder prune. Done means a nonexistent label leaves build cache untouched, with regression coverage for the filter behavior.