Multi-keys setup: Diff + Checkout from other branch impossible when using different keys per branch

We have a setup with 3 branches. Each branch has a separate git-crypt key. The feature we are using for that is documented in https://github.com/AGWA/git-crypt/blob/master/doc/multiple_keys.md

When checking out and unlocking branch A (which uses key A), I can neither
* checkout single files from branch B (which uses key B) nor
* diff files between branch A and branch B nor
* show files from branch B

```
# git checkout B -- myfile
git-crypt: error: encrypted file has been tampered with!
error: external filter "/usr/bin/git-crypt" smudge --key-name=A failed 1
error: external filter "/usr/bin/git-crypt" smudge --key-name=A failed
fatal: myfile: smudge filter git-crypt-A failed
```