Merge and Rebase with remote repo fail

When a encrypted file is changed in a remote repo, I am facing issues with merge and rebase.
### Merge

With merge, the issue happens when there is a conflict in a secret file between my repo and a remote repo.
The merge itself succeeds with showing there is a conflict in the secret file.
But I can't see the diffs in the file.
The lines like "<<<<<<< HEAD" are supposed to be in the file.

I did this with commands below.
In this situation, I am in a branch "feature/a" with changes in a secret file.
And "origin/master" also has a change in the secret file

```
$ git fetch
$ git pull origin master
```

If there is a change in a secret file in the remote repo but no conflict with my local repo, it succeeds :)
### Rebase

Whether there is a conflict or not, if there is a change in a secret file in a remote repo, rebase always fails as follows.

This is a same situation as the merge above.
I did this with commands below.

```
$ git fetch
$ git rebase origin/master
```

The rebase seems to succeed but I am getting messages below.

```
File not encrypted
error: external filter "git-crypt" smudge "key name here" failed 1
error: external filter "git-crypt" smudge "key name here" failed
```

This message is shown for each secret file that were changed.
Then if I do "git status", it shows the secret file is modified.
But once I got the messages above, I can't do "git stash" or "git reset".
The secret file keeps failing to encrypt and there is no way to turn the file back to encrypted one without just remove the file from the repo and add this again.

Thanks