git-crypt breaks git partial staging

The problems can be reproduced like this:
- from command line: 'git add -p' -> Only binary files changed.
- from git gui: right-click -> 'stage lines for commit' is greyed out

The problem seems to be that if a file is encrypted, it is seen as binary, and patch staging only works for text files so git can only stage the full file.
Maybe this is intrinsic to how git-crypt works, but maybe there are workarounds, maybe something can be done changing settings in .gitattributes?