truncation vs rejection by NT and other formats with utf-16

It was found in #4032. NT format rejects some candidates and truncates others:

>>> passlib.hash.nthash().hash(u'a' * 27)
'3f9798b4e3c435593074a9ef81662507'

$ echo '$NT$3f9798b4e3c435593074a9ef81662507' > nt.pw

$ # Ok
$ perl -le 'print "a" x 27' | ./john/run/john --pot=nt.pot nt.pw --stdin ; rm -f nt.pot
[...]
Using default input encoding: UTF-8
[...]
aaaaaaaaaaaaaaaaaaaaaaaaaaa (?)     
1g 0:00:00:00  6.666g/s 6.666p/s 6.666c/s 6.666C/s aaaaaaaaaaaaaaaaaaaaaaaaaaa

$ # Truncate
$ perl -le 'print "a" x 81' | ./john/run/john --pot=nt.pot nt.pw --stdin ; rm -f nt.pot
[...]
Using default input encoding: UTF-8
[...]
aaaaaaaaaaaaaaaaaaaaaaaaaaa (?)     
1g 0:00:00:00  5.882g/s 5.882p/s 5.882c/s 5.882C/s aaaaaaaaaaaaaaaaaaaaaaaaaaa

$ # Reject
$ perl -le 'print "a" x 82' | ./john/run/john --pot=nt.pot nt.pw --stdin ; rm -f nt.pot
[...]
Using default input encoding: UTF-8
[...]
0g 0:00:00:00  0g/s 0p/s 0c/s 0C/s

$ ./john/run/john --format=nt --list=format-all-details
[...]
Min. password length                 0
Max. password length                 27
[...]
 Case sensitive                      yes
 Truncates at max. length            no
 Supports 8-bit characters           yes
 Converts internally to UTF-16/UCS-2 yes
 Honours --encoding=NAME             yes

So maximum length in bytes for utf-8 is 81 bytes (27 * 3). NT format rejects everything longer than 81 bytes. But candidates shorter than 81 would get truncated to 27 utf-16 chars.

I guess other formats with internal encoding conversion have similar behaviour.

The info is not accurate. Also format hashes more than it should. It does not seem a big problem at the moment.

Reproduce the NT-format examples using passlib, john --stdin, and --list=format-all-details, focusing on UTF-8 input converted to UTF-16/UCS-2. Compare the reported truncation and rejection behavior with the displayed length and truncation metadata, then check whether other internally converted formats show the same inconsistency; done means the behavior and format information agree.