openwall / openwall/john

bcrypt-opencl is much slower than should be on NVIDIA Maxwell & Pascal

Open
#2,620 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
C
Stars
13.6k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Description

Per my testing, our bcrypt-opencl is roughly same speed as hashcat's on Tahiti (which was the primary target when Sayantan worked on our bcrypt-opencl), but is much slower on NVIDIA Maxwell (and per others' testing also on Pascal). These newer NVIDIA GPUs matter for bcrypt-opencl especially because they're the ones where bcrypt-opencl's speeds would finally be reasonable and competitive vs. CPUs.

Titan X Maxwell JtR:

Device 6: GeForce GTX TITAN X
Benchmarking: bcrypt-opencl ("$2a$05", 32 iterations) [Blowfish OpenCL]... DONE
Speed for cost 1 (iteration count) of 32
Raw:	4735 c/s real, 4708 c/s virtual

Titan X Maxwell hashcat:

* Device #7: GeForce GTX TITAN X, 3071/12287 MB allocatable, 24MCU
* Device #8: GeForce GTX TITAN, skipped.

Hashtype: bcrypt $2*$, Blowfish (Unix)

Speed.Dev.#7.....:     6679 H/s (56.21ms)

(On a related note, hashcat's device numbering starts with 1 and JtR's with 0 - maybe this is something to adjust as well.)

Others report even bigger differences in favor of hashcat on Pascal cards, but that may be because our MULTIPLIER in opencl_bf_std.h isn't large enough for those bigger GPUs.

I tried tweaking DEFAULT_LWS and MULTIPLIER, but could not achieve much of an improvement on our Titan X Maxwell card in this way alone. Per a quick glance at hashcat, it appears to use LWS=8 (I tried, this made little difference), local memory (not private, and we also do the same on these GPUs), different layout of the S-boxes in local memory than ours (would take some effort to try, but perhaps we should), and slightly different code (ditto). Besides trying things, we could also review the ptx assembly.

Also, hashcat appears to compute the hashes on GPU only (first 128 bits only, that is doing 2 out of 3 blocks 64-bit for the final 64 Blowfish encryptions?), whereas we compute the first 64 bits on GPU and transfer the resulting S-boxes to host in order to allow for computing of the rest from that point on in cmp_exact(). I don't know if that transfer to host is causing us much speed loss - probably it is not, but it's worth trying to comment it out and remove the check in cmp_exact() to see how much faster it could go with the 64-bit check only (which should be sufficient in practice).

Tahiti JtR:

Device 2: Tahiti [AMD Radeon HD 7900 Series]
Benchmarking: bcrypt-opencl ("$2a$05", 32 iterations) [Blowfish OpenCL]... DONE
Speed for cost 1 (iteration count) of 32
Raw:	4484 c/s real, 614400 c/s virtual

Tahiti hashcat:

Hashtype: bcrypt $2*$, Blowfish (Unix)

Speed.Dev.#3.....:     4315 H/s (57.65ms)

While at it, we should add FMT_TRUNC to opencl_bf_fmt_plug.c.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with opencl_bf_std.h and opencl_bf_fmt_plug.c, then benchmark bcrypt-opencl on the reported Maxwell and Pascal GPUs against the Tahiti results and hashcat figures. Investigate DEFAULT_LWS, MULTIPLIER, local-memory S-box layout, PTX assembly, and the host transfer used before cmp_exact(). Done means documenting or achieving a meaningful performance improvement and adding FMT_TRUNC.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.