Policies for packages that use OpenCL

This issue is a followup to https://github.com/conda-forge/openmm-feedstock/pull/7#. We are trying to move OpenMM to be distributed through conda-forge, and some policy questions came up that need to be resolved. In particular, we have been told conda-forge has an established policy that any package that uses OpenCL is required to list ocl-icd as a dependency. We do not want to list it as a dependecy: given our particular needs, there doesn't exist any situation where it would benefit our users, and there do exist situations where it would cause problems.

Let me give some background information, since it will help to clarify the issues we're dealing with. OpenMM is a widely used package for molecular dynamics simulation. It can use OpenCL, but it doesn't require it. It comes with multiple computational backends: CUDA, OpenCL, and CPU. It automatically selects which one to use at runtime based on the available hardware and software (whether or CUDA and/or OpenCL is installed). The CUDA and OpenCL backends exist mainly for users with high end GPUs, which can be much faster than CPUs. For users who don't have one of those, it's generally better to use the CPU backend.

Accurate results are critical. This is scientific software, and errors could lead to users having to retract papers and losing months or years of work. We only support a small number of OpenCL implementations. On Linux, we support the ones from AMD and NVIDIA. Neither of those is conda installable. They are included with the video drivers. On macOS, we support Apple's implementation, which is built into the OS. These are the only ones we test. Using any other implementation would have a high risk of producing incorrect results. Our installation instructions tell users how to install appropriate OpenCL implementations. We are very concerned about anything that might lead to a user inadvertently using a different implementation they didn't even know was there.

OpenMM has APIs for C++, C, Fortran, and Python. So although we use conda as a distribution mechanism, it is part of a much larger ecosystem than just conda-forge, conda, or even Python. It needs to work correctly with all other libraries on the OS, regardless of how they are installed or what language they are used from.

So that's the background. Now let me describe our particular concerns.

1. Given our needs, there simply is no benefit to installing ocl-icd. None of the OpenCL implementations we support is conda installable.

2. Installing it creates new opportunities for the user to get incorrect results by silently using a different OpenCL implementation than the supported one we told them to install.

3. ocl-icd loads ICD files from a nonstandard location. Whereas all standard ICD loaders on Linux look for them in /etc/OpenCL/vendors, ocl-icd looks for them in a location inside the conda installation. If users want to use any OpenCL that was installed system-wide (such as the ones from the GPU vendors), they have to add a symlink pointing to the standard location. So including ocl-icd makes life harder for users. (A workaround is to also install ocl-icd-system, which creates the symlink for them, but it appears few packages currently do this.)

4. On macOS there is an additional, very serious problem: ocl-icd is written in a way which is incompatible with Apple's OpenCL framework. If two packages are loaded at the same time, one that links to ocl-icd and the other that links directly to Apple's framework (as nearly all OpenCL-based codes do on macOS), this will result in a crash. Linking to ocl-icd would therefore make OpenMM incompatible with nearly all other OpenCL based software that comes from any source other than conda-forge.

Given these issues, I suggest the existing policy should be reconsidered. Packages should not be required to depend on ocl-icd unless they have a specific reason to depend on it. Even better, I would suggest it should be a dependency for *implementations* of OpenCL, not for packages that *use* OpenCL. This is how the ICD mechanism was designed to work in the first place. ICD loaders are normally included with the implementation. They are never included with individual applications that just happen to use OpenCL. Tying the loader to the application rather than the implementation is contrary to the design of the mechanism.

This would make life easier for users of all OpenCL based packages. If they haven't specifically conda installed an OpenCL implementation, ocl-icd would not block the standard implementation that came with their GPU drivers from being found.

I also suggest that any use at all of ocl-icd on macOS should be carefully reconsidered. Until it can be made to coexist properly with the standard framework built into the OS, it just isn't appropriate for any package that is intended to be widely distributed.

cc: @jchodera @jaimergp

Start by reading the linked openmm-feedstock pull request and the 68-comment discussion on this issue. Compare the proposed ocl-icd dependency policy with the described Linux and macOS compatibility concerns, then establish and record a clear decision about when OpenCL users or implementations should depend on it.

Written by the indexing model from the issue text.