Prepare to run studies against EMIS

In order to run studies against EMIS with the same workflow that we use with TPP, we need to make changes to several parts of the system.  This is my initial braindump of what we need to do and I'm sure I've missed things. Please edit to add your notes, and then let's discuss in a call on Tuesday or Wednesday this week.

• job-server (@ghickman)
  • users need to be able to select which backend they run jobs
    • initially, we want to hide the ability to run against EMIS by default, and have specific opt-ins for supported repos. Longer term we will need to bake-in the concept of which backends are required or able to run a given study definition (see below). Initial implementation could just be a database flag which is false by default and we manually enable for specific repos.
  • /status/ will need to show information about multiple backends
  • Think about error reporting / sentry / etc
  • questions:
    • should a user be able to run a job against multiple backends? (@sebbacon's best placed to know what users will expect)
      • my (Seb's) view is that UI support would be via an extension of the usual "run" mechanism; "run all" would select everything as it currently does, but there are now N * M tickboxes where N is backends and M is actions. Users could uncheck a specific backend if they wanted (actions would be grouped visually by backend in the UI); if they want to completely exclude a backend from the UI they'd do this by editing their project.yaml (see below). "Supported backends" would show somewhere obvious on a workspace header.
• job-runner (@evansd)
  • possibly no code changes?
  • longer term we will need to bake-in the concept of which backends are required or able to run a given study definition. This will probably be on a per-column basis; you might be able to extract patient ages in emis but not SGSS status, for example. Users should also be able to define which backends are included or excluded in their project.yaml, i.e. to be able to skip TPP backend completely (for example, they just don't need it, so it's faster)
  • we'll need to flesh out playbooks/EMIS.md
• EMIS infrastructure (@bloodearnest)
  • user management (lower priority)
    • set up unix groups for level 2/3/4 access
    • create user accounts
      • level 2 access should be limited to a handful of engineers with NHSE contracts
      • level 3 access should be limited to researchers with appropriate NHSE contracts
      • level 4 access can be wider
        • what are the requirements here?
    • set up directory structure necessary for high/medium privacy outputs
  • ensure that we can pull repos from GH
  • ensure that we can push output to GH
  • ensure users can install opensafely cli tool and os-release script (see docs.opensafely.org)
  • harden and buildout software installations
    • consider asking for an ubuntu container within which we have root?
    • job runner as a service - systemd?
    • scripted installation - at least the basics we can build on, supplemented by installation narrative if needed as stopgap
    • backups if necessary
    • log rotation, disk space monitoring, root cron emails setup, etc, if we have root access. If not, conversation with EMIS about what they've set up
  • work out how to support viewing, editing, publishing outputs
    • For viewing outputs, a web browser should be fine (pdfs, svgs, html and text)
    • For diffing outputs, command line git may be sufficient but visual would be ideal: Github Desktop or at a push, gitweb?
    • For redacting outputs, a text editor is needed. We could consider mandating VS Code for simplicity, but worth canvassing
    • For publishing outputs, we may need to install Github Desktop, although again we might be able to mandate command line git (if we provide adequate documentatin)
    • I think it boils down to either (a) provisioning a Windows review server with access to L4 data or (b) providing a web browser and expecting command line tool usage. And I think (a) is probably unavoidable
  • questions:
    • can we be responsible for creating user accounts?
• cohort-extractor (@inglesp)
  • questions:
    • do we need to pass EMIS_ORGANISATION_HASH in as an environment variable, or can it be hard-coded?
    • how do we support studies that use backends with different coding systems?
      • it would be wonderful if we could use SNOMED in TPP
    • how can we support more features from the TPP backend (eg date expressions) in EMIS?
    • is there a half-way house for things we're willing to implement now while we await refactor?

The issue names no specific files or tests and spans job-server, job-runner, EMIS infrastructure, and cohort-extractor. Start by reviewing those sections and playbooks/EMIS.md; this needs an agreed, bounded scope before implementation, with each selected area having defined setup, backend, or workflow requirements.