Dataverse in Docker in the cloud (with Keycloak)

Years ago we wrote a bash script that spins up Dataverse in EC2. Here's the PR where added it as well as the docs:

• #5063
• https://guides.dataverse.org/en/6.10.1/developers/deployment.html#deploying-the-dataverse-software-to-amazon-web-services-aws

It works great for the following use cases:

• CI, called by Jenkins
• one-off spin ups of an env for a demo (JSF only)

These days we have a new React UI at https://github.com/IQSS/dataverse-frontend that requires Keycloak, which means we can't use our bash script, which knows nothing about Keycloak.

In the "dev-env" directory of the frontend repo, we have a way to spin up the backend, frontend, Keycloak (and other necessary services) all configured and allowing login. The idea is to get something similar working in the cloud, like we have now using the bash script above.

I'm not picky about the technology we use, as long as it's open source. Originally, I figured we'd just add a "container mode" to the bash script with -c or whatever, as I've described here:

• https://github.com/gdcc/dataverse-ansible/issues/268

However, @srmanda-cs and @poikilotherm seem to favor modern tools, as discussed at https://dataverse.zulipchat.com/#narrow/channel/375812-containers/topic/Docker.20mode.20when.20spinning.20up.20EC2.20instances/near/591807528

Whatever tools we use, part of the definition of done is to have sufficient documentation (as above) that clearly steps through how to both get set up with tooling and do the spin up.

We can certainly create sub-issues for smaller chunks of this work. I'll make some suggestions below.

Phase 1: backend

Get just the backend (which still includes JSF, currenly) spun up. I know I mentioned the "dev-env" directory of the frontend repo above, but we could even start with the compose.yml file at https://guides.dataverse.org/en/6.10.1/container/running/demo.html#quickstart

We're planning to try OpenTofu ( https://opentofu.org ) first. Let's put the OpenTofu files in /docker/opentofu.

Phase 2: HTTPS

The spun-up environment should have a valid HTTPS cert.

Related:

• https://github.com/IQSS/dataverse/issues/10359

Phase 3: frontend (no auth), arbitrary branches for both backend and frontend

Add the frontend but don't let people log into it yet.

Let's say we want to spin up this branch from the frontend:

• https://github.com/IQSS/dataverse-frontend/pull/955

It's already merged but let's say it requires this branch from the backend:

• https://github.com/IQSS/dataverse/pull/11753

Phase 3: frontend (with auth)

Add Keycloak and let the frontend log in with users that are preconfigured in Keycloak. For example, see below from https://github.com/IQSS/dataverse-frontend/blob/v.0.3.0/dev-env/keycloak/test-realm.json#L439

  "users": [
    {
      "id": "e5531496-cfb8-498c-a902-50c98d649e79",
      "createdTimestamp": 1684755721064,
      "username": "curator",
      "enabled": true,
      "totp": false,
      "emailVerified": true,
      "firstName": "Dataverse",
      "lastName": "Curator",
      "email": "dataverse-curator@mailinator.com",
      "credentials": [
        {
          "id": "664546b4-b936-45cf-a4cf-5e98b743fc7f",
          "type": "password",
          "userLabel": "My password",
          "createdDate": 1684755740776,
          "secretData": "{\"value\":\"AvVqybCNtCBVAdLEeJKresy9tc3c4BBUQvu5uHVQw4IjVagN6FpKGlDEKOrxhzdSM8skEvthOEqJkloPo1w+NQ==\",\"salt\":\"2em2DDRRlNEYsNR3xDqehw==\",\"additionalParameters\":{}}",
          "credentialData": "{\"hashIterations\":27500,\"algorithm\":\"pbkdf2-sha256\",\"additionalParameters\":{}}"
        }
      ],
      "disableableCredentialTypes": [],
      "requiredActions": [],
      "realmRoles": ["default-roles-test"],
      "notBefore": 0,
      "groups": ["/curators"]
    }
  ],

Future plans/ideas

• Add workflow dispatch like https://github.com/IQSS/dataverse-frontend/actions/workflows/deploy.yml (idea from @ekraffmiller)
• Build part of the compose file on the fly.
• Support slash comment on PR to deploy

Start with the existing EC2 bash script, the compose.yml quickstart, and the frontend repository's dev-env directory, including keycloak/test-realm.json. Compare the backend, frontend, and Keycloak setup before deciding how the proposed /docker/opentofu layout should cover the phases. Done includes a cloud spin-up workflow and documentation that explains the tooling, deployment steps, HTTPS, authentication, and branch selection.