Job workflow: use case 2

Description

We want to define a job template that allows a user to transfer a list of selected dataset/files to a globus folder where they can access them. This job should place a message to a defined topic on a kafka server which gets read by the remote service which performs the transfer. Updates are allowed only through a POST to the job update endpoint with a specific user and token. When the transfer has been completed, than an email to the user needs should be sent.

Workflow Draft

1. User is logged in as user_1. She selects the following two datasets:
   • 20.500.12269/e7cc85c0-3fb3-4add-8e3b-6524192d9e71
   • 20.500.12269/22c45309-878b-410a-ba01-29b754c7a406
2. User clicks the transfer icon on the cart
3. The transfer job is defined with the following job template in configuration:

{
 "type": "user_transfer",
 "create": {
  "authorization": [“#has_access”],
  "method": {
   "type" : "kafka",
   "server" : "kafka.myinstitution.ext",
   "topic" : "transfer_data"
  }
 },
 "update": {
  "authorization": "transfer_service",
  "status" : {
   "allowed" : ["TRANSFERRED", "FAILED"],
   "transitions" : {
    "CREATED" : ["TRANSFERRED","FAILED"]
   }
  },
  "method": {
   "type" : "url",
   "server" : "scicat.myinstitution.ext",
   "endpoint" : "jobs/status/update"
  },
  "template" : {
   "status" : "<STATUS>",
   "message" : "<MESSAGE>",
   "job" : "<JOB_ID>",
   "token" : "<JWT_TOKEN>"
  }
 },
 "post": [
  {
   "method": {
    "type" : "url”,
    "url" : "myemailserver.myinstitution.ext",
    "action" : "POST"
    “arguments” : {
     “emails” : [“#user_email”]
    }
   },
   “body” : “Transfer job ${pid} has completed. Results: ${status}”
  },
 ]
}

4. A request is posted on the job create endpoint with the following information:

{
 "authenticated_user": "user_1",  ???
 "type": "user_transfer",
 "files": [ 
  { 
   "dataset.pid" : "20.500.12269/e7cc85c0-3fb3-4add-8e3b-6524192d9e71", 
   "file" : "*" 
  }, 
  { 
   "dataset.pid" : "20.500.12269/22c45309-878b-410a-ba01-29b754c7a406", 
   "file" : "*" 
  } 
 ]
}

5. The job creation function receives the request and check if user_1 has access to the datasets requested. We assume that it has the permissions and her token is valid
6. The new job is created with the following structure:

{
 "id" :"d0eb23b8-52e3-11ee-815f-7f1d70eaebe6",
 "type": "user_transfer",
 "created_by" : "user_1",
 "created_at" : <timestamp>,
 "updated_by" :  "user_1",
 "updated_at" : <timestamp>,
 "status" : "CREATED",
 "status_message" : "Job created",
 "update_history" : [
  {
   "updated_by" :  "user_1",
   "updated_at" : <timestamp>,
   "status" : "CREATED",
   "status_message" : "Job created",
  }
 ],
 "message_sent" : { <copy of the message posted, see #7> },
 "update_configuration": {
  "authorization": "transfer_service",
  "status" : {
   "allowed" : ["TRANSFERRED", "FAILED"],
   "transitions" : {
    "CREATED" : ["TRANSFERRED","FAILED"]
   }
  },
  "method": {
   "type" : "url",
   "server" : "scicat.myinstitution.ext",
   "endpoint" : "jobs/status/update"
  }
  "template" : {
   "status" : "<STATUS>",
   "message" : "<MESSAGE>",
   "job" : "<JOB_ID>",
   "token" : "<JWT_TOKEN>"
  }
 },
 "post_configuration": [
  {
   "method": {
    "type" : "url”,
    "url" : "myemailserver.myinstitution.ext",
    "action" : "POST"
    “arguments” : {
     “emails” : [“#user_email”]
    }
   },
   “template” : “Transfer job ${pid} has completed. Results: ${status}”
  }
 ]
}

7. A message is posted on topic transfer_data of server kafka.myinstitution.ext with body:

{ 
 “job_id” : d0eb23b8-52e3-11ee-815f-7f1d70eaebe6”,
 "type" : "archive", 
 “files” : [ 
  { 
   “pid” : “20.500.12269/e7cc85c0-3fb3-4add-8e3b-6524192d9e71”, 
   “file” : “<file_full_path>” 
  }, 
  ... 
  { 
   “pid” : “20.500.12269/22c45309-878b-410a-ba01-29b754c7a406”, 
   “file” : “<full_file_path>”
  },
  ... 
 ],
 “updates_configuration” : { 
  "method" : {
   “type: “url”, 
   “authorization”: “archiver_service”, 
   “server” : “scicat.myinstitution.ext”, 
   “endpoint” : “jobs/status/update” 
  }, 
  “template” : { 
   “job_id” : “d0eb23b8-52e3-11ee-815f-7f1d70eaebe6”, 
   "status" : "<STATUS>", 
   "message" : "<MESSAGE>", 
   "token" : "<JWT_TOKEN>" 
  } 
 }

8. the backend returns 200 on the job creation request
9. the transfer service reads the request from the kafka topic and initiate the action
10. the transfer is completed
11. the transfer service sends a request to endpoint https://scicat.myinstitution.ext/jobs/status/update with the following data:

{ 
 “job_id” : “d0eb23b8-52e3-11ee-815f-7f1d70eaebe6”, 
 "status" : "TRANSFERRED", 
 "message" : "The requested files have been transferred", 
 "token" : "<JWT_TOKEN_FOR_TRANSFER_SERVICE>" 
} 

12. credetianls are verified and they are valid
13. The job is updated as follow:

{
 "id" :"d0eb23b8-52e3-11ee-815f-7f1d70eaebe6",
 "type": "user_transfer",
 "created_by" : "user_1",
 "created_at" : <timestamp>,
 "updated_by" :  "archive_service",
 "updated_at" : <timestamp>,
 "status" : "TRANSFERRED",
 "status_message" : "The requested files have been transferred",
 "update_history" : [
  {
   "updated_by" :  "user_1",
   "updated_at" : <timestamp>,
   "status" : "CREATED",
   "status_message" : "Job created",
  },
  {
   "updated_by" :  "archive_service",
   "updated_at" : <timestamp>,
   "status" : "TRANSFERRED",
   "status_message" : "The requested files have been transferred",
  }
 ],
 "message_sent" : { <copy of the message posted, see #7> },
 "update_configuration": ...
 "post_configuration": ...
}

14. The status reached by the job is an end state, so the job is completed and we need to run the post actions
15. The only post action is executed. It is defined as:

{
 "method": {
  "type" : "url”,
  "url" : "https://myemailserver.myinstitution.ext",
  "action" : "POST"
  “arguments” : {
   “emails” : ["#user_email"]
  },
 }
 “template” : “The requested files have been transferred: ${status}”
}

16. A POST request to the URL https://myemailserver.myinstitution.ext?email=email_user with the correct data associated.
17. The external service receive the request and return a status code 200 and message "Email sent"
18. The job is update accordingly

{
 "id" :"d0eb23b8-52e3-11ee-815f-7f1d70eaebe6",
 "type": "user_transfer",
 "created_by" : "user_1",
 "created_at" : <timestamp>,
 "updated_by" :  "archive_service",
 "updated_at" : <timestamp>,
 "status" : "TRANSFERRED",
 "status_message" : "The requested files have been transferred",
 "update_history" : [
  {
   "updated_by" :  "user_1",
   "updated_at" : <timestamp>,
   "status" : "CREATED",
   "status_message" : "Job created",
  },
  {
   "updated_by" :  "archive_service",
   "updated_at" : <timestamp>,
   "status" : "TRANSFERRED",
   "status_message" : "The requested files have been transferred",
  }
 ],
 "message_sent" : { <copy of the message posted, see #7> },
 "update_conmfiguration": ...
 "post_configuration": [
  {
   "method": {
    "type" : "url”,
    "url" : "myemailserver.myinstitution.ext",
    "action" : "POST"
    “arguments” : {
     “emails” : [“#user_email”]
    }
   },
   “template” : “Transfer job ${pid} has completed. Results: ${status}”
   "status_code" : 200,
   "message" : "Email sent"
  },
 ]
}

19. the job is completed