Workflows: Phase 2

[Protected](https://docs.google.com/spreadsheets/d/14nmvNcvKJ8WP3aQS67rkZXc9Nc3fGzQ9vJcneb4CLVk/edit#gid=0)

## UI Issues

- To create a Studio, users go to the Studio sidebar tab, not the admin
- To see their Studio, users go to the Studio sidebar tab, but it's not showing because it has to be registered manually.
- `Copy ID` and `Copy nexus Address` are similar though not the same, and confusing for users. We should add an infobox or think of a better way to present it.

## Workflow Bugs

- [ ] Projects are not showing in Workflow sub-app in production after 1.5.0 deployment
- [ ] Data Table: when highlighting code in the edit form, the workflow canvas moves
- [ ] Data Table: use would like to have better column widths, even change it
- [ ] Data Table: when writing a query for a table, the lines of code can go out of the horizontal scrolling limit
- [ ] Data Table: when creating a new table in a step, it happens that the query changes back to the default query
- [ ] Data Table: new tables appear and disappear randomly during refresh after table creation, it stabilizes after approximately 30 seconds
- [ ] Data Table: when a new table is created. It breaks the arrangement of all the other tables in the canvas
- [ ] Data Table: when several tables in a step, new tables appear at "random" positions, sometimes off-canvas/screen
- [ ] Data Table: they sometimes disappear from the step/canvas
- [x] Canvas: Links between parents and children steps disappear when adding a second parent https://github.com/BlueBrain/nexus/issues/2496
- [ ] Steps: impossible to delete a step from the UI
- [ ] Canvas: when creating a step, it appears at a "random" position, and can be quite hard to find on the canvas
- [ ] Canvas: on macOS, when swiping left to move on the canvas, it changes the webpage, as swiping left is associated to return to the last page in the web browser
- [x] Data Table: deprecated resources are showing => filter deprecated resource in SPARQL query
- [x] Data Table: when removing the LIMIT 20 for a query and clicking on preview, it loads forever and doesn't display anything anymore => wrong SPARQL query
- [x] Data Table: displayed resources change at each refresh/randomly and the information displayed in the additional column becomes incorrect (it display other unrelated fields) => wrong SPARQL query
- [x] Default ES Query: https://github.com/BlueBrain/nexus/issues/2536
- [ ] Fix the create resource form to update the proper payload in the step (using input): I get two versions of the inputs but at least the context resolves properly in both cases
- [ ] Fix the activity link to update the proper payload in the step (using activity): the payload is saved with the extended version and not the shorthand, thus not leveraging the context, which is okay-ish

## Workflow Feature Requests

- [ ] A plugin to display pdfs in resources. I put the pdfs under the field "pdfs" for now, (for Emodel we can discuss, it might be better to have a plugin that plots itself some of the scores or the traces) see https://bbpcode.epfl.ch/code/#/c/53491/
- [ ] Be able to select several steps and drag&drop them to move them around as a group
- [ ] Be able to zoom and de-zoom
- [ ] Be able to click and drag between two steps to create a link
- [ ] Have a "delete" button on the steps and tables
- [x] Be able to programmatically change the status of a task based on the Luigi status of the task => edit the workflow step resource and its status property
- [ ] Be able to trigger Luigi tasks through the UI (using REST?)
- [ ] Be able to pass configuration parameters through the UI
- [ ] Be able to visualize the iterative nature of the e-model building process (to discuss further)

## Ideas to be Investigated

- ↓Manage resources and data across projects
- Is it useful for BBP? They often work in one Nexus project.
- Pro-actively register notebook/code
- How many people use notebooks, code, etc?
- Detect correct commit?
- Link code to many activities?
- Capture command line?
- What's the minimum to capture?
- ↓Infer new workflow steps (propose to the user to create them) from activities present in the project
- Motivation: Create visibility for activities in projects and bring structure
- List and compare activities (output entities) JDC, MR
- Motivation: Quick comparison for decision-making
- Select and compare entities/resources?
- ↑Support the Luigi workflows, based on provenance use
- Motivation: See the artifacts of the Luigi workflows, view running workflows, run new workflows
- What are the computational steps?
- What about Snakemake?
- Docs about Luigi/Nexus integration: developing Nexus-compatible workflows
- Investigate outputs across multiple activities (the full provenance of one output) JDC, MR
- Motivation: Transparency over how an artifact was created and introspection
- Provenance visualizer for single resources (plugin)
- Management of scientific artifacts aligned with DLM
-Motivation: enabling easy cleanup and organization
- Archival of resources/datasets
- Spawn a full coding environment from the activity and record everything that happens in that environment MR
- Motivation: one-click coding environment, speed, ease-of-use, automatically capture provenance of software agent
- Introspect what was done by the scientists
- Integration with DVC
- Motivation: streamlined CLI to ingest and manage data
- Consider integration with Delta
- ↓↓Autonomously manage permissions for projects (team, exploratory, Ph.D.)
- Who should be in charge of setting permission? Governance/PMO/DKE?
- Add members to projects (read/write/...)
- Members management
- ? Expected input and output for a step/activity (to compare with real input/output registered by users)
- Motivation: Proactive error messages for users when they mess up
- Introspect from activity type, add it in workflow step metadata (input entity type, output entity type)
- Allow users to define rules to automatically link activities to steps (not every single one manually)
- Motivation: Avoid linking hundreds of activities manually
- Non-explicit linking?
- Similar to outlook? Based on users' choices?
- Fully implicit? What do we need? What can we use?
- Mobile version
- Motivation: publishing data should be available on mobile
- Should we do part of the app? All of it?
- It implies a big screen re-design (mobile-first)
- Schema.org Dataset ingestion (incl. basic provenance)
- Motivation: users should be able to register datasets without using the Forge, in particular for generic use cases
- It will not be customizable for now
- Search to power-up Studios completely
- Motivation: ?? removing technical complexity
- Studio is a kind of static, customized search
- Nexus.js improvements (ongoing)
- Plugins dev experience improvements
- App refactor (smooth the sharp edges)
- Make canvas visually appealing by adding expected inputs and outputs
- List the number of views on a studio by counting both registered users and public users
- Motivation: easily accessible and transparent statistics for users
- Number of downloads of file or resource at the resource level
- Motivation: easily accessible and transparent statistics for users

This is a broad Phase 2 tracking issue covering UI bugs, workflow features, and investigation ideas, with no files, tests, or entry points identified. Start by selecting one unchecked item, verify whether it still applies, and define a separate issue with a specific affected area and completion criteria.

Written by the indexing model from the issue text.