Infrastructure connectivity checks

Problem

Receiving reports that are not actually bugs but connectivity issues with components, for example slow LDAP servers, or unstable connections to the database or redis.

Solution

Having automated checks in place that makes it possible for admins to verify their infrastructure connectivity before reporting issues. The goal is to prevent people from sending issues that are not actual bugs.

Implementation idea

Nextcloud already has a "setup check" section in the settings where various settings are getting verified.

This section should be extended to include a table with two columns connectivity and reliability, and with rows representing the various components that are to be checked.

With connectivity we mean "is it accessible at all" and with reliability we mean "how many connection failures / timeouts in past time intervals (hours, days, weeks)"

Possible components to check that would appear as rows / grouped rows:

• Database
• Redis
• Local filesystem / Primary object store
• LDAP servers (multiple)
• external storages (multiple)
• office document servers
• ...

Implementation details

Status provider service

• Provide a new PHP service class in the server for apps to register "connectivity status providers"
  • Status provider method getType(): string for displaying in the table as prefix for a component type (and for grouping)
  • Status provider method getDisplayName(): string for displaying in the table, it must be useful enough for the admin to find out which exact component needs attention
  • Status provider a method checkConnectivity(): void that does an immediate connectivity check to one given component type, like for example a specific LDAP server. Only a connection is done, the measurement is done by the caller.
  • Status provider method getFailures(): array, returns array of timestamps for last failures.
    • ❓ should we include error messages to be able to aggregate there or at least show it somewhere ?
  • If multiple components of the same type, the app/implementor must register multiple status providers, one for each (ex: one provider per LDAP server, but using the same type name)
  • ❓ Make the registration lazy, maybe through an event that we only trigger from the connectivity check controller ?
• All the connectivity status provider will be queried whenever the settings page is queried
  • For each status provider, call "checkConnectivity()" and measure the time taken to respond and put it in the table
  • For each status provider, call "getFailures()" and summarize the frequency of failures in the table
  • The table needs time interval labels for "absent", "slow", "ok" (depending on how much time it took)
  • Implement UI + table to show results

Intermittent failure tracking

• Every app that manages components (ex: LDAP) must catch connection failures like timeouts and send it to the status provider service

  • Status provider service will store the event with timestamp somewhere (database?)

• Component types (raise tickets when ready)

  • Database: implement in core
  • Redis: implement in core
  • Object Store Primary storage: implement in core
  • LDAP: user_ldap app must return one status provider per LDAP server
  • External storage: files_external must return one status provider per mount
    • ❓ what to do when the list is very long ? aggregate ?
    • ❓ what about personal mounts ?
  • Document server: OnlyOffice / Collabora apps
  • Others ?

Development phases

• Phase 1: implement the status provider service in the server. This qualifies as API changes/addition so should ideally be released as part of a major relase
• Phase 2 / parallel: implement the status providers for each component, can be done in parallel and released independently in minor/patch releases

Open issues

• See "❓" entries in checklist
• The name "status providers" not very catching, any better ideas ?
• Should the responsibility of tracking and measuring be done by the apps / implementors or rather by the connectivity service itself ?
• Expand with more technical clarifications in the concept

Start with the existing setup check section in the settings and the Phase 1 status provider service requirements. Review the listed open questions and component responsibilities before implementation; the work is done when the provider API, connectivity and failure measurements, storage approach, and table scope are defined and implemented.