Information on Fail Over for Persistent EJB Timers

A new page can be created under:
https://github.com/OpenLiberty/docs/tree/master/ref/general

This is for https://github.com/OpenLiberty/open-liberty/issues/7774 which is currently targeted for beta.

It looks like there is currently not any information on persistent EJB timers in general, so I included some brief background as well.

Persistent EJB Timers in Liberty

Persistent EJB timers differ from non-persistent EJB timers in that persistent timers are transactional and written to a database. Operations such as scheduling, cancelling, and even running a timer happen within a transaction. The transaction can commit or roll back. When an execution of a timer rolls back, that timer execution can be retried. Timers that are persisted survive server restart and can optionally be configured to fail over to other servers.

Configuring persistent EJB timers for a single server

The default behavior of the ejbPersistentTimer-3.2 feature is such that a timer is only permitted to run on the server that scheduled the timer.
The only configuration necessary to enable persistent timers is the configuration of a data source. EJB persistent timers use the default data source:

<server>
  <featureManager>
    <feature>ejbPersistentTimer-3.2</feature>
    <feature>jdbc-4.2</feature>
  </featureManager>

  <dataSource id="DefaultDataSource">
    <jdbcDriver libraryRef="OracleLib"/>
    <properties.oracle URL="jdbc:oracle:thin:@//localhost:1521/SAMPLEDB"/>
    <containerAuthData user="dbuser1" password="dbpwd1"/>
  </dataSource>

  <library id="OracleLib">
    <file name="C:/Oracle/lib/ojdbcx.jar"/>
  </library>
</server>

If you prefer to avoid writing timers to the default data source, you can configure a different one,

  <databaseStore id="EJBTimerDatabaseStore" createTables="true" tablePrefix="EJBTimer_" dataSourceRef="TimerDB">
    <authData user="timersUser1" password="timersPwd1"/>
  </databaseStore>

  <dataSource id="TimerDB">
    <jdbcDriver libraryRef="OracleLib"/>
    <properties.oracle URL="jdbc:oracle:thin:@//localhost:1521/TIMERDB"/>
  </dataSource>

Finally, you can adjust various settings, such as how often retries should occur,

  <persistentExecutor id="defaultEJBPersistentTimerExecutor" retryInterval="30m" retryLimit="50"/>

Configuring persistent EJB timers for fail over across multiple servers

To enable fail over of timers, in addition to configuring the data source, you configure a missed task threshold, which is the maximum expected length for a timer execution before another server should take over the attempt to run it.

  <persistentExecutor id="defaultEJBPersistentTimerExecutor" missedTaskThreshold="5m"/>

Each server periodically polls the database for timers that ought to run within the next poll cycle which are not already claimed by another server. When it finds timers to run, it claims them for the duration of the missed task threshold beyond their next scheduled execution time. Claims on timer execution are relinquished, either when the duration expires, or on best-effort basis when the timer execution attempt ends (whether successfully or unsuccessfully). When timer execution rolls back and needs to be retried, the retries happen when servers poll the database and find the missed timer. Retries do not happen at the retryInterval, which is only for single-server mode. Polling is enabled and automatically computed by Liberty if you enable the missed task threshold. To explicitly specify the poll interval for a server, use the following configuration,

  <persistentExecutor id="defaultEJBPersistentTimerExecutor" missedTaskThreshold="5m" pollInterval="30m"/>

It is important to note that an explicitly configured poll interval applies to each server upon which it is configured. In the above example, if you have 4 servers, expect to see around 4 polls of the database within a given 30 minute interval. In cases where you have a large number of servers, it might be undesirable to have all of the servers polling the database. In that case, you can omit the poll interval configuration and let Liberty coordinate it across servers. Or, you can configure some of the servers to avoid polling by disabling their ability to execute timer tasks. Timers that are scheduled a server that cannot run timers are still able execute on different servers where execution of timer tasks is enabled.

  <!-- only do this on servers where you don't want timers to run -->
  <persistentExecutor id="defaultEJBPersistentTimerExecutor" missedTaskThreshold="5m" enableTaskExecution="false"/>

Switching from single-server mode to fail over mode

It is generally possible to switch from single-server mode to fail over mode even if you have active or pending timers. This involves removing the retryInterval (if specified), which is not valid when fail over is enabled, and configuring missedTaskThreshold. It is even possible to switch if you have multiple servers running in single-server mode which are pointing at the same database tables and schema. The servers can be updated one by one if necessary while the environment is being switched, but do not leave the environment in a mixed state where some servers are trying to operate in single-server mode while others are operating in fail over mode. Also, once servers are switched to fail over mode, do not switch back to single-server mode while there are active or pending timers, because these timers will remain in fail over mode without any servers capable of running them.

Configuring missed persistent timer behavior

Since persistent timers survive a server restart, scheduled timer expirations for repeating and calendar based timers may be missed while a server is not actively running. The action performed when multiple persistent timer expirations have been missed may be configured on the timer service using the "missedPersistentTimerAction" attribute. Note that the default behavior is different depending on whether fail over has been enabled.

The configurable actions are:

ALL
The timeout method is invoked immediately for all missed expirations. When multiple expirations have been missed for the same timer, each invocation will occur synchronously until all missed expirations have been processed, then the timer will resume with the next future expiration.

ALL is the default when fail over is not enabled.

ONCE
The timeout method is invoked once immediately. All other missed expirations are skipped and the timer will resume with the next future expiration. When the timer runs on server start, calling getNextTimeout() will return the next timeout in the future, accounting for all the expirations that will be skipped, not the next timeout based on the missed expiration (i.e. so not a time in the past)

ONCE is the default behavior when failover is enabled.

Note: The missed persistent timer action does not apply to single action timers. Single action timers will always run once on server start, and then removed.

To change the missed timer action when fail over is not enabled to invoke the timer once and then skip all other missed expirations, set the missed persistent timer action to ONCE:

    <ejbContainer>
        <timerService missedPersistentTimerAction="ONCE"/>
    </ejbContainer>

When using the ONCE action, if a persistent timer is scheduled to expire every 30 seconds, and the server is down for 20 minutes, then 40 expirations will be missed during the 20 minutes, but the timer will run just once on server start to catch up on all the missed expirations and then resume at the next scheduled 30 second interval.

To change the missed timer action when fail over is enabled to invoke the timer for all missed expirations, set the missed persistent timer action to ALL:

    <ejbContainer>
        <timerService missedPersistentTimerAction="ALL"/>
    </ejbContainer>

When using the ALL action, if a persistent timer is scheduled to expire every 30 seconds, and the server is down for 20 minutes, then 40 expirations will be missed during the 20 minutes and so the timer will run 40 times, immediately, on server start to catch up on all the missed expirations and then continue at the next 30 second interval.

Start in the ref/general directory of the OpenLiberty/docs repository and review nearby pages before creating the new page. Use the issue text as the source for persistent EJB timer background, single-server configuration, failover, mode switching, and missed-timer behavior. Done means the page is added under ref/general and covers the supplied configuration examples and behavior.