Transient Processes

Process run information about all activity instances and process data values that are part of a business process instance is stored in the Audit Trail. There are situations in which this detailed information is not necessarily needed for all process types. Pure BPM-driven solutions are mainly requiring a full and detailed Audit Trail on an abstract level or business view. There are also many use cases in which Stardust is used more as a routing and transformation engine, whereas the persistence of all process run information is even less important. For these use cases the performance like latency or throughput is more critical. To reduce the amount of data stored in the database and increase the overall runtime performance of the engine, Stardust provides the option to use transient processes.

Performance Enhancements by using Transient Processes

Transient processes keep instances of the process in-memory only during the course of execution. No audit trail is persisted at all during or after the execution of the process instance. The corresponding objects are created during process execution and are available in the context of the activity threads running for that process instance.

Almost no remote calls to the database and no persisting of data is necessary, thus transient processes are executed faster and with lower latency.

If there is system downtime during process execution, the instances of a transient process do not leave a trace in the audit trail. However in case of unhandled faults, the information is written to the audit trail.

To summarize, using project transient processes provides the following performance enhancements:

There is almost no contention like Java synchronization or database locks, when running transient process instances due to:

Using Transient Processes

The process run information of transient processes is only kept in an internal in-memory storage, e.g. Hazelcast, during execution of the process instance and will be discarded when the process instance is completed.

For details on how to use transient processes refer to chapter Tuning for low Latency and high Throughput in the Operation Guide.

Serialization of activity threads

In usual processing of process instances several activity threads can be running in parallel in the scope of one process hierarchy. This is typically the case if several parallel transitions originate from one gateway of type AND -split. This could result in a very large number of parallel running threads. To avoid such a possible overhead, all activity threads in a transient process instance are executed sequentially. As in the normal (persistence Immediate ) case, the first transition modeled for that AND split will be executed first and in the same transaction. All other activity threads will be invoked via Messaging. The corresponding messages will be processed:

Mandatory Requirements for Transient Processing

Due to technical restrictions, several events can occur during the lifetime of a process instance that are incompatible with transient processing. These events will force the engine to persist the audit trail for this process instance in order to avoid permanent loss of data. This is the case when the process instance cannot be further processed without asynchronous actions of an external party, e.g. a participant or external system. An example for such a required interaction is the processing of an interactive activity or the triggering of a recovery. The current state of the process instance audit trail will be persisted and any subsequent processing of the process instance will be done in persistence mode Immediate for the following cases:

Note
Currently, asynchronous subprocesses are not supported for transient processing. Whenever the workflow of a transient or deferred process instance reaches a subprocess activity that invokes an asynchronous sub-process, the audit trail persistence is set to Immediate for both the parent as well as the sub-process.

Whenever a process instance changes persistence mode from Transient to Immediate or Deferred for one of the reasons listed above, this change in persistence mode will propagate and be applied to the entire process hierarchy from that point of execution onwards.

Further Requirements and Restrictions

Behavior in case of Application Exceptions

In case the property Carnot.Engine.ErrorHandling.ApplicationExceptionPropagation is set to value always and an ApplicationException occurs, the whole transient process instance graph is purged from the in-memory storage and written to the audit trail database such that a subsequent recovery action may recover the process instance. This also applies if the property is set to onRollback and an activity thread rolls back due to an uncaught exception and there are other activity threads already completed, e.g. committed against the internal cache.

Pull events are ignored for Transient Process Instance Execution

As long as a process instance is executed transiently, pull events, i.e. events triggered by the event daemon, are not processed. Instead a warning is logged whenever a transient process instance having a pull trigger event bound is written to the in-memory storage. This can be the case for example if its transaction is committed. The according log message looks as follows:

Event binding 'Event binding [oid = 1, type = 2, model oid = 2, object oid = 1, handler oid = 2]' applies to a transient process instance and will therefore not be processed by the event daemon.

Platforms supported for Transient Processing

Note that the usage of any audit trail storage mode other than Immediate requires a second level cache configured in the runtime, e.g. Hazelcast. Currently, the following platforms are covered:

Visibility of Transient Workflow Objects in API

Retrieving a Transient Process Instance via API

Retrieving a process instance or its related data via the WorkflowService or the AdministrationService will always return a snapshot of the currently executed process instance. After completion the snapshot is completely gone (unless its deferred). Since a transient process instance by definition is executed without waiting in a defined state, e.g. a suspended activity instance, there are no means to modify the state of the process instance safely. In general, to perform a safe modification you have to first determine the current state and afterwards modify it based on the retrieved state. However in case of transient processes as soon as the state has been retrieved it might have already changed. Therefore the outcome of any modifying operations is undefined.

Query Results

All QueryService methods operate against the persisted Audit Trail database exclusively. Hence any transient process instances do not influence any QueryService method results. Transient processes will never change any query results, even after their completion.

Use Case - Executing A Transient Process Instance

This use case describes the execution of a transient process instance, for example a process instance that executes without leaving a trace in the database.

Precondition

The precondition for this use case is one of the following:

Postcondition

The transient process instance has been completed without leaving a trace in the database.

Basic Flow of Events

Alternative Flow of Events

  1. Interactive/Asynchronous Activity:
    The process instance hits an activity instance that requires interaction or is executed asynchronously:
    The process instance (including all related information) will be written to the database - exactly as it would have been done for non-transient process instances.
  2. Execution Error:
    Something prevents the process instance from completing successfully (e.g. an error during process instance execution):
    The process instance (including all related information) will be written to the database - exactly as it would have been done for non-transient process instances. This will be done regardless of whether the process instance execution has been interrupted or the activity thread rescheduling retry count exceeded.
  3. Abort:
    The process instance is aborted during its execution:
    The process instance (including all related information) will be written to the database - exactly as it would have been done for non-transient process instances.
  4. Deferred Persist:
    The process instance execution is configured to allow for deferred persistence (either modeled via the underlying process definition or configured with the global setting):
    The process instance (including all related information) will be written to the database, but not until all related process instances are completed.