The following image illustrates the components involved in the import process as well as basic interactions between them:

During execution, the following steps are performed:

1. Start an import round
2. Prepare the import
   • 2.1 Poll a new page
   • 2.2 Map entities and add an import job
3. Execute the import
   • 3.1 Poll a job
   • 3.2 Persist the new entities to Elasticsearch

Start an import round

The import process is triggered automatically after startup of Optimize in a separate thread, implemented by the ImportScheduler class. The scheduler works in rounds. In each import round, multiple import services are scheduled, which are responsible for fetching one specific type of entity each, conducted by a dedicated ImportService. E.g., one service is responsible for importing the historic activity instances and another one for the process definitions.

For each service, it is checked if new data was imported. Once all entities for one import service have been imported, the service starts to backoff. To be more precise, before it can be scheduled again it stays idle for a certain period oftime, controlled by the “backoff” interval and a “backoff” counter. After the idle time has passed , the service can perform another try to import new data. Each round in which no new data could be imported, the counter is incremented. Thus, the backoff counter will act as a multiplier for the backoff time and increase the time to be idle between two import rounds. This mechanism is configurable using the following properties:

import:
  handler:
    backoff:
      # Interval which is used for the backoff time calculation.
      interval: 5000
      # Once all pages are consumed, the import service component will
      # start scheduling fetching tasks in increasing periods of time,
      # controlled by "backoff" counter.
      max: 15

If you would like to rapidly update data imported into Optimize, you have to reduce this value. However, this will cause additional strain on the engine an might influence the performance of the engine if you set the value to low.

Prepare the import

The preparation of the import is executed by the ImportService. Thereby, every ImportService implementation performs several steps:

Poll a new page

The whole polling/preparation workflow of the engine data is done in pages, meaning we only fetch a certain amount of entities. For example, we could assume that the engine has 1000 historic activity instances (HAI) and the page size is 100. As a consequence, the engine is polled 10 times. This prevents running out of memory and overloading the network.

Polling a new page does not only consist of the ImportService, but the IndexHandler and the EntityFetcher are also involved. The following image depicts how those components are connected with each other:

First, the ImportScheduler retrieves the newest index, which is saying where to start fetching lines in a table. This index is passed to the ImportService to order it to import a page of data. With the index and the page size, the fetching of the engine data is deligated to the EntitityFetcher.

Map entities and add an import job

All fetched entities are mapped to a representation that allows Optimize to query the data very quickly. Subsequently, an import job is created and added to the queue to persist the data in Elasticsearch.

Execute the import

Full aggregation of the data is performed by the ImportJobExecutor, which waits for ImportJob instances to be added to the execution queue. As soon as a job is in the queue, the executor

• polls the job with the new Optimize entities
• persists the new entities to Elasticsearch.

The data from the engine and Optimize do not have a one-to-one relationship, i.e., one entity type in Optimize may consist of data aggregated from different data types of the engine. For example, the historic process instance is first mapped to an Optimize ProcessInstance. However, for the heatmap analysis it is also necessary for ProcessInstance to contain all activities that were executed in the process instance. Therefore, the Optimize is an aggregation of the engine’s historic process instance and its historic activity instances (and more, but we leave that here aside for the sake of simplicity). It is important to note that data in Elasticsearch is never updated after initial persistence.

Also note that the Start/Preparation and the execution are actually independent from another. They follow a producer-consumer-pattern, where the ImportService is the producer and the ImportJobExecutor is the consumer. So, both are executed in different threads. To adjust the processing speed of the executor, the queue size and the number of threads that process the import jobs can be configured:

import:
  # Number of threads being used to process the import jobs that are writing data to elasticsearch.
  elasticsearchJobExecutorThreadCount: 2
  # Adjust the queue size of the import jobs that store data to elasticsearch.
  # A too large value might cause memory problems.
  elasticsearchJobExecutorQueueSize: 100