Import Overview

Architecture Overview

In general, the import assumes the following setup:

  • A Camunda engine, where the data is imported from.
  • The Optimize back-end, where the data is transformed to an appropriate format for efficient data analysis.
  • Elasticsearch, which is the database of Optimize, to where the formatted data is persisted.

The following depicts the setup and how the components communicate with each other:

Hereby, the main idea is that Optimize queries the engine data using the REST-API and transforms the data such that it can be quickly and easily queried by Optimize. To prevent the engine from producing too much load from the Optimize queries and at the same time to make the import fast, Optimize adapts the amount of queries automatically to the engine’s response time.

Furthermore, one should be aware of what the general requirements for the data in Optimize are:

  • Optimize does not own the data of the engine. The Optimize dataset can always be removed and reimported or adapted to the needs of Optimize.
  • The data is only a near real-time representation of the engine database. That means, Elasticsearch may not contain the data of the most recent time frame, e.g. the last two minutes, but all the previous data should be synchronized.

If you are interested in the details of the import, have a look at the designated section Import Procedure.

Import performance overview

This section gives an overview of how fast Optimize imports certain datasets to get a feeling of the import speed of Optimize and if it meets certain demands. However, this presumably changes on different datasets and onhow all involved components are set up. E.g., if you deploy the Camunda platform on a different machine than Optimize and Elasticsearch the process is likely to speed up.

Setup

The following components were used within the import:

Component Version
Camunda Platform 7.8.0 on a Tomcat 8.0.47
Elasticsearch 5.4.0
Optimize 1.5.0

The Optimize configuration with the default settings was used, as described in detail in the configuration overview.

All three of the following components were running on a single laptop:

  • Processor: Intel® Core™ i5 (6. Generation),6440HQ Processor,4x 2.60 GHz
  • Working Memory: 16 GB (DDR4)
  • Storage: 192 SSD (SSD)

The time was measured from the start of Optimize until the whole import of the data to Optimize was finished.

Medium size data set

This dataset contains the following amount of instances:

Number of Process Definitions Number of Activity Instances Number of Process Instances Number of Variable Instances
46 1 427 384 261 106 1 273 324

Here you can see how the data is distributed over the different process definitions:

Results:

  • Duration of importing the whole data set: ~23 minutes
  • Speed of the import: 1700-2200 database rows per second during the import process

Small data set

This dataset contains the following amount of instances:

Number of Process Definitions Number of Activity Instances Number of Process Instances Number of Variable Instances
13 777 340 73 487 2 387 146

Here you can see how the data is distributed over the different process definitions:

Results:

  • Duration of importing the whole data set: ~10 minutes
  • Speed of the import: 1700-2200 database rows per second during the import process

On this Page: