Engine Data Deletion

The engine slows down if the historic data grows significantly over time, particularly in cases where the amount of data streaming in each day is large. One solution to this is to remove old data from the engine on a regular basis, still importing the data to Optimize so it can still be kept for deeper analytics.

To support the described use-case, Optimize does not care if you delete any data on the engine side. Specifically, Optimize does not sync with the engine on data deletion. If you want to remove any data from Optimize you can either erase the data from ElasticSearch or use the Optmize History Cleanup Feature .

The subsections below describe the ways in which Optimize handles data deletion from the engine.

Deletion of historic Data

There are two possible ways of doing this:

• Historic Cleanup: If you have enabled the history cleanup, historic data will be removed in the engine based on configurable time-to-live.
• Manual Deletion: You trigger a manual deletion, e.g. via REST-API.

Optimize can handle this case as it imports the old data first and once it has imported everything it will only add to its database new data streaming in.

Manual Deletion of Deployments/Definitions

In most cases, the deletion of deployments and definitions will not cause any problems. If you have several versions of a definition deployed, e.g. 1-4, and you delete the definition/deployment with definition version 1, then this wouldn’t cause any issues. The assumption here is that Optimize has imported the definitions and related historical data beforehand. Otherwise, the deleted definition is lost.

However, there are two scenarios, Optimize can have troubles with. The first is depicted as follows:

1. You deploy a (process/decision) definition A the first time.
2. Optimize imports definition A with version 1.
3. You delete the definition/deployment of the definition without having added another version of it. Definition A with version 1 is removed from the engine.
4. You deploy the definition A with the same ID again.
5. Optimize imports another definition A with version 1.

In this case, Optimize will import the same process twice. The reason is that Optimize identifies the unique definitions by the combination of definition key, version and tenant. As a result, if you select definition A in Optimize the same version would show up multiple times. To prevent this from happening, avoid deleting and redeploying the same definition (same definition key) to the engine.

Secondly, Optimize can have troubles with deleted decision definitions, where the corresponding decision instance data is still present in the history tables of the engine. In this case, Optimize will get stuck importing the instance data that hasn’t been imported yet, because it depends on the decision definition to be available. To avoid Optimize getting caught in the import, make sure the deletion of a decision definition/deployment is cascading to the respective history data.