Learn how to set up multiple engines with Optimize and which scenarios are supported.

Possible multiple engines scenarios

There are two possible setups where multiple engines can be used:

• Multiple engines with a shared database
• Multiple engines with a distributed database

Please check which scenario corresponds to your setup because the configuration of multiple engines to Optimize is not always suited for the best import performance.

Multiple engines with a distributed Database

In this scenario you have multiple engines distributed in a cluster, but this time each engine has its own database. See the following diagram for an illustration:

Now you are able to connect each engine to Optimize. The data will then automatically be imported into Optimize. The following diagram depicts the setup:

In order to set up the connections to the engines, you need to add the information to the configuration file. For the sake of simplicity, let’s assume we have two departments, sales and marketing, each having their own engine with its own database and processes. Both are accessible in the local network. The sales engine has the port 8080 and the marketing engine the port 1234. Now an excerpt of the configuration could look as follows:

engines:
  sales:
    name: default
    rest: http://localhost:8080/engine-rest
    authentication:
      enabled: false
      password: ""
      user: ""
    enabled: true
  marketing:
    name: default
    rest: http://localhost:1234/engine-rest
    authentication:
      enabled: false
      password: ""
      user: ""
    enabled: true

sales and marketing are custom names that were choosen in order to later on distinguish where the data was originally imported from.

Multiple engines with a shared Database

In this scenario you have multiple engines distributed in a cluster, where each engine instance is connected to a shared database. See the following diagram for an illustration:

Now it could be possible to connect each engine to Optimize. Since every engine accesses the same data through the shared database, Optimize would import the engine data multiple times. There is also no guarantee that importing the same data multiple times will cause any data corruption. For this reason it is highly recommended to not use the setup depicted above.

Instead of a multiple engine scenario with a shared database, it might make sense to balance the work load on each engine during the import. To do that place a load balancer between the engines and Optimize, which ensures that the data is imported only once and the load is distributed among all engines. Thus, Optimize would only communicate to the load balancer. The following diagram depicts the described setup:

In general, tests have shown that the strain Optmize puts on the engine is very low and in almost all cases neglectable.

Authentication/Authorization in the multiple engine setup

When you configured multiple engines in Optimize, each engine can host different users with a different set of authorizations. If a user is logging in, Optimize needs to decide which engine to authenticate against and retrieve the authorizations from. In case you are not familiar of how the the authorization/authentication works for a single engine scenario, please checkout the User Access Management and Authorization Management documentation first.

To determine if a user is allowed to log in and which resources he is allowed to access within the multiple engine scenario, Optimize uses the following algorithm:

Given the user X logs into Optimize, go through the list of engines in a round robin fashion and as soon as X could be authenticated against one of the engines and is authorized to access Optimize, fetch the permissions for X from that engine and allow X to access Optimize.

To give you a better understanding of how that works, let’s take the following multiple engine scenario:

- Engine `marketing`:
  - User without Optimize Permission: Scooter, Walter
  - User with Optimize Permission: Kermit, Gonzo
  - Authorized Definitions for Kermit/Gonzo/Scooter: Lead Qualification

- Engine `sales`:
  - User with Optimize Permission: Piggy, Scooter
  - Authorized Definitions for Piggy, Scooter: Invoice Process

- Engine `hr`:
  - User with Optimize Permission: Gonzo
  - Authorized Definitions for Gonzo: Hiring

Here are some examples that might help you to understand the authentication/authorization procedure:

• If Piggy logged in to Optimize, she would be granted access to Optimize and create reports for the definition Invoice Process.
• If Walter or Rizzo logged in to Optimize, they would be rejected because Rizzo is not known to any engine and Walter is known but does not have the permission to access Optimize.
• If Scooter logged in to Optimize, he would be granted access to Optimize and create reports for the definition Invoice Process. He wouldn’t get permissions for the Lead Qualification definition, since he doesn’t have Optimize permissions on the marketing engine.
• If Gonzo logged in to Optimize, he would be granted access to Optimize and create reports for the definition Lead Qualification. He wouldn’t get permissions for the Hiring definition, since the authentication check is done in a round robin fashion. Optimize would first check the marketing engine and find out that Gonzo is already a valid user with permissions and stop continuing to check the other engines.