This guide walks you through migrating your Optimize installation from Elasticsearch 8.x to Elasticsearch 9.x.

Prerequisites

Before starting the migration, ensure:

1. Optimize version: You are running Optimize 3.15.15 or later. ES9 support was introduced in this version.
2. Elasticsearch version: Your current Elasticsearch version is 8.19 or later. Elasticsearch enforces sequential major version upgrades — you cannot upgrade directly from 8.17 or 8.18 to 9.x. If you are on an earlier version, upgrade to at least 8.19 first.
3. Snapshot: You have created a full Elasticsearch snapshot of your cluster. This is mandatory — it is your only rollback mechanism.
4. Disk space: Ensure sufficient disk space for the snapshot and the upgrade process.
5. Plugins: If you use custom Optimize plugins that directly import Apache HttpClient 4 (HC4) classes, recompile them against the new Optimize version. The DatabaseCustomHeaderSupplier interface is unchanged, but direct HC4 imports will no longer resolve.

Migration Steps

Step 1: Create an Elasticsearch snapshot

Create a full snapshot of your Elasticsearch 8 cluster:

# Register a snapshot repository (if not already configured)
curl -X PUT "http://localhost:9200/_snapshot/my_backup" -H 'Content-Type: application/json' -d '{
  "type": "fs",
  "settings": {
    "location": "/path/to/backup/location"
  }
}'

# Create a snapshot
curl -X PUT "http://localhost:9200/_snapshot/my_backup/pre_es9_migration?wait_for_completion=true"

Verify the snapshot completed successfully:

curl -X GET "http://localhost:9200/_snapshot/my_backup/pre_es9_migration"

The response should show "state": "SUCCESS".

Step 2: Shut down Optimize

Stop all running Optimize instances to ensure no writes occur during migration.

Step 3: Upgrade Elasticsearch from 8.x to 9.x

Follow the official Elasticsearch upgrade guide to upgrade your Elasticsearch cluster from 8.x to 9.x.

For single-node setups:

1. Stop Elasticsearch 8.
2. Install Elasticsearch 9.x (9.0 or later).
3. Start Elasticsearch 9.

For clustered setups:

Follow the rolling upgrade procedure documented by Elastic. Upgrade one node at a time to minimize downtime.

Step 4: Deploy and run Optimize with --upgrade

Deploy the new Optimize version (3.15.15+) and start it with the --upgrade flag:

# Linux
./optimize-startup.sh --upgrade

# Windows
./optimize-startup.bat --upgrade

Or in Docker environments:

services:
  optimize:
    image: registry.camunda.cloud/optimize-ee/optimize:3.15.15
    command: ["./optimize.sh", "--upgrade"]

What --upgrade does

The upgrade process: - Removes legacy /_template/optimize-* index template entries (deprecated in ES9). - Recreates them as composable /_index_template/optimize-* index templates. - Does NOT upgrade your Elasticsearch server — that must be done separately (Step 3).

Step 5: Verify the migration

After Optimize starts successfully, verify:

1. Health check:

curl -s -o /dev/null -w "%{http_code}" http://localhost:8090/api/readyz
# Expected: HTTP 200 when Optimize is ready; HTTP 503 means Optimize is not ready yet
# HTTP 000 means Optimize has not started yet

2. Composable index templates are in place:

curl -X GET "http://localhost:9200/_index_template/optimize-*"
# Expected: Returns your Optimize index templates

3. Legacy templates are removed:

curl -X GET "http://localhost:9200/_template/optimize-*"
# Expected: HTTP 404 or empty result

4. Data integrity: - Open the Optimize UI and verify your reports and dashboards load correctly. - Spot-check process instance data and decision data.

Rollback Procedure

If you encounter issues after migrating to Elasticsearch 9, you can restore from your pre-migration snapshot. Direct downgrade from ES9 to ES8 is not supported — ES9 writes data using Lucene 10 format, which ES8 (Lucene 9) cannot read. Attempting to start ES8 against an ES9 data directory will fail with a fatal indexCreatedVersionMajor is in the future error. Rollback requires a fresh ES8 installation with snapshot restore.

Steps to rollback

1. Stop Optimize and Elasticsearch 9:

# Linux — stop Optimize (if started with optimize-startup.sh)
pkill -f "optimize-backend.*\.jar"

# Linux — stop Elasticsearch (systemd)
sudo systemctl stop elasticsearch

# Docker — stop Elasticsearch 9 container (do NOT use -v; keep snapshot volume)
docker compose -f docker-compose.es9.yml down

1. Install Elasticsearch 8 with a fresh data directory:

You cannot reuse the Elasticsearch 9 data directory — ES9 writes Lucene 10 format which ES8 cannot read. Install or start a fresh Elasticsearch 8.19+ instance with an empty data directory.

# Docker — remove the ES9-written data volume, then start ES8 fresh
docker volume rm <compose-project>_es-data
docker compose -f docker-compose.es8.yml up -d

# Wait until ES8 is healthy
until curl -s "http://localhost:9200/_cluster/health" | grep -qE '"status":"(green|yellow)"'; do
  echo "Waiting for ES8..."; sleep 3
done
echo "ES8 ready"

1. Register the snapshot repository and restore:

# Register the same snapshot repository used before migration
curl -X PUT "http://localhost:9200/_snapshot/my_backup" \
  -H 'Content-Type: application/json' -d '{
  "type": "fs",
  "settings": {
    "location": "/path/to/backup/location"
  }
}'

# Verify the snapshot is accessible
curl -s "http://localhost:9200/_snapshot/my_backup/pre_es9_migration" | grep '"state":"SUCCESS"'

# Restore the pre-migration snapshot (restores cluster state; only use this on a dedicated cluster or where overwriting global state is acceptable)
curl -X POST "http://localhost:9200/_snapshot/my_backup/pre_es9_migration/_restore?wait_for_completion=true" \
  -H 'Content-Type: application/json' -d '{
  "indices": "optimize-*",
  "include_global_state": true
}'

Expected response contains "shards":{"total":...,"successful":...,"failed":0}.

1. Restart Optimize (without --upgrade):

# Linux
./optimize-startup.sh

# Windows
./optimize-startup.bat

# Docker
docker compose up optimize

1. Verify that data is intact and Optimize is operational:

# Confirm ES8 version
curl -s "http://localhost:9200" | grep '"number"'

# Check legacy templates are restored
curl -s "http://localhost:9200/_template/optimize-*" | grep '"index_patterns"'

# Open the Optimize UI and confirm reports and dashboards load correctly

FAQ

Can I skip Elasticsearch 8 and go directly to Elasticsearch 9?

No. Elasticsearch enforces sequential major version upgrades. You must be on Elasticsearch 8.19+ before upgrading to 9.x. Upgrading from 8.17 or 8.18 directly to 9.x will fail at startup with a fatal exception.

Can I run Elasticsearch 9 without the --upgrade flag?

• Fresh installations (no pre-existing Optimize data): Yes, --upgrade is not needed.
• Upgrades from previous Optimize versions: No. You must run --upgrade to migrate legacy index templates.

How long does the migration take?

• The Elasticsearch server upgrade time depends on your data size and cluster configuration.
• The Optimize --upgrade step (index template migration) typically completes in under 5 minutes.

What if --upgrade fails?

The upgrade process is resumable. If interrupted, simply re-run ./optimize-startup.sh --upgrade and previously completed steps will be skipped. If issues persist, restore from your snapshot and contact support.

Do I need to update my Optimize plugins?

Only if your plugins directly import Apache HttpClient 4 classes. The DatabaseCustomHeaderSupplier plugin interface is unchanged. Plugins that only use the documented plugin API will work without changes.