Resolving Global Sync Issues with Concurrent Changes and Parallel Processing

Overview

Global Sync may encounter issues when multiple requests for the same user or related entries arrive simultaneously or in rapid succession. This can result in:

• Changes being missed or skipped when coming in too quickly
• Incorrect attribute assignments across different users
• Role additions not completing the full process to update dependent attributes
• ConcurrentModificationException errors in sync engine logs

Environment

• Product: Global Sync
• Version: 7.4.x
• Scenario: Multiple role additions, password resets, or other changes sent through pipelines to the same user or topology

Root Cause

In earlier ICS versions, Apply processing was sequential, allowing dependent changes to be applied one after another. The second change could wait until the first request was fully applied before processing.

In the current Global Sync engine, the Apply process is parallelized:

• Polls for change events in batches (up to 100 changes by default)
• Applies changes concurrently within each batch
• Does not maintain ordering between events in the same batch

When multiple dependent updates (such as roles and authorities for different users) fall into the same batch and are applied in parallel, one event may start before the previous object's change is fully applied, causing:

• Timing conflicts between dependent entries in the same topology
• Shared state issues in transformation logic
• Cross-user attribute assignments when static/shared variables are used

Solution

1. Refactor Transformation Code to Eliminate Shared State

Remove static variables that can be shared across multiple concurrent events:

• Move variables into method scope instead of class-level static declarations
• Avoid reusing lists or collections across different user updates
• Ensure each event has its own isolated state

Example of problematic code patterns to avoid:

textprivate static String opType
private static ListObject valuesPatch
private static ListString patchValues

Instead, declare variables within methods or pass them as parameters to ensure isolation between concurrent processing threads.

2. Maintain Single Connection Per Processing Cycle

Keep database/LDAP connections open for the duration of processing rather than connecting and disconnecting after each change. This improves consistency and reduces timing issues.

3. Apply Topology Changes Properly

When making code changes to transformation scripts:

• Stop the topology
• If simple restart doesn't reflect changes, delete and recreate the topology
• Start the topology after recreation
• Verify changes are reflected by testing with sample data

4. Verify Object Classes and Prerequisites

Ensure target entries have required object classes before attempting attribute modifications. Missing object classes can cause silent failures where changes don't apply.

5. Configuration Settings

While sequential processing is not configurable in Global Sync, consider these settings:

• Polling Interval: Default is 5000ms; can be adjusted (e.g., 10000ms)
• Message Batch Size: Default is 100; reducing this value decreases the number of concurrent changes processed together but may cause queue buildup if Apply is slow
• Max Retries On Error: Default is 5
• Retry Interval On Error: Default is 10000ms

Important Limitations

• Global Sync does not provide configuration to force sequential processing within a batch
• The Apply Batch MAX size of 100 is hardcoded
• Setting Message Batch Size to 1 may help process events more sequentially, but can cause queue piling if Apply is slow
• Adding artificial delays (Thread.sleep) in transformation code is not recommended as a primary solution

Verification Steps

After implementing the solution:

• Monitor sync engine logs for ConcurrentModificationException errors
• Verify all role assignments, authorities, and product attributes are correctly applied
• Run comparison scripts to validate source and target data consistency
• Check that changes for multiple users processed simultaneously are all reflected correctly

Additional Notes

If an apply event fails due to backend or view errors, it will be returned to the queue for retry (5 attempts with 5-second intervals) before being moved to the dead letter queue.

Related Error Messages

• java.util.ConcurrentModificationException - message left in queue - will retry
• java.lang.ArrayIndexOutOfBoundsException - message left in queue - will retry
• Failure when processing sync event