Orchestration Steps Stuck Due to Custom Handler Exceeding SOQL Governor Limit

Symptom

Orchestration processes that work in Foreground mode get stuck when running in Background mode. Steps remain in "In Progress" status indefinitely. Switching the orchestration process from Background to Foreground causes steps to progress immediately.

Other orchestration processes in the same org may complete normally in Background mode, indicating the issue is intermittent and specific to certain processes.

Cause

A custom orchestration step handler (e.g., a custom Apex class invoked by the Orchestrator Accelerator) is exceeding the Salesforce SOQL governor limit (101 queries per transaction). When the step fails due to this limit, the transaction rolls back, leaving the step in "In Progress" status. The background engine retries the step on the next cycle, but it fails again with the same error, creating an infinite retry loop.

This single stuck step blocks the entire custom step processing queue, preventing other orchestration processes from progressing their custom steps.

Resolution

Step 1: Identify the stuck step

Check the orchestration queue for steps that have been stuck for an extended period:

SELECT LastModifiedDate, Id, Name, CSPOFA__Status__c,
  CSPOFA__Type__c, CSPOFA__Class__r.Name,
  CSPOFA__Orchestration_Process__r.Name
FROM CSPOFA__Orchestration_Step__c
WHERE CSPOFA__Status__c = 'In Progress'
  AND CSPOFA__Step_On_Hold__c = false
  AND CSPOFA__Orchestration_Process__r.CSPOFA__Process_On_Hold__c = false
  AND CSPOFA__Orchestration_Process__r.CSPOFA__State__c = 'ACTIVE'
ORDER BY LastModifiedDate ASC
LIMIT 50

Look for steps that have been in "In Progress" for hours or days without progressing.

Step 2: Check debug logs for the error

Enable debug logging for the Orchestrator Accelerator user and reproduce the error. Look for the SOQL governor limit exception in the logs:

System.LimitException: Too many SOQL queries: 101

The stack trace will identify the custom handler class and method causing the issue.

Step 3: Put the stuck step on hold

Set CSPOFA__Step_On_Hold__c = true on the stuck step to remove it from the processing queue. This immediately unblocks the custom step queue for other processes.

Step 4: Fix the custom handler code

Optimize the custom Apex handler to reduce the number of SOQL queries. Common fixes include:
- Bulkifying queries (querying outside loops)
- Using collections to reduce query count
- Caching query results

Step 5: Verify the queue is cleared

After putting the stuck step on hold, verify that other orchestration processes resume processing normally.

Additional Notes

• A single stuck step in "In Progress" blocks the entire custom step processing queue because the engine processes custom steps sequentially
• Foreground mode processes steps directly in the user's transaction, which is why switching to Foreground appears to "fix" the issue -- it bypasses the background queue entirely
• The background engine's step processing limit per transaction is configurable via Orchestrator_Constants__c custom settings (custom_steps_per_batch)
• Enable History_Trigger_On in Orchestrator_Constants__c to track step state transitions for easier debugging of future issues