Ad Servers Inventory Sync Jobs Stuck in Pending Status

Problem

Ad Server Inventory synchronization jobs are completing their runs but the status remains "Pending", and no changes are applied to the Ad Servers. When attempting to manually synchronize, users encounter an error message: "There is a pending message for the Ad Server Inventory Synch on this record. New request cannot be sent before the response is processed."

Symptoms:
- Inventory sync jobs show status "Pending" indefinitely
- Manual sync attempts fail with "pending message" error
- No inventory updates are reflected on Ad Server records
- Sync jobs appear to complete (no errors in logs) but status never changes
- Issue persists across multiple Ad Server records
- Affects both scheduled and manual inventory synchronization

Affected CloudSense Modules:
- Digital Fulfilment (csdf)
- Ad Server Management (csam)

Root Cause

The csam batch engine (messaging engine) is not running. The Ad Server Inventory sync process uses asynchronous messaging to communicate with external ad servers. When the batch engine is not active, outgoing messages are sent but incoming responses are never processed, leaving jobs stuck in "Pending" status.

Technical Details:

The Ad Server Inventory sync workflow:
1. User triggers sync (manual or scheduled)
2. System creates an outgoing message to the ad server
3. Ad server processes the request and sends a response
4. Response processing requires the csam batch engine to be running
5. If the engine is not running, responses accumulate but are never processed
6. Job status remains "Pending" indefinitely

Resolution

Start the csam batch engine (messaging engine) to process pending responses and enable future synchronization.

Step 1: Access the Admin Console

1. Navigate to the Admin Console:
2. In your Salesforce org, go to: /apex/csam__AdminConsole

3. Or search for "Admin Console" in the App Launcher

4. Verify you have admin access:

5. You need System Administrator permissions or equivalent
6. If you don't see the Admin Console, contact your Salesforce administrator

Step 2: Start the Messaging Engine

1. Locate the Messaging Engine section:
2. In the Admin Console, find the "Messaging Engine" or "Batch Engine" section

3. This section shows the current status of the csam batch engine

4. Check current status:

5. If status shows "Not Running" or "Stopped", proceed to start it

6. If status shows "Running", the issue may be elsewhere (see Troubleshooting)

7. Start the engine:

8. Click the "Start" button next to the Messaging Engine
9. Wait for confirmation message: "Messaging Engine started successfully"

10. Verify status changes to "Running"

11. Verify batch job is scheduled:

12. Go to Setup > Apex Jobs > Scheduled Jobs
13. Look for a scheduled job named similar to: csam__AsyncMessagingBatchScheduler or AsyncMessagingBatch
14. Verify it's scheduled to run at regular intervals (e.g., every 5 minutes)
15. If not present, the Admin Console Start button should have created it

Step 3: Process Pending Messages

Once the messaging engine is running, pending responses will be processed automatically on the next batch run.

1. Wait for next batch execution:
2. The batch typically runs every 5 minutes

3. Wait 5-10 minutes for the first batch to execute

4. Monitor job status:

5. Go to Setup > Apex Jobs > Apex Jobs
6. Look for recent executions of the messaging batch job
7. Verify jobs complete successfully (Status: "Completed")

8. Check for any errors in the job details

9. Verify inventory sync status changes:

10. Navigate to your Ad Server records
11. Check the inventory sync status field
12. Status should change from "Pending" to "Completed" or "Failed" (with error details)
13. Verify inventory data is updated on the Ad Server record

Step 4: Retry Manual Sync (If Needed)

If you had attempted manual sync before starting the engine:

1. Wait for pending messages to clear:
2. Allow the batch engine to process all pending responses (5-10 minutes)

3. Verify Ad Server sync status is no longer "Pending"

4. Trigger manual sync again:

5. Navigate to the Ad Server record
6. Click the "Sync Inventory" button (or equivalent custom button)

7. Verify sync initiates successfully (no "pending message" error)

8. Monitor new sync job:

9. Check sync status field updates to "In Progress" then "Completed"
10. Verify inventory data is refreshed on the Ad Server record

Step 5: Verify Scheduled Syncs Work

If you use scheduled inventory synchronization:

1. Check scheduled sync configuration:
2. Navigate to Setup > Custom Settings
3. Find csam__AsyncMessOptions__c (if applicable)

4. Verify sync schedule settings are correct

5. Wait for next scheduled sync:

6. Allow the next scheduled sync to execute

7. Monitor Ad Server records for status updates

8. Verify syncs complete successfully:

9. Check that scheduled syncs no longer get stuck in "Pending"
10. Verify inventory data is updated regularly

Verification

After starting the messaging engine and processing pending messages:

1. Messaging Engine Status:
2. Go to /apex/csam__AdminConsole
3. Verify Messaging Engine shows "Running"

4. Verify scheduled batch job exists in Setup > Apex Jobs > Scheduled Jobs

5. Pending Messages Cleared:

6. Navigate to Ad Server records that were stuck
7. Verify sync status is no longer "Pending"

8. Check that inventory data has been updated

9. New Syncs Work:

10. Trigger a new manual sync on an Ad Server
11. Verify sync completes successfully (status changes to "Completed")

12. Verify no "pending message" errors occur

13. Batch Job Executes Regularly:

14. Go to Setup > Apex Jobs > Apex Jobs
15. Verify the messaging batch job executes every 5 minutes

16. Check recent executions show "Completed" status

17. No Errors in Logs:

18. Check Salesforce Debug Logs for any errors related to csam__AsyncMessagingBatch
19. Verify no exceptions or failures during message processing

Troubleshooting

Messaging Engine Won't Start

Symptoms: Clicking "Start" in Admin Console fails or shows error

Possible Causes:
- Insufficient permissions
- Apex governor limits reached
- Existing scheduled job conflicts

Resolution:
1. Verify you have System Administrator permissions
2. Check for existing scheduled jobs with the same name:
- Go to Setup > Apex Jobs > Scheduled Jobs
- Look for duplicate csam__AsyncMessagingBatchScheduler jobs
- Delete duplicates if found
3. Try starting the engine again

Status Still Pending After Starting Engine

Symptoms: Messaging engine is running but sync status remains "Pending"

Possible Causes:
- Batch hasn't executed yet
- External ad server is down or unreachable
- Authentication issues with ad server
- Message processing errors

Resolution:
1. Wait 10-15 minutes for batch to execute multiple times
2. Check Setup > Apex Jobs > Apex Jobs for batch execution logs
3. Review Debug Logs for errors during message processing
4. Verify ad server credentials are correct
5. Test ad server connectivity (ping, API test)
6. Check for errors in the csam__Incoming_Message__c or csam__Outgoing_Message__c objects

Manual Sync Still Shows "Pending Message" Error

Symptoms: Error persists even after engine is running

Possible Causes:
- Old pending message not yet processed
- Message stuck in processing state
- Corrupted message record

Resolution:
1. Wait for batch to process all pending messages (10-15 minutes)
2. If error persists, check the csam__Outgoing_Message__c object:
- Find messages related to your Ad Server record
- Look for messages with status "Pending" or "In Progress"
- Note the message IDs
3. Contact CloudSense support with message IDs for manual cleanup

Batch Job Not Executing

Symptoms: Scheduled job exists but doesn't execute

Possible Causes:
- Job is scheduled but inactive
- Apex governor limits preventing execution
- Org has reached scheduled job limit

Resolution:
1. Go to Setup > Apex Jobs > Scheduled Jobs
2. Find the csam__AsyncMessagingBatchScheduler job
3. Verify it's Active (not paused)
4. Check the "Next Run" time is in the future
5. If inactive, delete and restart the messaging engine from Admin Console
6. If org has reached scheduled job limit (100 max), delete unused scheduled jobs

Inventory Data Not Updating

Symptoms: Sync status changes to "Completed" but inventory data is unchanged

Possible Causes:
- Ad server returned empty response
- Inventory mapping configuration incorrect
- Field-level security preventing updates

Resolution:
1. Check the csam__Incoming_Message__c object for the response payload
2. Verify the response contains inventory data
3. Review inventory mapping configuration in Ad Server settings
4. Verify field-level security allows updates to inventory fields
5. Check Debug Logs for any field update errors

Prevention

To prevent the messaging engine from stopping in the future:

1. Monitor Engine Status:
2. Periodically check /apex/csam__AdminConsole to verify engine is running

3. Set up a monitoring alert if engine stops (if available)

4. Scheduled Job Monitoring:

5. Regularly review Setup > Apex Jobs > Scheduled Jobs
6. Verify csam__AsyncMessagingBatchScheduler is active and executing

7. Check for any failed executions and investigate errors

8. Org Maintenance:

9. Avoid deleting or modifying the messaging batch scheduled job
10. After org refreshes or deployments, verify the engine is still running

11. Include engine status check in post-deployment verification

12. Apex Governor Limits:

13. Monitor Apex CPU time and heap size usage
14. If batch jobs frequently hit limits, consider optimizing message processing logic

15. Review csam__AsyncMessOptions__c custom settings for batch size configuration

16. Documentation:

17. Document the messaging engine requirement in your org's runbook
18. Train administrators on how to start/stop the engine
19. Include engine status check in troubleshooting guides

Related Issues

• All Async Operations Stuck: If other async operations (not just inventory sync) are stuck, the messaging engine is likely stopped
• Outgoing Messages Not Sent: If outgoing messages are not being sent at all, check network connectivity and ad server credentials
• Batch Job Errors: If batch jobs fail with errors, review Debug Logs and check Apex governor limits

Important Notes

• Messaging engine must remain running for all asynchronous ad server operations
• Do not delete the scheduled job - it's required for message processing
• Pending messages will be processed automatically once engine is started (no manual intervention needed)
• Sync frequency: Batch typically runs every 5 minutes; adjust via csam__AsyncMessOptions__c if needed
• Multiple Ad Servers: Starting the engine once fixes all Ad Servers in the org
• No data loss: Pending messages are preserved and will be processed once engine starts