Introduction: The Challenge of Handling Large JSON Payloads
Working with large datasets in Salesforce can be challenging, especially when dealing with JSON payloads that push the limits of Salesforce’s heap memory. In Salesforce, synchronous processing only allows up to 6MB of heap memory, while asynchronous processing extends this to 12MB. Recently, I encountered a situation where even the 12MB asynchronous limit was insufficient for a client project. We needed a way to process and display this data efficiently without running into memory issues.
To tackle this, we re-engineered our approach using Salesforce Platform Events and Lightning Web Components (LWC). This solution enabled us to break down the data, process it in manageable batches, and update the UI in real-time, ensuring a responsive and scalable experience. Here’s how Platform Events and LWC saved the day.
Problem Statement: JSON Payload and Heap Memory Limits in Salesforce
When dealing with data-intensive applications, Salesforce’s heap memory limits can quickly become a roadblock:
- Synchronous Processing Limit: 6MB of heap memory
- Asynchronous Processing Limit: 12MB of heap memory
In this case, the client required large JSON payloads to be processed within Salesforce, but the payloads were regularly exceeding the 12MB heap limit. Traditional approaches were insufficient, as they led to errors and an inconsistent user experience.
Why This Was a Problem
The heap memory limits meant that:
- Larger payloads couldn’t be processed in a single transaction, causing failures and slowing down data processing.
- The UI was not updating in real time, leaving users with delayed information and a lack of responsiveness.
It was clear that we needed a solution that could handle large datasets without hitting memory limits while keeping the UI responsive.
Solution: Leveraging Platform Events and LWC for Scalable, Real-Time Processing
To address these limitations, we turned to Platform Events and LWC. Here’s how each part of this solution helped overcome the challenge.
Platform Events for Asynchronous Data Processing
Salesforce Platform Events allow for asynchronous communication, making them an ideal tool for handling large data in distributed systems. Here’s how we used them to tackle the memory issue:
- Breaking Down Large Payloads: Instead of processing the entire payload in a single transaction, we divided it into smaller, manageable batches.
- Publishing Platform Events: Each batch was published as a Platform Event, triggering downstream processes that could handle data incrementally.
- Batch Processing: Platform Events allowed us to process each batch independently, keeping memory usage low and within Salesforce’s limits.
This approach enabled us to bypass the 12MB heap limit by treating each event as a discrete transaction, making it possible to handle large JSON payloads without triggering errors.
Lightning Web Components (LWC) for Real-Time UI Updates
With Lightning Web Components, we created a responsive front-end that subscribed to Platform Events and updated in real time. Here’s how LWC enhanced the solution:
- Event Subscriptions in LWC: LWC components subscribed to the Platform Events, allowing them to listen for updates as each batch was processed.
- Real-Time UI Refresh: Each time a batch was processed and an event was published, the UI automatically refreshed, displaying the latest data without requiring a full page reload.
- Improved User Experience: Users could see updates as they happened, creating a seamless experience where they could trust that the data was current.
By combining Platform Events and LWC, we ensured that the UI remained in sync with data processing, giving users real-time insights without memory-related interruptions.
Benefits of Using Platform Events and LWC in This Solution
1. Asynchronous Processing for Efficiency
Platform Events allowed us to shift from a synchronous, memory-intensive process to a scalable asynchronous one. By handling data in batches, we minimized the risk of hitting memory limits and achieved more efficient data processing.
2. Enhanced Scalability
This solution can scale to handle even larger datasets in the future. As data volumes grow, Platform Events can be configured to handle additional records in parallel, making it a robust solution for high-growth environments.
3. Real-Time User Experience
With LWC’s real-time subscription to Platform Events, users received immediate updates as data was processed. This enhanced responsiveness ensured that users always had access to the latest information.
4. Future-Proof Design
Using Platform Events and LWC lays the groundwork for future enhancements, such as integrating with other external systems or adding more complex data processing flows without changing the core architecture.
