Data Mirroring using Events

Overview

This guide provides best practices and patterns for processing lemon.markets webhook events to maintain a synchronized mirror system. A mirror system replicates the state of user accounts, orders, transactions, and other entities in your own database by consuming webhook events in real-time.

Architecture Overview
Event Processing Principles
Entity State Management
Event Processing Patterns
Error Handling and Recovery
Data Consistency Strategies
Implementation Checklist

Architecture Overview

Core Components

A robust webhook processing system requires:

Webhook Receiver: HTTP endpoint to receive events from lemon.markets
Event Queue: Message queue for reliable async processing
Event Processor: Workers that process events and update the mirror database
Mirror Database: Your local database storing the replicated state
Reconciliation Service: Periodic sync to detect and fix inconsistencies

graph LR
    A[lemon.markets] -->|Webhook| B[Webhook Receiver]
    B -->|Store Event| C[Event Queue]
    B -->|200 OK| A
    C -->|Process| D[Event Processor]
    D -->|Update State| E[Mirror Database]
    F[Reconciliation Service] <-->|Sync| A
    F -->|Fix Inconsistencies| E

Event Processing Principles

1. Acknowledge Immediately

Always return a 200 OK response to lemon.markets within less than 10 seconds to avoid retries:

POST /webhooks/lemon
→ Validate signature
→ Store event in queue
→ Return 200 OK immediately

Never process events synchronously in the webhook endpoint.

2. Process Idempotently

Events may be delivered multiple times. Your processor must be idempotent:

Use the event id as a unique key
Store processed event IDs to detect duplicates
Make database updates conditional based on current state

3. Process in Order (Per Entity)

Events for the same entity (e.g., order, account) must be processed in order:

Use entity ID as the partition key for your queue
Events for different entities can be processed in parallel
Respect the event created_at timestamp for ordering

4. Handle Out-of-Order Events

Despite best efforts, events may arrive out of order:

Process all events - Never skip or ignore events
Store event timestamps and sequence information
When events arrive out of order, fetch the current entity state from the lemon.markets API
Use the API response as the source of truth to resolve conflicts
Update your mirror to match the API state, then mark the event as processed
Log out-of-order events for investigation and monitoring

Important: Events provide notifications of changes, but the lemon.markets API is the authoritative source of truth. When in doubt, fetch and trust the API state.

Entity State Management

Database Schema Design

Your mirror system should maintain two types of data:

Current entity state - The latest state of each entity (accounts, orders, positions, etc.)
Complete event history - An audit log of all webhook events received

For each entity, track:

The entity's current status/state
When it was last updated
Which event last modified it (for ordering and debugging)

For each webhook event, store:

The complete event payload
When it was received and processed
Processing status (pending, processed, failed)
Retry attempts and error information

Event Processing Pattern

When processing a webhook event, follow this sequence:

1. Inspect the Event Type and Context

Every webhook event contains:

type - The event type (e.g., order.executed, account.opened)
context - An object containing identifiers for related entities

Always check the type field first to understand what happened, then extract the relevant identifiers from the context object based on that event type.

2. Fetch Current Entity State from the API

For each entity referenced in the event context, fetch the current state from the lemon.markets API. This ensures you have the authoritative, up-to-date state of each entity.

Examples of API calls to fetch entity state:

GET /accounts/{account_id} - Fetch account details
GET /orders/{order_id} - Fetch order details

If an event references multiple entities, you can fetch them in parallel. The lemon.markets API supports HTTP/2 multiplexing with 100+ streams per TCP+TLS connection, so the API is built to handle concurrent requests efficiently.

Do not rely on the event payload alone - the event is a notification that something changed, not a complete representation of the current state.

3. Make Decisions Based on API State and Event Type

Use the combination of:

The event type - tells you what kind of change occurred
The current entity state from the API - provides the authoritative state to update your mirror with

Update your mirror database to reflect the current state returned by the API.

Important Considerations

Never use event context structure for decision-making logic - The context structure may evolve over time. Always fetch and use the actual entity state from the API.
The API is the source of truth - If there's any discrepancy between what you expect and what the API returns, trust the API.
Process events as triggers, not as data sources - Events tell you when to sync, not what to sync to.

Error Handling and Recovery

Retry Strategy

When event processing fails, implement a retry strategy with exponential backoff. Distinguish between:

Transient errors (network issues, temporary API unavailability, rate limits) - Retry with increasing delays
Permanent errors (invalid data, logic errors, unrecoverable states) - Don't retry, log and alert immediately

For transient errors, use exponential backoff (e.g., 1 minute, 2 minutes, 4 minutes, etc.) to avoid overwhelming your system or the API during incidents.

Dead Letter Queue

Events that fail after all retry attempts should be moved to a dead letter queue (DLQ) for manual investigation. The DLQ should:

Preserve the complete event payload
Track the error that caused the failure
Record the number of retry attempts
Trigger alerts for operational teams

Failed events in the DLQ indicate gaps in your mirror system that need manual resolution.

Monitoring and Alerts

Set up monitoring and alerts for:

Processing lag - Events taking longer than expected to process
High failure rate - Spike in failed event processing
Dead letter queue growth - New events failing permanently

Data Consistency Strategies

Periodic Reconciliation

Webhooks are highly reliable, but no event delivery system is perfect. Implement periodic reconciliation to ensure your mirror stays in sync with lemon.markets:

Periodically fetch entity state from the lemon.markets API (don't synchronize all of the data though – prefer implementing a sliding window or skipping entities in terminal state)
Compare API state with your mirror database
Identify and log any discrepancies (missing entities, state mismatches)
Update your mirror to match the API state

The frequency of reconciliation depends on your use case and tolerance for inconsistency. More critical entities may need more frequent reconciliation.

Remember: The lemon.markets API is always the source of truth. Reconciliation ensures your mirror catches up with any events that may have been missed or failed to process.

Implementation Checklist

Core Requirements

Set up webhook endpoint with signature verification
Implement event ingestion queue (Kafka, RabbitMQ, AWS SQS, etc.)
Create event processing workers
Implement idempotency checks
Store complete event history

Event Processing

Fetch current entity state from API for each event
Process all events (never skip events)
Handle out-of-order events by trusting API state
Process events in order per entity

Error Handling

Implement retry logic with exponential backoff
Set up dead letter queue for failed events
Distinguish between transient and permanent errors

Data Consistency

Implement periodic reconciliation with lemon.markets API
Always trust API as source of truth

Testing

Test duplicate event handling
Test out-of-order event handling
Test missing events
Test failure recovery scenarios

Best Practices Summary

Always acknowledge webhooks immediately - Don't lose events
Process idempotently - Duplicate events should be safe
Store complete history - Keep all events for audit and debugging
Fetch entity state from the API - The API is the source of truth, not the event payload
Use the history property - When available, use it to detect out-of-order events
Reconcile regularly - Catch missed events and fix inconsistencies
Handle failures gracefully - Retry transient errors, alert on permanent ones
Test edge cases - Duplicates, out-of-order, missing events
Plan for scale - Queue depth, worker count, database performance

Additional Resources

lemon.markets Webhook Events API Reference
lemon.markets Brokerage API Documentation
What is a Dead Letter Queue? - AWS guide explaining DLQ concepts and best practices

Document Version: 1.0 Last Updated: 2026-06-04 Target Audience: Backend engineers implementing lemon.markets integrations