Skip to main content

Real-time Ingestion

Real-time ingestion handles clickstream data, inventory updates, pricing changes, and promotions in real time to keep search results current.

Overview​

Real-time ingestion ensures that search results reflect the latest product data, inventory levels, prices, and promotions, providing accurate and up-to-date experiences.

Data Types​

Clickstream Data​

{
  "data_type": "clickstream",
  "ingestion": "real_time",
  "events": [
    "search",
    "click",
    "add_to_cart",
    "purchase"
  ],
  "latency": "< 1 second"
}

Inventory Updates​

{
  "data_type": "inventory",
  "ingestion": "real_time",
  "updates": [
    "stock_levels",
    "availability"
  ],
  "latency": "< 5 seconds"
}

Pricing Changes​

{
  "data_type": "pricing",
  "ingestion": "real_time",
  "updates": [
    "price_changes",
    "promotions"
  ],
  "latency": "< 10 seconds"
}

Best Practices​

  1. Ensure reliability: Use reliable data sources
  2. Handle failures: Implement retry logic
  3. Monitor latency: Track ingestion latency
  4. Validate data: Validate data before ingestion
  5. Scale appropriately: Handle peak loads