naman-kumar / journey / projects / observability-migrationsPublic

Back to all projects

ShippedCvent (Splunk to Datadog) · Latitude (Sumo Logic to Datadog)

Large-scale observability migrations

~880M and 20M+ log events per week moved to Datadog

Two log platform migrations to Datadog, both completed: one at Cvent, and one at Latitude Financial Services (client, via Viable Solutions and Synechron).

My role
Migrated at Cvent and at Latitude
Cvent
~880M log events per week, Splunk to Datadog (completed)
Latitude
20M+ log events per week, Sumo Logic to Datadog (completed)
Standards
SLI/SLO and instrumentation standards for Java microservices
Also
Moved a newly acquired company to Datadog (10+ dashboards)

Architecture

  1. Sources

    • MicroservicesJava services and more
    • Business systemsIncluding payment logs

  2. Before

    • SplunkCvent
    • Sumo LogicLatitude

  3. Ingest

    • Datadog log pipelinesGrok parsing

  4. Operate

    • Logs, metrics, tracesOne platform
    • Monitors and dashboardsSLIs/SLOs
Generalised before-and-after for both migrations.

Problem

Logs, metrics and traces split across platforms make incidents slower to diagnose. Both migrations consolidate them for single-pane dashboards.

My contribution

  • Cvent: migrated ~880M log events per week from Splunk to Datadog.
  • Cvent: owned SLI/SLO and custom instrumentation standards for Java microservices.
  • Cvent: led a newly acquired company's move to Datadog (10+ dashboards).
  • Latitude: migrated 20M+ log events per week from Sumo Logic to Datadog, including Grok parsing pipelines and detection monitors.

Engineering decisions

  • One platform for logs, metrics and traces instead of separate tools.
  • Parse at ingest with Grok pipelines, so payment logs arrive as structured fields.
  • Standards alongside the move: SLI/SLO and instrumentation standards for Java microservices (Cvent).

Reliability and security

  • Detection monitors for anomalies in payment transaction flows (Latitude).

Verified outcomes

  • Cvent: logs, metrics and traces on one platform.
  • Latitude: migration complete; 20M+ log events per week now on Datadog.

Stack

Pipelines and monitors are described by purpose only; log content and client system names are omitted.

git log work/cvent: see this in the career journey