System Design

PULSE architecture and system design overview

System Design

PULSE is built on a serverless, cloud-native architecture designed for scalability and cost-efficiency.

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                        Client Layer                         │
│  (Web, Mobile, Backend, JavaScript Tracker)                 │
└────────────────────────┬────────────────────────────────────┘
                         │
                    HTTP/HTTPS
                         │
┌────────────────────────▼────────────────────────────────────┐
│              Cloudflare Edge Network                         │
│  (Rate Limiting, CORS, TLS Termination)                     │
└────────────────────────┬────────────────────────────────────┘
                         │
┌────────────────────────▼────────────────────────────────────┐
│         Cloudflare Workers (Main Application)               │
│  ┌──────────────────────────────────────────────────────┐   │
│  │ Request Router                                       │   │
│  │  - /collect → Event Ingestion                        │   │
│  │  - /api/v1/stats → Query Handler                     │   │
│  │  - /api/v1/cohorts → Cohort Management              │   │
│  │  - /api/v1/alerts → Alert Rules                     │   │
│  │  - /api/v1/webhooks → Webhook Management            │   │
│  │  - /api/v1/imports → Bulk Imports                   │   │
│  │  - /api/v1/exports → Scheduled Exports              │   │
│  └──────────────────────────────────────────────────────┘   │
└────────────────────────┬────────────────────────────────────┘
         │               │               │               │
    ┌────▼─────┐    ┌────▼─────┐   ┌────▼─────┐   ┌────▼─────┐
    │  D1 DB   │    │ KV Cache │   │Durable   │   │Datadog   │
    │(SQLite)  │    │          │   │Objects   │   │(optional)│
    └──────────┘    └──────────┘   └──────────┘   └──────────┘

Component Architecture

1. Event Ingestion Layer

Receives events from clients and validates them:

• Validation — Check required fields, timestamp, schema
• Enrichment — Add server timestamp, compute session duration
• Buffering — Queue events for batch processing
• Error Handling — Return meaningful errors for invalid events

2. Data Layer (D1 Database)

SQLite database for persistent storage:

-- Core tables
events              -- Raw events
users               -- User attributes
sessions            -- Session tracking
cohort_members      -- Cohort membership
retention_snapshots -- Pre-computed retention

-- Feature tables
alert_rules         -- Alert rule definitions
alert_history       -- Alert trigger history
webhook_endpoints   -- Webhook configurations
import_jobs         -- Bulk import tracking
export_jobs         -- Scheduled export tracking

3. Query Engine

Aggregates and filters event data:

• Metrics — event_count, unique_users, conversion_rate
• Filtering — By date range, device, country, user segment
• Grouping — By hour, day, week, device_type, country
• Funnel Analysis — Multi-step user journeys
• Cohort Analysis — Segment tracking and retention

4. Caching Layer (KV)

Cloudflare KV for distributed caching:

• API Key Cache — Validate keys without DB query (60s TTL)
• Webhook Endpoints — Cache active webhook endpoints (5min TTL)
• Session Tokens — Store session data (1 hour TTL)
• Rate Limit State — Track request counts

5. Message Queue (Durable Objects)

Reliable message delivery for:

• WebSocket Broadcasting — Real-time analytics updates
• Alert Delivery — Send alerts to webhooks
• Background Jobs — Nightly snapshot generation
• Event Aggregation — Batch processing

6. Scheduled Jobs

Cron-triggered maintenance tasks:

• 2 AM UTC — Generate retention snapshots (nightly)
• Every Hour — Data cleanup, export file deletion
• Every 10 Minutes — Metric flush to Datadog

Data Flow

Event Ingestion Flow

Event from Client
    ↓
Validation
    ↓
Rate Limit Check
    ↓
Authentication
    ↓
D1 Insert (transactional)
    ↓
KV Cache Invalidation
    ↓
Webhook Dispatch (async)
    ↓
Response 202 Accepted

Query Flow

Query Request
    ↓
Authentication
    ↓
Parameter Validation
    ↓
KV Cache Check (hit → return)
    ↓
D1 Query
    ↓
Result Aggregation
    ↓
KV Cache Write (TTL)
    ↓
Response

WebSocket Streaming Flow

Client WebSocket Connect
    ↓
Durable Object Session Manager
    ↓
Listen for Event Broadcasts
    ↓
On New Event: Serialize → Send to Connected Clients
    ↓
Aggregation Broadcast (every 10s)
    ↓
Client Receives Real-Time Updates

Scaling Strategy

Horizontal Scaling

• Workers — Cloudflare automatically scales Workers across edges
• Database — D1 handles read replicas for scale
• KV — Global KV namespace automatically distributed

Vertical Scaling

• Query Optimization — Indexed queries, materialized views
• Caching — KV cache reduces DB queries by 90%+
• Batch Processing — Bulk inserts, async operations

Load Testing Results

PULSE handles:

• 1000+ events/second sustained throughput
• 10,000+ concurrent WebSocket connections
• 1 million events/day with <100ms p95 latency
• 100k cohort members without memory issues

Deployment Model

PULSE runs on Cloudflare infrastructure:

• Regions — Deployed globally across 300+ Cloudflare data centers
• Redundancy — Auto-failover and replication
• Uptime — 99.99% SLA
• Cost — Pay-as-you-go, no minimum commitments

Security Model

Authentication

• API Keys — Bearer token authentication
• Rate Limiting — Per-IP request rate limits
• HTTPS/TLS — All traffic encrypted

Data Protection

• Encryption at Rest — D1 encryption enabled
• Encryption in Transit — TLS 1.3+
• Access Control — Row-level security for multi-tenant

Compliance

• GDPR — Configurable retention, right to deletion
• CCPA — User data export and deletion
• SOC 2 — In progress certification

Performance Characteristics

Monitoring & Observability

• Cloudflare Analytics — Automatic request tracking
• Datadog Integration — Custom metrics and alerts
• Health Checks — /health and /health/ready endpoints
• Request Logging — Structured logs via Workers

Next Steps

• Data Model — Database schema and tables
• Components — Detailed component descriptions
• Performance — Optimization techniques
• Security — Security architecture