Performance Optimization

PULSE performance tuning and optimization strategies

Performance Optimization

PULSE is optimized for high-throughput, low-latency analytics.

Critical Path Optimizations

1. Event Import Batch Processing

Before: Loop-based inserts (20-50ms per record) After: Batch processing (1ms per record) Improvement: 20-25x faster

// Batch processing approach
const statements = records.map(r =>
  db.prepare('INSERT INTO events ...').bind(...)
)
await db.batch(statements)

2. API Key Caching

Before: DB query per event (100+ queries/sec) After: KV cache with 60s TTL (1 query per 100 events) Improvement: 100x faster

const cached = await env.CACHE.get(`key:${apiKey}`)
if (cached) return JSON.parse(cached)

3. Funnel Optimization

Before: In-memory processing (100-1000x slower) After: SQL window functions Improvement: 100-1000x faster

// SQL window function approach
SELECT
  user_id,
  event_name,
  LEAD(event_name) OVER (PARTITION BY user_id ORDER BY timestamp) as next_event
FROM events

Query Optimization

Indexing Strategy

Critical indexes for event queries:

-- Hot path (most queries)
CREATE INDEX idx_events_site_timestamp
  ON events(site_id, timestamp DESC)

-- User segmentation
CREATE INDEX idx_events_user
  ON events(user_id)

-- Cohort analysis
CREATE INDEX idx_cohort_members_cohort
  ON cohort_members(cohort_id)

Query Patterns

Good:

SELECT COUNT(*) FROM events
WHERE site_id = ? AND timestamp BETWEEN ? AND ?

Bad:

SELECT * FROM events  -- No WHERE clause
WHERE 1=1  -- Inefficient filter

Caching Strategies

Database Tuning

Connection Pooling

D1 automatically handles connection pooling.

Batch Operations

Use batch for multiple operations:

await db.batch([
  statement1,
  statement2,
  statement3
])

Transaction Isolation

// Prevents reading uncommitted data
await db.prepare('BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED').run()

Network Optimization

Payload Compression

Use gzip for large responses:

const headers = {
  'Content-Encoding': 'gzip'
}

Payload Reduction

Only send necessary fields:

{
  "timestamp": 1703001234567,
  "metric_id": "event_count",
  "value": 42
}

vs full event object (80% smaller)

Connection Reuse

All API calls reuse HTTP connections:

const response = await fetch(url, {
  method: 'POST',
  headers: { 'Connection': 'keep-alive' }
})

Memory Management

Large Dataset Handling

Never load full dataset into memory:

// Bad
const allEvents = await db.prepare('SELECT * FROM events').all()
const filtered = allEvents.filter(...)  // OOM risk

// Good
const filtered = await db.prepare(
  'SELECT * FROM events WHERE ...'
).all()

Stream Processing

Process events in batches:

const BATCH_SIZE = 1000
for (let i = 0; i < total; i += BATCH_SIZE) {
  const batch = await repo.findBatch(i, BATCH_SIZE)
  // Process batch
}

Performance Metrics

Latency Targets

Throughput Targets

Load Testing

Test with production-scale data:

npm run test:load

# Simulates:
# - 1000 concurrent users
# - 100 events/sec for 10 minutes
# - Query every 5 seconds

Monitoring Performance

Key Metrics

• API latency (p50, p95, p99)
• Database query count
• Cache hit rate
• Memory usage
• CPU usage

Alerts

• Latency p95 > 500ms
• Cache hit rate < 70%
• Query count increasing
• Memory > 100MB

Performance vs. Cost Trade-offs

Caching

Cost: KV usage increases by 5% Benefit: 90% query reduction, better UX

Batching

Cost: Higher latency (1-2s batch time) Benefit: 25x throughput improvement

Snapshots

Cost: 5 seconds/night CPU Benefit: 50x faster retention queries

Next Steps

• Security — Security architecture
• Components — Component details