Prometheus Metrics

Rindexer exposes Prometheus metrics for production observability. These metrics help you monitor indexing performance, RPC health, database operations, and stream delivery in real-time.

Overview

The metrics endpoint provides insight into:

• Indexing progress - Events processed, blocks indexed, sync status
• RPC performance - Request latencies, success/error rates, in-flight requests
• Database operations - Query durations, operation counts, connection pool status
• Stream delivery - Message counts, delivery latencies by stream type
• Chain state - Reorg detection, block lag behind chain head

Metrics Endpoint

Metrics are served on the same port as the health server (default 8080) at the /metrics path.

# Metrics available when indexer is running
curl http://localhost:8080/metrics

The endpoint returns metrics in Prometheus text exposition format, compatible with Prometheus, Grafana, Datadog, and other monitoring tools.

Configuration

Configure the metrics port using health_port in your rindexer.yaml:

global:
  health_port: 9090  # Metrics available at http://localhost:9090/metrics

Available Metrics

Indexing Metrics

rindexer_events_processed_total{contract="USDC",event="Transfer",network="ethereum"} 19971
rindexer_last_synced_block{contract="USDC",event="Transfer",network="ethereum"} 21901064
rindexer_blocks_behind{contract="USDC",event="Transfer",network="ethereum"} 2425731

RPC Metrics

Histogram buckets: 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s

rindexer_rpc_requests_total{method="eth_getLogs",network="mainnet",status="success"} 156
rindexer_rpc_requests_total{method="eth_getLogs",network="mainnet",status="error"} 2
rindexer_rpc_request_duration_seconds_sum{method="eth_getLogs",network="mainnet"} 45.23
rindexer_rpc_request_duration_seconds_count{method="eth_getLogs",network="mainnet"} 156

Database Metrics

Operation types: query, insert, update, delete, batch_insert, batch_execute

Atomic batch labels: status is committed or rolled_back; reason is ok on commit and otherwise deadlock, cursor_missing or error. A rolled_back batch is harmless (nothing was written and the batch is retried), so alert on the rate of reason="deadlock" rather than on rollbacks as such. Recorded only on the Postgres-only custom-tables path (see Delivery Semantics).

Histogram buckets: 1ms, 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s

Stream Metrics

Stream types: sns, webhook, rabbitmq, kafka, redis, cloudflare_queues

Chain State Metrics

Shared Tip Logs Metrics

Live indexing reads each new tip block once per network with an unfiltered eth_getLogs and serves every stream from that answer (see shared_tip_logs). rindexer_rpc_requests_total{method="eth_getLogs"} keeps counting both the shared and the per-stream calls.

The network label of these series is the chain name, the same label rindexer_rpc_requests_total carries (polygon for chain id 137), so the two can be subtracted to count the per-stream tip calls that remain.

Histogram buckets: 50ms, 100ms, 250ms, 500ms, 1s, 2s, 3s, 5s, 8s, 12s

Build Info

Prometheus Integration

Basic Prometheus Config

Add rindexer to your prometheus.yml:

scrape_configs:
  - job_name: 'rindexer'
    static_configs:
      - targets: ['localhost:8080']
    scrape_interval: 15s
    metrics_path: /metrics

Docker Compose Example

services:
  rindexer:
    image: your-rindexer-image
    ports:
      - "8080:8080"
 
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
 
  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

Useful PromQL Queries

Indexing Progress

# Events per second (rate over 5 minutes)
rate(rindexer_events_processed_total[5m])

# Blocks behind chain head
rindexer_blocks_behind

# Sync progress percentage
(rindexer_last_synced_block / rindexer_latest_chain_block) * 100

RPC Health

# RPC error rate
rate(rindexer_rpc_requests_total{status="error"}[5m])
  / rate(rindexer_rpc_requests_total[5m])

# P99 RPC latency
histogram_quantile(0.99, rate(rindexer_rpc_request_duration_seconds_bucket[5m]))

# RPC requests per second by method
rate(rindexer_rpc_requests_total[5m])

Database Performance

# Database operation latency (P95)
histogram_quantile(0.95, rate(rindexer_db_operation_duration_seconds_bucket[5m]))

# Database error rate
rate(rindexer_db_operations_total{status="error"}[5m])

# Atomic batch deadlock rate (each one is a rolled back, retried batch)
rate(rindexer_atomic_batches_total{reason="deadlock"}[5m])

Stream Delivery

# Stream delivery rate by type
rate(rindexer_stream_messages_total{status="success"}[5m])

# Stream error rate
rate(rindexer_stream_messages_total{status="error"}[5m])

Alerting Examples

Prometheus Alerting Rules

groups:
  - name: rindexer
    rules:
      - alert: RindexerHighBlockLag
        expr: rindexer_blocks_behind > 1000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Rindexer falling behind chain head"
          description: "{{ $labels.contract }} is {{ $value }} blocks behind"
 
      - alert: RindexerHighRPCErrorRate
        expr: rate(rindexer_rpc_requests_total{status="error"}[5m]) > 0.1
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High RPC error rate detected"
 
      - alert: RindexerReorgDetected
        expr: increase(rindexer_reorgs_detected_total[5m]) > 0
        labels:
          severity: warning
        annotations:
          summary: "Chain reorg detected on {{ $labels.network }}"

Grafana Dashboard

Import these panels for a basic rindexer dashboard:

1. Sync Progress - Gauge showing rindexer_blocks_behind
2. Events/sec - Graph of rate(rindexer_events_processed_total[1m])
3. RPC Latency - Heatmap of rindexer_rpc_request_duration_seconds
4. Error Rates - Graph of error rates for RPC, DB, and streams
5. Active Tasks - Stat panel for rindexer_active_indexing_tasks

Best Practices

• Set appropriate scrape intervals - 15-30 seconds is typical for indexer metrics
• Monitor block lag - Alert when rindexer_blocks_behind exceeds acceptable thresholds
• Track RPC errors - High error rates may indicate provider issues or rate limiting
• Watch reorg metrics - Frequent reorgs may indicate network instability
• Use labels wisely - Filter by network, contract, event for granular insights