Skip to content

Add Prometheus metrics exporter #47

Description

@willibrandon

Description:
Export log event metrics to Prometheus for monitoring and alerting. This enables teams to monitor logging health and set up alerts based on log patterns.

Package: github.com/willibrandon/mtlog/adapters/prometheus

Proposed API:

import mtlogprom "github.com/willibrandon/mtlog/adapters/prometheus"

// Create metrics exporter
exporter := mtlogprom.NewExporter(logger,
    mtlogprom.WithCounters("log_events_total", "level", "source"),
    mtlogprom.WithHistogram("log_event_size_bytes"),
    mtlogprom.WithSinkMetrics(), // Track sink performance
)

// Expose at /metrics endpoint
http.Handle("/metrics", exporter.Handler())

// Or use as an enricher to add metrics
logger := mtlog.New(
    mtlog.WithConsole(),
    mtlog.WithEnricher(exporter),
)

Metrics to Export:

Event Metrics:

  • log_events_total{level="error",source="api"} - Count by level and source
  • log_event_size_bytes - Histogram of event sizes
  • log_properties_per_event - Histogram of property counts

Performance Metrics:

  • log_sink_latency_seconds{sink="seq"} - Sink write latency
  • log_sink_errors_total{sink="elasticsearch",error="timeout"} - Sink errors
  • log_sink_events_dropped_total{sink="file",reason="full"} - Dropped events

Template Cache Metrics:

  • log_template_cache_hits_total - Cache hit count
  • log_template_cache_misses_total - Cache miss count
  • log_template_cache_size - Current cache size
  • log_template_parse_duration_seconds - Template parsing time

Example Prometheus Queries:

# Alert on high error rate
rate(log_events_total{level="error"}[5m]) > 10

# Monitor sink performance
histogram_quantile(0.99, log_sink_latency_seconds)

# Track template cache efficiency
rate(log_template_cache_hits_total[5m]) / 
  (rate(log_template_cache_hits_total[5m]) + rate(log_template_cache_misses_total[5m]))

Implementation Notes:

  • Use prometheus/client_golang
  • Metrics should be zero-cost when not enabled
  • Consider cardinality limits for labels
  • Support custom metric namespaces
  • Provide pre-built Grafana dashboard JSON

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestobservabilityEnhancements that improve production observability and debugging capabilities

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions