feat: add FastAPI middleware for per-request emissions tracking by davidberenstein1957 · Pull Request #1203 · mlco2/codecarbon

davidberenstein1957 · 2026-05-19T19:33:42Z

Summary

Adds optional codecarbon[fastapi] integration: middleware to track CO₂ emissions per HTTP request on FastAPI/Starlette apps
Measurement runs after the response is sent, so clients are not blocked on hardware sampling
Includes create_codecarbon_lifespan for a shared app-level tracker, route include/exclude filters, optional save_to_api, and opt-in response_headers
Rebased on v3.2.9 (latest release)
Measured on a real HF embedder workload (paraphrase-MiniLM-L3-v2): default deferred middleware ~+4 ms per request; sync headers cost more. See docs/how-to/fastapi.md.

Test plan

uv run pytest tests/integrations/ -v
Manual smoke: uv run --extra fastapi uvicorn examples.fastapi_middleware:app --reload then curl localhost:8000/predict

Usage

from fastapi import FastAPI
from codecarbon.integrations.fastapi import add_codecarbon_middleware, create_codecarbon_lifespan

app = FastAPI(lifespan=create_codecarbon_lifespan(app, project_name="my-api"))
add_codecarbon_middleware(app)

Full guide: docs/how-to/fastapi.md

codecov · 2026-05-19T19:35:40Z

Codecov Report

❌ Patch coverage is 88.35878% with 61 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.12%. Comparing base (58c0b19) to head (76f4380).

Files with missing lines	Patch %	Lines
codecarbon/integrations/fastapi/middleware.py	85.97%	39 Missing ⚠️
codecarbon/emissions_tracker.py	85.96%	16 Missing ⚠️
codecarbon/core/powermetrics.py	0.00%	2 Missing ⚠️
codecarbon/output_methods/http.py	84.61%	2 Missing ⚠️
codecarbon/core/resource_tracker.py	93.75%	1 Missing ⚠️
codecarbon/integrations/fastapi/_routing.py	97.43%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1203      +/-   ##
==========================================
- Coverage   89.70%   89.12%   -0.59%     
==========================================
  Files          48       51       +3     
  Lines        4778     4945     +167     
==========================================
+ Hits         4286     4407     +121     
- Misses        492      538      +46

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

benoit-cty · 2026-05-20T09:25:15Z

Hello, thanks for this. There is a problem with your branch : there are many changes that are already merged. Can you do a rebase ?

davidberenstein1957 · 2026-05-20T09:38:29Z

@benoit-cty I have reached out to some people at FastAPI, if they would be interested in a quick review :) For visibility, we could also consider deploying it as a standalone integration, but let's see if people like it.

SaboniAmine

Thanks David this a great PR! Left a few questions, and saw that you have already prepared a benchmark script, do you have any numbers / graph already computed to share ?

Ship optional codecarbon[fastapi] integration with CodeCarbonMiddleware, configurable response headers, route-based task naming, and lifespan helper for shared app-level tracking. Co-authored-by: Cursor <cursoragent@cursor.com>

Add tests for app tracking mode, deprecated include_emissions_header, on_request_complete callbacks, header preset edge cases, and routing formatters so Codecov patch coverage meets the PR threshold. Co-authored-by: Cursor <cursoragent@cursor.com>

…iltering Update the FastAPI middleware to support include and exclude patterns for request tracking, allowing users to specify which endpoints to measure. Refactor routing helpers for improved clarity and add support for deferred measurement. Update documentation to reflect new features and usage examples. Co-authored-by: Cursor <cursoragent@cursor.com>

Refactor FastAPI middleware and routing code for improved readability by adjusting line breaks and indentation. Remove the product telemetry link from the documentation navigation. Update test cases for consistency in formatting and structure.

Refactor the test for FastAPI middleware to handle deferred task execution using a thread pool. This change enhances the test's reliability by ensuring that asynchronous tasks are properly awaited in a separate thread, improving overall test coverage and stability.

Refactor the exception handling in gpu_amd.py to specifically catch AttributeError when importing amdsmi. Update the warning message to provide clearer guidance on ensuring proper configuration of amdsmi for AMD GPU metrics.

Document latency for sync vs deferred logging on a MiniLM embedder workload and add a reproducible scripts/benchmark_fastapi_middleware.py runner. Co-authored-by: Cursor <cursoragent@cursor.com>

…ions logging Add support for tracking HTTP requests in the FastAPI middleware, including a new `HttpRequestBaseline` dataclass for capturing request metrics. Implement methods for starting and finishing HTTP request tracking, allowing emissions data to be logged after the response is sent. Update middleware to handle task naming and emissions data persistence, improving overall emissions tracking capabilities. Co-authored-by: Cursor <cursoragent@cursor.com>

…#1251)" This reverts commit 11374f4.

…2/4) (#1252)" This reverts commit 473351b.

#1253)" This reverts commit 553e852.

…4/4) (#1254)" This reverts commit 3161c53.

…path" This reverts commit 0c34475.

Introduce a new script for benchmarking GPU monitoring overhead, comparing heavyweight NVML calls with a lightweight utilization-only approach. The script measures unnecessary NVML calls in the _monitor_power() hot path and evaluates latency differences. It supports various modes, including quick runs, full benchmarks, and simulated multi-GPU environments, while providing detailed reporting on NVML call breakdowns and performance metrics.

Clarify the behavior of emissions measurement in FastAPI middleware, emphasizing that tracking occurs after the response is sent. Update descriptions for `create_codecarbon_lifespan` and logging options, and improve benchmark section to reflect average response times with and without middleware. Enhance readability and consistency throughout the document.

davidberenstein1957 · 2026-07-20T08:31:31Z

@inimaz This is ready for review. Rebased on v3.2.9 and benchmarked on a real HF embedder workload — middleware adds about 3 ms per request (~24 ms → ~27 ms). Details in the updated PR description and docs/how-to/fastapi.md.

Update GPU monitor tests to mock get_gpu_details after perf revert, reset logger level in FastAPI middleware test after CLI monitor sets ERROR level, and clear CODECARBON_TELEMETRY env vars in config tests.

Move FIELD_UNITS, HEADER_PRESETS, and HTTP methods into codecarbon.core.emission_fields so FastAPI headers/routing reuse shared StrEnums. Simplify resolve_header_mapping with match dispatch and keep backward-compatible dict exports.

davidberenstein1957 · 2026-07-20T11:10:13Z

@SaboniAmine Thanks again for the thorough review — I've addressed the inline threads in the latest pushes:

Core enums for emission fields / header presets / HTTP methods (codecarbon/core/emission_fields.py)
Simpler resolve_header_mapping + package-level Starlette import guard
Concurrent task naming documented/tested (stable route labels + UUID uniqueness on the lifespan path)
Measurement model docs + regression test (sync measure on tracker worker before callback)
compose_lifespans() for stacking with user-owned startup/shutdown
Benchmarks vs raw FastAPI + Logfire (--with-logfire); ~3 ms CodeCarbon overhead on the HF embedder workload

Local uv run task test-package: 552 passed. Ready for another look when you have a moment.

Remove unused header/enum stack. Opt-in response_headers measures before response.start. include_background_tasks toggles whether finalize waits for Response.background. Docs clarify BG vs WebSockets.

davidberenstein1957 · 2026-07-20T11:34:36Z

Ponytail cleanup on this PR:

Removed unused _headers.py / core emission_fields enums (never wired to middleware)
Added opt-in response_headers= (sync measure → X-CodeCarbon-*; costs client latency)
Added include_background_tasks (default True; set False to finalize at end-of-body)
Docs: BackgroundTasks are included by default; WebSockets stay unsupported (no implementation)

See docs/how-to/fastapi.md Limitations + config section.

Document ~+25 ms at c=1 and ~+65 ms at c=4 for sync headers with a mocked 20 ms sample, and keep placeholder project/experiment UUIDs.

davidberenstein1957 · 2026-07-20T11:43:33Z

Added a sync response_headers row to the Performance section (and --with-headers on the benchmark script):

Mode	Rough client latency (mocked 20 ms measure)
Deferred default	~+3 ms (embedder table)
`response_headers=True`, c=1	~~55 ms (~~+25 ms vs ~30 ms baseline)
`response_headers=True`, c=4	~95 ms (tracker worker serializes sync measures)

Reproduce: uv run --extra fastapi python scripts/benchmark_fastapi_middleware.py --quick --with-headers

Document measured deferred vs sync-headers latency on a real MiniLM workload, and harden tracker shutdown against cancelled futures.

Keep a static measured-overhead overview in the docs; remove the reproducibility commands and the large benchmark harness.

davidberenstein1957 force-pushed the feat/add-fastapi-middleware branch from f44740f to 15c0ccf Compare May 20, 2026 09:31

davidberenstein1957 marked this pull request as ready for review May 20, 2026 09:33

davidberenstein1957 requested a review from a team as a code owner May 20, 2026 09:33

davidberenstein1957 force-pushed the feat/add-fastapi-middleware branch 2 times, most recently from eabd526 to bb6995c Compare May 20, 2026 16:03

SaboniAmine reviewed Jun 9, 2026

View reviewed changes

davidberenstein1957 force-pushed the feat/add-fastapi-middleware branch from 250881f to deba2ce Compare July 15, 2026 15:46

davidberenstein1957 and others added 14 commits July 20, 2026 09:32

fix: improve error handling for AMD GPU metrics

7a8d6db

Refactor the exception handling in gpu_amd.py to specifically catch AttributeError when importing amdsmi. Update the warning message to provide clearer guidance on ensuring proper configuration of amdsmi for AMD GPU metrics.

docs: add HF embedder middleware benchmarks and benchmark script

48a35c4

Document latency for sync vs deferred logging on a MiniLM embedder workload and add a reproducible scripts/benchmark_fastapi_middleware.py runner. Co-authored-by: Cursor <cursoragent@cursor.com>

Revert "perf: defer tracker initialization and slim import path (1/4) (…

cb7da29

…#1251)" This reverts commit 11374f4.

Revert "perf: cache hardware detection and optimize warm-path reuse (…

b1f4bf0

…2/4) (#1252)" This reverts commit 473351b.

Revert "perf: defer API run creation until first emission upload (3/4) (

f05cd35

#1253)" This reverts commit 553e852.

Revert "perf: speed up CLI monitor startup and fix wrapped commands (…

6baf441

…4/4) (#1254)" This reverts commit 3161c53.

Revert "perf(gpu): skip heavyweight NVML calls in _monitor_power hot …

83cbe75

…path" This reverts commit 0c34475.

davidberenstein1957 force-pushed the feat/add-fastapi-middleware branch from deba2ce to 86cc355 Compare July 20, 2026 07:38

davidberenstein1957 assigned inimaz Jul 20, 2026

davidberenstein1957 added 2 commits July 20, 2026 11:45

fix: stabilize tests after perf revert and env isolation

44eb894

Update GPU monitor tests to mock get_gpu_details after perf revert, reset logger level in FastAPI middleware test after CLI monitor sets ERROR level, and clear CODECARBON_TELEMETRY env vars in config tests.

refactor: slim FastAPI middleware and add sync headers / BG flag

4b31052

Remove unused header/enum stack. Opt-in response_headers measures before response.start. include_background_tasks toggles whether finalize waits for Response.background. Docs clarify BG vs WebSockets.

docs: add response_headers sync-measure latency to FastAPI benchmarks

d532221

Document ~+25 ms at c=1 and ~+65 ms at c=4 for sync headers with a mocked 20 ms sample, and keep placeholder project/experiment UUIDs.

davidberenstein1957 added 2 commits July 20, 2026 15:33

docs: add HF embedder performance overview for FastAPI middleware

f34793a

Document measured deferred vs sync-headers latency on a real MiniLM workload, and harden tracker shutdown against cancelled futures.

chore: drop FastAPI middleware benchmark script

76f4380

Keep a static measured-overhead overview in the docs; remove the reproducibility commands and the large benchmark harness.

Uh oh!

Uh oh!

Conversation

davidberenstein1957 commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Usage

Uh oh!

codecov Bot commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

benoit-cty commented May 20, 2026

Uh oh!

davidberenstein1957 commented May 20, 2026

Uh oh!

SaboniAmine left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

davidberenstein1957 commented Jul 20, 2026

Uh oh!

davidberenstein1957 commented Jul 20, 2026

Uh oh!

davidberenstein1957 commented Jul 20, 2026

Uh oh!

davidberenstein1957 commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

davidberenstein1957 commented May 19, 2026 •

edited

Loading

codecov Bot commented May 19, 2026 •

edited

Loading