feat(bedrock): add TTL support to auto-injected tool and system/user cache points#2232
feat(bedrock): add TTL support to auto-injected tool and system/user cache points#2232kpx-dev wants to merge 4 commits into
Conversation
|
Added 3 integration tests covering the new paths. Ran locally against Coverage matrix:
|
6a74d89 to
797f359
Compare
…cache points Extends prompt caching TTL coverage beyond user-supplied cachePoint blocks (PR strands-agents#1660) to the two SDK-managed auto-injected paths on BedrockModel: - Adds cache_tools_ttl config option so the toolConfig auto-injected cache point can carry a TTL (e.g. '5m' or '1h'). - Adds ttl field to CacheConfig dataclass so _inject_cache_point propagates TTL into the cache point appended to the last user message when strategy='auto'. Together, these let users align all three cache checkpoint TTLs (toolConfig -> system -> messages) to satisfy Bedrock's non-increasing TTL ordering rule -- which was previously impossible because cache_tools hardcoded an implicit 5m TTL. Partially addresses strands-agents#2121 (Bug 2: cache_tools ordering violation with 1h TTL). Bug 1 from strands-agents#2121 was resolved by strands-agents#1660. Tests: - 4 unit tests covering cache_tools_ttl and CacheConfig.ttl with and without TTL (backward-compat). - 3 integration tests against Claude Haiku 4.5 (officially documented for 1h TTL on Bedrock), including a regression test that sets 1h TTL on all three cache checkpoints simultaneously. - Model ID extracted into a _CACHE_TTL_MODEL_ID module constant so future model bumps are a one-line change.
797f359 to
0835dc3
Compare
…user-cache-points # Conflicts: # tests/strands/models/test_bedrock.py
| cache_tools: str | None | ||
| cache_tools_ttl: str | None |
There was a problem hiding this comment.
I would recommend expanding the type of cache_tools to ensure that ttl is configured with the cache tools type. So something like:
cache_tools: str | CacheToolsConfig | None
...
@dataclass
class CacheToolsConfig:
type: str = "default"
ttl: str | None = NoneThere was a problem hiding this comment.
Good call — refactored in cc32ea8. Added a CacheToolsConfig dataclass with type (default "default") and ttl fields, and changed cache_tools to str | CacheToolsConfig | None. Removed the standalone cache_tools_ttl field so TTL can only be set together with the type. The plain str form still works for back-compat.
BedrockModel(cache_tools=CacheToolsConfig(type="default", ttl="1h"))
# or, unchanged:
BedrockModel(cache_tools="default")…class Address review feedback: couple `type` and `ttl` together in a single config object so users can't set a TTL without a type. - Add CacheToolsConfig(type, ttl) dataclass - Change cache_tools to `str | CacheToolsConfig | None` (str preserved for back-compat) - Drop standalone cache_tools_ttl field - Export CacheToolsConfig from strands.models
|
|
||
|
|
||
| @dataclass | ||
| class CacheToolsConfig: |
There was a problem hiding this comment.
I would recommend defining and exporting this from model.py for consistency with CacheConfig.
There was a problem hiding this comment.
Resolved by 8759076 — went with your other suggestion (drop CacheToolsConfig entirely and reuse cache_config.ttl for the tool checkpoint), so there's no new dataclass to move.
| cache_tools=CacheToolsConfig(type="default", ttl="1h"), | ||
| cache_config=CacheConfig(strategy="auto", ttl="1h"), |
There was a problem hiding this comment.
Does this mean you can't have say 5m for tools and 1h for messages? But the opposite would be allowed since it would be decreasing? Do you have a link to the docs that outlines these rules?
There was a problem hiding this comment.
Good point — pivoted in 8759076. Dropped CacheToolsConfig entirely. The toolConfig cache point now picks up its TTL from cache_config.ttl, so a single knob covers both SDK-managed checkpoints. Bedrock's non-increasing rule is satisfied trivially since both checkpoints share one value.
BedrockModel(
cache_tools="default",
cache_config=CacheConfig(strategy="auto", ttl="1h"),
)
# toolConfig cache point AND last-user-message cache point both get ttl="1h"This also resolves your other comment about moving the dataclass to model.py — there's no new dataclass to move now.
…config Address review feedback: instead of a separate CacheToolsConfig, the toolConfig cache point now picks up its TTL from cache_config.ttl. This keeps a single TTL knob for SDK-managed checkpoints, which satisfies Bedrock's non-increasing-TTL ordering rule trivially. - Revert cache_tools type to `str | None` - _build_tools_cache_point reads ttl from self.config["cache_config"].ttl - Remove CacheToolsConfig class and export - Update tests to set TTL via cache_config
Description
Extends prompt caching TTL coverage beyond the user-supplied
cachePointpath (already fixed in #1660) to the two SDK-managed auto-injected paths onBedrockModel.Previously, only cache points that users passed directly inside message content could carry a
ttlfield — the formatter preserved it end-to-end. However, when the SDK auto-injected cache points on the user's behalf, thettlwas hardcoded away:cache_tools—BedrockModel(cache_tools="default")appends{"cachePoint": {"type": "default"}}totoolConfig.toolswith no way to set a TTL._inject_cache_point— whencache_config=CacheConfig(strategy="auto")is set, the SDK appends{"cachePoint": {"type": "default"}}to the last user message with no way to set a TTL.Because Bedrock processes cache checkpoints in order
toolConfig → system → messagesand requires TTLs to be non-increasing, a user who wanted to run 1h caching end-to-end could not — the SDK-managed checkpoints would implicitly be 5m and violate the ordering rule (see #2121 Bug 2).Changes
BedrockModel: addedcache_tools_ttl: str | Noneconfig option. When set together withcache_tools, the TTL is propagated into thetoolConfigcache point.CacheConfig: addedttl: str | Nonefield. When set,_inject_cache_pointincludes the TTL in the cache point appended to the last user message.Users can now align all three cache checkpoint TTLs consistently:
API reference: https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_CachePointBlock.html
Launch announcement: https://aws.amazon.com/about-aws/whats-new/2026/01/amazon-bedrock-one-hour-duration-prompt-caching/
Related Issues
Closes #1243 (longer TTL for prompt caching) — together with the already-merged #1660 (user-supplied
cachePoint.ttl), this PR completes TTL coverage across all three Bedrock cache checkpoint paths (toolConfig, system, messages), so users can opt into Bedrock's 1h TTL end-to-end.Partially addresses #2121 (Bug 2:
cache_toolsordering violation with 1h TTL). Bug 1 from #2121 (user-suppliedttlbeing dropped) was resolved by #1660.Documentation PR
Type of Change
New feature
Testing
Added 4 unit tests:
test_inject_cache_point_with_ttl— verifiesCacheConfig(ttl="5m")propagates into the auto-injected message cache point.test_inject_cache_point_without_ttl— backward compat:CacheConfig(strategy="auto")with no TTL still works.test_format_request_cache_tools_with_ttl— verifiescache_tools_ttlpropagates intotoolConfigcache point.test_format_request_cache_tools_without_ttl— backward compat:cache_toolswithout TTL still works.Verified:
hatch run prepare(lint, format, mypy all clean; pre-existing local env test failures unrelated to this change)Checklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.