An extension for SillyTavern that caches API tokenizer requests to heavily reduce pre-generation latency.
SillyTavern often sends a multitude of small tokenizer requests (especially when dealing with massive lorebooks). If you are accessing your PC remotely via Tailscale, tunnels, or a slow mobile connection, this can slow down the interface to a crawl. This extension was built to eliminate that network roundtrip.
- Smart Context Approximation: This is the core driver of the extension. Because prompt text changes constantly, exact matches in the raw cache are rare. Instead, the extension uses a mathematical heuristic to guess token counts based on previous API responses.
- Accuracy Warning: It is always inaccurate, though the built-in auto-calibration tries its best to compensate for this. Don't push your luck when you're at the limit of the context length.
- Note on Disabling: If you decide to turn off Smart Context Approximation, please refresh your browser tab afterward to ensure the active prediction state is fully cleared.
- Open SillyTavern.
- Go to the Extensions menu.
- Select "Install extension".
- Paste the link to this repository:
https://github.com/Nidelon/st-tokenizer-cache - Click Install.
You can configure the extension in the Extensions -> Tokenizer Cache tab.
- Enabled: Toggles the caching system on or off.
- Smart Context Approximation: Attempts to predict the token count for text based on previous API responses.
- Max Cache Entries: The maximum number of entries to keep in cache (default
5000). - Clear Cache: Manually wipes the stored cache (in the extension) and resets prediction stats.
MIT