Skip to content

Say how many source types there are, and how to find them - #26

Open
leggetter wants to merge 1 commit into
mainfrom
docs/source-types-are-not-just-the-famous-ones
Open

Say how many source types there are, and how to find them#26
leggetter wants to merge 1 commit into
mainfrom
docs/source-types-are-not-just-the-famous-ones

Conversation

@leggetter

Copy link
Copy Markdown
Collaborator

authentication.md described source types as "Provider presets (Stripe, Shopify, GitHub, etc.)". There are 151.

What this was doing

Measured in hookdeck/evals, on a scenario that asks an agent to receive ElevenLabs transcription callbacks and verify them.

Configuration Result
No skills Created an ELEVENLABS source. 4/5
With this skill, before Created a generic WEBHOOK source, hand-rolled verification. 0/5
With this skill, after Created an ELEVENLABS source. 4/5

With the skill installed, the model never mentioned ELEVENLABS once, while following this file's own --source-type pattern twelve times. Three examples read as the complete list, so it concluded its provider had no preset and took the generic path.

That path is worse in every way: it hand-rolls an algorithm, header key and encoding the preset already knows, and gets them wrong.

So this line made an agent perform worse than no guidance at all.

The change

Name the count, say the examples are examples, and give a command that lists them:

curl -s https://api.hookdeck.com/2025-07-01/openapi \
  | jq -r '.components.schemas | keys[] | select(startswith("SourceTypeConfig")) | ltrimstr("SourceTypeConfig")'

ELEVENLABS mentions went from 0 to 33, and the scenario from 0/5 back to 4/5.

Worth checking elsewhere

An agent has no other source for where a boundary lies, so an example list is read as an exhaustive one. Anywhere these skills enumerate a few of something and mean "for instance", that is a candidate for the same failure. Being illustrative where a reader needs complete is not neutral.

Scope

This repairs a regression rather than producing a lift: it restores the no-skills result. A separate finding, not addressed here, is that hookdeck gateway connection upsert accepts an empty --source-webhook-secret and creates a source that can never verify anything. That is the remaining cause of the 4/5, and it belongs to the CLI rather than to this repo.

`authentication.md` described source types as "Provider presets (Stripe,
Shopify, GitHub, etc.)". There are 151. Three examples read as the whole set.

Measured in hookdeck/evals, on a scenario that asks an agent to receive
ElevenLabs callbacks. With this skill installed, a weaker model concluded
ElevenLabs had no preset, built a generic WEBHOOK source and hand-rolled the
verification, and never mentioned ELEVENLABS once while following this file's
own `--source-type` pattern twelve times. The same model with no skill at all
created an ELEVENLABS source correctly.

So this line was making an agent worse than no guidance, and the generic path it
chose is worse in every way: it hand-rolls an algorithm, header and encoding the
preset already knows.

Naming the count, saying the examples are examples, and giving a command that
lists them took ELEVENLABS from zero mentions to thirty-three, and the scenario
from 0/5 back to 4/5.

The general point is worth keeping in mind elsewhere in these skills: an agent
has no other source for where a boundary is, so an example list is read as an
exhaustive one. Illustrative where a reader needs complete is not neutral.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant