Add `serving-llms-on-epyc` walkthrough by amd-lalithnc · Pull Request #83 · amd/skills

amd-lalithnc · 2026-07-01T10:48:45Z

Adds a walkthrough for serving-llms-on-epyc at walkthroughs/serving-llms-on-epyc.md, with a link in walkthroughs/README.md. Resolves #82

It follows the same sections as the existing walkthroughs: prerequisites, checking which skills are visible, installing the skill, running it, calling the endpoint, and a comparison of doing the task without the skill.

It also re-registers the skill in the marketplace (the entry was removed in #76) and folds in a few fixes found while testing the walkthrough on real EPYC hosts:

Require AVX-512 (Zen4+) — the skill now stops on pre-Zen4 EPYC.
Drop --shm-size from the container launch — it conflicts with --ipc=host on podman.
Recognize lettered EPYC SKUs in detect.py (e.g. 9B45 → Turin).

Tested end-to-end on two EPYC hosts: the endpoint comes up and answers.

Change-Id: Ia2bf2b8f40c2c709f8ad3b3d394a7946d4949b26

- Add walkthroughs/serving-llms-on-epyc.md (+ README link) for issue amd#82. - Re-register the skill in the marketplace (needs a walkthrough to be listed). - Require AVX-512 (Zen4+): hard gate in Step 1; scope in the description. - Fix launch: drop --shm-size (conflicts with --ipc=host on podman). - detect.py: recognize lettered EPYC SKUs (e.g. 9B45 -> Turin/Zen5). - Note re-run name collision, rootless-podman cpuset, and HF_HOME cache mount. Signed-off-by: Lalithnarayan C <Lalithnarayan.C@amd.com> Change-Id: Ia2bf2b8f40c2c709f8ad3b3d394a7946d4949b26

danielholanda · 2026-07-08T00:35:03Z

@shailensobhee Can you please help review this here as well?

danielholanda · 2026-07-20T20:32:23Z

@shailensobhee Any updates here?

shailensobhee · 2026-07-28T14:14:54Z

@amd-lalithnc Preparing to approve this. Some tasks still needs to be done:

The code drifted a bit and needs to be adapted to resolve conflicts. Can you please check?
I see that other skills have an eval. Can we have one too? See this: https://github.com/amd/skills/tree/main/skills/local-ai-use/evals
Can yo ucomment if all "EPYC" works? Example, EPYC 4000 series have AVX-512, so technically would work, but we know that this is not ofcicially listed in the ZenDNN release notes. The skill must align accordingly.
Also, please manually read the SKILL.md file. Some inconsistencies (which AI may overlook):
or launch an LLM on AMD EPYC, Zen CPU - what exactly is a Zen CPU here (can lead to ambiguity). All our x86 CPUs have Zen architecture, and because of that part, the AI may think the skill would work OOB on Ryzen without issues.
Also, that skills do not use on Instinct systems. I may argue this here; with semantic routers becoming popular, we may start seeing the channeling of small queries to small models, freeing a beefy model served on the GPU from handling simple tasks. On the same system, you could have multiple serving engines, on CPU and GPU. With this skill right now, you may completely strike out this scenario.
Venice went public at Advancing AI Day. Have the skill cover this as well (even if you have "or newer"). Tops at Turin currently.
in the walkthrough, first occurrence of AMD EPYC CPU, you may need the (TM) [legal]

shailensobhee · 2026-07-28T14:38:39Z

+## Step 1 - Understanding which skills are available
+
+* Run `claude "Which skills can you see?" --model sonnet`. You should see a list of skills that does **not** include anything about serving LLMs on EPYC / CPU.
+* Make sure there is no `AGENTS.md` file in your local folder.


Can you comment on why this line is important? What happens if you have an AGENTS.md file?

Agents.md files may contain additional instructions for claude. Asking the user to remove additional instructions their agents may have is helpful to make this process more reliable and easy to understand.

Integrate the upstream single-bundle marketplace and repo restructuring: Claude/Cursor/Codex plugin manifests and their generators, workflow and behavioral-eval relocations, and the other skills' updates. Resolve the marketplace and walkthrough-index conflicts by registering serving-llms-on-epyc (and its walkthrough) in the merged single bundle. Signed-off-by: Lalithnarayan C <Lalithnarayan.C@amd.com> Change-Id: I2cb6cbcbd82abbe76f432fefb09480bbbb712d76

Recognize 6th Gen EPYC (Venice/Zen6) and EPYC 4004/4005 by SKU, expose is_supported_epyc, and restrict the recipe to documented ZenDNN server families. Clarify the skill still serves a CPU endpoint on hosts that also have AMD Instinct GPUs. Signed-off-by: Lalithnarayan C <Lalithnarayan.C@amd.com> Change-Id: I9ec29f060d2e8fd6331eb6e635d58592bb04fcfa

Pin the validated vLLM 0.25.1 / zentorch 2.11.0.3 image (drop the redundant --shm-size), and probe the selected runtime for its exact versions and active vLLM platform. Venice on a non-pinned vLLM requires explicit confirmation; a stock (non-Zen) CPU platform is blocked. Signed-off-by: Lalithnarayan C <Lalithnarayan.C@amd.com> Change-Id: I4977cd5176df0bbf06db19bbae9c0d4569bf80c9

check_model.py inspects the model's chat template (standalone jinja or tokenizer_config) and picks /v1/chat/completions when one is present, else /v1/completions for raw prompts. A multimodal model with no usable template is rejected before launch. Signed-off-by: Lalithnarayan C <Lalithnarayan.C@amd.com> Change-Id: Iec5681d1a001f5c2968d492d95a21abbf1ceff3e

Document the compatibility gate and endpoint selection, make health verification endpoint-aware (/v1/models + non-error choices), and give a full client handoff (curl + OpenAI Python, parameter guidance). Add deterministic tests (gate policy, template classification, endpoint selection, detector, stack pin) and behavioral tests. Signed-off-by: Lalithnarayan C <Lalithnarayan.C@amd.com> Change-Id: I21a0474373ef784b1d5ada363c51baccb66ea51b

Limit the supported targets to AMD EPYC 9000-series server CPUs -- Genoa (9004), Turin (9005), and Venice (9006). Bergamo, Siena, and the AM5 EPYC 4004/4005 are still detected and named, but reported is_supported_epyc: false so the skill stops on them. Update the gate, reference, and walkthrough wording and add a regression test. Signed-off-by: Lalithnarayan C <Lalithnarayan.C@amd.com> Change-Id: I8593a33ffdf549b81fa0a157fe8a19cb8f7b41dd

amd-lalithnc · 2026-08-04T12:31:17Z

hi @shailensobhee - please take a look, have addressed your comments.

In addition, have made some minor changes to the core skill - giving client side info - chat templates, args to help the user better with sending requests to the user

danielholanda · 2026-08-05T23:44:21Z

@amd-lalithnc Can you take a look at the following link as part of your skill/walkthrough? It seems to be dead:

[404] https://github.com/AMD-AGI/TraceLens/blob/main/docs_original/Inference_analysis.md (at 27:1) | Rejected status code: 404 Not Found

danielholanda · 2026-08-06T00:29:25Z

      "skills": [
        "./skills/local-ai-use",
        "./skills/local-ai-app-integration",
+        "./skills/serving-llms-on-epyc",


Please remove marketplaces changes here. Skills only graduate to those "showcase" marketplaces after dev trials and marketing guidance.

Signed-off-by: Lalithnarayan C <Lalithnarayan.C@amd.com> Change-Id: I815a4dbf65594712f0533a2caf9135deceaa7220

Signed-off-by: Lalithnarayan C <Lalithnarayan.C@amd.com> Change-Id: I5e1075932c21aa34e11929f901d2932764b95580

amd-lalithnc · 2026-08-06T05:20:42Z

@amd-lalithnc Can you take a look at the following link as part of your skill/walkthrough? It seems to be dead:

[404] https://github.com/AMD-AGI/TraceLens/blob/main/docs_original/Inference_analysis.md (at 27:1) | Rejected status code: 404 Not Found

@danielholanda looks like this isn't part of this skill/walkthrough. That link is in walkthroughs/tracelens-analysis-orchestrator.md (the TraceLens skill's walkthrough, added in #89), not in serving-llms-on-epyc. The target is in the private AMD-AGI/TraceLens repo, so the external link-checker gets a 404 even though it resolves for authorized users.

danielholanda requested a review from shailensobhee July 8, 2026 00:34

danielholanda assigned amd-lalithnc Jul 20, 2026

shailensobhee reviewed Jul 28, 2026

View reviewed changes

amd-lalithnc added 6 commits August 4, 2026 03:00

danielholanda changed the title ~~Add serving-llms-on-epyc walkthrough~~ Add serving-llms-on-epyc walkthrough Aug 6, 2026

danielholanda reviewed Aug 6, 2026

View reviewed changes

Move serving-llms-on-epyc code changes and tests to a separate PR

7c2b5bd

Signed-off-by: Lalithnarayan C <Lalithnarayan.C@amd.com> Change-Id: I815a4dbf65594712f0533a2caf9135deceaa7220

amd-lalithnc mentioned this pull request Aug 6, 2026

serving-llms-on-epyc: behavioral tests and supporting checks #123

Open

Remove serving-llms-on-epyc from the marketplace manifests

bf5301f

Signed-off-by: Lalithnarayan C <Lalithnarayan.C@amd.com> Change-Id: I5e1075932c21aa34e11929f901d2932764b95580

Merge branch 'main' into add-serving-llms-on-epyc-walkthrough

95cd661

Navigation Menu

Uh oh!

Add `serving-llms-on-epyc` walkthrough - #83

Add `serving-llms-on-epyc` walkthrough#83
amd-lalithnc wants to merge 10 commits into
amd:mainfrom
amd-lalithnc:add-serving-llms-on-epyc-walkthrough

amd-lalithnc commented Jul 1, 2026

Uh oh!

danielholanda commented Jul 8, 2026

Uh oh!

danielholanda commented Jul 20, 2026

Uh oh!

shailensobhee commented Jul 28, 2026 •

edited

Loading

Uh oh!

shailensobhee Jul 28, 2026

Uh oh!

danielholanda Jul 31, 2026

Uh oh!

amd-lalithnc commented Aug 4, 2026

Uh oh!

danielholanda commented Aug 5, 2026

Uh oh!

danielholanda Aug 6, 2026

Uh oh!

amd-lalithnc Aug 6, 2026

Uh oh!

amd-lalithnc commented Aug 6, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

amd-lalithnc commented Jul 1, 2026

Uh oh!

danielholanda commented Jul 8, 2026

Uh oh!

danielholanda commented Jul 20, 2026

Uh oh!

shailensobhee commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shailensobhee Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

danielholanda Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

amd-lalithnc commented Aug 4, 2026

Uh oh!

danielholanda commented Aug 5, 2026

Uh oh!

danielholanda Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

amd-lalithnc Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

amd-lalithnc commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shailensobhee commented Jul 28, 2026 •

edited

Loading

amd-lalithnc commented Aug 6, 2026 •

edited

Loading