fix(pd): stop/eos must end request instead of continuing to next split block - #1440
Merged
sufubao merged 1 commit intoAug 6, 2026
Merged
Conversation
sufubao
force-pushed
the
fix/pd-master-stop-ignored-in-split-blocks
branch
from
August 6, 2026 11:28
2f16bd6 to
a3a063c
Compare
PD master splits max_new_tokens into blocks (LIGHTLLM_PD_SPLIT_MAX_NEW_TOKENS, default 2048). The block loop only special-cased finish_reason=="length", so when an early block ended on a real stop/eos the outer loop continued into the next block and resumed generation past the stop string. Break out of the split loop when a block finishes for any non-length reason.
sufubao
force-pushed
the
fix/pd-master-stop-ignored-in-split-blocks
branch
from
August 6, 2026 12:28
a3a063c to
d27699d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题
PD 分离部署下,当
max_new_tokens > LIGHTLLM_PD_SPLIT_MAX_NEW_TOKENS(默认 2048)时,stop 字符串 / eos 失效,生成会越过停止串继续吐字。根因
pd_master 的
_generate会把max_new_tokens按段(默认 2048)切分循环推理。分段循环里只特判了finish_reason == "length"(命中分段上限时转成 NoFinished 续算下一段):当某一段因 stop 字符串 / eos / abort / error 结束时,
_wait_to_token_package正常返回,但外层分段循环没有 break,于是用prompt + history_gen_token_strs(已含停止串文本)再起一段继续生成。而lightllm_generate_stream是把 generator 抽干的,不会因 finish 提前停止消费,所以续算真的发生,表现为停止串被无视。当
max_new_tokens <= 2048(单段)时不触发,这也是该 bug 非必现的原因。修复
某段以非
length原因结束时,跳出分段循环。只有命中分段长度上限(length)才允许续算下一段。abort/error同样会触发 break,避免在异常结束后还继续下一段。影响范围
仅 pd_master 路径,单段场景(
max_new_tokens <= LIGHTLLM_PD_SPLIT_MAX_NEW_TOKENS)行为不变。