Skip to content

fix(pd): stop/eos must end request instead of continuing to next split block - #1440

Merged
sufubao merged 1 commit into
ModelTC:mainfrom
sufubao:fix/pd-master-stop-ignored-in-split-blocks
Aug 6, 2026
Merged

fix(pd): stop/eos must end request instead of continuing to next split block#1440
sufubao merged 1 commit into
ModelTC:mainfrom
sufubao:fix/pd-master-stop-ignored-in-split-blocks

Conversation

@sufubao

@sufubao sufubao commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

问题

PD 分离部署下,当 max_new_tokens > LIGHTLLM_PD_SPLIT_MAX_NEW_TOKENS(默认 2048)时,stop 字符串 / eos 失效,生成会越过停止串继续吐字。

根因

pd_master 的 _generate 会把 max_new_tokens 按段(默认 2048)切分循环推理。分段循环里只特判了 finish_reason == "length"(命中分段上限时转成 NoFinished 续算下一段):

async for sub_req_id, request_output, metadata, finish_status in results_generator:
    assert sub_req_id == block_group_request_id
    if finish_status.get_finish_reason() == "length" and (not is_last_block):
        finish_status = FinishStatus()  # 转换为NoFinished
    ...
    yield origin_group_request_id, request_output, metadata, finish_status

await self.remove_req(group_request_id=block_group_request_id)
# ← 缺少 break

当某一段因 stop 字符串 / eos / abort / error 结束时,_wait_to_token_package 正常返回,但外层分段循环没有 break,于是用 prompt + history_gen_token_strs(已含停止串文本)再起一段继续生成。而 lightllm_generate_stream 是把 generator 抽干的,不会因 finish 提前停止消费,所以续算真的发生,表现为停止串被无视。

max_new_tokens <= 2048(单段)时不触发,这也是该 bug 非必现的原因。

修复

某段以非 length 原因结束时,跳出分段循环。只有命中分段长度上限(length)才允许续算下一段。

block_real_finished = False
async for sub_req_id, request_output, metadata, finish_status in results_generator:
    assert sub_req_id == block_group_request_id
    orig_finish_reason = finish_status.get_finish_reason()
    if orig_finish_reason == "length" and (not is_last_block):
        finish_status = FinishStatus()  # 转换为NoFinished
    ...
    yield origin_group_request_id, request_output, metadata, finish_status
    if orig_finish_reason not in ("length", None, ""):
        block_real_finished = True

await self.remove_req(group_request_id=block_group_request_id)
if block_real_finished:
    break

abort / error 同样会触发 break,避免在异常结束后还继续下一段。

影响范围

仅 pd_master 路径,单段场景(max_new_tokens <= LIGHTLLM_PD_SPLIT_MAX_NEW_TOKENS)行为不变。

@sufubao
sufubao force-pushed the fix/pd-master-stop-ignored-in-split-blocks branch from 2f16bd6 to a3a063c Compare August 6, 2026 11:28
PD master splits max_new_tokens into blocks (LIGHTLLM_PD_SPLIT_MAX_NEW_TOKENS,
default 2048). The block loop only special-cased finish_reason=="length", so
when an early block ended on a real stop/eos the outer loop continued into the
next block and resumed generation past the stop string. Break out of the split
loop when a block finishes for any non-length reason.
@sufubao
sufubao force-pushed the fix/pd-master-stop-ignored-in-split-blocks branch from a3a063c to d27699d Compare August 6, 2026 12:28
@sufubao
sufubao merged commit ff896c2 into ModelTC:main Aug 6, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant