测试修复:🐛 修复测试运行互斥、错误阶段记录与 CASE 日志恢复 - #710
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
本 PR 将
dev合入upstream/main以来的测试基建修改统一提交,覆盖后台任务互斥、测试阶段错误分类、CASE 日志持久化恢复和take_along_axis输入生成。原实现按脚本名生成 PID 文件,多个入口可能同时操作同一输出目录;CASE 日志在进程中断或重试时可能留下未闭合记录;错误日志中的阶段信息也不统一,影响定位和复测分类。问题定位
take_along_axis的函数入口参数名为arr,Tensor method 绑定后接收者参数为x,旧规则直接读取arr.shape会触发NoneType异常log_inorder.log实现方案
后台任务以实际
output.log_dir作为任务身份,使用同目录 PID 文件和 flock 锁串行化启动、停止、状态查询和 PID 清理。CASE 日志继续使用每 worker 一个临时文件和增量 offset;完整CASE_END后执行fsync,聚合通过.log_inorder.build和.inorder_state.json保存安全 offset,使用发布临时文件原子替换最终日志。只有最终派生输出发布成功后才清理 source,主进程重启时可丢弃未闭合尾部并恢复完整 CASE。测试阶段统一使用
Stage类型和预定义阶段常量,保留错误类型与执行阶段的独立性。输入规则通过arr/x回退兼容函数和 Tensor method 两种绑定形式。主要变更
1. 输出目录级任务互斥
run.py、run-example.sh和test_pipeline/V4/*.sh统一使用output.log_dir/.paddleapitest.pid及对应锁文件,增加进程组停止等待、旧 PID 保护和自然退出清理保护。CLI 文档与 run 配置 schema 同步移除脚本自定义pid_file。2. CASE 日志持久化与恢复
tester/reporting/log_aggregation.py增加安全 build、source offset、fsync和原子发布流程,支持首次 baseline、主进程中断恢复、worker crash synthetic end 和 cleanup 重试幂等。tester/reporting/log_runtime.py、log_worker.py负责结果分片和 CASE_END 的落盘。3. 统一错误阶段和结果日志
tester/reporting/log_schema.py定义输入、Paddle、Torch、同步、比较和内存预检查阶段;accuracy、paddle-only、稳定性和性能 tester 使用统一阶段名称,删除旧的 Paddle error dismiss 兼容路径,并扩展结果日志结构化字段。4. 输入生成规则修复
tester/input_generation/generation_rules.py兼容paddle.take_along_axis和paddle.Tensor.take_along_axis的参数绑定差异。改动文件
run.py,run-example.sh,test_pipeline/V4/*.shdocs/CLI_REFERENCE.md,test_pipeline/run_config.schema.json,test_pipeline/run_config.yamltester/reporting/log_aggregation.py,tester/reporting/log_runtime.py,tester/reporting/log_worker.pytester/reporting/log_schema.py,tester/reporting/log_comparison.pytester/base.py,tester/accuracy.py,tester/accuracy_stable.py,tester/paddle_only.py,tester/paddle_cinn_vs_dygraph.py,tester/paddle_device_vs_cpu.py,tester/paddle_device_vs_gpu.py,tester/paddle_gpu_performance.py,tester/paddle_torch_gpu_performance.py,tester/torch_gpu_performance.pytester/input_generation/generation_rules.py