fix: align eval user_id default and support recursive evalset discovery#5425
Draft
brucearctor wants to merge 2 commits intogoogle:mainfrom
Draft
fix: align eval user_id default and support recursive evalset discovery#5425brucearctor wants to merge 2 commits intogoogle:mainfrom
brucearctor wants to merge 2 commits intogoogle:mainfrom
Conversation
Fixes two server-side issues from google#5423: 1. Align default user_id fallback from 'test_user_id' to 'user' to match the adk-web frontend default, preventing session 404 errors when eval sets don't specify a user_id. 2. Log a warning when .evalset.json files are found in subdirectories of the agent directory. These files are not discovered by the UI and should be moved to the root agent directory. This addresses the 'fragile and undocumented' evalset discovery reported in the issue. Note on port change (Issue 3): adk web now serves on port 8000 instead of the previous 8501. This is not a bug but should be called out in migration/upgrade notes.
9120947 to
e9215a2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes two server-side issues from #5423:
1. Align default
user_idfallback (test_user_id→user)When an eval set doesn't specify a
user_id, the eval runner was defaulting to"test_user_id", while the adk-web frontend defaults to"user". This mismatch caused session 404 errors when the UI tried to fetch eval result sessions.Changed in:
evaluation_generator.pylocal_eval_service.py2. Warn on undiscoverable evalset files in subdirectories
LocalEvalSetsManager.list_eval_sets()only scans the immediate agent directory..evalset.jsonfiles in subdirectories (e.g.,app/eval_sets/my_test.evalset.json) are silently ignored, causing confusion.Now logs a clear warning message when subdirectory evalset files are detected, telling the user to move them to the root directory:
Testing
test_local_eval_sets_manager.pytest_local_eval_sets_manager_list_eval_sets_subdirectoriesto verify the warning log behaviorlist_eval_sets_successtest to use real filesystem instead of mockingos.listdirRelated