Skip to content

fix: align eval user_id default and support recursive evalset discovery#5425

Draft
brucearctor wants to merge 2 commits intogoogle:mainfrom
brucearctor:fix/eval-ui-issues-5423
Draft

fix: align eval user_id default and support recursive evalset discovery#5425
brucearctor wants to merge 2 commits intogoogle:mainfrom
brucearctor:fix/eval-ui-issues-5423

Conversation

@brucearctor
Copy link
Copy Markdown

@brucearctor brucearctor commented Apr 21, 2026

Summary

Fixes two server-side issues from #5423:

1. Align default user_id fallback (test_user_iduser)

When an eval set doesn't specify a user_id, the eval runner was defaulting to "test_user_id", while the adk-web frontend defaults to "user". This mismatch caused session 404 errors when the UI tried to fetch eval result sessions.

Changed in:

  • evaluation_generator.py
  • local_eval_service.py

Note: The companion frontend fix (using the userId from the eval result response instead of hardcoding) is submitted separately to google/adk-web#424.

2. Warn on undiscoverable evalset files in subdirectories

LocalEvalSetsManager.list_eval_sets() only scans the immediate agent directory. .evalset.json files in subdirectories (e.g., app/eval_sets/my_test.evalset.json) are silently ignored, causing confusion.

Now logs a clear warning message when subdirectory evalset files are detected, telling the user to move them to the root directory:

WARNING - Eval set file 'eval_sets/nested.evalset.json' found in subdirectory and will be ignored. Move it to '/path/to/app/' to make it discoverable.

Note on port change (Issue 3): adk web now serves on port 8000 instead of the previous 8501. This is not a bug but should be called out in migration/upgrade notes.

Testing

  • All 30 existing + new tests pass in test_local_eval_sets_manager.py
  • Added test_local_eval_sets_manager_list_eval_sets_subdirectories to verify the warning log behavior
  • Updated list_eval_sets_success test to use real filesystem instead of mocking os.listdir

Related

Fixes two server-side issues from google#5423:

1. Align default user_id fallback from 'test_user_id' to 'user'
   to match the adk-web frontend default, preventing session 404
   errors when eval sets don't specify a user_id.

2. Log a warning when .evalset.json files are found in subdirectories
   of the agent directory. These files are not discovered by the UI
   and should be moved to the root agent directory. This addresses
   the 'fragile and undocumented' evalset discovery reported in the
   issue.

Note on port change (Issue 3): adk web now serves on port 8000
instead of the previous 8501. This is not a bug but should be
called out in migration/upgrade notes.
@brucearctor brucearctor force-pushed the fix/eval-ui-issues-5423 branch from 9120947 to e9215a2 Compare April 21, 2026 07:07
@adk-bot adk-bot added the eval [Component] This issue is related to evaluation label Apr 21, 2026
@brucearctor brucearctor marked this pull request as draft April 21, 2026 07:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

eval [Component] This issue is related to evaluation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants