[DRAFT] BREAKING FEAT: Scenario Core Refactor Proposal

.github/instructions/scenarios.instructions.md

@@ -34,9 +34,13 @@ class MyScenario(Scenario):
         return DatasetConfiguration(dataset_names=["my_dataset"])
 ```
 
-4. **Optionally override `_get_atomic_attacks_async()`** — the base class provides a default
+4. **Optionally override `_get_steps_async()`** — the base class provides a default
    that uses the factory/registry pattern (see "AtomicAttack Construction" below).
-   Only override if your scenario needs custom attack construction logic.
+   Only override if your scenario needs custom step construction logic.
+
+   > **Deprecation note:** The legacy hook `_get_atomic_attacks_async()` still works as a
+   > passthrough but is deprecated and will be removed in 0.16.0. Migrate overrides to
+   > `_get_steps_async()` — the body is identical; only the name changes.
 
 ## Constructor Pattern
 
@@ -53,7 +57,7 @@ def __init__(
     if not objective_scorer:
         objective_scorer = self._get_default_scorer()
 
-    # 2. Store config objects for _get_atomic_attacks_async
+    # 2. Store config objects for _get_steps_async
     self._scorer_config = AttackScoringConfig(objective_scorer=objective_scorer)
 
     # 3. Call super().__init__ — required args: version, strategy_class, objective_scorer
@@ -139,9 +143,12 @@ Note: `atomic_attack_name` must remain unique per `AtomicAttack` for correct res
 
 ## AtomicAttack Construction — Default Base Class Behaviour
 
-The `Scenario` base class provides a default `_get_atomic_attacks_async()` that uses the
+The `Scenario` base class provides a default `_get_steps_async()` that uses the
 factory/registry pattern.  Scenarios that register their techniques via `_get_attack_technique_factories()`
-get atomic-attack construction **for free** — no override needed.
+get step construction **for free** — no override needed.
+
+> The legacy hook `_get_atomic_attacks_async()` still works as a passthrough but is
+> deprecated and will be removed in 0.16.0. Use `_get_steps_async()` for new code.
 
 The default implementation:
 1. Calls `self._get_attack_technique_factories()` to get name→factory mapping
@@ -150,13 +157,13 @@ The default implementation:
 4. Uses `self._build_display_group()` for user-facing grouping
 5. Builds `AtomicAttack` with unique `atomic_attack_name` = `"{technique}_{dataset}"`
 
-### Customization hooks (no need to override `_get_atomic_attacks_async`):
+### Customization hooks (no need to override `_get_steps_async`):
 - **`_get_attack_technique_factories()`** — override to add/remove/replace factories
 - **`_build_display_group()`** — override to change grouping (default: by technique)
 
-### When to override `_get_atomic_attacks_async`:
+### When to override `_get_steps_async`:
 Only override when the scenario **cannot** use the factory/registry pattern — e.g., scenarios
-with custom composite logic, per-strategy converter stacks, or non-standard attack construction.
+with custom composite logic, per-strategy converter stacks, or non-standard step construction.
 
 Overrides that want baseline support must emit it themselves by calling `self._build_baseline_atomic_attack(seed_groups=...)` with the same seeds used for the strategy attacks and prepending the result. The base implementation emits baseline automatically; passing freshly resolved seeds reintroduces ADO 9012 (baseline-vs-strategy population divergence under `max_dataset_size`).
 
@@ -185,4 +192,4 @@ New scenarios must be registered in `pyrit/scenario/__init__.py` as virtual pack
 - Forgetting `@apply_defaults` on `__init__`
 - Empty `seed_groups` passed to `AtomicAttack`
 - Missing `VERSION` class constant
-- Missing `_async` suffix on `_get_atomic_attacks_async`
+- Missing `_async` suffix on `_get_steps_async`

doc/code/scenarios/0_scenarios.py

@@ -64,8 +64,10 @@
 # 2. **Scenario Class**: Extend `Scenario` and implement these abstract methods:
 #    - `get_strategy_class()`: Return your strategy enum class
 #    - `get_default_strategy()`: Return the default strategy (typically `YourStrategy.ALL`)
-#    - The base class provides a default `_get_atomic_attacks_async()` that uses the factory/registry
-#      pattern. Override it only if your scenario needs custom attack construction logic.
+#    - The base class provides a default `_get_steps_async()` that uses the factory/registry
+#      pattern. Override it only if your scenario needs custom step construction logic.
+#      (The legacy hook `_get_atomic_attacks_async()` still works but is deprecated and will
+#      be removed in 0.16.0.)
 #
 # 3. **Default Dataset**: Implement `default_dataset_config()` to specify the datasets your scenario uses out of the box.
 #    - Returns a `DatasetConfiguration` with one or more named datasets (e.g., `DatasetConfiguration(dataset_names=["my_dataset"])`)
@@ -155,8 +157,8 @@ def __init__(
     def _build_display_group(self, *, technique_name: str, seed_group_name: str) -> str:
         return seed_group_name
 
-    # No _get_atomic_attacks_async override needed!
-    # The base class builds attacks from the (technique x dataset) cross-product
+    # No _get_steps_async override needed!
+    # The base class builds steps from the (technique x dataset) cross-product
     # using the factory/registry pattern automatically.
 
 

doc/code/scenarios/3_adaptive_scenarios.ipynb

@@ -0,0 +1,242 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "0",
+   "metadata": {},
+   "source": [
+    "# Adaptive Scenarios\n",
+    "\n",
+    "An **adaptive scenario** doesn't run every attack technique against every objective.\n",
+    "Instead, it picks which technique to try next per-objective, learns from what worked,\n",
+    "and stops as soon as one technique succeeds. This concentrates spend on techniques\n",
+    "that actually work on your target.\n",
+    "\n",
+    "## How it works (high level)\n",
+    "\n",
+    "For each objective, the scenario tries up to `max_attempts_per_objective` techniques:\n",
+    "\n",
+    "- With probability `epsilon`, it **explores** — picks a random technique.\n",
+    "- Otherwise it **exploits** — picks the technique with the highest observed success\n",
+    "  rate so far.\n",
+    "- It records the outcome and stops early on success.\n",
+    "\n",
+    "Unseen techniques are tried first, so the first few objectives effectively round-robin\n",
+    "through every technique before the scenario settles on the best performers.\n",
+    "\n",
+    "## Adaptive vs. static scenarios\n",
+    "\n",
+    "| Feature             | Static scenarios                  | Adaptive scenarios                 |\n",
+    "|---------------------|-----------------------------------|------------------------------------|\n",
+    "| Technique selection | Run every selected technique      | Pick per-objective from outcomes   |\n",
+    "| Early stopping      | No                                | Yes — stops on first success       |\n",
+    "| Cost                | O(techniques × objectives)        | O(max_attempts × objectives)       |\n",
+    "\n",
+    "`AdaptiveScenario` is the modality-agnostic base class.\n",
+    "[`TextAdaptive`](../../../pyrit/scenario/scenarios/adaptive/text_adaptive.py) is the\n",
+    "text subclass used in the examples below."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1",
+   "metadata": {},
+   "source": [
+    "## Setup"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "\n",
+    "from pyrit.registry import TargetRegistry\n",
+    "from pyrit.scenario import DatasetConfiguration\n",
+    "from pyrit.scenario.printer.console_printer import ConsoleScenarioResultPrinter\n",
+    "from pyrit.scenario.scenarios.adaptive import TextAdaptive, harm_category_context\n",
+    "from pyrit.setup import initialize_from_config_async\n",
+    "\n",
+    "await initialize_from_config_async(config_path=Path(\"../../scanner/pyrit_conf.yaml\"))  # type: ignore\n",
+    "\n",
+    "objective_target = TargetRegistry.get_registry_singleton().get_instance_by_name(\"openai_chat\")\n",
+    "printer = ConsoleScenarioResultPrinter()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3",
+   "metadata": {},
+   "source": [
+    "## Basic usage\n",
+    "\n",
+    "Defaults: `epsilon=0.2`, `max_attempts_per_objective=3`, the subclass's default datasets."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "scenario = TextAdaptive()\n",
+    "\n",
+    "await scenario.initialize_async(  # type: ignore\n",
+    "    objective_target=objective_target,\n",
+    ")\n",
+    "result = await scenario.run_async()  # type: ignore\n",
+    "await printer.write_async(result)  # type: ignore"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5",
+   "metadata": {},
+   "source": [
+    "## Configuring a run\n",
+    "\n",
+    "All the knobs below are constructor or `initialize_async` arguments — combine whichever\n",
+    "you need on a single scenario instance:\n",
+    "\n",
+    "- **`epsilon`** — exploration probability. `0.0` is pure exploit, `1.0` is pure random,\n",
+    "  `0.2` (default) is 20% exploration.\n",
+    "- **`max_attempts_per_objective`** — caps techniques tried per objective. Higher means\n",
+    "  more chances to succeed and more API calls.\n",
+    "- **`context_extractor`** — partitions the success-rate table. The default\n",
+    "  `global_context` keeps one shared table; `harm_category_context` learns each harm\n",
+    "  category independently. Custom callables of type `Callable[[SeedAttackGroup], str]`\n",
+    "  are supported.\n",
+    "- **`seed`** — makes every selection decision deterministic.\n",
+    "- **`scenario_strategies`** (on `initialize_async`) — restricts which techniques the\n",
+    "  selector can pick from. Use `TextAdaptive.get_strategy_class()` to access the enum.\n",
+    "\n",
+    "The cell below exercises all of them at once."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "strategy_class = TextAdaptive.get_strategy_class()\n",
+    "\n",
+    "configured_scenario = TextAdaptive(\n",
+    "    epsilon=0.3,\n",
+    "    max_attempts_per_objective=5,\n",
+    "    context_extractor=harm_category_context,\n",
+    "    seed=42,\n",
+    ")\n",
+    "\n",
+    "await configured_scenario.initialize_async(  # type: ignore\n",
+    "    objective_target=objective_target,\n",
+    "    scenario_strategies=[strategy_class(\"single_turn\")],\n",
+    "    dataset_config=DatasetConfiguration(\n",
+    "        dataset_names=[\"airt_hate\", \"airt_violence\"],\n",
+    "        max_dataset_size=4,\n",
+    "    ),\n",
+    ")\n",
+    "configured_result = await configured_scenario.run_async()  # type: ignore\n",
+    "await printer.write_async(configured_result)  # type: ignore"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7",
+   "metadata": {},
+   "source": [
+    "## Resuming a run\n",
+    "\n",
+    "Adaptive scenarios are resumable — pass `scenario_result_id=...` to the `TextAdaptive`\n",
+    "constructor and the run picks up where it left off, with prior outcomes replayed into\n",
+    "the selector. Resume must use the same configuration as the original run."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "resumed_scenario = TextAdaptive(\n",
+    "    epsilon=0.3,\n",
+    "    max_attempts_per_objective=5,\n",
+    "    context_extractor=harm_category_context,\n",
+    "    seed=42,\n",
+    "    scenario_result_id=str(configured_result.id),\n",
+    ")\n",
+    "\n",
+    "await resumed_scenario.initialize_async(  # type: ignore\n",
+    "    objective_target=objective_target,\n",
+    "    scenario_strategies=[strategy_class(\"single_turn\")],\n",
+    "    dataset_config=DatasetConfiguration(\n",
+    "        dataset_names=[\"airt_hate\", \"airt_violence\"],\n",
+    "        max_dataset_size=4,\n",
+    "    ),\n",
+    ")\n",
+    "resumed_result = await resumed_scenario.run_async()  # type: ignore\n",
+    "await printer.write_async(resumed_result)  # type: ignore"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9",
+   "metadata": {},
+   "source": [
+    "## Inspecting which techniques were tried\n",
+    "\n",
+    "The dispatcher stamps every objective's `AttackResult.metadata` with:\n",
+    "\n",
+    "- `adaptive_context` — the bucket key from the `context_extractor`.\n",
+    "- `adaptive_attempts` — the ordered list of `{\"technique\", \"outcome\"}` dicts\n",
+    "  recording exactly which techniques the selector picked and what happened.\n",
+    "\n",
+    "Walk that metadata to see the per-objective trail and aggregate counts."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "10",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from collections import Counter\n",
+    "\n",
+    "# Per-objective trail\n",
+    "for results in resumed_result.attack_results.values():\n",
+    "    for r in results:\n",
+    "        attempts = r.metadata.get(\"adaptive_attempts\", [])\n",
+    "        trail = \" → \".join(f\"{a['technique']}({a['outcome']})\" for a in attempts)\n",
+    "        print(f\"[{r.outcome.value:7s}] {r.objective!r}: {trail}\")\n",
+    "\n",
+    "# Aggregate per-technique pick counts and success rate across the run\n",
+    "picks: Counter[str] = Counter()\n",
+    "wins: Counter[str] = Counter()\n",
+    "for results in resumed_result.attack_results.values():\n",
+    "    for r in results:\n",
+    "        for step in r.metadata.get(\"adaptive_attempts\", []):\n",
+    "            picks[step[\"technique\"]] += 1\n",
+    "            if step[\"outcome\"] == \"success\":\n",
+    "                wins[step[\"technique\"]] += 1\n",
+    "\n",
+    "print(\"\\nTechnique             wins / picks   rate\")\n",
+    "for technique, n in picks.most_common():\n",
+    "    print(f\"{technique:20s}  {wins[technique]:>4} / {n:<4}   {wins[technique] / n:.0%}\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "jupytext": {
+   "main_language": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}