digest

Data digest: localized-calibration study data as parquet

A standalone export of the study data behind "Localized Calibrated Uncertainty in Code Language Models" (the code in the rest of this repository). It is deliberately self-contained: copy this folder into another project and use the data + helpers without installing anything from the parent repo.

What this is

For each problem, a code-generation model produced a solution (base_code, possibly buggy) and there is a ground-truth corrected version (gt_fix_code). base_code is tokenized; for each token, gt_keeps[i] is True if that token survives unchanged into the fix and False if the intent aligning patch changed it. The localization task is to predict the False tokens.

2723 problems: 1368 from gpt-4o, 1355 from Qwen/Qwen2.5-Coder-7B-Instruct, across datasets humaneval_plus, livecodebench, mbpp_plus, repocod.

Files

ground_truth.parquet

The reusable core for testing any localization technique.

column	type	notes
problem_id	str	dataset-local id
dataset	str	humaneval_plus / livecodebench / mbpp_plus / repocod
gen_model	str	model that generated base_code
prompt_text	str	the prompt given to gen_model to produce base_code (flat string form; note it prefixes the role, e.g. user: ...)
prompt_messages	str (JSON)	same prompt as a list of {role, content} chat turns — prefer this for programmatic use
base_code	str	the (possibly buggy) generated solution = "".join(tokens)
gt_fix_code	str	ground-truth corrected version
base_success	bool	did base_code pass the tests as-is
tokenizer_key	str	HF tokenizer used (e.g. Qwen/Qwen2.5-Coder-0.5B)
tokens	list[str]	tokenization of base_code
gt_keeps	list[bool]	per token: True=kept in fix, False=changed in diff
line_token_spans	list[[int,int]]	[start,end) token ranges per line (study's def)
n_tokens, n_lines	int
gen_model_properties	str (JSON)	extra generation metadata

base_code/gt_fix_code are kept verbatim so you can apply an alternative tokenization and recompute keeps yourself — see Re-tokenizing below.

multis_estimates.parquet — the multi-sample technique (1 row/problem)

The existing study's localization method, quarantined from the ground truth. Join to ground_truth on (problem_id, dataset, gen_model).

column	type	notes
estimated_keeps	list[float]	per-token P(kept), mean over resamples; aligns with tokens
keep_tallys	list[list[int]]	per-sample 0/1 keep arrays
n_samples	int	valid resamples (2–5)
multi_temperature	float	0.8
multi_mode	str	from_prompt

Install

No package to install — copy this folder into your project and import loccalib_digest.py directly. Requirements (Python ≥ 3.10):

pip install pandas pyarrow       # or: uv add pandas pyarrow

Optional extras: scikit-learn only for platt_scale()/calibration_metrics(), transformers only for retokenize()/tokenize(). The original research repo and its dependencies (synthegrator, lmwrapper, torch, etc.) are not needed.

Usage

import loccalib_digest as lcd
tables = lcd.load("path/to/digest")     # dict of DataFrames
gt = tables["ground_truth"]

row = gt.iloc[0]
spans = [tuple(s) for s in row["line_token_spans"]]
lcd.line_texts(row["tokens"], spans)              # per-line source
lcd.line_is_buggy(row["gt_keeps"], spans)         # per-line ground-truth bug flag
# line-level estimate = aggregate token estimates over each line span:
# lcd.aggregate_line_estimate(est_row["estimated_keeps"], spans, method="min"|"mean"|"gmean")

Metrics

loccalib_digest includes the paper's calibration metrics, replicated from the study's calipy usage. Needs scikit-learn for Platt scaling.

import numpy as np
p = np.concatenate(est_rows["estimated_keeps"].to_list())   # P(token kept)
y = np.concatenate(gt_rows["gt_keeps"].to_list()).astype(int)
lcd.calibration_metrics(p, y)
# -> {"bss", "ece", "auc", "scaled_bss", "scaled_ece"}

Definitions (matching the paper):

• BSS brier_skill_score: (ref - brier) / ref with ref = base_rate * (1 - base_rate) (Brier of always predicting the label base rate))
• ECE ece: 10 uniform bins over [0,1]
• AUC auroc: standard ROC AUC (rank-based; positive class = kept).
• Scaled variants: Platt scaling (platt_scale) = logistic regression on the log-odds of the predictions (clip 1e-6, sklearn defaults). The paper's "Scaled" numbers fit the calibrator on each evaluation fold's own predictions.

Conventions: predictions are P(token kept), labels are gt_keeps (1 = kept). For line level, aggregate token predictions with aggregate_line_estimate and use not line_is_buggy as the label. The paper reports metrics per fold (fold = source dataset) and averages the folds.

Re-tokenizing with a different tokenizer

ground_truth ships a Qwen-0.5B tokenization, but you can swap in any tokenizer and recompute the keep labels from the raw base_code/gt_fix_code — no dependency on the original research repo (needs transformers):

out = lcd.retokenize(row["base_code"], row["gt_fix_code"],
                    "Qwen/Qwen2.5-Coder-7B-Instruct")
# -> {"tokens", "gt_fix_tokens", "gt_keeps", "line_token_spans", "tokenizer_key"}

lcd.tokenize(text, key) and lcd.compute_gt_keeps(base_tokens, fix_tokens) are the underlying pieces, lifted verbatim from the study (HF tokenize + marker detok; difflib keep-or-insert-before). Verified to reproduce the shipped tokens and gt_keeps exactly when called with the original tokenizer_key. Supported detok mappings: Qwen, Llama, Mistral.

Line structure is derived purely from tokens (a line break is a token containing \n), so lines stay in context with their problem — there is intentionally no separate per-line table.

Filtering note: nothing is filtered below the problem level. All tokens (including whitespace/newline tokens) are present, and whitespace-only lines (~5% of lines) are kept — the paper's line-level metrics included them, so keep them for comparable numbers. The only span-level exclusion is zero-token spans (ignore_empty_lines=True, the study's definition); spans still cover every token contiguously. At the problem level, only localizations passing the study's own validity filters were exported (see export_stats_*.json).

Citation

@misc{gros2025localizedcalibrateduncertaintycode,
      title={Localized Calibrated Uncertainty in Code Language Models},
      author={David Gros and Prem Devanbu},
      year={2025},
      eprint={2512.24560},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2512.24560},
}

base_code rows are model generations (GPT-4o, Qwen2.5-Coder-7B-Instruct) for problems drawn from the HumanEval+ / MBPP+ (EvalPlus), LiveCodeBench, and RepoCod benchmarks; see those projects for their own licenses and terms.

Some of this stuff could ideally be a bit cleaner. Let me know if any questions!