This repository contains the data processing and model training code used for ML studies on MINERvA events.
Evaluation plots — browse the latest model-comparison figures (classification, regression, training curves) by configuration: minerva-ml plots · Event viewer (work in progress)
The processed dataset used by this project is available on Hugging Face:
- gregorkrzmanc/minerva-ml
- It is a preprocessed version of the MINERvA open data release for ML/physics tasks such as available-energy estimation and event tagging.
- Source data comes from MINERvA open data: MINERvA Open Data.
- This is a derived dataset and is not an official MINERvA collaboration product.
For detailed data fields and semantics, see DATASET.md. For model architecture details, see MODELS.md. For NERSC-specific paths and SLURM commands, see INTERNAL.md.
The typical workflow is:
- Download raw playlists
- Preprocess ROOT files into ML-ready tensors
- Split into train/val/test
- Train models locally or submit SLURM jobs
- Run test evaluation (
eval) on checkpoints, then produce figures withsrc.eval(or notebooks)
Use the gkrz/minerva_ml:v1 container (see the provided Dockerfile).
Choose one of the following:
- Option A (from scratch): Download raw MINERvA playlists, then preprocess locally.
- Option B (quick start): Download the already preprocessed dataset from Hugging Face and skip local preprocessing.
Set SCRATCH first, then run:
# Monte Carlo playlists
python -m src.scripts.download_data
# Recorded data playlists
python -m src.scripts.download_data --prefix MediumEnergy_FHC_Data_PlaylistIf you want to skip raw playlist processing, download the preprocessed dataset snapshot:
pip install -U "huggingface_hub[cli]"
huggingface-cli download gregorkrzmanc/minerva-ml \
--repo-type dataset \
--local-dir <HF_DATA_DIR>After download, point your training/splitting commands to the downloaded folder structure.
Skip this section if you used Option B and already have the preprocessed files you need.
Minimal invocation (creates .pb files with event-wise particle tensors and labels):
python -m src.scripts.preprocess_dataset --output-dir <OUTPUT_DIR>For a full pipeline on this project’s layout—preprocess, split playlists 1A / 1B, and extract baselines—edit paths in the script if needed, then run:
bash src/scripts/preprocess.shsrc/scripts/preprocess.sh sets DATA_DIR, runs preprocess_dataset with blob/prong limits and playlist selection, runs split_dataset per playlist (with different val/test ratios for 1B vs 1A), and runs extract_baselines against the raw playlist directories under scratch.
python -m src.scripts.split_dataset \
--input-dir <PREPROCESSED_DIR> \
--output-dir <SPLIT_OUTPUT_DIR>To inspect created features quickly, see notebooks/stats.ipynb.
After splitting, summarize the ML-ready .pb tensors with src.scripts.plot_dataset_distributions. Run from the repository root with your project Python environment activated (see section 0).
Full run (reads the split, writes all PDFs and a JSON summary):
python -m src.scripts.plot_dataset_distributions \
--data-path <SPLIT_OUTPUT_DIR> \
--playlist 1A \
--split train \
--out-dir plots/dataset_distributionsQuick check on a subsample (faster; good for testing):
python -m src.scripts.plot_dataset_distributions \
--data-path <SPLIT_OUTPUT_DIR> \
--playlist 1A --split train \
--max-events 50000 \
--out-dir plots/dataset_distributionsOutputs live under plots/dataset_distributions/ (per playlist + split, tag like 1A_train):
| Path | Contents |
|---|---|
particle_features_<tag>.pdf |
Per-particle columns (η, φ, log pT, log E, PID, …) |
logE_by_PID_<tag>.pdf |
log(E) histogram per reconstruction PID type |
global_features_<tag>.pdf |
16 event-level conditioning features |
truth_labels_<tag>.pdf |
All 15 MC truth columns |
n_particles_per_event_<tag>.pdf |
Token multiplicity |
summary_<tag>.json |
min/max/mean/std/percentiles per variable |
poster/ |
Publication-style step histograms (see below) |
poster_v2/ |
Same figures, two-red palette + Matter font |
Poster figures (poster/ and poster_v2/):
| File | Description |
|---|---|
tokens_per_event_<tag>.pdf |
Tokens per MINERvA event |
energy_comparison_<tag>.pdf |
True neutrino energy vs sum of token energies |
e_available_spectrum_<tag>.pdf |
MC E_available and E_available + E_μ |
e_available_by_class_<tag>.pdf |
E_ν plus E_available per Pi_labels_v2 class (CC 1π±, CC Nπ±, CC 1π⁰, CC other, NC) |
joint/tokens_per_event_<tag>.pdf |
MINERvA vs JetClass-II token multiplicity overlay |
joint/energy_comparison_<tag>.pdf |
MINERvA neutrino energy vs JetClass-II jet energy (log–log) |
Poster plots use density-normalized step histograms. The first full run also writes pickle caches under poster/poster_bins_<tag>.pkl (and poster_bins_jetclass2_<tag>.pkl when JetClass-II is enabled).
After a full run has built the poster caches, restyle or tweak poster code without reloading .pb files:
python -m src.scripts.plot_dataset_distributions \
--playlist 1A --split train \
--out-dir plots/dataset_distributions \
--poster-from-cacheThis regenerates both poster/ and poster_v2/. If a cache predates newer plots (e.g. E_available spectra), the script reloads truth labels from the dataset only for those missing sections.
Joint MINERvA/JetClass-II figures need preprocessed JetClass-II h5 shards. By default one file is sampled from the OmniLearned path; skip with --no-jetclass2:
python -m src.scripts.plot_dataset_distributions \
--data-path <SPLIT_OUTPUT_DIR> \
--playlist 1A --split train \
--jetclass2-path /path/to/jetclass2/train \
--jetclass2-n-files 1 \
--max-events 50000 \
--out-dir plots/dataset_distributions| Flag | Purpose |
|---|---|
--playlist |
Playlist subfolder (1A, 1B, …) |
--split |
train, val, or test |
--max-events |
Subsample events (deterministic with --seed) |
--bins |
Histogram bins for overview grids (default 80) |
--log-y |
Log y-axis on overview continuous histograms |
--poster-from-cache |
Regenerate poster PDFs from cached bins only |
--no-jetclass2 |
Skip JetClass-II and joint overlays |
Models are launched via src/jobs/submit_new_dataset_jobs.py, which submits the standard comparison model set (one SLURM job per (model, task, seed)) or, for MLP, prints a local training command.
Supported models (plot keys): HyperScale-small, HyperScale-small-rw, OmniLearned-small, OmniLearned-small-rw, OmniLearned-medium, HyperScale-medium, HyperScale-medium-rw, BERT-tiny, BERT-tiny-rw, BERT-small, BERT-small-rw, plus MLP.
# List available model keys
python src/jobs/submit_new_dataset_jobs.py --list-models
# Submit all models, both tasks, for the given seeds
python src/jobs/submit_new_dataset_jobs.py --seeds 55 56 57 58
# Submit a subset of models/tasks
python src/jobs/submit_new_dataset_jobs.py --seeds 55 --models HyperScale-small --tasks regression
# Ablation: zero muon-token kinematics (tags run names with _noMuonKin)
python src/jobs/submit_new_dataset_jobs.py --seeds 55 --models HyperScale-small --remove-muon-kinematics
# Write .slurm files without calling sbatch
python src/jobs/submit_new_dataset_jobs.py --seeds 55 --dry-runDataset/output paths default to this project's NERSC layout; pass --data-path to point at a different preprocessed dataset root. See INTERNAL.md for the exact hardcoded paths, SLURM account, and cluster layout.
After training, group the runs you want to compare in Weights & Biases by assigning the same tag to each run (in the run’s overview or via the API). The tools below use that tag against the minerva-models project under your W&B entity (set WANDB_ENTITY and use wandb login as needed).
Run or preview eval for checkpoints that still need test_results (skipped if an .npz for that dataset already exists). Activate your project Python environment first; child evals use the same interpreter (sys.executable).
# Preview commands
python -m src.scripts.evaluate_single_gpu --wandb-flag <TAG> --dry-run
# Run evals sequentially on the current node (login or single-GPU session)
python -m src.scripts.evaluate_single_gpu --wandb-flag <TAG>--wandb-flag only considers runs whose checkpoint folder name matches a wandb run name with that tag; omit it to evaluate every run under --ckpt-dir (default: see --help). Evaluation is very small and fast—it is fine to run on login nodes when a GPU is available locally.
On a multi-GPU node, pass --num-gpus N (alias -n N) to spread the eval jobs across N GPUs. One job runs per GPU concurrently, each pinned via CUDA_VISIBLE_DEVICES, and idle GPUs immediately pick up the next pending job. Eval jobs are deduplicated the same way as the single-GPU path (a (folder, dataset) is skipped if its .npz already exists), so the GPUs never repeat work.
python -m src.scripts.evaluate_single_gpu --wandb-flag <TAG> --num-gpus 4When CUDA_VISIBLE_DEVICES is already set (as SLURM does on an allocation), the script only schedules onto those allocated GPUs and caps --num-gpus to the number visible (warning if you ask for more). --num-gpus 1 (the default) keeps the original sequential behavior. See INTERNAL.md for the SLURM allocation command used on NERSC.
Offline plotting reads cached pickles and writes PDFs under plots/. Run from the repository root with your project Python environment activated (see section 0).
W&B runs are selected by the wandb tag passed as --flag (see fetch_runs_from_wandb in src/utils/utils.py). Model keys in plot configs must match the grouped model names in those pickles (e.g. OmniLearned-small, BERT-tiny).
export FLAG=Run_2703
export OUT=<CACHE_DIR>
python -m src.eval.collect_eval_data \
--flag $FLAG --out-dir $OUTWrites classification_<TAG>.pkl and regression_<TAG>.pkl under $OUT. Alternatively, download pre-built pickles from gregorkrzmanc/minerva-ml-eval and point $OUT at that directory.
Use scripts/rebuild_plot_caches.sh to rebuild all caches after new eval data:
bash scripts/rebuild_plot_caches.sh --force --sync-canonicalCaches are written outside the repo, next to $OUT (see INTERNAL.md for the exact NERSC path).
| Cache file | Built by | Used for |
|---|---|---|
classification_metrics.pkl |
build_classification_cache |
q3, W, pions (--plots-only) |
classification_light.pkl |
plot_classification_light |
light appendix + q3 light panels |
steps.pkl |
build_steps_cache |
training-curve plots (plot_steps --plots-only) |
regression.pkl |
plot_regression / collect_eval_data |
energy-regression plots |
One-shot (rebuild caches + render + publish):
bash scripts/rebuild_plot_caches.sh --force --config default --publishPDFs only (caches already fresh):
bash scripts/generate_comparison_plots.sh --config default
bash scripts/publish_plots.shA per-tag wrapper script that chains eval → collect → rebuild → replot for a single
W&B tag also exists (scripts/rebuild_plots_Run_0707.sh); see INTERNAL.md
for its usage and SLURM invocation.
Copies selected PDFs from plots/<config>/ into figures_latex/<config>/ with
stable names. Pass the same config name used for generate_comparison_plots.sh:
python -m src.scripts.copy_figures_for_paper --config V1Paper
# -> figures_latex/V1Paper/
# or: python -m src.scripts.copy_figures_for_paper -c V1Paper --dry-run
# optional: --plots-root plots/V1Paper --figures-root figures/V1PaperMoved to EVENT_VIEWER.md: a lightweight demo pipeline that carves the ML-ready dataset into small per-kinematic-bin samples, renders an interactive HTML event viewer, and scores those samples with trained models.
If you use this code or dataset, please cite:
@misc{krzmanc2026,
title={Cross-Domain Transfer with Particle Physics Foundation Models: From Jets to Neutrino Interactions},
author={Gregor Krzmanc and Vinicius Mikuni and Benjamin Nachman and Callum Wilkinson},
year={2026},
eprint={2604.12364},
archivePrefix={arXiv},
primaryClass={hep-ex},
url={https://arxiv.org/abs/2604.12364},
}See also CITATION.bib.
This project is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0) — see LICENSE.