ASAREE

Analytical Sandbox for Agentic Research, Engineering, and Experimentation — a workbench for running LLM agents as designed experiments rather than one-off prompts.

You build a pipeline of agents on a visual protocol canvas, declare the factors you want to vary across it (model, effort, whether a critic gate is enabled, anything else bound to a node's config), and ASAREE materializes the full factorial design as cells, runs every replicate, and collects each replicate's metrics so the comparison is a measured result instead of an impression. Datasets are registered, split, and versioned as they pass between agents, and the tools the agents reach for are MCP servers, so a run is reproducible end to end.

ASAREE is built on top of Motoro, which provides the agent runtime, execution patterns, LLM service, and MCP integration. Motoro ships no HTTP layer, no auth, and no UI; ASAREE adds those, plus the experiment/protocol/dataset model, and depends on Motoro as a pinned library — in-process, not a service call.

Releases and development

For installation, reproducible research, and production deployments, use a tagged GitHub Release. The latest release is the recommended version for new installations. The project treats every published release tag as permanent so a deployment or experiment can be recreated from the same source later. Repository administrators enforce this policy with GitHub's release immutability setting; ordinary Git tags are not inherently immutable.

The main branch contains the latest development version. It may include changes that have not yet been released or fully validated for production, so do not use main when an exact, stable version matters.

Get started

You need git and Docker with Compose v2 (docker compose version), about 10 GB of free disk, and 10–20 minutes for the first build.

1. Install a release and start it. Open the releases page, choose a tag, and replace vX.Y.Z below with that tag (for example, v0.3.0).

git clone --branch vX.Y.Z --depth 1 https://github.com/EpistasisLab/ASAREE.git
cd ASAREE
cp .env.example .env
docker compose up -d --build

Contributors who intentionally want the current development version can clone main instead:

git clone --branch main https://github.com/EpistasisLab/ASAREE.git

That brings up Postgres, Redis, both migration steps, the API, the run worker, and the frontend.

2. Check it came up.

docker compose ps       # the two migrate services read "Exited (0)" — that's success
curl localhost:8000/health

3. Register. Open http://localhost:5173 and create an account.

4. Add an LLM API key for Anthropic, OpenAI, or Azure Foundry, under Profile → LLM credentials.

To keep them safe, put your own ASAREE_ENCRYPTION_KEY in .env before you save your first one — the sample value shipped in .env.example is public, and rotating the key later means re-entering every stored credential.

python3 -c "import secrets; print(secrets.token_urlsafe(32))"
docker compose up -d    # picks up the changed .env

5. Run an experiment. Follow the worked myocardial-infarction use case in publications/BDM/README.md — five agents in series building a classifier on a public dataset. It picks up exactly where this step leaves off.

Persona library

Add a Persona node and choose a built-in preset or a saved persona in its inspector. Uploading a plain .md file saves a reusable library entry named from the filename and copies its text into the node or factor level. No frontmatter, description, or personality metadata is required. The original local file is unchanged.

Use Manage personas / Save current in the node inspector or persona factor editor to save typed instructions, edit library entries, or delete them. Existing nodes and factor levels retain their copied text; choose an entry again to use its latest revision. Published experiments keep their frozen persona text even if the library entry is later edited or deleted.

Measurements and Results

The Design panel offers built-in runtime and supervised-ML metrics with their producer, required inputs, units, aggregation, direction, and current readiness. Custom metrics can read a typed numeric/Boolean structured-output field or use a model judge with an explicit model, rubric, bounds, and selected run inputs. A judge is an extra model call on every replicate, so its recurring call and cost estimate is shown before save. The optional wizard creates the same typed draft as the manual editor and cannot bypass the normal validator.

Every completed attempt records one state for every declared metric: measured, unavailable (the required value was absent), failed (the producer could not evaluate it), or not_applicable (the metric does not apply to that task). Results keeps those states distinct. Confusion matrices, calibration data, and per-class reports are evaluation artifacts: they appear in attempt detail but are never ranked or averaged as scalar metrics.

The Results CSV exports scalar values as ordinary analysis columns. Its observation_statuses JSON column retains each metric's state and error, and evaluation_artifacts retains structured diagnostics without flattening them into misleading scalar columns. The legacy_values JSON column keeps old text or other non-rankable values with legacy.unknown provenance when their original producer was never recorded; choose an explicit producer before running that legacy declaration again.

Automatic statistics reports

In an experiment’s Results panel, open Statistics and select Analyze all metrics. Statistics use only the experiment version being viewed, including when viewing history. The action is enabled once that version has completed whole-dataset replicates with recorded measurements. Reports are filtered to the selected version; rerun older unscoped reports to attach them to a version. Reanalyzing replaces that version’s previous automatic reports; older report copies are not retained. Each report shows the selected model distribution and the stored observed summaries (replicate count, minimum, maximum, mean, standard deviation, distinct values, and zero count), both in the panel and the HTML download. The observed distribution uses a histogram or category-frequency chart. Older reports recover observations from their immutable input snapshot, including ordinal categories and binomial counts; no experiment rerun or statistical refit is needed. Missing evidence is explained instead of leaving an empty chart. “What ran and why” presents the model, tests, uncertainty checks, and interpretation in a responsive overview with optional technical details. Findings use plain-language templates; statistics and report generation make no LLM calls. Available reports appear in a collapsible list; reports without valid statistical conclusions are hidden. New reports show analytic factor-test p-values alongside separately Holm-adjusted permutation sensitivity p-values, and analytic comparison intervals alongside bootstrap intervals. Defaults are 1,000 bootstrap refits and 1,999 permutations per term. Existing saved reports must be rerun to include the new results.

Native-model intervals use a cell-stratified bootstrap of independent runs; specialized custom models retain their parametric bootstrap. Model selection is held fixed. Bootstrap intervals require all requested attempts and at least 90% successful refits; native refitting has a 60-second budget per metric and reports incomplete attempts explicitly. Analytic results remain available if resampling cannot produce a reliable interval.

Permutation checks use working-scale OLS statistics, not the model's Wald statistic. Main effects use within-other-factor-stratum exchanges; interactions use approximate Freedman–Lane residual permutation. Exchangeability is an extra assumption: agreement does not validate a model, and different methods can test different null hypotheses. Known paired/repeated designs are withheld until a dependence model is supported; unequal trial totals withhold proportion permutation checks. Residual-permutation interaction checks are withheld for non-Gaussian models and when Gaussian cell variances show strong heterogeneity. Passing that screen does not prove exchangeability. Pointwise intervals are not multiplicity-adjusted, and separate metrics are exploratory rather than jointly corrected.

Download HTML report includes the same results, test names, rationale, requested/completed/failed counts, seed, limitations, and a script-free observed-distribution chart. To regenerate the labeled synthetic examples without a database or credentials:

python scripts/stats_example.py artifacts/statistics-demo

Everyday commands

docker compose logs -f asaree-app     # or asaree-worker, asaree-frontend
docker compose up -d --build          # rebuild after pulling new code
docker compose restart asaree-app     # apply an edited .env
docker compose down                   # stop, keep all data

Run the backend test suite in Compose when you do not already have the dev database exposed to the host. The one-shot runner waits for both migration chains and connects to Postgres over the internal Compose network, so it does not depend on localhost, POSTGRES_PORT, or a host PostgreSQL installation:

docker compose run --rm --build asaree-tests

The tests themselves still run under pytest; only pytest and its real Postgres dependency are placed on the same network. The test profile keeps this service out of a normal docker compose up.

CI selects checks from the complete PR diff or every commit in a push:

Changes	Checks
Documentation only	Dependency-free CI selector tests
Frontend	Type check, frontend tests and lint
Backend	Lint, type check, migrations and regression tests
Statistics source/tests or Python dependencies	Backend checks plus statistical validation
CI configuration, shared build files, or unknown paths	All checks

Statistical validation includes the repeated null simulations, non-Gaussian and custom-model uncertainty checks, and repeated interaction/bootstrap integration checks. Fast formula/reference checks, failure handling, API and migration tests remain in the regression suite. No tests are removed. Tests unrelated to resampling use small explicit simulation counts rather than production defaults.

With the test environment installed and any required PostgreSQL URLs configured:

uv run pytest tests/ -m 'not stats_validation' -q  # regression suite
uv run pytest tests/ -m stats_validation -q      # expensive validation
uv run pytest tests/ -q                         # both suites
python3 -m unittest discover -s .github/ci -p 'test_*.py'

Before a release, run the CI workflow manually using GitHub Actions' Run workflow control; manual runs always select every suite. Unknown or unavailable diffs also select every suite. Skipped backend/frontend jobs retain their existing required check names; a failed selector blocks the backend check. CI prints the slowest test durations for both backend tiers. New expensive tests should use @pytest.mark.stats_validation; keep their input paths covered by .github/ci/scope.py and its tests.

The frontend hot-reloads from your checkout; backend changes need a rebuild.

docker compose up -d --build also runs pending database migrations. The one-shot motoro-migrate service upgrades Motoro's motoro database first; asaree-migrate then upgrades ASAREE's asaree database. The API and worker start only after both migration services exit successfully. docker compose restart asaree-app restarts only that service and does not rerun the migration services.

Before upgrading an existing production deployment, back up both databases. Then check out the desired release tag and recreate the stack:

git fetch --tags
git switch --detach vX.Y.Z
docker compose up -d --build
docker compose ps

To run the migrations explicitly against an external PostgreSQL server, build the release's migration image, supply URLs for both databases, and run the two chains in order. URL-encode any special characters in the credentials.

docker compose build motoro-migrate asaree-migrate

MOTORO_DB_URL='postgresql+asyncpg://USER:PASSWORD@HOST:PORT/motoro'
ASAREE_DB_URL='postgresql+asyncpg://USER:PASSWORD@HOST:PORT/asaree'

docker compose run --rm --no-deps motoro-migrate deploy --url "$MOTORO_DB_URL"
docker compose run --rm --no-deps asaree-migrate upgrade --url "$ASAREE_DB_URL"

docker compose run --rm --no-deps motoro-migrate current --url "$MOTORO_DB_URL"
docker compose run --rm --no-deps asaree-migrate current --url "$ASAREE_DB_URL"

The stack binds ports 8000 (API), 5173 (frontend), 5453 (Postgres), and 6381 (Redis). If one is taken, set POSTGRES_PORT or REDIS_PORT in .env; the first two are in compose.yml.

Every bundled MCP server — asaree-workspace, motoro-okf, and the six domain servers (asaree-sklearn-dc, -eda, -fs, -fte, -model, -stats, from mcp-servers/) — ships as a dependency of the app and registers itself each time the app or worker starts (see asaree.services.system_mcp_servers).

Large experiment results

For large experiment results, the dedicated result worker handles summaries, batch commands and exports. Deployment, recovery, API/SDK usage and pending PostgreSQL acceptance checks are documented in RESULTS_OPERATIONS.md.

Resetting your dev environment

One Postgres server hosts two databases in a single volume — motoro (core's schema) and asaree (this repo's). Wiping it wipes both at once: every user, agent, experiment, dataset, MCP server registration, and LLM credential.

docker compose down -v
docker compose up -d --build

You're now at true zero. To get back to a working state:

1. Register a user in the GUI again (and, for SDK/notebook work, issue a token — see the SDK's Auth bootstrap).
2. Re-add the LLM credential, then re-run a use case notebook's early setup cells (experiment, dataset, agent creation) — or, for the public myocardial-infarction use case, follow its walkthrough: publications/BDM/README.md.