Skip to content

feature(Raw): Add raw data downloads - #20

Merged
szemyd merged 1 commit into
mainfrom
claude/youthful-thompson-1ysfn8
Sep 29, 2026
Merged

szemyd merged 1 commit into
mainfrom
claude/youthful-thompson-1ysfn8

Conversation

@szemyd

@szemyd szemyd commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds raw data (Prime + Raw plan) to the CLI. It mirrors download_raw and get_raw_coverage from the Python client in aperiodic-io/client#32. get_raw loads data into a DataFrame, so the CLI has nothing that matches it and it is left out.

New commands

  • aperiodic raw <dataset> streams raw Parquet files for trades, quotes, mark_price, index_price, funding_rate or open_interest.
  • aperiodic raw coverage [--dataset] [--exchange] [--symbol] calls the public /metadata/raw endpoint. With no filters it prints one summary row per dataset and exchange, plus the preview symbols. With filters it prints one row per symbol: first and last day, day count, missing-day count and size. It needs no API key, and it sends no X-API-KEY header.
  • aperiodic raw help shows the datasets, where each one is available, the file layout and the flags.

Download behaviour, matching the Python client

  • File layout: files are written in the bucket's own Hive layout, {output-dir}/{dataset}/exchange=…/symbol=…/year=YYYY/month=MM[/day=DD]/data.parquet. Before 2026-08-01 there is one file per month; from that date there is one per day. On Windows, : is written as %3A. I checked the paths against the Python raw_file_path and they match exactly.
  • Long ranges: a range over 366 days is split into several requests, which avoids the range_too_long error. A monthly file that shows up in two of those requests is downloaded once.
  • Existing files: a file already on disk with the listed size is skipped, so running the same command again picks up where it stopped. --overwrite downloads everything again.
  • Expired URLs: if storage refuses a presigned URL with a 403, the CLI asks for that one period again and retries once with the new URL. A 403 is not retried with backoff. Transport errors and other bad statuses are retried with exponential backoff (1s, 2s, 4s).
  • Partial files: each file is streamed to data.parquet.part and renamed once it is complete. A glob for **/data.parquet never picks up a half-written file.
  • Validation: the CLI checks the dataset, the exchange and the dataset × exchange pair before it calls the API. Hyperliquid only has trades and quotes, and the error says so.
  • Plan errors: a raw_not_in_plan 403 prints the server's message, the upgrade URL and a hint to try --preview.
  • --preview: uses /data/raw/preview/{dataset} with the shared DEMO-KEY, the same way --preview already works for metrics. Dates are not needed.
  • Concurrency: --max-concurrent defaults to 4, the same as the Python DEFAULT_RAW_DOWNLOADS.

Changes to shared code

  • 403 errors: handleAPIError now reads the body of a 403 and keeps the message, details, code and upgrade_url. Before, it always reported Forbidden. A 403 with an empty body still reports Forbidden. APIError has two new fields, Code and UpgradeURL.
  • Concurrency helper: the semaphore and WaitGroup loop in DownloadFilesConcurrently moved into runConcurrently, which both downloaders use now. Metric downloads behave the same, with one exception: --max-concurrent 0 used to hang forever and now runs one download at a time.
  • Timeouts: raw transfers don't use the HTTP client's 60s overall timeout, because a multi-gigabyte file would hit it. A transfer is cancelled only if no data arrives for 60s.
  • README: new "Raw data" section covering the datasets, where each is available, the Prime + Raw plan requirement, the flags, the file layout, preview and coverage. The install section is unchanged.

How to try it

go build -o aperiodic ./cmd/aperiodic

# No account needed
./aperiodic raw coverage
./aperiodic raw trades --preview --exchange binance-futures --symbol perpetual-BTC-USDT:USDT --output-dir ./raw

# Needs a key on the Prime + Raw plan
export APERIODIC_API_KEY=...
./aperiodic raw quotes \
  --exchange okx-perps \
  --symbol perpetual-BTC-USDT:USDT \
  --start-date 2025-01-01 \
  --end-date 2025-12-31 \
  --output-dir ./raw

Testing

  • go build ./..., go vet ./... and gofmt -l . are all clean. GOOS=windows go vet ./... and GOOS=darwin go build ./... also pass.
  • New tests in raw_test.go run offline against httptest stubs. They cover:
    • splitting ranges into 366-day windows, including the leap-year case the Python tests use
    • parsing periods and working out their start and end days
    • the file layout, with and without Windows escaping
    • dataset × exchange validation
    • keeping the 403 reason
    • download end to end: the Hive layout, splitting plus de-duplication, skip-if-present plus --overwrite, refreshing an expired URL, giving up after one refresh, retrying transient 500s, raw_not_in_plan, showing validation details, preview with DEMO-KEY, reporting missing periods, and flags placed before or after the dataset
    • the coverage summary, symbol listing and rejected filters
    • transfer timeouts: a slow but steady transfer finishes after the client's timeout has passed, and a stalled transfer is aborted with nothing left on disk
  • go test -race ./...: every new test passes. Five existing tests fail locally only because APERIODIC_API_KEY is not set in this environment (requireAPIKey calls t.Fatal): TestCLI_Symbols, TestCLI_OHLCV_Download, TestCLI_Symbols_HyperliquidPerps, TestCLI_OHLCV_HyperliquidPerps_Download and TestCLI_VTWAP_Download. They fail the same way on main. CI sets the secret.
  • The environment could not reach aperiodic.io, so nothing was tested against the real API. I also ran the built binary by hand against a local stub server.

🤖 Generated with Claude Code

https://claude.ai/code/session_016HQcoGs28SNTU4LwSW19kE


Generated by Claude Code

Add `aperiodic raw <dataset>` to stream raw Parquet files (trades, quotes,
mark_price, index_price, funding_rate, open_interest) into the bucket's Hive
layout, matching the Python client's download_raw: ranges over 366 days are
split into several requests, files already present with the expected size are
skipped, an expired presigned URL is re-requested once, and --preview fetches
the free 2025-06 file with the shared demo key. Add `aperiodic raw coverage`
for the public coverage endpoint.

A 403 now keeps the server's message, code and upgrade_url, so a plan without
raw data points to the upgrade page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016HQcoGs28SNTU4LwSW19kE
@szemyd
szemyd merged commit 7961173 into main Sep 29, 2026
9 checks passed
@szemyd
szemyd deleted the claude/youthful-thompson-1ysfn8 branch September 29, 2026 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants