Repository navigation
feature(Raw): Add raw data downloads - #20
Merged
Merged
Conversation
Add `aperiodic raw <dataset>` to stream raw Parquet files (trades, quotes, mark_price, index_price, funding_rate, open_interest) into the bucket's Hive layout, matching the Python client's download_raw: ranges over 366 days are split into several requests, files already present with the expected size are skipped, an expired presigned URL is re-requested once, and --preview fetches the free 2025-06 file with the shared demo key. Add `aperiodic raw coverage` for the public coverage endpoint. A 403 now keeps the server's message, code and upgrade_url, so a plan without raw data points to the upgrade page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016HQcoGs28SNTU4LwSW19kE
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds raw data (Prime + Raw plan) to the CLI. It mirrors
download_rawandget_raw_coveragefrom the Python client in aperiodic-io/client#32.get_rawloads data into a DataFrame, so the CLI has nothing that matches it and it is left out.New commands
aperiodic raw <dataset>streams raw Parquet files fortrades,quotes,mark_price,index_price,funding_rateoropen_interest.aperiodic raw coverage [--dataset] [--exchange] [--symbol]calls the public/metadata/rawendpoint. With no filters it prints one summary row per dataset and exchange, plus the preview symbols. With filters it prints one row per symbol: first and last day, day count, missing-day count and size. It needs no API key, and it sends noX-API-KEYheader.aperiodic raw helpshows the datasets, where each one is available, the file layout and the flags.Download behaviour, matching the Python client
{output-dir}/{dataset}/exchange=…/symbol=…/year=YYYY/month=MM[/day=DD]/data.parquet. Before 2026-08-01 there is one file per month; from that date there is one per day. On Windows,:is written as%3A. I checked the paths against the Pythonraw_file_pathand they match exactly.range_too_longerror. A monthly file that shows up in two of those requests is downloaded once.--overwritedownloads everything again.data.parquet.partand renamed once it is complete. A glob for**/data.parquetnever picks up a half-written file.tradesandquotes, and the error says so.raw_not_in_plan403 prints the server's message, the upgrade URL and a hint to try--preview.--preview: uses/data/raw/preview/{dataset}with the sharedDEMO-KEY, the same way--previewalready works for metrics. Dates are not needed.--max-concurrentdefaults to 4, the same as the PythonDEFAULT_RAW_DOWNLOADS.Changes to shared code
handleAPIErrornow reads the body of a 403 and keeps the message, details,codeandupgrade_url. Before, it always reportedForbidden. A 403 with an empty body still reportsForbidden.APIErrorhas two new fields,CodeandUpgradeURL.DownloadFilesConcurrentlymoved intorunConcurrently, which both downloaders use now. Metric downloads behave the same, with one exception:--max-concurrent 0used to hang forever and now runs one download at a time.How to try it
Testing
go build ./...,go vet ./...andgofmt -l .are all clean.GOOS=windows go vet ./...andGOOS=darwin go build ./...also pass.raw_test.gorun offline againsthttpteststubs. They cover:--overwrite, refreshing an expired URL, giving up after one refresh, retrying transient 500s,raw_not_in_plan, showing validation details, preview withDEMO-KEY, reporting missing periods, and flags placed before or after the datasetgo test -race ./...: every new test passes. Five existing tests fail locally only becauseAPERIODIC_API_KEYis not set in this environment (requireAPIKeycallst.Fatal):TestCLI_Symbols,TestCLI_OHLCV_Download,TestCLI_Symbols_HyperliquidPerps,TestCLI_OHLCV_HyperliquidPerps_DownloadandTestCLI_VTWAP_Download. They fail the same way onmain. CI sets the secret.🤖 Generated with Claude Code
https://claude.ai/code/session_016HQcoGs28SNTU4LwSW19kE
Generated by Claude Code