diff --git a/README.md b/README.md index 3bbb2d2..9bbb9d3 100644 --- a/README.md +++ b/README.md @@ -44,14 +44,18 @@ Get your API key at [aperiodic.io](https://aperiodic.io). For [preview data](#preview-data) (`--preview`), no API key is required — the CLI uses the shared public demo key automatically. +[Raw data](#raw-data) needs a key on the Prime + Raw plan; `aperiodic raw coverage` and `aperiodic raw --preview` work without one. + ## Usage ``` aperiodic [flags] aperiodic symbols [flags] +aperiodic raw [flags] +aperiodic raw coverage [flags] ``` -The first argument is the metric name. Use `symbols` to list available symbols for an exchange. +The first argument is the metric name. Use `symbols` to list available symbols for an exchange, and `raw` for [raw data](#raw-data). ## Available Metrics @@ -177,7 +181,7 @@ aperiodic ohlcv --preview \ ## Output -All data commands download **Parquet files** to `--output-dir`, fetched concurrently (tunable via `--max-concurrent`). +All metric commands download **Parquet files** to `--output-dir`, fetched concurrently (tunable via `--max-concurrent`). Raw data uses its own [file layout](#raw-file-layout). History up to **2026-07-31** is one file per month; from **2026-08-01** onwards it is one file per day. You always ask for a date range and get back every file covering it, so a range spanning the changeover downloads the earlier months as monthly files followed by a daily file per day. @@ -189,6 +193,82 @@ Filenames follow the granularity, zero-padded so a directory listing sorts chron 2026-08-02.parquet ``` +## Raw data + +Raw trades, top-of-book quotes and derivative ticks for Binance, OKX and Hyperliquid perpetuals: the data the metrics are built from. Raw data needs the **Prime + Raw** plan ([pricing](https://aperiodic.io/pricing)); it uses the same API key and symbols as the metrics. To try it without an account, use [`--preview`](#raw-preview). + +| Dataset | Contents | `binance-futures` | `okx-perps` | `hyperliquid-perps` | +|-----------------|----------------------------------------------|:-----------------:|:-----------:|:-------------------:| +| `trades` | Every trade: id, taker side, price, amount | yes | yes | yes | +| `quotes` | Top of book: best bid/ask price and amount | yes | yes | yes | +| `mark_price` | Mark price, one row per change | yes | yes | — | +| `index_price` | Index price, one row per change | yes | yes | — | +| `funding_rate` | Funding rate and next funding time | yes | yes | — | +| `open_interest` | Open interest, one row per change | yes | yes | — | + +Hyperliquid serves `trades` and `quotes` only; the CLI rejects the other combinations before calling the API. Raw L2 order books are not offered. + +Every file starts with `exchange_timestamp` (the venue's time), `local_timestamp` (when the event reached the capture machine) and `local_timestamp_kind`: `"measured"`, or `"modelled"` for days before the feed was captured directly, where the local time is the exchange time plus a latency drawn from the measured distribution. Don't use modelled days for latency research. Timestamps are UTC. + +**Download a year of trades:** +```bash +aperiodic raw trades \ + --exchange binance-futures \ + --symbol perpetual-BTC-USDT:USDT \ + --start-date 2025-01-01 \ + --end-date 2025-12-31 \ + --output-dir ./raw +``` + +**List what is available (no API key needed):** +```bash +aperiodic raw coverage # one row per dataset and exchange +aperiodic raw coverage --dataset quotes --exchange okx-perps # every symbol: first/last day, days, size +aperiodic raw coverage --symbol perpetual-BTC-USDT:USDT # one symbol across datasets +``` + +### Raw flags + +| Flag | Default | Description | +|--------------------|-------------------|-----------------------------------------------------------------------------| +| `--exchange` | `binance-futures` | Exchange name | +| `--symbol` | | Trading pair symbol (Atlas unified symbology) | +| `--start-date` | | First day, UTC (`YYYY-MM-DD`) | +| `--end-date` | | Last day, inclusive (`YYYY-MM-DD`); ranges over 366 days are split into several requests for you | +| `--output-dir` | | Root folder for the file layout below (required) | +| `--max-concurrent` | `4` | Maximum concurrent downloads | +| `--overwrite` | `false` | Download files again even if already present with the expected size | +| `--preview` | `false` | Fetch the free preview file with the shared demo key (dates are ignored) | + +`aperiodic raw coverage` takes `--dataset`, `--exchange` and `--symbol` as filters; with none it prints the summary. + +### Raw file layout + +Files land in the bucket's own Hive layout, the same one the Python client's `download_raw` writes, so either can top up a folder the other started. History before **2026-08-01** is one file per month; from then on it is one file per day: + +``` +raw/trades/exchange=binance-futures/symbol=perpetual-BTC-USDT:USDT/year=2025/month=06/data.parquet # monthly +raw/trades/exchange=binance-futures/symbol=perpetual-BTC-USDT:USDT/year=2026/month=08/day=01/data.parquet # daily +``` + +On Windows, `:` is not allowed in file names, so it is written as `%3A`, which Hive-partition readers decode back. + +- Monthly files are written whole, so they can reach past the requested range: filter on `exchange_timestamp` when reading. Monthly and daily files sit at different depths, so read a folder with a recursive glob, e.g. `pl.scan_parquet("raw/trades/**/data.parquet")`. +- A file already on disk with the expected size is skipped, so running the same command again resumes an interrupted download. Each file is streamed to `data.parquet.part` and renamed once complete. +- Download URLs are valid for one hour; one that has expired is re-requested automatically. +- A key whose plan lacks raw data gets a `raw_not_in_plan` error with the link to upgrade. + +### Raw preview + +`--preview` needs no account and no API key (the shared public demo key is used automatically). It returns the **2025-06** file of each venue's BTC perpetual: `perpetual-BTC-USDT:USDT` on `binance-futures` and `okx-perps`, `perpetual-BTC-USDC:USDC` on `hyperliquid-perps`, for any dataset the venue serves. `--start-date` and `--end-date` are not needed. + +```bash +aperiodic raw quotes --preview \ + --exchange okx-perps \ + --symbol perpetual-BTC-USDT:USDT \ + --output-dir ./raw +``` + ## Build from Source Requires Go 1.24+. diff --git a/cli.go b/cli.go index 577c09c..5c91551 100644 --- a/cli.go +++ b/cli.go @@ -28,11 +28,15 @@ func (c *CLI) Run(args []string) int { } cmd := args[0] - if cmd == "help" || cmd == "-h" || cmd == "--help" { + if isHelpArg(cmd) { c.printUsage() return 0 } + if cmd == "raw" { + return c.runRaw(args[1:]) + } + fs := flag.NewFlagSet("aperiodic", flag.ContinueOnError) fs.SetOutput(c.Stderr) @@ -50,15 +54,9 @@ func (c *CLI) Run(args []string) int { return 2 } - apiKey := c.Env("APERIODIC_API_KEY") - if apiKey == "" { - if *previewFlag { - // Preview data is served against the shared demo key, so no key is required. - apiKey = DemoAPIKey - } else { - fmt.Fprintln(c.Stderr, "Error: APERIODIC_API_KEY environment variable not set (pass --preview to use the shared demo key)") - return 1 - } + apiKey, ok := c.resolveAPIKey(*previewFlag) + if !ok { + return 1 } client := NewAperiodicClient(apiKey) @@ -75,12 +73,32 @@ func (c *CLI) Run(args []string) int { return c.handleData(client, cmd, *timestampFlag, *intervalFlag, *exchangeFlag, *symbolFlag, *startDateFlag, *endDateFlag, *maxConcurrentFlag, *outputDirFlag, *previewFlag) } +func isHelpArg(arg string) bool { + return arg == "help" || arg == "-h" || arg == "--help" +} + +// resolveAPIKey returns APERIODIC_API_KEY or, for a preview request without +// one, the shared demo key. It reports the missing key itself. +func (c *CLI) resolveAPIKey(preview bool) (string, bool) { + if apiKey := c.Env("APERIODIC_API_KEY"); apiKey != "" { + return apiKey, true + } + if preview { + // Preview data is served against the shared demo key, so no key is required. + return DemoAPIKey, true + } + fmt.Fprintln(c.Stderr, "Error: APERIODIC_API_KEY environment variable not set (pass --preview to use the shared demo key)") + return "", false +} + func (c *CLI) printUsage() { fmt.Fprintln(c.Stdout, "Aperiodic CLI Client") fmt.Fprintln(c.Stdout) fmt.Fprintln(c.Stdout, "Usage:") fmt.Fprintln(c.Stdout, " aperiodic [flags]") fmt.Fprintln(c.Stdout, " aperiodic symbols [flags]") + fmt.Fprintln(c.Stdout, " aperiodic raw [flags]") + fmt.Fprintln(c.Stdout, " aperiodic raw coverage [flags]") fmt.Fprintln(c.Stdout) fmt.Fprintln(c.Stdout, "Metrics:") fmt.Fprintln(c.Stdout, " ohlcv OHLCV (open/high/low/close/volume)") @@ -105,6 +123,7 @@ func (c *CLI) printUsage() { fmt.Fprintln(c.Stdout) fmt.Fprintln(c.Stdout, "Commands:") fmt.Fprintln(c.Stdout, " symbols List available symbols for an exchange") + fmt.Fprintln(c.Stdout, " raw Download raw trades, quotes and derivative ticks (Prime + Raw plan; see 'aperiodic raw help')") fmt.Fprintln(c.Stdout, " help Show this help") fmt.Fprintln(c.Stdout) fmt.Fprintln(c.Stdout, "Environment:") diff --git a/cli_raw.go b/cli_raw.go new file mode 100644 index 0000000..f19808d --- /dev/null +++ b/cli_raw.go @@ -0,0 +1,447 @@ +package aperiodic + +import ( + "cmp" + "errors" + "flag" + "fmt" + "net/http" + "slices" + "strings" + "sync" + "text/tabwriter" + "time" +) + +var rawDatasetDescriptions = map[RawDataset]string{ + RawTrades: "Every trade: id, taker side, price, amount", + RawQuotes: "Top of book: best bid/ask price and amount", + RawMarkPrice: "Mark price, one row per change", + RawIndexPrice: "Index price, one row per change", + RawFundingRate: "Funding rate and next funding time", + RawOpenInterest: "Open interest, one row per change", +} + +type rawDownloadFlags struct { + exchange string + symbol string + startDate string + endDate string + outputDir string + maxConcurrent int + overwrite bool + preview bool +} + +func newRawDownloadFlagSet() (*flag.FlagSet, *rawDownloadFlags) { + f := &rawDownloadFlags{} + fs := flag.NewFlagSet("aperiodic raw", flag.ContinueOnError) + fs.StringVar(&f.exchange, "exchange", string(ExchangeBinanceFutures), "Exchange name") + fs.StringVar(&f.symbol, "symbol", "", "Trading pair symbol, e.g. perpetual-BTC-USDT:USDT") + fs.StringVar(&f.startDate, "start-date", "", "First day, UTC (YYYY-MM-DD)") + fs.StringVar(&f.endDate, "end-date", "", "Last day, inclusive (YYYY-MM-DD); ranges over 366 days are split for you") + fs.StringVar(&f.outputDir, "output-dir", "", "Root folder for the Hive layout (mandatory)") + fs.IntVar(&f.maxConcurrent, "max-concurrent", DefaultRawMaxConcurrent, "Maximum concurrent downloads") + fs.BoolVar(&f.overwrite, "overwrite", false, "Download files again even if already present with the expected size") + fs.BoolVar(&f.preview, "preview", false, "Fetch the free preview file (2025-06, each venue's BTC perpetual) with the shared demo key; dates are ignored") + return fs, f +} + +type rawCoverageFlags struct { + dataset string + exchange string + symbol string +} + +func newRawCoverageFlagSet() (*flag.FlagSet, *rawCoverageFlags) { + f := &rawCoverageFlags{} + fs := flag.NewFlagSet("aperiodic raw coverage", flag.ContinueOnError) + fs.StringVar(&f.dataset, "dataset", "", "List the symbols of this raw dataset") + fs.StringVar(&f.exchange, "exchange", "", "List the symbols on this exchange") + fs.StringVar(&f.symbol, "symbol", "", "Show this symbol's coverage") + return fs, f +} + +func (c *CLI) runRaw(args []string) int { + if len(args) == 0 || isHelpArg(args[0]) { + c.printRawUsage() + return 0 + } + if args[0] == "coverage" { + return c.runRawCoverage(args[1:]) + } + return c.runRawDownload(args) +} + +// parseRawArgs parses flags wherever they sit among the positional arguments; +// the flag package alone stops at the first positional one. On failure it +// returns the exit code, 0 when the user asked for help. +func (c *CLI) parseRawArgs(fs *flag.FlagSet, args []string) (positional []string, exitCode int, ok bool) { + fs.SetOutput(c.Stderr) + fs.Usage = func() {} + + for { + if err := fs.Parse(args); err != nil { + if errors.Is(err, flag.ErrHelp) { + c.printRawUsage() + return nil, 0, false + } + fmt.Fprintln(c.Stderr, "Run 'aperiodic raw help' for usage.") + return nil, 2, false + } + args = fs.Args() + if len(args) == 0 { + return positional, 0, true + } + positional = append(positional, args[0]) + args = args[1:] + } +} + +func (c *CLI) runRawDownload(args []string) int { + fs, f := newRawDownloadFlagSet() + positional, exitCode, ok := c.parseRawArgs(fs, args) + if !ok { + return exitCode + } + + switch { + case len(positional) == 0: + fmt.Fprintf(c.Stderr, "Error: a raw dataset is required (%s)\n", joinNames(RawDatasets)) + return 1 + case len(positional) > 1: + fmt.Fprintf(c.Stderr, "Error: unexpected arguments: %s\n", strings.Join(positional[1:], " ")) + return 1 + } + + query := RawQuery{ + Dataset: RawDataset(positional[0]), + Exchange: Exchange(f.exchange), + Symbol: f.symbol, + Preview: f.preview, + } + if err := ValidateRawQuery(query.Dataset, query.Exchange); err != nil { + fmt.Fprintf(c.Stderr, "Error: %v\n", err) + return 1 + } + if query.Symbol == "" { + fmt.Fprintln(c.Stderr, "Error: --symbol is required") + return 1 + } + if f.outputDir == "" { + fmt.Fprintln(c.Stderr, "Error: --output-dir is mandatory") + return 1 + } + if f.maxConcurrent < 1 { + fmt.Fprintln(c.Stderr, "Error: --max-concurrent must be at least 1") + return 1 + } + + var start, end time.Time + if query.Preview { + if f.startDate != "" || f.endDate != "" { + fmt.Fprintln(c.Stderr, "Note: --preview ignores --start-date and --end-date; the preview is a fixed file") + } + } else { + var err error + if start, end, err = parseRawDateRange(f.startDate, f.endDate); err != nil { + fmt.Fprintf(c.Stderr, "Error: %v\n", err) + return 1 + } + } + + apiKey, ok := c.resolveAPIKey(query.Preview) + if !ok { + return 1 + } + client := NewAperiodicClient(apiKey) + + listing, err := client.FetchRawFiles(query, start, end) + if err != nil { + c.reportRawAPIError("Error fetching raw file URLs", err, query.Preview) + return 1 + } + + if len(listing.Files) == 0 { + c.printMissingPeriods(listing.MissingPeriods) + fmt.Fprintln(c.Stdout, "No data found for the given criteria") + return 0 + } + + plan, err := PlanRawDownloads(f.outputDir, query, listing.Files, f.overwrite) + if err != nil { + fmt.Fprintf(c.Stderr, "Error: %v\n", err) + return 1 + } + + pending := slices.DeleteFunc(slices.Clone(plan), func(d RawDownload) bool { return d.Present }) + present := len(plan) - len(pending) + + fmt.Fprintf(c.Stdout, "Found %d %s files (%s) for %s on %s\n", len(plan), query.Dataset, formatBytes(totalRawBytes(plan)), query.Symbol, query.Exchange) + c.printMissingPeriods(listing.MissingPeriods) + if present > 0 { + fmt.Fprintf(c.Stdout, "Skipping %d already present with the expected size (pass --overwrite to download them again)\n", present) + } + if len(pending) == 0 { + fmt.Fprintf(c.Stdout, "Nothing to download: every file is already in %s\n", f.outputDir) + return 0 + } + + fmt.Fprintf(c.Stdout, "Downloading %d files (%s) to %s...\n", len(pending), formatBytes(totalRawBytes(pending)), f.outputDir) + + var mu sync.Mutex + done := 0 + err = client.DownloadRawFiles(query, pending, f.maxConcurrent, func(d RawDownload) { + mu.Lock() + defer mu.Unlock() + done++ + fmt.Fprintf(c.Stdout, " [%d/%d] %s (%s)\n", done, len(pending), d.Path, formatBytes(d.File.Size)) + }) + if err != nil { + fmt.Fprintf(c.Stderr, "Error downloading files: %v\n", err) + if done > 0 { + fmt.Fprintf(c.Stderr, "%d files finished and are kept; run the same command again to resume.\n", done) + } + return 1 + } + + fmt.Fprintf(c.Stdout, "Successfully downloaded %d files\n", len(pending)) + return 0 +} + +func (c *CLI) printMissingPeriods(periods []string) { + if len(periods) > 0 { + fmt.Fprintf(c.Stdout, "No file for %d period(s) in the range: %s\n", len(periods), summarizePeriods(periods)) + } +} + +func parseRawDateRange(startDate, endDate string) (start, end time.Time, err error) { + if startDate == "" || endDate == "" { + return start, end, errors.New("--start-date and --end-date are required (or pass --preview)") + } + if start, err = time.Parse(rawDateLayout, startDate); err != nil { + return start, end, fmt.Errorf("--start-date %q is not a YYYY-MM-DD date", startDate) + } + if end, err = time.Parse(rawDateLayout, endDate); err != nil { + return start, end, fmt.Errorf("--end-date %q is not a YYYY-MM-DD date", endDate) + } + if end.Before(start) { + return start, end, errors.New("--end-date must be on or after --start-date") + } + return start, end, nil +} + +func (c *CLI) reportRawAPIError(prefix string, err error, preview bool) { + var apiErr *APIError + if !errors.As(err, &apiErr) { + fmt.Fprintf(c.Stderr, "%s: %v\n", prefix, err) + return + } + + if apiErr.Code == "raw_not_in_plan" { + fmt.Fprintf(c.Stderr, "Error: %s\n", apiErr.Message) + fmt.Fprintf(c.Stderr, "Raw data needs the Prime + Raw plan: %s\n", cmp.Or(apiErr.UpgradeURL, RawUpgradeURL)) + fmt.Fprintln(c.Stderr, "To try it without an account, add --preview for the free 2025-06 file of each venue's BTC perpetual.") + return + } + + fmt.Fprintf(c.Stderr, "%s: %v\n", prefix, err) + for _, detail := range apiErr.Details { + fmt.Fprintf(c.Stderr, " - %s\n", detail) + } + if preview && apiErr.StatusCode == http.StatusBadRequest { + fmt.Fprintln(c.Stderr, "The raw preview serves only the 2025-06 file of each venue's BTC perpetual; 'aperiodic raw coverage' lists the preview symbols.") + } +} + +func (c *CLI) runRawCoverage(args []string) int { + fs, f := newRawCoverageFlagSet() + positional, exitCode, ok := c.parseRawArgs(fs, args) + if !ok { + return exitCode + } + if len(positional) > 0 { + fmt.Fprintf(c.Stderr, "Error: unexpected arguments: %s (filter with --dataset, --exchange or --symbol)\n", strings.Join(positional, " ")) + return 1 + } + + if err := validateRawCoverageFilters(RawDataset(f.dataset), Exchange(f.exchange)); err != nil { + fmt.Fprintf(c.Stderr, "Error: %v\n", err) + return 1 + } + + // Coverage is public; the key, if any, is not sent. + coverage, err := NewAperiodicClient("").GetRawCoverage() + if err != nil { + c.reportRawAPIError("Error fetching raw coverage", err, false) + return 1 + } + + if f.dataset == "" && f.exchange == "" && f.symbol == "" { + c.printRawCoverageSummary(coverage) + return 0 + } + return c.printRawCoverageSeries(coverage, f) +} + +// validateRawCoverageFilters checks whichever of the two filters are set. +func validateRawCoverageFilters(dataset RawDataset, exchange Exchange) error { + switch { + case dataset != "" && exchange != "": + return ValidateRawQuery(dataset, exchange) + case dataset != "": + return validateRawDataset(dataset) + case exchange != "": + return validateExchange(exchange) + } + return nil +} + +func (c *CLI) printRawCoverageSummary(coverage *RawCoverageResponse) { + if coverage.Coverage.GeneratedAt != "" { + fmt.Fprintf(c.Stdout, "Raw coverage as of %s\n\n", coverage.Coverage.GeneratedAt) + } + + w := tabwriter.NewWriter(c.Stdout, 0, 0, 2, ' ', 0) + fmt.Fprintln(w, "DATASET\tEXCHANGE\tSYMBOLS\tFIRST\tLAST\tSIZE") + for _, row := range coverage.Summary { + fmt.Fprintf(w, "%s\t%s\t%d\t%s\t%s\t%s\n", row.Dataset, row.Exchange, row.Symbols, row.First, row.Last, formatBytes(row.Bytes)) + } + w.Flush() + + if len(coverage.Preview.Symbols) > 0 { + previews := make([]string, 0, len(coverage.Preview.Symbols)) + for _, exchange := range orderedKeys(coverage.Preview.Symbols, toStrings(Exchanges)) { + previews = append(previews, exchange+" "+coverage.Preview.Symbols[exchange]) + } + fmt.Fprintf(c.Stdout, "\nFree preview (--preview, no account): %s of %s\n", coverage.Preview.Period, strings.Join(previews, ", ")) + } + fmt.Fprintln(c.Stdout, "Filter with --dataset, --exchange or --symbol to list symbols.") +} + +func (c *CLI) printRawCoverageSeries(coverage *RawCoverageResponse, f *rawCoverageFlags) int { + w := tabwriter.NewWriter(c.Stdout, 0, 0, 2, ' ', 0) + rows := 0 + + byDataset := coverage.Coverage.Datasets + for _, dataset := range orderedKeys(byDataset, toStrings(RawDatasets)) { + if f.dataset != "" && dataset != f.dataset { + continue + } + byExchange := byDataset[dataset] + for _, exchange := range orderedKeys(byExchange, toStrings(Exchanges)) { + if f.exchange != "" && exchange != f.exchange { + continue + } + bySymbol := byExchange[exchange] + for _, symbol := range orderedKeys(bySymbol, nil) { + if f.symbol != "" && symbol != f.symbol { + continue + } + if rows == 0 { + fmt.Fprintln(w, "DATASET\tEXCHANGE\tSYMBOL\tFIRST\tLAST\tDAYS\tMISSING\tSIZE") + } + s := bySymbol[symbol] + fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%d\t%d\t%s\n", dataset, exchange, symbol, s.First, s.Last, s.Days, len(s.Missing), formatBytes(s.Bytes)) + rows++ + } + } + } + w.Flush() + + if rows == 0 { + fmt.Fprintln(c.Stdout, "No raw coverage matches the given filters") + } + return 0 +} + +// orderedKeys returns m's keys with those in known first, in known's order, +// then the rest sorted. +func orderedKeys[V any](m map[string]V, known []string) []string { + keys := make([]string, 0, len(m)) + for _, k := range known { + if _, ok := m[k]; ok { + keys = append(keys, k) + } + } + + var rest []string + for k := range m { + if !slices.Contains(known, k) { + rest = append(rest, k) + } + } + slices.Sort(rest) + return append(keys, rest...) +} + +func totalRawBytes(downloads []RawDownload) int64 { + var total int64 + for _, d := range downloads { + total += d.File.Size + } + return total +} + +// formatBytes renders a size in binary units, e.g. "1.5 GiB". +func formatBytes(n int64) string { + const unit = 1024 + if n < unit { + return fmt.Sprintf("%d B", n) + } + div, exp := int64(unit), 0 + for m := n / unit; m >= unit; m /= unit { + div *= unit + exp++ + } + return fmt.Sprintf("%.1f %ciB", float64(n)/float64(div), "KMGTPE"[exp]) +} + +// summarizePeriods lists the first few periods and counts the rest. +func summarizePeriods(periods []string) string { + const shown = 5 + if len(periods) <= shown { + return strings.Join(periods, ", ") + } + return fmt.Sprintf("%s and %d more", strings.Join(periods[:shown], ", "), len(periods)-shown) +} + +func (c *CLI) printRawUsage() { + out := c.Stdout + fmt.Fprintln(out, "Aperiodic CLI Client: raw data (Prime + Raw plan)") + fmt.Fprintln(out) + fmt.Fprintln(out, "Usage:") + fmt.Fprintln(out, " aperiodic raw [flags] Download raw Parquet files") + fmt.Fprintln(out, " aperiodic raw coverage [flags] Show which symbols and days are available (no API key needed)") + fmt.Fprintln(out) + fmt.Fprintln(out, "Datasets:") + dw := tabwriter.NewWriter(out, 0, 0, 2, ' ', 0) + for _, dataset := range RawDatasets { + fmt.Fprintf(dw, " %s\t%s\t(%s)\n", dataset, rawDatasetDescriptions[dataset], joinNames(RawExchanges(dataset))) + } + dw.Flush() + fmt.Fprintln(out) + fmt.Fprintln(out, "Files land in the bucket's own Hive layout, one per month before 2026-08-01") + fmt.Fprintln(out, "and one per day from then on:") + fmt.Fprintln(out, " //exchange=/symbol=/year=YYYY/month=MM/data.parquet") + fmt.Fprintln(out, " //exchange=/symbol=/year=YYYY/month=MM/day=DD/data.parquet") + fmt.Fprintln(out, "Monthly files are written whole, so filter on exchange_timestamp when reading.") + fmt.Fprintln(out, "Files already present with the expected size are skipped, so re-running resumes.") + fmt.Fprintln(out) + fmt.Fprintln(out, "Environment:") + fmt.Fprintln(out, " APERIODIC_API_KEY API key on the Prime + Raw plan (not needed with --preview or for coverage)") + fmt.Fprintln(out) + fmt.Fprintln(out, "Download flags:") + downloadFlags, _ := newRawDownloadFlagSet() + downloadFlags.SetOutput(out) + downloadFlags.PrintDefaults() + fmt.Fprintln(out) + fmt.Fprintln(out, "Coverage flags:") + coverageFlags, _ := newRawCoverageFlagSet() + coverageFlags.SetOutput(out) + coverageFlags.PrintDefaults() + fmt.Fprintln(out) + fmt.Fprintln(out, "Example:") + fmt.Fprintln(out, " aperiodic raw trades --exchange binance-futures --symbol perpetual-BTC-USDT:USDT \\") + fmt.Fprintln(out, " --start-date 2025-01-01 --end-date 2025-12-31 --output-dir raw") +} diff --git a/client.go b/client.go index d2d64aa..9783f3c 100644 --- a/client.go +++ b/client.go @@ -23,6 +23,12 @@ type APIError struct { StatusCode int Message string Details []string + + // Code is the machine-readable reason some responses carry, e.g. + // "raw_not_in_plan" or "range_too_long" on raw data requests. + Code string + // UpgradeURL accompanies a "raw_not_in_plan" refusal. + UpgradeURL string } func (e *APIError) Error() string { @@ -77,7 +83,17 @@ func (c *AperiodicClient) handleAPIError(resp *http.Response) error { case http.StatusUnauthorized: apiErr.Message = "Unauthorized" case http.StatusForbidden: + // A 403 can carry a reason (e.g. raw_not_in_plan); keep it. apiErr.Message = "Forbidden" + var errResp APIErrorResponse + if err := json.NewDecoder(resp.Body).Decode(&errResp); err == nil { + if errResp.Error != "" { + apiErr.Message = errResp.Error + } + apiErr.Details = errResp.Details + apiErr.Code = errResp.Code + apiErr.UpgradeURL = errResp.UpgradeURL + } case http.StatusNotFound: apiErr.Message = "Not Found" case http.StatusTooManyRequests: @@ -87,6 +103,8 @@ func (c *AperiodicClient) handleAPIError(resp *http.Response) error { if err := json.NewDecoder(resp.Body).Decode(&errResp); err == nil { apiErr.Message = errResp.Error apiErr.Details = errResp.Details + apiErr.Code = errResp.Code + apiErr.UpgradeURL = errResp.UpgradeURL } else { body, _ := io.ReadAll(resp.Body) apiErr.Message = string(body) diff --git a/processor.go b/processor.go index 773c4fa..7fe7fc4 100644 --- a/processor.go +++ b/processor.go @@ -70,42 +70,62 @@ func (c *AperiodicClient) DownloadFilesConcurrently(files []FileInfo, maxConcurr } results := make([]DownloadedFile, len(files)) - errs := make([]error, len(files)) + err = runConcurrently(len(files), maxConcurrent, func(i int) error { + f := files[i] + filename := filenames[i] + destPath := filepath.Join(outputDir, filename) + + if err := c.downloadToFile(f.URL, destPath, 3); err != nil { + return fmt.Errorf("failed to download %s: %w", filename, err) + } + results[i] = DownloadedFile{ + Year: f.Year, + Month: f.Month, + Day: f.Day, + Filename: filename, + } + return nil + }) + if err != nil { + return nil, err + } + + return results, nil +} + +// runConcurrently calls task for every index in [0, n), at most maxConcurrent +// at a time, and once all have finished returns the error of the lowest index +// that failed. A maxConcurrent below 1 runs one at a time: an unbuffered +// semaphore would block every task forever. +func runConcurrently(n, maxConcurrent int, task func(i int) error) error { + if maxConcurrent < 1 { + maxConcurrent = 1 + } + + errs := make([]error, n) var wg sync.WaitGroup semaphore := make(chan struct{}, maxConcurrent) - for i, file := range files { + for i := 0; i < n; i++ { wg.Add(1) - go func(i int, f FileInfo) { + go func(i int) { defer wg.Done() semaphore <- struct{}{} defer func() { <-semaphore }() - filename := filenames[i] - destPath := filepath.Join(outputDir, filename) - - if err := c.downloadToFile(f.URL, destPath, 3); err != nil { - errs[i] = fmt.Errorf("failed to download %s: %w", filename, err) - return - } - results[i] = DownloadedFile{ - Year: f.Year, - Month: f.Month, - Day: f.Day, - Filename: filename, - } - }(i, file) + errs[i] = task(i) + }(i) } wg.Wait() for _, err := range errs { if err != nil { - return nil, err + return err } } - return results, nil + return nil } func (c *AperiodicClient) downloadToFile(url, destPath string, maxRetries int) error { diff --git a/raw.go b/raw.go new file mode 100644 index 0000000..665fb96 --- /dev/null +++ b/raw.go @@ -0,0 +1,506 @@ +package aperiodic + +// Raw per-tick data (Prime + Raw plan): trades, quotes and derivative ticks. +// +// History before 2026-08-01 is one Parquet file per calendar month, one per +// day from then on. Files are written in the bucket's own Hive layout, the same +// one the Python client's download_raw writes, so a folder filled by either can +// be topped up by the other. + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/url" + "os" + "path/filepath" + "runtime" + "slices" + "strings" + "time" +) + +const ( + // RawMaxRangeDays is the most days the API lists per request; longer + // ranges are split into several requests. + RawMaxRangeDays = 366 + + // DefaultRawMaxConcurrent is lower than the metrics default: raw files run + // to gigabytes, so a few streams already saturate most links. + DefaultRawMaxConcurrent = 4 + + // RawUpgradeURL is where a plan without raw data is pointed when the API + // refusal carries no upgrade_url of its own. + RawUpgradeURL = "https://aperiodic.io/pricing" + + rawDateLayout = "2006-01-02" + rawMonthLayout = "2006-01" + rawMaxRetries = 3 +) + +var ( + // RawDatasets lists every raw dataset in catalog order. + RawDatasets = []RawDataset{RawTrades, RawQuotes, RawMarkPrice, RawIndexPrice, RawFundingRate, RawOpenInterest} + + // Exchanges lists every supported exchange. + Exchanges = []Exchange{ExchangeBinanceFutures, ExchangeOkxPerps, ExchangeHyperliquidPerps} + + // Hyperliquid's derivative feed carries no exchange time, so the four + // derivative datasets are not served for it. + rawDerivativeExchanges = []Exchange{ExchangeBinanceFutures, ExchangeOkxPerps} + + // rawStallTimeout aborts a transfer that receives nothing for this long. + // Raw downloads cannot use the client's overall timeout: a multi-gigabyte + // file legitimately takes far longer than DefaultTimeout to stream. + rawStallTimeout = DefaultTimeout + + // rawRetryBackoff is the wait before the first retry of a failed + // transfer; it doubles on every further attempt. A var so tests can + // shrink it. + rawRetryBackoff = time.Second + + // ":" (in every symbol) is not a legal file-name character on Windows; + // there it is written as "%3A", which Hive-partition readers decode back. + // A var so tests can exercise both spellings on any platform. + rawEscapeColon = runtime.GOOS == "windows" + + // errURLRefused marks a 403 from storage: the presigned URL expired or was + // refused, so re-sending it cannot succeed and a fresh one is requested. + errURLRefused = errors.New("403 Forbidden (URL expired or refused)") +) + +// RawExchanges returns the exchanges a raw dataset is served for. +func RawExchanges(dataset RawDataset) []Exchange { + switch dataset { + case RawTrades, RawQuotes: + return Exchanges + default: + return rawDerivativeExchanges + } +} + +// ValidateRawQuery rejects an unknown dataset or exchange, or a combination the +// API does not serve, naming the valid choices. +func ValidateRawQuery(dataset RawDataset, exchange Exchange) error { + if err := validateRawDataset(dataset); err != nil { + return err + } + if err := validateExchange(exchange); err != nil { + return err + } + available := RawExchanges(dataset) + if !slices.Contains(available, exchange) { + return fmt.Errorf( + "raw dataset %s is not available for %s (available: %s); %s serves trades and quotes only", + dataset, exchange, joinNames(available), ExchangeHyperliquidPerps, + ) + } + return nil +} + +func validateRawDataset(dataset RawDataset) error { + if !slices.Contains(RawDatasets, dataset) { + return fmt.Errorf("unknown raw dataset %q (valid: %s)", dataset, joinNames(RawDatasets)) + } + return nil +} + +func validateExchange(exchange Exchange) error { + if !slices.Contains(Exchanges, exchange) { + return fmt.Errorf("unknown exchange %q (valid: %s)", exchange, joinNames(Exchanges)) + } + return nil +} + +func joinNames[T ~string](names []T) string { + return strings.Join(toStrings(names), ", ") +} + +func toStrings[T ~string](names []T) []string { + out := make([]string, len(names)) + for i, n := range names { + out[i] = string(n) + } + return out +} + +// RawQuery names one raw series. With Preview set, files come from the free +// preview endpoint, which serves one fixed period and ignores any date range. +type RawQuery struct { + Dataset RawDataset + Exchange Exchange + Symbol string + Preview bool +} + +// RawListing is every file overlapping a range, one per period, in period +// order, plus the periods in the range the API holds no file for. +type RawListing struct { + Files []RawFileInfo + MissingPeriods []string +} + +// RawDownload is one listed file and where it lands on disk. +type RawDownload struct { + File RawFileInfo + Path string + // Present is set when the file is already on disk with the listed size. + Present bool +} + +type rawWindow struct { + start, end time.Time +} + +// rawWindows splits an inclusive date range into windows the API accepts. +func rawWindows(start, end time.Time) ([]rawWindow, error) { + if end.Before(start) { + return nil, errors.New("end date must be on or after start date") + } + + var windows []rawWindow + for cursor := start; !cursor.After(end); { + last := cursor.AddDate(0, 0, RawMaxRangeDays-1) + if last.After(end) { + last = end + } + windows = append(windows, rawWindow{start: cursor, end: last}) + cursor = last.AddDate(0, 0, 1) + } + return windows, nil +} + +type rawPeriod struct { + first time.Time + daily bool +} + +// parseRawPeriod reads a "YYYY-MM" (monthly) or "YYYY-MM-DD" (daily) period. +func parseRawPeriod(period string) (rawPeriod, error) { + if day, err := time.Parse(rawDateLayout, period); err == nil { + return rawPeriod{first: day, daily: true}, nil + } + if month, err := time.Parse(rawMonthLayout, period); err == nil { + return rawPeriod{first: month}, nil + } + return rawPeriod{}, fmt.Errorf("unrecognised raw period %q (want YYYY-MM or YYYY-MM-DD)", period) +} + +// last is the final day the period covers. +func (p rawPeriod) last() time.Time { + if p.daily { + return p.first + } + return p.first.AddDate(0, 1, -1) +} + +// RawFilePath is where one raw file is written: the bucket's Hive layout, +// {outputDir}/{dataset}/exchange={exchange}/symbol={symbol}/year=YYYY/month=MM +// [/day=DD]/data.parquet. +func RawFilePath(outputDir string, dataset RawDataset, exchange Exchange, symbol, period string) (string, error) { + p, err := parseRawPeriod(period) + if err != nil { + return "", err + } + if rawEscapeColon { + symbol = strings.ReplaceAll(symbol, ":", "%3A") + } + + parts := []string{ + outputDir, + string(dataset), + "exchange=" + string(exchange), + "symbol=" + symbol, + fmt.Sprintf("year=%d", p.first.Year()), + fmt.Sprintf("month=%02d", int(p.first.Month())), + } + if p.daily { + parts = append(parts, fmt.Sprintf("day=%02d", p.first.Day())) + } + return filepath.Join(append(parts, "data.parquet")...), nil +} + +// FetchRawFiles lists every file overlapping [start, end] (inclusive days, +// UTC). Ranges over RawMaxRangeDays are split into several requests, and a +// monthly file listed by two of them is kept once. +func (c *AperiodicClient) FetchRawFiles(q RawQuery, start, end time.Time) (*RawListing, error) { + if q.Preview { + resp, err := c.fetchRawPage(q, nil) + if err != nil { + return nil, err + } + return newRawListing(resp.Files, resp.MissingPeriods), nil + } + + windows, err := rawWindows(start, end) + if err != nil { + return nil, err + } + + var files []RawFileInfo + var missing []string + for _, w := range windows { + resp, err := c.fetchRawPage(q, &w) + if err != nil { + return nil, err + } + files = append(files, resp.Files...) + missing = append(missing, resp.MissingPeriods...) + } + return newRawListing(files, missing), nil +} + +// newRawListing keeps one file per period (the last listed, whose URL is the +// freshest) and sorts both lists; the two period formats sort chronologically +// as plain strings. +func newRawListing(files []RawFileInfo, missing []string) *RawListing { + byPeriod := make(map[string]RawFileInfo, len(files)) + for _, f := range files { + byPeriod[f.Period] = f + } + + listing := &RawListing{Files: make([]RawFileInfo, 0, len(byPeriod))} + for _, f := range byPeriod { + listing.Files = append(listing.Files, f) + } + slices.SortFunc(listing.Files, func(a, b RawFileInfo) int { + return strings.Compare(a.Period, b.Period) + }) + + for _, p := range slices.Compact(slices.Sorted(slices.Values(missing))) { + if _, found := byPeriod[p]; !found { + listing.MissingPeriods = append(listing.MissingPeriods, p) + } + } + return listing +} + +func (c *AperiodicClient) fetchRawPage(q RawQuery, window *rawWindow) (*RawFilesResponse, error) { + dataPath := fmt.Sprintf("%s/data/raw/%s", c.BaseURL, q.Dataset) + if q.Preview { + dataPath = fmt.Sprintf("%s/data/raw/preview/%s", c.BaseURL, q.Dataset) + } + + u, err := url.Parse(dataPath) + if err != nil { + return nil, err + } + + query := u.Query() + query.Set("exchange", string(q.Exchange)) + query.Set("symbol", q.Symbol) + if window != nil { + query.Set("start_date", window.start.Format(rawDateLayout)) + query.Set("end_date", window.end.Format(rawDateLayout)) + } + u.RawQuery = query.Encode() + + var resp RawFilesResponse + if err := c.getJSON(u, c.getHeaders(), &resp); err != nil { + return nil, err + } + return &resp, nil +} + +func (c *AperiodicClient) getJSON(u *url.URL, header http.Header, out any) error { + req := &http.Request{ + Method: http.MethodGet, + URL: u, + Header: header, + } + + resp, err := c.HTTPClient.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + + if err := c.handleAPIError(resp); err != nil { + return err + } + return json.NewDecoder(resp.Body).Decode(out) +} + +// refreshRawFile re-lists a single period to get a fresh presigned URL. +func (c *AperiodicClient) refreshRawFile(q RawQuery, period string) (RawFileInfo, error) { + p, err := parseRawPeriod(period) + if err != nil { + return RawFileInfo{}, err + } + + listing, err := c.FetchRawFiles(q, p.first, p.last()) + if err != nil { + return RawFileInfo{}, err + } + for _, f := range listing.Files { + if f.Period == period { + return f, nil + } + } + return RawFileInfo{}, fmt.Errorf("%s is no longer listed by the API", period) +} + +// PlanRawDownloads resolves every file's path under outputDir and marks the +// ones already on disk with the listed size, unless overwrite is set. +func PlanRawDownloads(outputDir string, q RawQuery, files []RawFileInfo, overwrite bool) ([]RawDownload, error) { + plan := make([]RawDownload, len(files)) + for i, f := range files { + path, err := RawFilePath(outputDir, q.Dataset, q.Exchange, q.Symbol, f.Period) + if err != nil { + return nil, err + } + plan[i] = RawDownload{File: f, Path: path} + if overwrite { + continue + } + if info, err := os.Stat(path); err == nil && info.Mode().IsRegular() && info.Size() == f.Size { + plan[i].Present = true + } + } + return plan, nil +} + +// DownloadRawFiles streams each file to its path, at most maxConcurrent at a +// time. A URL refused with a 403 (typically expired) is re-requested once. +// onDone, if set, is called from the downloading goroutine as each file lands, +// so it must be safe for concurrent use. Files that finished before an error +// stay on disk, so running the same download again resumes it. +func (c *AperiodicClient) DownloadRawFiles(q RawQuery, downloads []RawDownload, maxConcurrent int, onDone func(RawDownload)) error { + return runConcurrently(len(downloads), maxConcurrent, func(i int) error { + d := downloads[i] + if err := c.downloadRawFile(q, d); err != nil { + return fmt.Errorf("failed to download %s: %w", d.File.Period, err) + } + if onDone != nil { + onDone(d) + } + return nil + }) +} + +func (c *AperiodicClient) downloadRawFile(q RawQuery, d RawDownload) error { + err := c.streamRawFile(d.File.URL, d.Path) + if !errors.Is(err, errURLRefused) { + return err + } + + fresh, err := c.refreshRawFile(q, d.File.Period) + if err != nil { + return fmt.Errorf("refreshing an expired URL: %w", err) + } + return c.streamRawFile(fresh.URL, d.Path) +} + +// streamRawFile downloads fileURL to path via a ".part" file renamed into place +// once complete, so a glob for data.parquet never matches a half-written file. +// Transport errors and bad statuses are retried with exponential backoff; a +// 403 is returned at once as errURLRefused. +func (c *AperiodicClient) streamRawFile(fileURL, path string) error { + if err := os.MkdirAll(filepath.Dir(path), 0755); err != nil { + return fmt.Errorf("failed to create directory: %w", err) + } + partial := path + ".part" + + var lastErr error + for attempt := 0; attempt <= rawMaxRetries; attempt++ { + if attempt > 0 { + time.Sleep(rawRetryBackoff << (attempt - 1)) + } + + retryable, err := c.streamRawOnce(fileURL, partial) + if err == nil { + return os.Rename(partial, path) + } + _ = os.Remove(partial) + if !retryable { + return err + } + lastErr = err + } + return lastErr +} + +// streamRawOnce makes a single attempt, like downloadOnce, but without the +// client's overall timeout: the transfer is cancelled only once it has +// received nothing for rawStallTimeout. +func (c *AperiodicClient) streamRawOnce(fileURL, destPath string) (retryable bool, err error) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + stall := time.AfterFunc(rawStallTimeout, cancel) + defer stall.Stop() + + req, err := http.NewRequestWithContext(ctx, http.MethodGet, fileURL, nil) + if err != nil { + return false, err + } + + client := *c.HTTPClient + client.Timeout = 0 + + resp, err := client.Do(req) + if err != nil { + return true, stalledOr(ctx, err) + } + defer resp.Body.Close() + + if resp.StatusCode == http.StatusForbidden { + return false, errURLRefused + } + if resp.StatusCode != http.StatusOK { + return true, fmt.Errorf("bad status: %s", resp.Status) + } + + out, err := os.Create(destPath) + if err != nil { + return false, fmt.Errorf("failed to create file: %w", err) + } + + _, copyErr := io.Copy(out, &stallReader{r: resp.Body, timer: stall}) + closeErr := out.Close() + if copyErr != nil { + return true, stalledOr(ctx, copyErr) + } + return true, closeErr +} + +func stalledOr(ctx context.Context, err error) error { + if ctx.Err() != nil { + return fmt.Errorf("no data received for %s", rawStallTimeout) + } + return err +} + +// stallReader pushes the stall deadline back every time data arrives. +type stallReader struct { + r io.Reader + timer *time.Timer +} + +func (s *stallReader) Read(p []byte) (int, error) { + n, err := s.r.Read(p) + if n > 0 { + s.timer.Reset(rawStallTimeout) + } + return n, err +} + +// GetRawCoverage returns the raw datasets and every symbol's coverage. The +// endpoint is public, so no API key is sent. +func (c *AperiodicClient) GetRawCoverage() (*RawCoverageResponse, error) { + u, err := url.Parse(fmt.Sprintf("%s/metadata/raw", c.BaseURL)) + if err != nil { + return nil, err + } + + header := c.getHeaders() + header.Del("X-API-KEY") + + var resp RawCoverageResponse + if err := c.getJSON(u, header, &resp); err != nil { + return nil, err + } + return &resp, nil +} diff --git a/raw_test.go b/raw_test.go new file mode 100644 index 0000000..ab528ad --- /dev/null +++ b/raw_test.go @@ -0,0 +1,972 @@ +package aperiodic + +// Offline tests for raw data: the API and the presigned file URLs are served +// by an httptest stub, so the real code runs end to end: chunking long ranges, +// deduplicating monthly files, re-requesting a URL after a 403, streaming to +// the Hive layout and skipping files that are already there. + +import ( + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "net/url" + "os" + "path/filepath" + "slices" + "strings" + "sync" + "testing" + "time" +) + +const rawTestSymbol = "perpetual-BTC-USDT:USDT" + +func rawBody(period string) []byte { + return []byte("parquet bytes for " + period) +} + +type rawAPICall struct { + path string + query url.Values + key string + // hasKey distinguishes an absent X-API-KEY from an empty one. + hasKey bool +} + +// rawStub serves the raw listing, the file blobs and the coverage endpoint. +// listing is called once per listing request, numbered from 0. +type rawStub struct { + *httptest.Server + + listing func(call int, baseURL string) (status int, body any) + coverage any + forbidden map[string]bool // ?v= versions answered with 403, like an expired URL + failFirst map[string]int // period → number of 500s to answer before serving it + + mu sync.Mutex + apiCalls []rawAPICall + blobCalls map[string]int +} + +func newRawStub(t *testing.T, listing func(call int, baseURL string) (int, any)) *rawStub { + t.Helper() + + stub := &rawStub{ + listing: listing, + forbidden: map[string]bool{}, + failFirst: map[string]int{}, + blobCalls: map[string]int{}, + } + stub.Server = httptest.NewServer(http.HandlerFunc(stub.serve)) + t.Cleanup(stub.Close) + + t.Setenv("APERIODIC_API_URL", stub.URL) + t.Setenv("APERIODIC_API_KEY", "key-1") + + previous := rawRetryBackoff + rawRetryBackoff = time.Millisecond + t.Cleanup(func() { rawRetryBackoff = previous }) + + return stub +} + +func (s *rawStub) serve(w http.ResponseWriter, r *http.Request) { + if period, isBlob := strings.CutPrefix(r.URL.Path, "/blob/"); isBlob { + s.mu.Lock() + s.blobCalls[period]++ + fail := s.failFirst[period] > 0 + if fail { + s.failFirst[period]-- + } + forbidden := s.forbidden[r.URL.Query().Get("v")] + s.mu.Unlock() + + switch { + case forbidden: + http.Error(w, "AccessDenied", http.StatusForbidden) + case fail: + http.Error(w, "upstream hiccup", http.StatusInternalServerError) + default: + _, _ = w.Write(rawBody(period)) + } + return + } + + key, hasKey := r.Header["X-Api-Key"] + s.mu.Lock() + call := len(s.apiCalls) + s.apiCalls = append(s.apiCalls, rawAPICall{ + path: r.URL.Path, + query: r.URL.Query(), + key: strings.Join(key, ","), + hasKey: hasKey, + }) + s.mu.Unlock() + + status, body := http.StatusOK, s.coverage + if r.URL.Path != "/metadata/raw" { + status, body = s.listing(call, s.URL) + } + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(status) + _ = json.NewEncoder(w).Encode(body) +} + +func (s *rawStub) calls() []rawAPICall { + s.mu.Lock() + defer s.mu.Unlock() + return slices.Clone(s.apiCalls) +} + +func (s *rawStub) totalBlobCalls() int { + s.mu.Lock() + defer s.mu.Unlock() + total := 0 + for _, n := range s.blobCalls { + total += n + } + return total +} + +func rawListing(baseURL string, version int, periods ...string) RawFilesResponse { + files := make([]RawFileInfo, len(periods)) + for i, p := range periods { + files[i] = RawFileInfo{ + Period: p, + URL: fmt.Sprintf("%s/blob/%s?v=%d", baseURL, p, version), + Size: int64(len(rawBody(p))), + } + } + return RawFilesResponse{ + Dataset: "trades", + Exchange: "binance-futures", + Symbol: rawTestSymbol, + SchemaVersion: 1, + ExpiresIn: 3600, + Files: files, + MissingPeriods: []string{}, + } +} + +// staticListing answers every listing request with the same files. +func staticListing(periods ...string) func(int, string) (int, any) { + return func(_ int, baseURL string) (int, any) { + return http.StatusOK, rawListing(baseURL, 1, periods...) + } +} + +func rawPath(t *testing.T, dir string, dataset RawDataset, exchange Exchange, period string) string { + t.Helper() + path, err := RawFilePath(dir, dataset, exchange, rawTestSymbol, period) + if err != nil { + t.Fatalf("RawFilePath(%s): %v", period, err) + } + return path +} + +// writtenFiles lists every file under dir, relative and slash-separated. +func writtenFiles(t *testing.T, dir string) []string { + t.Helper() + var files []string + err := filepath.WalkDir(dir, func(path string, d os.DirEntry, err error) error { + if err != nil || d.IsDir() { + return err + } + rel, err := filepath.Rel(dir, path) + files = append(files, filepath.ToSlash(rel)) + return err + }) + if err != nil { + t.Fatalf("walking %s: %v", dir, err) + } + slices.Sort(files) + return files +} + +func rawArgs(outputDir string, extra ...string) []string { + return append([]string{ + "raw", "trades", + "--exchange", "binance-futures", + "--symbol", rawTestSymbol, + "--output-dir", outputDir, + }, extra...) +} + +// --------------------------------------------------------------------------- +// Pure helpers +// --------------------------------------------------------------------------- + +func mustDate(t *testing.T, s string) time.Time { + t.Helper() + d, err := time.Parse(rawDateLayout, s) + if err != nil { + t.Fatal(err) + } + return d +} + +func TestRawWindows(t *testing.T) { + tests := []struct { + name string + start, end string + expected [][2]string + }{ + {"single day", "2025-06-01", "2025-06-01", [][2]string{{"2025-06-01", "2025-06-01"}}}, + {"exactly the limit", "2025-01-01", "2026-01-01", [][2]string{{"2025-01-01", "2026-01-01"}}}, + { + "one day over the limit splits", + "2025-01-01", "2026-01-02", + [][2]string{{"2025-01-01", "2026-01-01"}, {"2026-01-02", "2026-01-02"}}, + }, + { + // The same split the Python client's tests pin. + "across a leap year", + "2024-07-01", "2025-07-31", + [][2]string{{"2024-07-01", "2025-07-01"}, {"2025-07-02", "2025-07-31"}}, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + windows, err := rawWindows(mustDate(t, tt.start), mustDate(t, tt.end)) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + got := make([][2]string, len(windows)) + for i, w := range windows { + got[i] = [2]string{w.start.Format(rawDateLayout), w.end.Format(rawDateLayout)} + if days := int(w.end.Sub(w.start).Hours()/24) + 1; days > RawMaxRangeDays { + t.Errorf("window %d covers %d days, over the %d-day limit", i, days, RawMaxRangeDays) + } + } + if !slices.Equal(got, tt.expected) { + t.Errorf("expected %v, got %v", tt.expected, got) + } + }) + } + + if _, err := rawWindows(mustDate(t, "2025-06-02"), mustDate(t, "2025-06-01")); err == nil { + t.Error("expected an error when the end is before the start") + } +} + +func TestParseRawPeriod(t *testing.T) { + tests := []struct { + period string + first, last string + }{ + {"2025-06", "2025-06-01", "2025-06-30"}, + {"2025-02", "2025-02-01", "2025-02-28"}, + {"2024-02", "2024-02-01", "2024-02-29"}, + {"2026-08-10", "2026-08-10", "2026-08-10"}, + } + for _, tt := range tests { + p, err := parseRawPeriod(tt.period) + if err != nil { + t.Fatalf("%s: unexpected error: %v", tt.period, err) + } + if got := p.first.Format(rawDateLayout); got != tt.first { + t.Errorf("%s: expected first day %s, got %s", tt.period, tt.first, got) + } + if got := p.last().Format(rawDateLayout); got != tt.last { + t.Errorf("%s: expected last day %s, got %s", tt.period, tt.last, got) + } + } + + for _, bad := range []string{"", "2025", "2025-13", "2025-6", "2025/06", "2025-06-31", "../../etc"} { + if _, err := parseRawPeriod(bad); err == nil { + t.Errorf("expected %q to be rejected", bad) + } + } +} + +func TestRawFilePath(t *testing.T) { + previous := rawEscapeColon + t.Cleanup(func() { rawEscapeColon = previous }) + + rel := func(t *testing.T, period string) string { + t.Helper() + path, err := RawFilePath("raw", RawQuotes, ExchangeOkxPerps, rawTestSymbol, period) + if err != nil { + t.Fatal(err) + } + return filepath.ToSlash(path) + } + + rawEscapeColon = false + if got, want := rel(t, "2025-06"), "raw/quotes/exchange=okx-perps/symbol=perpetual-BTC-USDT:USDT/year=2025/month=06/data.parquet"; got != want { + t.Errorf("monthly: expected %s, got %s", want, got) + } + if got, want := rel(t, "2026-08-01"), "raw/quotes/exchange=okx-perps/symbol=perpetual-BTC-USDT:USDT/year=2026/month=08/day=01/data.parquet"; got != want { + t.Errorf("daily: expected %s, got %s", want, got) + } + + rawEscapeColon = true + if got := rel(t, "2025-06"); !strings.Contains(got, "symbol=perpetual-BTC-USDT%3AUSDT/") { + t.Errorf("expected the colon escaped as %%3A on Windows, got %s", got) + } + + if _, err := RawFilePath("raw", RawQuotes, ExchangeOkxPerps, rawTestSymbol, "junk"); err == nil { + t.Error("expected an unparseable period to be rejected") + } +} + +func TestValidateRawQuery(t *testing.T) { + for _, dataset := range RawDatasets { + for _, exchange := range []Exchange{ExchangeBinanceFutures, ExchangeOkxPerps} { + if err := ValidateRawQuery(dataset, exchange); err != nil { + t.Errorf("%s on %s: unexpected error: %v", dataset, exchange, err) + } + } + } + + for _, dataset := range []RawDataset{RawTrades, RawQuotes} { + if err := ValidateRawQuery(dataset, ExchangeHyperliquidPerps); err != nil { + t.Errorf("%s on hyperliquid-perps: unexpected error: %v", dataset, err) + } + } + for _, dataset := range []RawDataset{RawMarkPrice, RawIndexPrice, RawFundingRate, RawOpenInterest} { + err := ValidateRawQuery(dataset, ExchangeHyperliquidPerps) + if err == nil || !strings.Contains(err.Error(), "binance-futures, okx-perps") { + t.Errorf("%s on hyperliquid-perps: expected an error naming the available exchanges, got %v", dataset, err) + } + } + + if err := ValidateRawQuery("l2_book", ExchangeBinanceFutures); err == nil || !strings.Contains(err.Error(), "open_interest") { + t.Errorf("expected an unknown dataset to list the valid ones, got %v", err) + } + if err := ValidateRawQuery(RawTrades, "bybit"); err == nil || !strings.Contains(err.Error(), "okx-perps") { + t.Errorf("expected an unknown exchange to list the valid ones, got %v", err) + } +} + +func TestHandleAPIError_ForbiddenKeepsTheReason(t *testing.T) { + stub := newRawStub(t, func(int, string) (int, any) { + return http.StatusForbidden, map[string]string{ + "error": "Raw data is not included in the prime plan. It is part of Prime + Raw.", + "code": "raw_not_in_plan", + "upgrade_url": "https://aperiodic.io/pricing", + } + }) + + client := NewAperiodicClient("key-1") + _, err := client.FetchRawFiles(RawQuery{Dataset: RawTrades, Exchange: ExchangeBinanceFutures, Symbol: rawTestSymbol}, mustDate(t, "2025-06-01"), mustDate(t, "2025-06-30")) + + apiErr, ok := err.(*APIError) + if !ok { + t.Fatalf("expected *APIError, got %T (%v)", err, err) + } + if apiErr.StatusCode != http.StatusForbidden || apiErr.Code != "raw_not_in_plan" || apiErr.UpgradeURL != "https://aperiodic.io/pricing" { + t.Errorf("expected the 403's code and upgrade URL, got %+v", apiErr) + } + if !strings.Contains(apiErr.Message, "not included") { + t.Errorf("expected the server's message, got %q", apiErr.Message) + } + if len(stub.calls()) != 1 { + t.Errorf("expected one listing request, got %d", len(stub.calls())) + } +} + +// --------------------------------------------------------------------------- +// aperiodic raw +// --------------------------------------------------------------------------- + +func TestCLI_Raw_WritesTheHiveLayout(t *testing.T) { + stub := newRawStub(t, staticListing("2025-06", "2025-07")) + outputDir := t.TempDir() + + stdout, stderr, code := runCLI(rawArgs(outputDir, "--start-date", "2025-06-01", "--end-date", "2025-07-31")...) + if code != 0 { + t.Fatalf("expected exit code 0, got %d; stderr: %s", code, stderr) + } + if !strings.Contains(stdout, "Successfully downloaded 2 files") { + t.Errorf("expected two successes, got: %s", stdout) + } + + calls := stub.calls() + if len(calls) != 1 { + t.Fatalf("expected one listing request, got %d", len(calls)) + } + call := calls[0] + if call.path != "/data/raw/trades" { + t.Errorf("expected /data/raw/trades, got %s", call.path) + } + if call.key != "key-1" { + t.Errorf("expected X-API-KEY key-1, got %q", call.key) + } + for param, want := range map[string]string{ + "exchange": "binance-futures", + "symbol": rawTestSymbol, + "start_date": "2025-06-01", + "end_date": "2025-07-31", + } { + if got := call.query.Get(param); got != want { + t.Errorf("expected %s=%s, got %q", param, want, got) + } + } + + for _, period := range []string{"2025-06", "2025-07"} { + path := rawPath(t, outputDir, RawTrades, ExchangeBinanceFutures, period) + body, err := os.ReadFile(path) + if err != nil { + t.Fatalf("expected %s to be written: %v", path, err) + } + if string(body) != string(rawBody(period)) { + t.Errorf("expected %s to hold its own period's bytes, got %q", path, body) + } + } + + // Nothing but the two data files: no leftover .part files. + if got := writtenFiles(t, outputDir); len(got) != 2 { + t.Errorf("expected exactly two files, got %v", got) + } +} + +func TestCLI_Raw_LayoutIsTheBucketLayout(t *testing.T) { + // The on-disk layout must match the Python client's download_raw exactly, + // so pin the literal paths (where ":" is a legal file-name character). + if rawEscapeColon { + t.Skip("symbols are escaped on this platform; TestRawFilePath covers it") + } + newRawStub(t, staticListing("2026-07", "2026-08-01", "2026-08-02")) + outputDir := t.TempDir() + + _, stderr, code := runCLI(rawArgs(outputDir, "--start-date", "2026-07-15", "--end-date", "2026-08-02")...) + if code != 0 { + t.Fatalf("expected exit code 0, got %d; stderr: %s", code, stderr) + } + + prefix := "trades/exchange=binance-futures/symbol=perpetual-BTC-USDT:USDT/year=2026/" + expected := []string{ + prefix + "month=07/data.parquet", + prefix + "month=08/day=01/data.parquet", + prefix + "month=08/day=02/data.parquet", + } + if got := writtenFiles(t, outputDir); !slices.Equal(got, expected) { + t.Errorf("expected %v, got %v", expected, got) + } +} + +func TestCLI_Raw_SplitsLongRangesAndDedupesMonths(t *testing.T) { + stub := newRawStub(t, func(call int, baseURL string) (int, any) { + if call == 0 { + return http.StatusOK, rawListing(baseURL, 1, "2025-05", "2025-06") + } + return http.StatusOK, rawListing(baseURL, 1, "2025-06", "2025-07") + }) + outputDir := t.TempDir() + + stdout, stderr, code := runCLI(rawArgs(outputDir, "--start-date", "2024-07-01", "--end-date", "2025-07-31")...) + if code != 0 { + t.Fatalf("expected exit code 0, got %d; stderr: %s", code, stderr) + } + + var windows [][2]string + for _, c := range stub.calls() { + windows = append(windows, [2]string{c.query.Get("start_date"), c.query.Get("end_date")}) + } + expected := [][2]string{{"2024-07-01", "2025-07-01"}, {"2025-07-02", "2025-07-31"}} + if !slices.Equal(windows, expected) { + t.Errorf("expected windows %v, got %v", expected, windows) + } + + // 2025-06 was listed twice but downloaded once. + if got := stub.totalBlobCalls(); got != 3 { + t.Errorf("expected 3 downloads, got %d", got) + } + if !strings.Contains(stdout, "Found 3 trades files") { + t.Errorf("expected three files found, got: %s", stdout) + } +} + +func TestCLI_Raw_SkipsFilesAlreadyPresent(t *testing.T) { + stub := newRawStub(t, staticListing("2025-06", "2025-07")) + outputDir := t.TempDir() + + kept := rawPath(t, outputDir, RawTrades, ExchangeBinanceFutures, "2025-06") + stale := rawPath(t, outputDir, RawTrades, ExchangeBinanceFutures, "2025-07") + for path, body := range map[string][]byte{kept: rawBody("2025-06"), stale: []byte("truncated")} { + if err := os.MkdirAll(filepath.Dir(path), 0755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, body, 0644); err != nil { + t.Fatal(err) + } + } + + args := rawArgs(outputDir, "--start-date", "2025-06-01", "--end-date", "2025-07-31") + stdout, stderr, code := runCLI(args...) + if code != 0 { + t.Fatalf("expected exit code 0, got %d; stderr: %s", code, stderr) + } + if got := stub.totalBlobCalls(); got != 1 { + t.Errorf("expected only the wrong-sized file to be downloaded, got %d downloads", got) + } + if !strings.Contains(stdout, "Skipping 1 already present") { + t.Errorf("expected the skip to be reported, got: %s", stdout) + } + if body, _ := os.ReadFile(stale); string(body) != string(rawBody("2025-07")) { + t.Errorf("expected the wrong-sized file to be replaced, got %q", body) + } + + stdout, _, code = runCLI(args...) + if code != 0 || stub.totalBlobCalls() != 1 { + t.Errorf("expected a re-run to download nothing, got exit %d and %d downloads", code, stub.totalBlobCalls()) + } + if !strings.Contains(stdout, "Nothing to download") { + t.Errorf("expected a nothing-to-do message, got: %s", stdout) + } + + _, _, code = runCLI(append(args, "--overwrite")...) + if code != 0 || stub.totalBlobCalls() != 3 { + t.Errorf("expected --overwrite to download both again, got exit %d and %d downloads", code, stub.totalBlobCalls()) + } +} + +func TestCLI_Raw_RefreshesAnExpiredURL(t *testing.T) { + stub := newRawStub(t, func(call int, baseURL string) (int, any) { + return http.StatusOK, rawListing(baseURL, call+1, "2025-06") + }) + stub.forbidden["1"] = true + outputDir := t.TempDir() + + _, stderr, code := runCLI(rawArgs(outputDir, "--start-date", "2025-06-15", "--end-date", "2025-06-20")...) + if code != 0 { + t.Fatalf("expected exit code 0, got %d; stderr: %s", code, stderr) + } + + calls := stub.calls() + if len(calls) != 2 { + t.Fatalf("expected the listing to be re-requested once, got %d requests", len(calls)) + } + // The refresh asks for just that month. + refresh := calls[1].query + if refresh.Get("start_date") != "2025-06-01" || refresh.Get("end_date") != "2025-06-30" { + t.Errorf("expected the refresh to cover 2025-06-01..2025-06-30, got %s..%s", refresh.Get("start_date"), refresh.Get("end_date")) + } + // A 403 is not retried with backoff: one try per URL. + if got := stub.totalBlobCalls(); got != 2 { + t.Errorf("expected two download attempts, got %d", got) + } + + body, _ := os.ReadFile(rawPath(t, outputDir, RawTrades, ExchangeBinanceFutures, "2025-06")) + if string(body) != string(rawBody("2025-06")) { + t.Errorf("expected the file from the fresh URL, got %q", body) + } +} + +func TestCLI_Raw_GivesUpAfterOneRefresh(t *testing.T) { + stub := newRawStub(t, staticListing("2025-06")) + stub.forbidden["1"] = true + outputDir := t.TempDir() + + _, stderr, code := runCLI(rawArgs(outputDir, "--start-date", "2025-06-01", "--end-date", "2025-06-30")...) + if code != 1 { + t.Fatalf("expected exit code 1, got %d", code) + } + if !strings.Contains(stderr, "403") || !strings.Contains(stderr, "2025-06") { + t.Errorf("expected the 403 and the period in stderr, got: %s", stderr) + } + if got := stub.totalBlobCalls(); got != 2 { + t.Errorf("expected two download attempts, got %d", got) + } + if got := writtenFiles(t, outputDir); len(got) != 0 { + t.Errorf("expected nothing on disk, got %v", got) + } +} + +func TestCLI_Raw_RetriesTransientDownloadErrors(t *testing.T) { + stub := newRawStub(t, staticListing("2025-06")) + stub.failFirst["2025-06"] = 2 + outputDir := t.TempDir() + + _, stderr, code := runCLI(rawArgs(outputDir, "--start-date", "2025-06-01", "--end-date", "2025-06-30")...) + if code != 0 { + t.Fatalf("expected exit code 0, got %d; stderr: %s", code, stderr) + } + if got := stub.totalBlobCalls(); got != 3 { + t.Errorf("expected two failures then a success, got %d attempts", got) + } + if got := len(stub.calls()); got != 1 { + t.Errorf("expected a 500 to be retried without re-listing, got %d listing requests", got) + } +} + +func TestCLI_Raw_NotInPlanPointsToTheUpgrade(t *testing.T) { + newRawStub(t, func(int, string) (int, any) { + return http.StatusForbidden, map[string]string{ + "error": "Raw data is not included in the prime plan. It is part of Prime + Raw.", + "code": "raw_not_in_plan", + "upgrade_url": "https://aperiodic.io/pricing", + } + }) + + _, stderr, code := runCLI(rawArgs(t.TempDir(), "--start-date", "2025-06-01", "--end-date", "2025-06-30")...) + if code != 1 { + t.Fatalf("expected exit code 1, got %d", code) + } + for _, want := range []string{"not included in the prime plan", "Prime + Raw", "https://aperiodic.io/pricing", "--preview"} { + if !strings.Contains(stderr, want) { + t.Errorf("expected stderr to mention %q, got: %s", want, stderr) + } + } +} + +func TestCLI_Raw_ShowsValidationDetails(t *testing.T) { + newRawStub(t, func(int, string) (int, any) { + return http.StatusBadRequest, map[string]any{ + "error": "Invalid query parameters", + "details": []string{"symbol: Symbol must match format like perpetual-BTC-USDT:USDT"}, + } + }) + + _, stderr, code := runCLI(rawArgs(t.TempDir(), "--start-date", "2025-06-01", "--end-date", "2025-06-30")...) + if code != 1 { + t.Fatalf("expected exit code 1, got %d", code) + } + if !strings.Contains(stderr, "Invalid query parameters") || !strings.Contains(stderr, "Symbol must match") { + t.Errorf("expected the error and its details, got: %s", stderr) + } +} + +func TestCLI_Raw_PreviewUsesTheDemoKey(t *testing.T) { + stub := newRawStub(t, staticListing("2025-06")) + t.Setenv("APERIODIC_API_KEY", "") + outputDir := t.TempDir() + + _, stderr, code := runCLI(rawArgs(outputDir, "--preview")...) + if code != 0 { + t.Fatalf("expected exit code 0, got %d; stderr: %s", code, stderr) + } + + calls := stub.calls() + if len(calls) != 1 { + t.Fatalf("expected one listing request, got %d", len(calls)) + } + if calls[0].path != "/data/raw/preview/trades" { + t.Errorf("expected the preview path, got %s", calls[0].path) + } + if calls[0].key != DemoAPIKey { + t.Errorf("expected X-API-KEY %s, got %q", DemoAPIKey, calls[0].key) + } + if len(calls[0].query) != 2 || calls[0].query.Get("exchange") != "binance-futures" || calls[0].query.Get("symbol") != rawTestSymbol { + t.Errorf("expected only exchange and symbol, got %v", calls[0].query) + } + if _, err := os.Stat(rawPath(t, outputDir, RawTrades, ExchangeBinanceFutures, "2025-06")); err != nil { + t.Errorf("expected the preview file on disk: %v", err) + } +} + +func TestCLI_Raw_FlagsMayFollowOrPrecedeTheDataset(t *testing.T) { + stub := newRawStub(t, staticListing("2025-06")) + outputDir := t.TempDir() + + _, stderr, code := runCLI( + "raw", "--exchange", "okx-perps", "quotes", + "--symbol", rawTestSymbol, + "--start-date", "2025-06-01", "--end-date", "2025-06-30", + "--output-dir", outputDir, + ) + if code != 0 { + t.Fatalf("expected exit code 0, got %d; stderr: %s", code, stderr) + } + call := stub.calls()[0] + if call.path != "/data/raw/quotes" || call.query.Get("exchange") != "okx-perps" { + t.Errorf("expected quotes on okx-perps, got %s ?%s", call.path, call.query.Encode()) + } +} + +func TestCLI_Raw_ReportsMissingPeriods(t *testing.T) { + newRawStub(t, func(_ int, baseURL string) (int, any) { + listing := rawListing(baseURL, 1, "2025-06") + listing.MissingPeriods = []string{"2025-04", "2025-05"} + return http.StatusOK, listing + }) + + stdout, stderr, code := runCLI(rawArgs(t.TempDir(), "--start-date", "2025-04-01", "--end-date", "2025-06-30")...) + if code != 0 { + t.Fatalf("expected exit code 0, got %d; stderr: %s", code, stderr) + } + if !strings.Contains(stdout, "No file for 2 period(s) in the range: 2025-04, 2025-05") { + t.Errorf("expected the missing periods, got: %s", stdout) + } +} + +func TestCLI_Raw_NoFiles(t *testing.T) { + newRawStub(t, staticListing()) + + stdout, stderr, code := runCLI(rawArgs(t.TempDir(), "--start-date", "2025-06-01", "--end-date", "2025-06-30")...) + if code != 0 { + t.Fatalf("expected exit code 0, got %d; stderr: %s", code, stderr) + } + if !strings.Contains(stdout, "No data found") { + t.Errorf("expected a no-data message, got: %s", stdout) + } +} + +func TestCLI_Raw_ValidatesBeforeCallingTheAPI(t *testing.T) { + stub := newRawStub(t, staticListing("2025-06")) + outputDir := t.TempDir() + dates := []string{"--start-date", "2025-06-01", "--end-date", "2025-06-30"} + + tests := []struct { + name string + args []string + want string + }{ + {"no dataset", []string{"raw", "--symbol", rawTestSymbol, "--output-dir", outputDir}, "raw dataset is required"}, + {"unknown dataset", append([]string{"raw", "book", "--symbol", rawTestSymbol, "--output-dir", outputDir}, dates...), `unknown raw dataset "book"`}, + {"unknown exchange", append([]string{"raw", "trades", "--exchange", "bybit", "--symbol", rawTestSymbol, "--output-dir", outputDir}, dates...), `unknown exchange "bybit"`}, + {"derivative on hyperliquid", append([]string{"raw", "funding_rate", "--exchange", "hyperliquid-perps", "--symbol", "perpetual-BTC-USDC:USDC", "--output-dir", outputDir}, dates...), "trades and quotes only"}, + {"missing symbol", append([]string{"raw", "trades", "--output-dir", outputDir}, dates...), "--symbol is required"}, + {"missing output dir", append([]string{"raw", "trades", "--symbol", rawTestSymbol}, dates...), "--output-dir is mandatory"}, + {"missing dates", rawArgs(outputDir), "--start-date and --end-date are required"}, + {"bad date", rawArgs(outputDir, "--start-date", "2025-06-31", "--end-date", "2025-07-01"), `--start-date "2025-06-31"`}, + {"reversed dates", rawArgs(outputDir, "--start-date", "2025-07-01", "--end-date", "2025-06-01"), "on or after"}, + {"zero concurrency", rawArgs(outputDir, append(dates, "--max-concurrent", "0")...), "--max-concurrent"}, + {"extra argument", append(rawArgs(outputDir, dates...), "quotes"), "unexpected arguments: quotes"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + _, stderr, code := runCLI(tt.args...) + if code != 1 { + t.Fatalf("expected exit code 1, got %d; stderr: %s", code, stderr) + } + if !strings.Contains(stderr, tt.want) { + t.Errorf("expected stderr to contain %q, got: %s", tt.want, stderr) + } + }) + } + + if n := len(stub.calls()); n != 0 { + t.Errorf("expected no API calls for invalid input, got %d", n) + } +} + +func TestCLI_Raw_RequiresAKeyWithoutPreview(t *testing.T) { + stub := newRawStub(t, staticListing("2025-06")) + t.Setenv("APERIODIC_API_KEY", "") + + _, stderr, code := runCLI(rawArgs(t.TempDir(), "--start-date", "2025-06-01", "--end-date", "2025-06-30")...) + if code != 1 || !strings.Contains(stderr, "APERIODIC_API_KEY") { + t.Errorf("expected a missing-key error, got exit %d; stderr: %s", code, stderr) + } + if n := len(stub.calls()); n != 0 { + t.Errorf("expected no API calls, got %d", n) + } +} + +func TestCLI_Raw_Help(t *testing.T) { + for _, args := range [][]string{{"raw"}, {"raw", "help"}, {"raw", "--help"}, {"raw", "trades", "-h"}} { + stdout, _, code := runCLI(args...) + if code != 0 { + t.Errorf("%v: expected exit code 0, got %d", args, code) + } + for _, want := range []string{"funding_rate", "Prime + Raw", "year=YYYY/month=MM/day=DD/data.parquet", "-overwrite"} { + if !strings.Contains(stdout, want) { + t.Errorf("%v: expected help to mention %q", args, want) + } + } + } + + stdout, _, _ := runCLI("help") + if !strings.Contains(stdout, "aperiodic raw ") { + t.Error("expected the top-level help to list the raw command") + } +} + +// --------------------------------------------------------------------------- +// aperiodic raw coverage +// --------------------------------------------------------------------------- + +func rawCoverageBody() map[string]any { + series := func(first, last string, bytes int64) map[string]any { + return map[string]any{"first": first, "last": last, "days": 10, "missing": []string{"2020-01-02"}, "bytes": bytes} + } + return map[string]any{ + "schema_version": 1, + "daily_files_from": "2026-08-01", + "preview": map[string]any{ + "period": "2025-06", + "symbols": map[string]string{ + "binance-futures": "perpetual-BTC-USDT:USDT", + "okx-perps": "perpetual-BTC-USDT:USDT", + "hyperliquid-perps": "perpetual-BTC-USDC:USDC", + }, + }, + "datasets": []any{}, + "summary": []map[string]any{ + {"dataset": "trades", "exchange": "binance-futures", "symbols": 2, "first": "2019-12-01", "last": "2026-09-27", "bytes": 3 << 30}, + {"dataset": "quotes", "exchange": "okx-perps", "symbols": 1, "first": "2020-01-01", "last": "2026-09-27", "bytes": 7}, + }, + "coverage": map[string]any{ + "generated_at": "2026-09-28T03:00:00Z", + "datasets": map[string]any{ + "trades": map[string]any{ + "binance-futures": map[string]any{ + "perpetual-ETH-USDT:USDT": series("2019-12-08", "2026-09-27", 1<<30), + "perpetual-BTC-USDT:USDT": series("2019-12-01", "2026-09-27", 2<<30), + }, + }, + "quotes": map[string]any{ + "okx-perps": map[string]any{ + "perpetual-BTC-USDT:USDT": series("2020-01-01", "2026-09-27", 7), + }, + }, + }, + }, + } +} + +func TestCLI_RawCoverage_Summary(t *testing.T) { + stub := newRawStub(t, staticListing()) + stub.coverage = rawCoverageBody() + t.Setenv("APERIODIC_API_KEY", "") + + stdout, stderr, code := runCLI("raw", "coverage") + if code != 0 { + t.Fatalf("expected exit code 0, got %d; stderr: %s", code, stderr) + } + + calls := stub.calls() + if len(calls) != 1 || calls[0].path != "/metadata/raw" { + t.Fatalf("expected one /metadata/raw request, got %+v", calls) + } + if calls[0].hasKey { + t.Error("expected no X-API-KEY header on the public coverage endpoint") + } + + for _, want := range []string{ + "as of 2026-09-28T03:00:00Z", + "trades binance-futures 2 2019-12-01 2026-09-27 3.0 GiB", + "quotes okx-perps 1 2020-01-01 2026-09-27 7 B", + "2025-06 of binance-futures perpetual-BTC-USDT:USDT, okx-perps perpetual-BTC-USDT:USDT, hyperliquid-perps perpetual-BTC-USDC:USDC", + } { + if !strings.Contains(stdout, want) { + t.Errorf("expected output to contain %q, got:\n%s", want, stdout) + } + } +} + +func TestCLI_RawCoverage_ListsSymbols(t *testing.T) { + stub := newRawStub(t, staticListing()) + stub.coverage = rawCoverageBody() + + stdout, stderr, code := runCLI("raw", "coverage", "--dataset", "trades", "--exchange", "binance-futures") + if code != 0 { + t.Fatalf("expected exit code 0, got %d; stderr: %s", code, stderr) + } + + lines := strings.Split(strings.TrimSpace(stdout), "\n") + if len(lines) != 3 { + t.Fatalf("expected a header and two symbols, got:\n%s", stdout) + } + if !strings.HasPrefix(lines[1], "trades binance-futures perpetual-BTC-USDT:USDT 2019-12-01 2026-09-27 10 1 2.0 GiB") { + t.Errorf("expected BTC first with its coverage, got %q", lines[1]) + } + if !strings.Contains(lines[2], "perpetual-ETH-USDT:USDT") { + t.Errorf("expected ETH second, got %q", lines[2]) + } + + stdout, _, _ = runCLI("raw", "coverage", "--symbol", rawTestSymbol) + if strings.Count(stdout, rawTestSymbol) != 2 { + t.Errorf("expected the symbol in trades and quotes, got:\n%s", stdout) + } + + stdout, _, _ = runCLI("raw", "coverage", "--symbol", "perpetual-NOPE-USDT:USDT") + if !strings.Contains(stdout, "No raw coverage matches") { + t.Errorf("expected a no-match message, got:\n%s", stdout) + } +} + +func TestCLI_RawCoverage_RejectsUnknownFilters(t *testing.T) { + stub := newRawStub(t, staticListing()) + + for _, args := range [][]string{ + {"raw", "coverage", "--dataset", "book"}, + {"raw", "coverage", "--exchange", "bybit"}, + {"raw", "coverage", "--dataset", "open_interest", "--exchange", "hyperliquid-perps"}, + {"raw", "coverage", "trades"}, + } { + if _, _, code := runCLI(args...); code != 1 { + t.Errorf("%v: expected exit code 1, got %d", args, code) + } + } + if n := len(stub.calls()); n != 0 { + t.Errorf("expected no API calls for invalid filters, got %d", n) + } +} + +func TestFormatBytes(t *testing.T) { + tests := map[int64]string{ + 0: "0 B", + 1023: "1023 B", + 1024: "1.0 KiB", + 1536: "1.5 KiB", + 5 << 20: "5.0 MiB", + 3 << 30: "3.0 GiB", + 1<<40 + 1: "1.0 TiB", + } + for n, want := range tests { + if got := formatBytes(n); got != want { + t.Errorf("formatBytes(%d): expected %s, got %s", n, want, got) + } + } +} + +// --------------------------------------------------------------------------- +// Transfer timeouts +// --------------------------------------------------------------------------- + +func withRawStallTimeout(t *testing.T, d time.Duration) { + t.Helper() + previousStall, previousBackoff := rawStallTimeout, rawRetryBackoff + rawStallTimeout, rawRetryBackoff = d, time.Millisecond + t.Cleanup(func() { rawStallTimeout, rawRetryBackoff = previousStall, previousBackoff }) +} + +func TestStreamRawFile_OutlastsTheClientTimeout(t *testing.T) { + // A raw file can take far longer than the client's overall timeout; as + // long as bytes keep arriving the transfer must not be cut off. + withRawStallTimeout(t, 200*time.Millisecond) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + for i := 0; i < 8; i++ { + _, _ = w.Write([]byte("chunk ")) + w.(http.Flusher).Flush() + time.Sleep(25 * time.Millisecond) + } + })) + t.Cleanup(srv.Close) + + client := NewAperiodicClient("key-1") + client.HTTPClient.Timeout = 50 * time.Millisecond + dest := filepath.Join(t.TempDir(), "nested", "data.parquet") + + if err := client.streamRawFile(srv.URL, dest); err != nil { + t.Fatalf("expected a steady transfer to finish, got: %v", err) + } + if body, _ := os.ReadFile(dest); string(body) != strings.Repeat("chunk ", 8) { + t.Errorf("expected the whole body, got %q", body) + } +} + +func TestStreamRawFile_AbortsAStalledTransfer(t *testing.T) { + withRawStallTimeout(t, 50*time.Millisecond) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = w.Write([]byte("partial")) + w.(http.Flusher).Flush() + <-r.Context().Done() + })) + t.Cleanup(srv.Close) + + dir := t.TempDir() + err := NewAperiodicClient("key-1").streamRawFile(srv.URL, filepath.Join(dir, "data.parquet")) + if err == nil || !strings.Contains(err.Error(), "no data received") { + t.Fatalf("expected a stall error, got: %v", err) + } + if got := writtenFiles(t, dir); len(got) != 0 { + t.Errorf("expected no data.parquet or .part left behind, got %v", got) + } +} diff --git a/types.go b/types.go index 856836f..b561c9f 100644 --- a/types.go +++ b/types.go @@ -83,8 +83,10 @@ type AggregateDataResponse struct { } type APIErrorResponse struct { - Error string `json:"error"` - Details []string `json:"details"` + Error string `json:"error"` + Details []string `json:"details"` + Code string `json:"code"` + UpgradeURL string `json:"upgrade_url"` } type SymbolsResponse struct { @@ -92,3 +94,82 @@ type SymbolsResponse struct { Exchange string `json:"exchange"` Bucket string `json:"bucket"` } + +// RawDataset is a raw per-tick dataset (Prime + Raw plan). +type RawDataset string + +const ( + RawTrades RawDataset = "trades" + RawQuotes RawDataset = "quotes" + RawMarkPrice RawDataset = "mark_price" + RawIndexPrice RawDataset = "index_price" + RawFundingRate RawDataset = "funding_rate" + RawOpenInterest RawDataset = "open_interest" +) + +type RawFileInfo struct { + // Period is "YYYY-MM" for a monthly file, "YYYY-MM-DD" for a daily one. + Period string `json:"period"` + URL string `json:"url"` + Size int64 `json:"size"` +} + +type RawFilesResponse struct { + Dataset string `json:"dataset"` + Exchange string `json:"exchange"` + Symbol string `json:"symbol"` + SchemaVersion int `json:"schema_version"` + ExpiresIn int `json:"expires_in"` + Files []RawFileInfo `json:"files"` + MissingPeriods []string `json:"missing_periods"` +} + +type RawColumn struct { + Name string `json:"name"` + Type string `json:"type"` + Nullable bool `json:"nullable"` + Description string `json:"description"` +} + +type RawDatasetInfo struct { + ID string `json:"id"` + Name string `json:"name"` + Description string `json:"description"` + Exchanges []string `json:"exchanges"` + Columns []RawColumn `json:"columns"` +} + +// RawCoverageSeries is one symbol's coverage within a dataset and exchange. +type RawCoverageSeries struct { + First string `json:"first"` + Last string `json:"last"` + Days int `json:"days"` + Missing []string `json:"missing"` + Bytes int64 `json:"bytes"` +} + +// RawCoverageSummary is one row per dataset and exchange. +type RawCoverageSummary struct { + Dataset string `json:"dataset"` + Exchange string `json:"exchange"` + Symbols int `json:"symbols"` + First string `json:"first"` + Last string `json:"last"` + Bytes int64 `json:"bytes"` +} + +type RawCoverageResponse struct { + SchemaVersion int `json:"schema_version"` + DailyFilesFrom string `json:"daily_files_from"` + Preview struct { + Period string `json:"period"` + Symbols map[string]string `json:"symbols"` + } `json:"preview"` + Datasets []RawDatasetInfo `json:"datasets"` + Summary []RawCoverageSummary `json:"summary"` + Coverage struct { + GeneratedAt string `json:"generated_at"` + // Dataset → exchange → symbol. + Datasets map[string]map[string]map[string]RawCoverageSeries `json:"datasets"` + } `json:"coverage"` +}