Skip to content

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

1 watching

Forks

Repository files navigation

ORF Archiv Scraper

GitHub Actions Workflow Status GitHub package.json version GitHub License

ORF Archiv Scraper is a NodeJS application, which fetches and persists ORF News Stories from multiple RSS feeds.

RSS feeds

Local Development

Prerequisites

  1. Clone with git clone --recursive or run git submodule update --init --recursive in an existing clone
  2. Start and configure a local MongoDB document store (more info)
  3. Install NodeJS and npm

Run scraper

  1. Optionally: create .env.local (copy from .env file) and override the database targets (see below)
  2. npm install
  3. npm start -- scrape

scrape fetches every RSS feed and persists the stories to every database target. New stories are inserted, stories whose title, category or URL changed are updated. Title embeddings are computed once per run and written to all targets.

npm start -- scrape                                  # scrape once
npm start -- scrape --poll                           # scrape every minute until interrupted
npm start -- scrape --poll --cron "0 0 * * * *"      # scrape every hour
npm start -- scrape --target orfarchiv-db-2          # only write to one target
npm start -- targets                                 # list the configured target labels
Flag Default Description
--poll off Keep scraping on the --cron schedule
--cron 0 * * * * * Polling interval in cron syntax (with seconds)
--target all Only write to the target with this label
--debug off Show debug logs
  • Database targets: ORFARCHIV_DB_URLS lists one connection URL per line or separated by ;, highest priority first. If it is unset, ORFARCHIV_DB_URL is used as the only target (default: mongodb://localhost). Both variables also accept a _FILE suffix pointing to a file with the value. A target's label is its host[:port], without credentials.
  • Content embeddings: ORFARCHIV_CONTENT_EMBEDDING_EXCLUDED_TARGETS lists target labels separated by ; (e.g. the Atlas M0 <cluster>.mongodb.net) that never receive content embeddings. scrape never writes content fields on any target. An excluded label that matches no target logs a warning. npm start -- targets shows each target's capability.
  • Devcontainer: .env points to orfarchiv-db-1 and orfarchiv-db-2. When running on the host, override ORFARCHIV_DB_URLS in .env.local with localhost:27017 and localhost:27018.
  • Failures: a target that cannot be reached or written to is logged and skipped, and the other targets still get their writes. A run fails only if every target failed. Each target has a 1-minute timeout per step, each run a 5-minute timeout.
  • Embeddings: if the embedding server fails, stories are stored without an embedding; use backfill-embeddings to fill them in later.

Backfill embeddings

scrape stores each story's title embedding next to the story. If the embedding server is unreachable or rejects a request, the stories are still stored, just without an embedding. backfill-embeddings fills in those gaps: it finds stories with a non-empty title and no titleEmbedding, newest first, embeds them in batches and writes the vectors back.

npm start -- backfill-embeddings                              # every target, until nothing is missing
npm start -- backfill-embeddings --max-docs 1000              # at most 1000 stories per target
npm start -- backfill-embeddings --target orfarchiv-db-2      # only one target, e.g. a newly added one
Flag Default Description
--batch-size 100 Stories per batch
--max-docs no limit Stop after this many stories per target
--target all Only backfill the target with this label
--debug off Show debug logs
  • Embedding server: requires ORFARCHIV_EMBEDDING_URL; ORFARCHIV_EMBEDDING_TOKEN is optional. Requests are paced by ORFARCHIV_EMBEDDING_RATE_LIMIT titles per ORFARCHIV_EMBEDDING_RATE_WINDOW (default: 1000 per 1 minute).
  • Multiple targets: targets from ORFARCHIV_DB_URLS are processed one after another. A title missing on several targets is embedded only once. A target that fails is logged and skipped; the run fails only if every target failed. npm start -- targets lists the available labels.

Content embeddings

embed-content gives each story a second vector computed from its article text. It selects stories without a contentStatus from the first content-enabled target, newest first, fetches their orf.at pages, extracts the article text (read-more links are not followed, so teasers are embedded as they are), embeds title: <title> | text: <body> truncated to fit 1024 tokens (CONTENT_MAX_TOKENS, with a small margin for special tokens) and writes contentEmbedding to every content-enabled target. No article text is stored.

npm start -- embed-content --max-docs 200                    # at most 200 stories, then exit
npm start -- embed-content --poll                            # keep running, e.g. in its own container
npm start -- embed-content --poll --idle-interval "5 minutes"
npm start -- embed-content --target orfarchiv-db-1           # select from and write to one target only
Flag Default Description
--poll off Keep running; wait --idle-interval when there is nothing to do or a batch made no progress
--idle-interval 1 minute Wait before the next batch in those cases (positive duration)
--batch-size 20 Stories per batch (positive integer)
--max-docs no limit Stop after this many stories (positive integer)
--target all Only use the target with this label (excluded labels fail)
--debug off Show debug logs
  • Status: contentStatus: 'embedded' stories have a contentEmbedding. contentStatus: 'failed' is terminal: the page returned 404/410 (unavailable) or no article text was found (empty); the reason is only logged. Transient failures (timeouts and 5xx after 3 retries, other 4xx such as 429 without retries) write nothing; such a story is skipped by this process for one hour and then retried.

  • Pacing: in poll mode, batches run back to back while there is a backlog. The next batch waits --idle-interval instead when:

    • the embedding server is unreachable: the batch ends before any page is fetched (or as soon as it fails), and the same stories are retried;
    • embedding a story timed out: the batch ends, and that story is skipped for one hour;
    • every story in the batch failed transiently, e.g. orf.at answered 403 or 429 to every request.

    Without --poll, the command exits when the backlog is drained or --max-docs is reached, and fails in the cases above.

  • orf.at requests: ORFARCHIV_CONTENT_FETCH_RATE_LIMIT pages per ORFARCHIV_CONTENT_FETCH_RATE_WINDOW (default: 10 per 1 second), fetched with up to 10 concurrent requests. ORFARCHIV_CONTENT_USER_AGENT sets the User-Agent (default: orfarchiv-scraper/<version> (+https://orfarchiv.news)).

  • Embedding server: uses ORFARCHIV_EMBEDDING_URL and ORFARCHIV_EMBEDDING_TOKEN like titles. The URL must end in /v1 (for example http://embeddings:8080/v1); without it, llama.cpp answers in its own format and every story fails as Embedding response was malformed. Inputs are truncated via llama.cpp's /tokenize and checked via /health, both at the server root (a trailing /v1 is removed). Requests are paced by ORFARCHIV_CONTENT_EMBEDDING_RATE_LIMIT inputs per ORFARCHIV_CONTENT_EMBEDDING_RATE_WINDOW (default: 100 per 1 minute); the title rate limit does not apply. The server needs --batch-size and --ubatch-size of at least 1024, otherwise long inputs are rejected.

  • Targets: targets in ORFARCHIV_CONTENT_EMBEDDING_EXCLUDED_TARGETS are never selected from or written to. A target that fails is logged and skipped. Every 10 batches, the remaining backlog is logged.

  • Rate variables: every *_RATE_LIMIT must be a positive integer and every *_RATE_WINDOW a positive duration (e.g. 1 minute); an empty value means the default. All scraper commands check all of them at startup and exit with an error naming the variable, so a typo in a content variable also stops scrape.

Dependency overrides

bson is pinned to 7.2.0 in the overrides block of package.json. Without the pin, npm start fails immediately with:

NotImplementedError: node:v8 isBuildingSnapshot is not yet implemented in Bun.
    at node_modules/bson/lib/bson.cjs
    at node_modules/mongodb/lib/index.js

bson 7.3.0 started calling v8 startupSnapshot.isBuildingSnapshot() while initializing ObjectId. Bun defines that function but throws when it is called, so importing mongodb crashes before any application code runs. npm start and run.sh use Bun, so they are affected; npm test and npm run build use NodeJS and are not, which is why CI stays green either way.

This will not be fixed on the bson side — mongodb/js-bson#903 proposed a guard and was declined as too runtime-specific. The real fix is oven-sh/bun#32502, which is merged but not yet in a release (oven-sh/bun#32501 has the background).

Remove the pin once a Bun release newer than 1.3.14 ships that fix, then let bson follow mongodb again. Until then the pin holds bson a few patch releases behind.

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages