ORF Archiv Scraper is a NodeJS application, which fetches and persists ORF News Stories from multiple RSS feeds.
- News
- Sport
- Help
- Science
- OE3
- FM4
- Österreich
- Burgenland
- Wien
- Niederösterreich
- Oberösterreich
- Salzburg
- Steiermark
- Kärnten
- Tirol
- Vorarlberg
- Clone with
git clone --recursiveor rungit submodule update --init --recursivein an existing clone - Start and configure a local MongoDB document store (more info)
- Install NodeJS and npm
- Optionally: create .env.local (copy from .env file) and override the database targets (see below)
npm installnpm start -- scrape
scrape fetches every RSS feed and persists the stories to every database target. New stories are inserted, stories
whose title, category or URL changed are updated. Title embeddings are computed once per run and written to all
targets.
npm start -- scrape # scrape once
npm start -- scrape --poll # scrape every minute until interrupted
npm start -- scrape --poll --cron "0 0 * * * *" # scrape every hour
npm start -- scrape --target orfarchiv-db-2 # only write to one target
npm start -- targets # list the configured target labels| Flag | Default | Description |
|---|---|---|
--poll |
off | Keep scraping on the --cron schedule |
--cron |
0 * * * * * |
Polling interval in cron syntax (with seconds) |
--target |
all | Only write to the target with this label |
--debug |
off | Show debug logs |
- Database targets: ORFARCHIV_DB_URLS lists one connection URL per line or separated by
;, highest priority first. If it is unset, ORFARCHIV_DB_URL is used as the only target (default:mongodb://localhost). Both variables also accept a_FILEsuffix pointing to a file with the value. A target's label is itshost[:port], without credentials. - Content embeddings: ORFARCHIV_CONTENT_EMBEDDING_EXCLUDED_TARGETS lists target labels separated by
;(e.g. the Atlas M0<cluster>.mongodb.net) that never receive content embeddings.scrapenever writes content fields on any target. An excluded label that matches no target logs a warning.npm start -- targetsshows each target's capability. - Devcontainer: .env points to
orfarchiv-db-1andorfarchiv-db-2. When running on the host, override ORFARCHIV_DB_URLS in .env.local withlocalhost:27017andlocalhost:27018. - Failures: a target that cannot be reached or written to is logged and skipped, and the other targets still get their writes. A run fails only if every target failed. Each target has a 1-minute timeout per step, each run a 5-minute timeout.
- Embeddings: if the embedding server fails, stories are stored without an embedding; use backfill-embeddings to fill them in later.
scrape stores each story's title embedding next to the story. If the embedding server is unreachable or rejects a
request, the stories are still stored, just without an embedding. backfill-embeddings fills in those gaps: it finds
stories with a non-empty title and no titleEmbedding, newest first, embeds them in batches and writes the vectors back.
npm start -- backfill-embeddings # every target, until nothing is missing
npm start -- backfill-embeddings --max-docs 1000 # at most 1000 stories per target
npm start -- backfill-embeddings --target orfarchiv-db-2 # only one target, e.g. a newly added one| Flag | Default | Description |
|---|---|---|
--batch-size |
100 |
Stories per batch |
--max-docs |
no limit | Stop after this many stories per target |
--target |
all | Only backfill the target with this label |
--debug |
off | Show debug logs |
- Embedding server: requires ORFARCHIV_EMBEDDING_URL; ORFARCHIV_EMBEDDING_TOKEN is optional.
Requests are paced by ORFARCHIV_EMBEDDING_RATE_LIMIT titles per ORFARCHIV_EMBEDDING_RATE_WINDOW
(default: 1000 per
1 minute). - Multiple targets: targets from ORFARCHIV_DB_URLS are processed one after another. A title missing on several
targets is embedded only once. A target that fails is logged and skipped; the run fails only if every target failed.
npm start -- targetslists the available labels.
embed-content gives each story a second vector computed from its article text. It selects stories without a
contentStatus from the first content-enabled target, newest first, fetches their orf.at pages, extracts the article
text (read-more links are not followed, so teasers are embedded as they are), embeds title: <title> | text: <body>
truncated to fit 1024 tokens (CONTENT_MAX_TOKENS, with a small margin for special tokens) and writes
contentEmbedding to every content-enabled target. No article text is stored.
npm start -- embed-content --max-docs 200 # at most 200 stories, then exit
npm start -- embed-content --poll # keep running, e.g. in its own container
npm start -- embed-content --poll --idle-interval "5 minutes"
npm start -- embed-content --target orfarchiv-db-1 # select from and write to one target only| Flag | Default | Description |
|---|---|---|
--poll |
off | Keep running; wait --idle-interval when there is nothing to do or a batch made no progress |
--idle-interval |
1 minute |
Wait before the next batch in those cases (positive duration) |
--batch-size |
20 |
Stories per batch (positive integer) |
--max-docs |
no limit | Stop after this many stories (positive integer) |
--target |
all | Only use the target with this label (excluded labels fail) |
--debug |
off | Show debug logs |
-
Status:
contentStatus: 'embedded'stories have acontentEmbedding.contentStatus: 'failed'is terminal: the page returned 404/410 (unavailable) or no article text was found (empty); the reason is only logged. Transient failures (timeouts and 5xx after 3 retries, other 4xx such as 429 without retries) write nothing; such a story is skipped by this process for one hour and then retried. -
Pacing: in poll mode, batches run back to back while there is a backlog. The next batch waits
--idle-intervalinstead when:- the embedding server is unreachable: the batch ends before any page is fetched (or as soon as it fails), and the same stories are retried;
- embedding a story timed out: the batch ends, and that story is skipped for one hour;
- every story in the batch failed transiently, e.g. orf.at answered 403 or 429 to every request.
Without
--poll, the command exits when the backlog is drained or--max-docsis reached, and fails in the cases above. -
orf.at requests: ORFARCHIV_CONTENT_FETCH_RATE_LIMIT pages per ORFARCHIV_CONTENT_FETCH_RATE_WINDOW (default: 10 per
1 second), fetched with up to 10 concurrent requests. ORFARCHIV_CONTENT_USER_AGENT sets the User-Agent (default:orfarchiv-scraper/<version> (+https://orfarchiv.news)). -
Embedding server: uses ORFARCHIV_EMBEDDING_URL and ORFARCHIV_EMBEDDING_TOKEN like titles. The URL must end in
/v1(for examplehttp://embeddings:8080/v1); without it, llama.cpp answers in its own format and every story fails asEmbedding response was malformed. Inputs are truncated via llama.cpp's/tokenizeand checked via/health, both at the server root (a trailing/v1is removed). Requests are paced by ORFARCHIV_CONTENT_EMBEDDING_RATE_LIMIT inputs per ORFARCHIV_CONTENT_EMBEDDING_RATE_WINDOW (default: 100 per1 minute); the title rate limit does not apply. The server needs--batch-sizeand--ubatch-sizeof at least 1024, otherwise long inputs are rejected. -
Targets: targets in ORFARCHIV_CONTENT_EMBEDDING_EXCLUDED_TARGETS are never selected from or written to. A target that fails is logged and skipped. Every 10 batches, the remaining backlog is logged.
-
Rate variables: every
*_RATE_LIMITmust be a positive integer and every*_RATE_WINDOWa positive duration (e.g.1 minute); an empty value means the default. All scraper commands check all of them at startup and exit with an error naming the variable, so a typo in a content variable also stopsscrape.
bson is pinned to 7.2.0 in the overrides block of package.json. Without the pin, npm start fails immediately
with:
NotImplementedError: node:v8 isBuildingSnapshot is not yet implemented in Bun.
at node_modules/bson/lib/bson.cjs
at node_modules/mongodb/lib/index.js
bson 7.3.0 started calling v8 startupSnapshot.isBuildingSnapshot() while initializing ObjectId. Bun defines
that function but throws when it is called, so importing mongodb crashes before any application code runs. npm start
and run.sh use Bun, so they are affected; npm test and npm run build use NodeJS and are not, which is
why CI stays green either way.
This will not be fixed on the bson side — mongodb/js-bson#903 proposed a
guard and was declined as too runtime-specific. The real fix is
oven-sh/bun#32502, which is merged but not yet in a release
(oven-sh/bun#32501 has the background).
Remove the pin once a Bun release newer than 1.3.14 ships that fix, then let bson follow mongodb again. Until
then the pin holds bson a few patch releases behind.