scrapy-sqlite

SQLite-backed persistent resume and retry for Scrapy spiders.

What It Solves

Scrapy spiders lose all crawl progress when the process stops — whether from a crash, a deploy, or a deliberate shutdown. Restarting means re-crawling every URL from scratch.

scrapy-sqlite persists every crawled URL and its outcome in a SQLite database. When the spider starts again, it picks up exactly where it left off — already-done URLs are skipped, in-flight URLs are retried, and failed URLs with retry budget are re-attempted.

No external services. No configuration. Just a single-file SQLite database.

How It Works

flowchart TD
    S[Spider yields Request] --> DF[SQLiteDupeFilter]
    DF -->|new URL| I[INSERT request row<br/>state=running]
    DF -->|already done| SKIP[Skip]
    DF -->|failed + retries remain| R[Re-allow]
    
    I --> DL[Downloader]
    R --> DL
    
    DL --> CB[Spider callback]
    CB --> PL[SQLitePipeline]
    PL -->|2xx| DONE[UPDATE state=done]
    PL -->|error| FAIL[UPDATE state=failed<br/>increment retry_count]
    
    DONE --> RESP[INSERT response row]
    FAIL --> RESP

Loading

Three components replace Scrapy's defaults:

Component	Scrapy default	scrapy-sqlite replacement
Dupefilter	RFPDupeFilter (in-memory)	SQLiteDupeFilter (persistent)
Pipeline	None	SQLitePipeline (stores responses)
Spider base	scrapy.Spider	SQLiteSpider (resumes unfinished requests)

On first run

1. Seed URLs are inserted into the request table with state = 'pending'.
2. The spider's start() reads unfinished rows, deserializes them into Request objects, and yields them.
3. Each request passes through SQLiteDupeFilter.request_seen() — new fingerprints get an INSERT (state = running), known fingerprints check their existing state.
4. When the spider yields an item with sqlite_request_id, SQLitePipeline writes the response and marks the request as done (or failed).

On resume

1. The spider starts. start() reads rows where state IN ('pending', 'running') or (state = 'failed' AND retry_count < max_retries).
2. Those requests are deserialized and re-crawled.
3. Already-done requests are skipped by the dupefilter.

Callback resolution

When a request is serialized for storage, its callback is recorded as a fully-qualified name (module.Class.method). On resume, scrapy-sqlite resolves that name back to the actual callable via importlib:

Callback type	Example	Resolution
Instance method	self.parse_item	Resolved from class, bound to self
@classmethod	cls.parse_class	Resolved already bound to the class
@staticmethod	parse_static	Resolved as plain function
Module function	module.parse	Resolved via importlib

No callback registry to maintain — callbacks work by default as long as the spider class is importable.

Database Schema

request                          response
─────────                        ────────
id (PK)                          id (PK)
url                              request_id (FK → request.id)
fingerprint (UNIQUE, run_id)     status
pickled_request (BLOB)           body (BLOB)
parent_id (FK → request.id)      created_at
depth
state          -- pending | running | done | failed
retry_count
max_retries
run_id         -- isolates multiple crawl runs
created_at
updated_at

The UNIQUE(fingerprint, run_id) constraint makes fingerprint collisions per-run — different runs can crawl the same URL independently.

Installation

pip install scrapy-sqlite

Or for development:

git clone <repo-url>
cd scrapy-sqlite-ext
pip install -e .

Usage

Minimal spider

from scrapy import Request
from scrapy.http import Response
from scrapy_sqlite.mixin import SQLiteSpider

class MySpider(SQLiteSpider):
    name = "my_spider"
    custom_settings = {
        "DUPEFILTER_CLASS": "scrapy_sqlite.dupefilter.SQLiteDupeFilter",
        "SQLITE_DB_PATH": "my_spider.db",
        "ITEM_PIPELINES": {
            "scrapy_sqlite.pipeline.SQLitePipeline": 300,
        },
    }

    def parse(self, response: Response):
        request_id = response.meta.get("sqlite_request_id")
        yield MyItem(
            scrape_endpoint=response.url,
            result=response.body,
            status=response.status,
            sqlite_request_id=request_id,
        )
        # Follow links with crawl-chain tracking
        for link in response.css("a::attr(href)").getall():
            if request_id is not None:
                yield self.child_request(
                    request_id,
                    url=response.urljoin(link),
                    callback=self.parse,
                )

Pipeline item contract

Items processed by SQLitePipeline must have these attributes:

Attribute	Type	Required
scrape_endpoint	str	Yes
result	bytes	Yes
sqlite_request_id	int	Yes — ties the response to its request row
status	int	No — 2xx marks the request done, anything else marks it failed

Items without both scrape_endpoint and result pass through unchanged.

Seeding the database

Before running the spider, insert seed URLs:

import os
import sqlite3
from scrapy import Request
from scrapy.utils.request import RequestFingerprinter
from scrapy_sqlite.serialization import serialize_request
from scrapy_sqlite.backend import init_db

conn = sqlite3.connect("my_spider.db")
init_db(conn)

fp = RequestFingerprinter().fingerprint(Request("https://example.com")).hex()
conn.execute(
    "INSERT OR IGNORE INTO request "
    "(url, fingerprint, pickled_request, state, max_retries, run_id) "
    "VALUES (?, ?, ?, 'pending', 3, ?)",
    ("https://example.com", fp, serialize_request(Request("https://example.com")), "run-1"),
)
conn.commit()
conn.close()

Or use SQLiteSpider.insert_seed() from within the spider:

spider.insert_seed(Request("https://example.com", callback=self.parse), max_retries=3)

Run isolation

Set SQLITE_RUN_ID in the environment to isolate crawl runs. Requests from a previous run won't interfere with the current one:

SQLITE_RUN_ID=2026-07-27 scrapy crawl my_spider

Settings

Setting	Default	Description
SQLITE_DB_PATH	(required)	Path to the SQLite database file
SQLITE_MAX_RETRIES	3	Default max_retries for new requests
DUPEFILTER_CLASS	(required)	Set to scrapy_sqlite.dupefilter.SQLiteDupeFilter

Request meta keys

Set on individual requests for fine-grained control:

Meta key	Purpose
sqlite_parent_id	Links this request to its parent (set via child_request())
sqlite_run_id	Override the environment's run ID for this request
sqlite_max_retries	Override the dupefilter's default max_retries
sqlite_request_id	Set automatically — the row ID after request_seen()

Sample Project

The sample_project/ directory contains a working spider that scrapes quotes.toscrape.com.

Modes

cd sample_project/

# Normal end-to-end run (DB resets each time)
python run.py

# Test callback resolution across resume
python run.py resume-test

# Inspect the database
python run.py status

Normal run

Crawls all 10 pages of quotes and every author detail page. Each run starts fresh — the database is reset automatically.

Resume test

Inserts three simulated "interrupted" requests with different callback types, then resumes them:

[instance]    http://quotes.toscrape.com                → resolved by: quotes
[classmethod] http://quotes.toscrape.com/page/2/         → resolved by: quotes
[staticmethod] http://quotes.toscrape.com/author/...     → resolved by: staticmethod

This proves the callback resolution chain works for instance methods, classmethods, and staticmethods — all three survive the serialize → deserialize → bind cycle.

What it demonstrates

• Crawl chains: child_request() links child URLs to their parent, tracking depth and lineage in the database.
• Custom callbacks: parse_author survives resume via fully-qualified name resolution.
• All binding styles: Instance methods are bound to self, classmethods are already class-bound, staticmethods are left as plain functions.
• Pipeline integration: ProcessResult items carry sqlite_request_id so the pipeline can map responses back to request rows.

Limitations

• Response body storage. Every response body is stored as a BLOB in the response table. For large crawls (millions of pages, large payloads) the database can grow substantially. The body is stored for audit and debugging; if you don't need it, subclass SQLitePipeline and override _write to skip the INSERT INTO response.
• One database per process. The connection is a process-global singleton. Two spiders in the same process with different SQLITE_DB_PATH settings will raise ValueError. Each spider should run in its own process with its own database file.
• Single-threaded model. All writes run on the reactor thread. A slow _write (e.g. a very large response body on a spinning disk) will briefly block the event loop. In practice SQLite writes are microsecond-latency; this is the same tradeoff Scrapy's own RFPDupeFilter makes.

Design Decisions

• SQLite, zero infrastructure. A single file works for any spider — local dev, CI, production.
• WAL mode. Write-Ahead Logging allows concurrent reads (the dupefilter) and writes (the pipeline) without locking.
• Single-threaded access. All database writes run on Scrapy's reactor thread, avoiding the cross-thread commit races that plague asyncio.to_thread() patterns.
• Fully-qualified callback names. Storing module.Class.method strings instead of pickling bound methods avoids the "cannot pickle sqlite3.Connection" problem — spider instances carry unpicklable state.
• Backward compatible. Old serialized data without cb_kwargs deserializes correctly.

License

MIT