Repository navigation
Conversation
Lt-Flash
added a commit
to Lt-Flash/opensips
that referenced
this pull request
Jul 24, 2026
The isolated-cache, 50k end-to-end and 100k three-way benchmarks are all published in PR OpenSIPS#4118; only the two-socket huge-page-arena end-to-end number is still outstanding.
Lt-Flash
force-pushed
the
feature/cachedb-perf-devel
branch
from
July 25, 2026 00:05
cf2d1e4 to
642c828
Compare
Lt-Flash
marked this pull request as ready for review
July 26, 2026 07:34
This was referenced Jul 26, 2026
topology_hiding: keep the dialog-less state in a shared store (fixes long Contact / SUBSCRIBE)
#4114
Open
Lt-Flash
force-pushed
the
feature/cachedb-perf-devel
branch
3 times, most recently
from
August 10, 2026 09:12
d51c274 to
2fc31a9
Compare
Lt-Flash
force-pushed
the
feature/cachedb-perf-devel
branch
from
August 21, 2026 02:47
0585839 to
16790b0
Compare
Lt-Flash
added a commit
to Lt-Flash/opensips
that referenced
this pull request
Aug 23, 2026
…penSIPS#4118 The upstream PR keeps the module description and the headline numbers; the index-structure shootout, concurrency and read-protocol experiments, memory-backing tiers, cachedb_local / cachedb_redis comparisons, the 50k and 100k topology-hiding runs, the huge-page arena measurements and the soak findings, with their graphs, live here.
Lt-Flash
force-pushed
the
feature/cachedb-perf-devel
branch
from
August 24, 2026 15:35
0fc5ee2 to
1eaa330
Compare
|
Tested and verified on commit 1eaa330 (branch feature/cachedb-perf-latest).
|
bogdan-iancu
requested changes
Oct 2, 2026
bogdan-iancu
left a comment
bogdan-iancu
left a comment
Member
There was a problem hiding this comment.
@Lt-Flash , is this PR finalized? IF as code is done, please note that the documentation must be conversted from xml to md. Thank you !
A local cachedb backend (URL scheme perf://) built for the SIP hot path: lock-free reads (per-bucket seqlock + copy-out validation), short locked writes, named collections that grow at runtime under a maintenance timer, instant expiry with cheap background reclaim, and a size-class slab arena with optional huge-page backing (memory_backing=auto|core|own-hg|own, with full reclaim - retire, re-cut, give-back - in the own modes). Optional surfaces, each degrading explicitly when absent: persistence to any db_* backend (perf_save/perf_load, db_mode), cluster warm-up sync over a sharing tag, and cross-node pull-on-miss over the clusterer bin links, the clusterer_controller plane, or the module's own udp/tcp sockets (pull_transport, with bin-capability HELLO peer discovery). Keys are hashed with MurmurHash3. One addition outside the module: an optional CACHEDB_CAP_GET_BUF cachedb capability - an allocation-free read into the caller's buffer - that backends and consumers may ignore. Ships with perf_stats/introspection MI, statistics, events, admin docs, a benchmark/soak harness (bench/), and the full measurement study (doc/STUDY.md, doc/PR-NOTES.md).
Three complete deployment shapes - standalone, clustered over the stock clusterer (bin/udp/tcp pull transports), and clusterer_controller-managed (clctr) - with the pull modes, persistence, memory backings and the failover hooks explained around each.
The earlier cross-node latency/convergence figures came from three containers on one host - a single-kernel bridge with no real wire. Re-run on three physical hosts (one container per host, host networking, same binary): 30k-key thirds convergence takes 10.7-14.2 s to 99.9% (udp/tcp/ bin), warm p50 is ~197 us including one real LAN round trip, and the transport differences live entirely in the miss-path tail, where the module-owned udp/tcp sockets halve bin's p99. Zero failures across 25M requests. STUDY.md gains the authoritative section + 4 figures; the archived PR-NOTES numbers are banner-marked as superseded.
Every MI command's exact invocation syntax and response shape, pulled directly from the mi_export_t recipes and their handlers rather than paraphrased - all 16 commands across read/introspect, single-key ops, bulk ops, cross-node pull (CP-15), and persistence/cluster sync (CP-19), plus the two invocation surfaces and the named-params-only / glob-first gotchas that cost real debugging time otherwise.
…ot written past pcache_htable_new() computed nbuckets = 1U << size_log2 and then allocated nbuckets / PCACHE_SEG_SIZE segments into ht->seg[], a FIXED array of PCACHE_NSEGS pointers sized for 2^24 buckets. A size_log2 of 25 asks for 8192 segments and writes past a 4096-entry directory; 32 or more is not a defined shift at all. Every caller in the tree is gated below the ceiling today, so this is latent rather than live - which is exactly the kind of latent a later caller trips, since nothing in the function's contract says where the limit is. Refuse above the ceiling and name it, and derive PCACHE_NSEGS from the new PCACHE_MAX_SIZE_LOG2 rather than repeating 24 in two places, so the guard and the directory cannot drift apart. (cherry picked from commit 58cd08c224c0b4a3204b4676c8b14e3b3248df44)
A table at its target load factor keeps almost nothing in the leg; a table that has stopped growing puts everything there, and the leg is a chain per hash bucket under ONE lock, so its occupancy is the difference between a table that performs and one that does not. ovf_count was already maintained and reported nowhere, so a table sitting at 81 entries per bucket looked identical from the outside to one at 4. An accessor beside pcache_ht_nbuckets(), which is where a caller already goes for the shape of a table. (cherry picked from commit 81080fe7c6735fb0542d8b39c0b4de151814d25b)
A key lives in its bucket OR in the overflow leg, never both, so any operation that misses the bucket must walk the chain for its hash under the one leg lock - and that includes every store, not only operations on records that are actually in the leg. A table whose leg grew during a write burst therefore taxes all of its subsequent writes. Profiled on a node applying replicas at about 118,000 records a second: the leg lookup was 17.0% of the applying thread, the largest cost outside the cipher, ahead of the store itself at 12.1%. Chain length is leg size divided by the number of heads, so widening the head array shortens every chain in proportion. 16,384 heads is 131,072 bytes, the largest power of two that still fits inside the single 256 KB region slot that 1,024 heads already occupied - sixteen times the chains for no additional memory. Nothing else changes: the array is one allocation, the index is a mask, and both follow the constant. (cherry picked from commit 20fc9c0b3ae0705a0178a952030a7830900d5a41)
pcache_ht_probe() reads through the fetch path with a NULL, zero-length destination - existence and metadata, no copy. The bucket scan honours that and stops before the copy; the overflow leg's ovf_fetch() tested the value length against the zero-length buffer first and returned PCACHE_E_TOOSMALL, which every caller testing for 0 read as absent. 2,000 records in 256 buckets of six slots put a third of them in the leg, and EXISTS answered 0 for every one of those. The probe now stops before the copy in the leg exactly as it does in the bucket.
…raining it in its final call A table at load factor four keeps some percent of its records in the overflow chains - a million-record table about nine percent, 90,000 records - and pcache_ht_scan() drained all of them in the call that reached the last bucket, unbounded by max_buckets. The cursor space now has a position inside the leg (PCACHE_CURSOR_OVF | overflow bucket), and each call emits about max_buckets buckets' worth of records from the leg, whole chains at a time, before handing the cursor back. Callers treat the cursor as opaque, as they always did.
…e, so no call is the last buckets plus a leg budget on top
scan_bucket() caught tag_matches()'s 64-bit word in an unsigned long and scanned it with ctzl. On ILP32 unsigned long is 4 bytes, so the match bits of slots 4 and 5 of every bucket were truncated away: the write-side find_slot (a plain loop) saw the record, lock-free readers did not, and a key that landed in slot 4 vanished right after its own store. The earlier ILP32 fix widened tag_matches() itself but not this caller. uint64_t m and ctzll. No change on LP64.
… callback iter_overflow_from() held ovf_lock across every callback. A callback that touches an overflow-resident key - removing the record it was handed takes ovf_lock - deadlocked against itself, and every other process's overflow lookup or store waited out whatever the callback did: a perf_sync DB save performs one SQL insert per record from that callback. The chain is walked by index, re-resolved under the lock per entry, and the record copied out before the lock is released; after the callback the node at the same index is re-checked, and if it changed the emitted node was removed and its successor slid into place, so the index does not advance. Self-removal stays exactly-once, other concurrent mutation is at-least-once, which the walk already promised. The budgeted resume added for the cursored walk is unchanged. The walk contract in the header now says no lock is held across the callback.
… under its lock carve_chunk() logged an ERROR for every allocation a full arena refused - every refused write - with the arena lock held, so every allocating process queued behind the one formatting the line. It now logs the first refusal and then at most once every 10 seconds per process, with the running count. The limiter uses get_ticks(): the arena's own tick only advances under the own-memory backing, so it would have silenced the default shm backing after the first line.
…pipe ipc_send_rpc_all() also visits free and never-used process slots - the headroom of an auto-scaling profile - and busy-waits 20000 usleep rounds on each before failing with "Bad file descriptor". With chunks lingering, the reclaim process spent nearly all its time in those waits and logged an error for every empty slot on every round. The flush now goes only to processes marked running, that take IPC and have a pipe.
A chunk counts as lingering while it is partly free for a whole quiet window - which is also what a chunk holding live records looks like, so on any cache with data the flush went out every reclaim_quiet_s seconds forever and interrupted every process for nothing. When a round leaves at least as many chunks lingering, the next waits twice as long, up to 600 s; a round that helps, or a tick with nothing lingering, resets it. The interval is reported as flush_interval_s.
Author
|
Hi Bogdan, a short follow-up. Running cachedb_perf on a box with an auto-scaled TCP profile, I found that the core's cachedb_perf's reclaim process was one of its callers, so I pushed two commits here that do not depend on that fix:
On a test box with an auto-scaled TCP profile (6 empty slots) and Thanks! |
…emitted The walk drops ovf_lock around every callback and found its place again by position, assuming that when the node there had changed exactly the emitted one had gone. Another process removing records from the chain in that window - the callback itself, a remove, the expiry sweep - could leave the position naming the node after the next one, which was never handed over to perf_scan, iter_keys or a glob delete. Now, under one hold of the lock, the walk finds the most recently emitted node still in the chain and takes the one after it. Nodes are only inserted at a chain's head and unlinked, so nothing after an emitted node is skipped; with every remembered node gone the chain is walked again from its head, a repeat that every walker tolerates. (From perfcached S330, e87258b.)
… authoritative A store whose value equals the stored one only re-arms the TTL, and that path kept the record's flags. A key first pulled from a peer (PASSIVE) and then written locally with the same value stayed PASSIVE, so with pull_authoritative_serve this node kept answering peers "held" for a key it now writes itself. A local write on that path now takes the writer's flags, as the in-place path already does; a pulled write still keeps them, so re-pulling the owner's bytes never demotes its copy. (From perfcached c1f58bb. Its accumulate half is not needed here: a native counter is never served to a peer, so it is never PASSIVE.)
An add or sub with expires 0 re-armed the record with expiry 0, so a counter that had a TTL silently became immortal. cachedb_local keeps an existing counter's expiry on an add, and cachedb_redis sends EXPIRE only for a non-zero TTL; a script switching backends must not see its rate limiter window disappear. pcache_ht_add() takes PCACHE_EXP_PRESERVE: every accumulate path keeps the record's own expiry (a converted string record passes it on), and a counter created under it never expires. The cachedb add/sub callbacks pass it for expires 0; a non-zero TTL still re-arms, as with redis. The stores_immortal statistic counts the expiry the record ends up with. (From perfcached 221bff8.)
A removal moves the bucket's last record into the freed slot, so a walk already past that slot missed the moved record, though it was stored for the whole walk. A bucket whose version moved during a pass is walked again, emitting only keys not emitted yet, up to four passes.
Every walk ran all six slot snapshots of every bucket, each with its own seqlock reads, before finding a bucket empty. Slots are packed and a tag is never 0, so one load of tags[6] answers both: a zero word is an empty bucket, a zero byte a free slot. A remove copies the last tag down before clearing it, so a record stable across the walk never reads as free. The bytes are tested in array order, so the check does not depend on the byte order. (From perfcached 221c643.)
Four cases, each on its own table, all run before the selftest fails: a leg walk whose callback removes the record before it and its own; a walk and a scan whose callback removes an already emitted record from a full bucket, every key exactly once; an unchanged-value write that promotes a pulled copy and never demotes a local one; and a counter add without a TTL that keeps the record's expiry.
…ve-back The kernel refuses MADV_REMOVE on locked memory, and the reservation of the THP and 4K tiers is mlock-pinned whenever the limit allows, so on such a node the first group punch failed and set giveback_off. That latch was tested at the top of the whole give-back, so the empty shm pages past the reservation, which go back through shm_free(), were never returned either. The latch now covers the reservation only, and perf_stats reports it as arena.giveback_off. (From perfcached 7a68906.)
After a successful MADV_REMOVE the give-back rebuilt the eight slot headers "cold" - eight writes into the memory it had just released, each faulting a page straight back in. On the hugetlb tier, which is never mlock-pinned and so punches even where the other tiers are pinned and refuse to, that took the whole 2 MB page back from the pool: cold_bytes said released while the memory stayed resident. An unpinned 4K or THP reservation got eight pages, or a huge page, back the same way. A punched slot is now known by the cold map alone. The carve that takes it writes a fresh header, and the two walks that read every reservation header (perf_stats and the huge-page capacity) skip cold slots. (From perfcached 12d67df, S101.)
Taking a cold slot wrote its header straight away. On the hugetlb tier the punch had returned that page to the kernel's pool, and if the pool was empty by then the write faulted with SIGBUS - there is no ENOMEM on that path - and took down whichever process was carving. A cold group is now committed again before anything writes to it: MADV_POPULATE_WRITE (Linux 5.14) reports an empty pool instead of faulting, and an older kernel's EINVAL falls back to the pool's free count. A refusal is counted (perf_stats arena.pool_empty), logged at most every 10 s, and the carve goes on to the frontier and to shm pages as it does past the reservation. The group comes back whole, all eight slots warm, since on hugetlb the first fault returns all 2 MB; the other tiers skip the check, their re-faults draw on ordinary memory rather than on a fixed pool. (From perfcached fc08e70, S94.)
… per run The give-back held the arena lock - the carve lock every writer needing a fresh chunk takes - across one MADV_REMOVE per 2 MB group and every shm_free() of an empty page, so a drain of a few hundred MB stalled every carve for the whole sweep. It now decides under the lock, makes the syscalls with it dropped and does the bookkeeping under it again. Groups being punched sit on an in-flight map, where no carve takes them and no walk reads their headers; empty pages are unlinked before the lock is dropped. A run of consecutive groups goes out in one madvise, counted in perf_stats as arena.punch_calls and arena.punch_groups. (From perfcached 12d67df, S95 and S96.)
The selftest carves past the free slots (into new shm pages when the reservation is small), drains them, ages them past the quiet and cool-off windows and runs one reclaim tick. Empty shm pages must go back even when the reservation refused its punch; a punched group must have no page resident (mincore); and the carve that takes a punched slot again must take its whole group. The punch checks need an unpinned reservation and are skipped, with a notice, otherwise.
A record never moves from a bucket into the leg: new records arrive there, and the leg drain that follows moves records out of it, into their bucket. Walked buckets-then-leg, a record moved into a bucket the walk had already passed would never be seen. pcache_ht_iter() and pcache_ht_scan() now take the leg first, so a leg record is seen there or in a bucket the ascending walk has not reached yet. The scan call that finishes the leg goes on into the buckets from 0 with its own budget, so a cursor of 0 still means begin and done, and a count that covers the table still completes in one call. (From perfcached 58acdab and 7536e1e, S281.)
…the table A record stored while its bucket was full went to the overflow leg and stayed there: a split makes room in the buckets but never looks at the leg. A burst that outran growth so left part of the table behind the one leg lock for good, paid on every lookup that misses its bucket. pcache_ht_drain_leg() walks leg chains from its own cursor and moves every record whose bucket now has a free slot; a record whose bucket is full is stepped over within its chain. The maintenance tick calls it with a budget of 1,024 steps, a step being a record or a chain found done, and perf_stats reports leg_drained. The drain takes the bucket lock before the leg lock, the store path's order, holds both across the move and lowers the bucket's expiry hint, or the sweep would not find a moved record that expires (perfcached S349). A lock-free reader that missed in the bucket and then in the leg could fall exactly between the two halves of a move. The move bumps the bucket version before it unlinks the record, so a reader whose leg lookup misses re-checks that version and looks again if it changed. (From perfcached 4df3886 and 37cce16, S172 and S349; the reader's re-check is not in perfcached.)
A scan stopped in its bucket phase, the keys it saw removed and the leg drained into that room must still meet every remaining key. A burst of 300 keys into 16 buckets, grown and drained, must leave the leg shorter by exactly what was moved, every value intact, and its expiring records reaped by one sweep. The selftest now runs its cases from a table.
pcache_region_alloc() put a 16-byte header and 64 bytes of alignment slack in front of every region before rounding up to whole 256 KB slots. A segment of 4,096 buckets is exactly 256 KB, so it took two slots, and its 16 KB expiry-hint block a third: 768 KB of the reservation for every 4,096 buckets, and a whole slot for each of a table's small regions. The header was never read on that path - the reservation's regions are neither listed nor freed. Reservation regions are now header-less, so a segment takes one slot, and a region smaller than the unused end of an earlier region slot is placed there: 272 KB per 4,096 buckets. The shm and HG paths are unchanged. (From perfcached cfed699, S155, which packs the hint blocks alone.)
A chunk retires only when every cell in it is free, so the survivors of a mass expiry - a few records spread over every chunk that held the expired ones - kept all of those chunks, and their memory, indefinitely. The reclaim tick now marks a chunk sparse when at most an eighth of it is live and it has been quiet for reclaim_quiet_s. While any chunk is, the maintenance timer walks 4,096 buckets of each collection per tick and moves up to 512 records out of sparse chunks: a copy into a cell of the densest other partly used chunk of the class, the slot swapped under the bucket lock with the version bumped around it, the old cell sent home. The emptied chunks retire on a later reclaim tick. perf_stats reports arena.sparse_chunks and arena.evacuated. The target is never the record's own chunk or an empty one: a move there frees nothing, and a class's last few survivors would go back and forth between two chunks for good. When every chunk of the class is sparse the fullest other one takes the records. (From perfcached 7586fa0, S190, whose target choice allows the empty and own chunk.)
The arena selftest carves a one-slot region, which must take exactly one slot of the reservation, and two 16 KB regions, which must share one. The htable selftest stores 420 records of one class, removes 15 of every 16, and requires the arena to find their chunks sparse, the evacuation to leave at least five fewer chunks with every survivor intact, a second evacuation to move nothing, and a class reduced to three records in one chunk to stay where it is.
…d checks later ticks With a small reservation the chunks kept per class and the index regions at the top can leave no whole free 2 MB group, and the case then failed although nothing was wrong. It now counts the whole free groups below the frontier before the tick and requires a punch only when there is one; otherwise it says that punches were not exercised. The tick that meets a refused punch decides its shm pages before the punch, so it returned them even with the latch covering everything; only later ticks showed the fault. After a refusal the case runs a second round and requires its empty pages to go back as well.
The drain examines one candidate per step - a record, or a chain found done - so the 1,024 steps the maintenance tick gave it bounded it to about a thousand records a second, not to 1,024 of the 16,384 chains: a burst that left tens of thousands of records in the leg took tens of seconds to drain. The tick now drains in chunks of 1,024 steps until 10 ms are spent, the leg is empty, or the cursor has completed a whole walk of the leg (pcache_ht_drain_passes()), whichever comes first; a count is the wrong bound, since what a split has just made placeable sits wherever the cursor is not. The selftest cases drain one whole pass. (From perfcached 8228a9c, S176.)
perf_keys, perf_scan, the glob TTL and the glob delete read nothing but the key, yet every record's value - up to 64 KB - was copied out of the table for them. pcache_ht_iter_ex() and pcache_ht_scan() take PCACHE_WALK_KEYS, under which neither the buckets nor the leg copy the value out and the callback gets an empty one; those four callers pass it, and perf_dump still walks with the values. A selftest case checks that a keys-only walk and a keys-only scan hand over every key once and no value. (From perfcached 7f86625, S40.)
A lookup reads tags[6] and the meta field as one 8-byte word, finds the matching bytes with a SWAR test masked to the low six bytes, and takes the slot index from the lowest set bit. That holds only when tags[i] is byte i of the value counted from the low end, which is little-endian order. On a big-endian host the mask kept tags[2..5] and the meta bytes, and the bit index named the mirrored slot, so a stored key was mostly reported absent. The word is now loaded with tags[i] in byte i on either byte order: byte-swapped on big-endian builds, unchanged on little-endian ones. A unit test of the tag code under qemu-s390x reported the right slot for 4,146 of 69,996 lookups before and for all of them after, and the meta bytes no longer match; the htable selftest gains the same check.
Author
|
Thank you for your patience with this PR. I'm still actively developing cachedb_perf, and as I go I keep finding bugs and fixing them. If that's all right with you, I would like to keep adding these fixes here, so that the version you review has all of them. I've just pushed the latest batch: 21 commits on top of 2e713f3, all inside
The README documents the new Please let me know if you would rather I hold further fixes for a separate PR. |
The reader re-checked its bucket after a leg miss only when it had looked in the leg. When the drain moved the leg's last record, the reader found the leg empty, skipped it and reported the key absent. It now re-checks the bucket version after any miss. The drain decrements the leg count with release and the reader loads it with acquire, so a reader that sees the leg shorter also sees the version change. After four re-looks with the bucket still changing, the lookup is decided under the bucket lock, which a move needs. pcache_ht_touch() released the bucket lock before it looked in the leg and could report a key absent in the same way; it now looks in the leg under the bucket lock, the order store and remove already use. A selftest hook runs the drain between a lookup's bucket miss and its leg lookup: a fetch with another record left in the leg, a fetch of the leg's only record and a touch must all find the key.
A walk can hand the same key over twice when a split or the leg drain moves a record while it runs. The save inserted every row it was given, so a table with a primary key on (collection, pkey) failed the save on the repeat - it rolled back, keeping the previous snapshot - and one without such a key stored the record twice. The startup and shutdown saves run with nothing else touching the table; a perf_save or perf_sync taken while the maintenance timer is draining could hit it. The save now remembers each key it writes by a 64-bit fingerprint, so the count it reports is of keys. Where the backend has replace (db_sqlite, db_mysql) every row is written with it and a repeat rewrites its own row; elsewhere a repeat is skipped. If no pkg memory is left for the fingerprints the save goes on as before and says so once. With db_url set, htable_selftest also saves a collection while a hook drains the leg between the walk's leg and bucket phases, once with replace and once with insert, and requires every key to come back from the table exactly once.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cachedb_perfis a local in-memorycachedbbackend, selected by URL scheme (perf://). It implements the standardcachedb_funcsvtable, so every module that takes acachedb_url(topology_hiding,dialogprofiles,sql_cacher, ...) and the corecache_*script functions work unchanged — changing the URL is the whole migration.It exists because
cachedb_local's single fixed-size table, per-bucket locking and per-write allocation become the bottleneck at high churn: with 8 concurrent writerscachedb_perfis 27× faster and its p99 stays under a microsecond (see Performance), and in the earlier end-to-end study topology hiding sustained 6,000 CPS wherecachedb_localbroke at ~3,900.How: lock-free reads (per-bucket version + copy-out validation; writers hold a bucket lock for ~hundreds of ns), named collections that grow at runtime, instant expiry with a cheap background sweep, and records in a size-class slab arena with full reclaim, in OpenSIPS shm or in a separate reservation of its own. Keys hash with MurmurHash3 (the core's hash collides on sequential keys).
Memory: inside or outside the OpenSIPS memory model. By default the records live in OpenSIPS shared memory, next to everything else, sized by
-mand counted in the core'sshmem:statistics. Settingarena_hugepage_mbinstead gives the cache a separate arena of its own, reserved at startup outside the OpenSIPS memory model. The cache then cannot exhaust the shm the rest of the server needs, and the arena is sized and reclaimed on its own. That arena is backed by huge pages where the host allows it (a hugetlb pool, else transparent huge pages, else plain 4 KB pages, chosen at startup by what the kernel actually grants), which is usually a little faster on a large cache because of fewer TLB misses. The figures below are all measured on plain shared memory, the same memorycachedb_localuses.Optional, each degrading explicitly when absent: persistence to any
db_*backend, cluster warm-up sync, and cross-node pull-on-miss — over the clusterer's bin links or the module's own udp/tcp sockets. The only cluster dependency is the stockclusterermodule.Outside the module, one optional cachedb capability:
CACHEDB_CAP_GET_BUF, an allocation-free read into the caller's buffer. Backends and consumers that do not know it are unaffected.Cross-node pull, in brief
Each node keeps its own cache; nothing is replicated per operation. With the stock
clustererloaded andsync_cluster_idset, a node can instead pull a key it does not have:replicate_collections). Only the operator knows which keys mean the same thing on every node (e.g. topology-hiding state), so nothing is pulled by default.pull_timeout_ms, default 50 ms) is never taken as absence.pull_negative_ms, default 300 ms) remembers a whole-cluster "not here", and any local write to the key clears it.pull_on_miss=1repairs misses transparently on the normal read path. It is off by default because it blocks the worker until the cluster answers. Otherwise the pull is explicit, through the module's pull API (usable fromasync()) or theperf_pullMI command.pull_bind), which skips the core TCP dispatcher and is faster under load. Theclctrtransport depends on clusterer_controller: zero-config HA clusterer control module with automatic sharing tag management via encrypted UDP multicast #4074, see below.cache_add), because a counter only means something on the node that holds it.Separately,
perf_syncrefreshes a whole collection cluster-wide through the shared DB (save, then signal the peers to reload), for single-writer / read-replica setups; withsync_shtagthe same happens automatically on a sharing-tag failover.Performance of this version
cdbbenchdriving each backend through the cachedb API from inside OpenSIPS:-a F_MALLOC -m 2048, 20,000 resident keys, 600-byte values, the same 2^15-bucket collection for both local modules, median of 5 runs (Redis: 3 runs, a localredis:8on its own cores).cachedb_perfcachedb_localcachedb_rediscachedb_localloses throughput as writers are added: every write is ashm_malloc+shm_freeon the allocator's global lock. Thecachedb_rediscolumn is that module's synchronous round trip per operation (about 2.4 commands per write), not Redis's own capacity. Built on master147ddf1e59.Usage
From the script the backend id is
perf(orperf:<group>when a grouped URL is declared):cache_store("perf", "call-$ci", "$var(state)", 3600);Complete worked configurations - standalone, clustered over the stock clusterer, and clusterer_controller-managed, with every pull mode and transport explained: Configuration examples.
clusterer_controller(#4074). The module can also carry pulls over the encrypted multicast plane ofclusterer_controller, proposed separately in #4074 (pull_transport=clctr). That support is compiled only when the controller module is in the tree (CLUSTERER_CTRL_SUPPORT), so on current master it builds out entirely, is not documented in the README, and an explicitpull_transport=clctrlogs a warning and falls back tobin. If #4074 is merged it becomes available with no change here.Parameters
cache_collectionsdefault=14name[=log2(buckets)];...— the size is only a starting point, tables growcachedb_urlperf://perf:///namepicks the collection; repeatablearena_hugepage_mb00= records in OpenSIPS shmreclaim_keep,reclaim_quiet_s,reclaim_cooloff_s,reclaim_givebackexpiry_sweep_period1growth_load_factor/growth_budget2/4096db_url,db_table,db_mode,persist_collectionssync_cluster_id,sync_shtagreplicate_collections,pull_on_misspull_transportbin(udponcepull_bindis set)binor module-ownedudp/tcpsockets (clctronce #4074 is in)pull_bind,pull_portpull_timeout_ms,pull_negative_mspull_slots,pull_max_key,pull_max_valuepull_linger_ms,pull_authoritative_serveevent_expired_collections,arena_selftest,htable_selftestMI (
perf_stats, introspection,perf_save/perf_load/perf_sync), statistics and events are documented in the module README.md. All 16 MI commands with real invocation + response examples: MI examples.Compatibility
Linux is the target platform. The module is in the default build and also builds on other systems, where the Linux-only parts drop out: no huge pages (plain 4 KB pages), freed slots stay resident because
MADV_REMOVEis Linux-only, and the cross-node pull (replicate_collections), built oneventfdandtimerfd, refuses to start with an error saying so. On Linux, a kernel withoutMAP_HUGETLB,MADV_HUGEPAGEorMADV_COLLAPSEfalls down the same ladder to the next tier. I have not run it on other systems; the non-Linux build was checked by compiling the module against FreeBSD 14.5 headers, also witheventfd/timerfdhidden as on macOS or OpenBSD.Docs, testing
The module docs are
modules/cachedb_perf/README.md, in markdown as per the docs migration; the DocBook sources are gone. The measurement material is kept out of the module tree and linked at the commit that last carried it: doc/STUDY.md (the study, with graphs), doc/PR-NOTES.md (progress notes), bench/README.md (the test/benchmark harness). Builds clean with gcc and clang under-Werror; no new dependencies.