123456789_123456789_123456789_123456789_123456789_

Repository layout

This repository ships two gems from one tree:

Cluster code, tests, and CHANGELOG live under cluster/. When making changes that span both, edit both — the cluster gem reuses lib/redis/commands/** from the main gem but has its own client class (cluster/lib/redis/cluster/client.rb) and transaction adapter.

::Redis::XXH3 — pure-Ruby local digest computation

lib/redis/xxh3.rb provides Redis::XXH3.hexdigest, local XXH3-64 digest computation for SET's IFDEQ/IFDNE and DELEX's IFDEQ/IFDNE options (Redis 8.4+ CAS/CAD), so a match-digest can be built without a round trip to DIGEST. It's plain Ruby, always available — no native extension, no build step, no optional flag: an earlier version of this vendored the reference xxHash C library as a native extension, but that made every gem install redis depend on make being present (a RubyGems property of any gem declaring s.extensions, regardless of whether the extension actually compiles anything), so it was replaced with a direct, function-by-function port of the same reference algorithm (https://github.com/Cyan4973/xxHash, v0.8.3 — the version and secret table the server itself uses). Each private method in lib/redis/xxh3.rb is named after the upstream C function it mirrors, to keep that mapping checkable; the 128-bit multiply-and-fold XXH3 needs throughout — the main source of complexity in a C port, which has no native 128-bit integer type — is trivial in Ruby thanks to native arbitrary-precision integers. test/redis/xxh3_test.rb cross-checks known-answer vectors at every algorithm size-class boundary (0–16, 17–128, 129–240, >240 bytes) against a live server's own DIGEST, and test/redis/digest_consistency_test.rb drives real SET/DELEX CAS calls with a locally computed digest end to end.

Common commands

The dev workflow runs Redis in Docker containers via docker-compose.yml using the prebuilt redislabs/client-libs-test image. Topologies are selected by Docker profiles (standalone, sentinel, cluster, all). The makefile is a thin shim around docker compose --profile X up -d --wait and down -v, so the historical target names still work:

# bring up everything: standalone, replica, 3 sentinels, 6-node cluster
make start_all

# run the full suite (all four test groups)
make test

# stop everything
make stop_all

# one shot: start, test, stop
make all

make test shells out to bundle exec rake test, which runs four Rake::TestTask groups defined in Rakefile:

bundle exec rake test:redis        # lib/redis core
bundle exec rake test:distributed  # lib/redis/distributed (client-side sharding)
bundle exec rake test:sentinel     # sentinel-mode tests
bundle exec rake test:cluster      # cluster gem (loads cluster/lib + cluster/test)

To run a single test file or method, use Minitest's options via TESTOPTS:

bundle exec rake test:redis TEST=test/redis/commands_on_strings_test.rb
BUNDLE_GEMFILE=cluster/Gemfile bundle exec rake test:cluster TEST=cluster/test/commands_on_strings_test.rb TESTOPTS="--name=/get/"

The cluster group needs BUNDLE_GEMFILE=cluster/Gemfile (the root bundle doesn't include redis-cluster-client); CI does the same.

Other useful knobs:

You can also drive docker compose directly when you want a single profile up:

docker compose --profile standalone up -d --wait
docker compose --profile sentinel   up -d --wait
docker compose --profile cluster    up -d --wait
docker compose --profile all down -v

The cluster profile's healthcheck waits for cluster_state:ok (not just PING) so tests can connect without hitting the historical InitialSetupError race. Pre-configured sentinel node directories live under test/support/sentinel-config/ and are bind-mounted into the sentinel container; the image's entrypoint starts each as a sentinel because their directory names begin with node-sentinel.

macOS / Docker Desktop note

The compose stack uses network_mode: host so sentinel and cluster nodes report 127.0.0.1 addresses the test runner on the host can reach. On Linux this works natively. On macOS, Docker Desktop's "host networking" beta must be enabled (Settings → Resources → Network → Enable host networking); without it, the containers are healthy but their ports aren't visible on 127.0.0.1. AF_UNIX sockets bind-mounted out of the standalone container also don't route through Docker Desktop's VM on macOS, so test_connecting_to_unix_domain_socket fails locally but passes on Linux CI.

Architecture

Layering

Redis (lib/redis.rb)                ergonomics: keyword DSL, reply reshaping, error translation,
   ├ Commands (lib/redis/commands)  pub/sub second-socket, pipelined/multi wrappers,
   ├ Monitor (lock)                 RESP3→RESP2 fallback, HIMPORT fieldset registry
   └ @client : Redis::Client < RedisClient
                                    ↓
              redis-client gem (external, vendored as runtime dep)
                                    ↓
              TCP/TLS/Unix socket

The Redis class is the public surface. It delegates all network I/O to ::Redis::Client, which inherits from RedisClient (in the redis-client gem) and only adds:

  1. Error translation: maps RedisClient.* exceptions to Redis.* via ERROR_MAPPING in lib/redis/client.rb. Every public method on ::Redis::Client is wrapped in a rescue that calls Client.translate_error!.
  2. A default of protocol: 3 (RESP3) in lib/redis/client.rb; callers can pass protocol: 2. Servers without RESP3 (Redis < 6.0, or anything replying NOPROTO) are detected on connect and Redis#with_protocol_fallback (lib/redis.rb) transparently rebuilds @client for RESP2 — every @client access funnels through Redis#synchronize, which is what applies the fallback, so pipelines/multi/watch fall back too. Return values are protocol-invariant except GEO coordinates (Float under RESP3).
  3. Trivial config delegators (#host, #port, #db, …).

The full command execution flow is: Redis#some_command (defined in lib/redis/commands/<category>.rb) builds an array → Redis#send_command grabs @monitor → Redis::Client#call_v rescues + re-raises → RedisClient#call_v serializes RESP and reads the reply → optional reshape lambda runs → result returned.

Commands as a module composition

Every Redis command category is a module under lib/redis/commands/ (strings, lists, hashes, sets, sorted_sets, streams, arrays, scripting, transactions, pubsub, etc.). Module-provided command families live under lib/redis/commands/modules/ (json.rb for JSON.*, search.rb + search/ for the Query Engine FT.*, whose replies reshape into Search::SearchResult/Search::AggregateResult objects). They are all included into a parent ::Redis::Commands module (lib/redis/commands.rb), which is in turn mixed into:

This is the single most important pattern in the codebase. To add a new command, find the matching commands/.rb and add a method that calls send_command([:cmd, ...]) — that method automatically becomes available on every client type. The mixin classes each provide their own send_command and synchronize so the same Commands methods work in direct calls, pipelines, and transactions.

There is also a catch-all method_missing in lib/redis/commands.rb that forwards any unknown method as a Redis command — so unwrapped commands "just work."

Reply reshaping (protocol-aware)

The top of lib/redis/commands.rb defines a family of lambdas — Boolify, BoolifySet, Hashify, Floatify, FloatifyPairs, HashifyInfo, HashifyStreamEntries, HashifyClusterNodes, … — that reshape raw replies into idiomatic Ruby (Hash, Float, boolean, etc.). They are passed as blocks to send_command:

def incrbyfloat(key, increment)
  send_command([:incrbyfloat, key, Float(increment)], &Floatify)
end

Since 6.0 the client negotiates RESP3 by default, so these lambdas are protocol-aware: they must accept both the RESP2 wire shape (flat arrays, string-encoded numbers) and the RESP3 one (native maps, doubles, pairs) and converge on the same Ruby value. The pattern is "detect the already-final shape and pass it through" — e.g. Hashify returns a Hash unchanged and each_slice(2).to_h's a flat array; FloatifyPairs skips re-mapping when the reply is already [[member, Float], ...]. When adding or changing a lambda, keep both branches, and test the command under both protocols (PROTOCOL=2 / PROTOCOL=3, see below).

Boolify is the exception: it assumes an integer reply (value != 0 unless value.nil?) and would invert a native RESP3 boolean — Boolify.call(false) returns true. That's unreachable for commands that reply with integers under both protocols (the case for all current uses), but confirm the command's reply_schema says "type": "integer" before reaching for it.

When adding a command that needs reply transformation, write or reuse one of these lambdas; do not coerce in the command method itself. See specs/adding-commands.md for the full catalog and the end-to-end checklist (the add-new-command skill automates this flow from a spec file).

Connection lifecycle (long-lived, lazy, fork-safe)

Pub/Sub — separate socket, same process

subscribe / psubscribe / ssubscribe open a second dedicated socket via @client.pubsub (in lib/redis.rb) wrapped in SubscribedClient (lib/redis/subscribe.rb). This keeps the command socket usable from other threads while one thread is blocked in the next_event loop. The subscription loop on the calling thread is synchronous — if you want it off the main thread, the caller spawns a Thread. There's a separate write-monitor on the subscription socket. Sharded pub/sub (SSUBSCRIBE) subscribes one channel at a time to avoid cross-slot errors in cluster mode.

Pipelines and transactions

pipelined and multi both yield a ::Redis::PipelinedConnection (or MultiConnection) that re-includes Commands (lib/redis/pipeline.rb). Each command inside the block returns a ::Redis::Future that resolves when the batch flushes. MultiFuture (in lib/redis/pipeline.rb) splits the EXEC reply array back across individual command futures. Inside a MULTI, blocking commands degrade to non-blocking — that's intentional, matching Redis server semantics.

HIMPORT — server session state with client-side recovery (experimental)

The HIMPORT command family (Redis 8.10) is the one place a command's state outlives a single call: fieldsets are per-connection server session state, destroyed by reconnect/failover/RESET. Redis#initialize keeps a registry of prepared schemas (@himport_fieldsets) and the himport_* overrides in lib/redis.rb repair a lost fieldset reactively — on the server's "no such fieldset" error, re-prepare from the registry and retry the SET exactly once. Recovery is per-fieldset and lazy (there is no reconnect hook); disable with himport_auto_prepare: false, in which case re-preparing is the caller's responsibility (the registry has no public reader). The overrides hold @monitor across command + registry mutation — don't split them. The cluster subclass adds reply aggregation on top (prepare/discard fan out to all masters).

Client identification (CLIENT SETINFO)

::Redis::LibIdentity (lib/redis/lib_identity.rb) reports lib-name=redis-rb lib-ver= on every connection by prepending onto RedisClient::Config / RedisClient::SentinelConfig and appending a second SETINFO pair to the connection prelude (last-write-wins overrides redis-client's own pair; zero extra round trips). The override is gated by a redis-rb_v marker in driver_info, so applications using redis-client directly are untouched. Downstream gems extend the name via Redis.new(driver_info: "my-gem_v1.0.0"); driver_info: false disables identification. This couples to redis-client's connection_prelude — one of the reasons the gemspec pins redis-client to an exact version; re-audit on every driver bump.

Cluster gem differences

cluster/lib/redis/cluster.rb defines Redis::Cluster < ::Redis, so it inherits the full Commands surface but swaps initialize_client to build a RedisClient::Cluster via the redis-cluster-client gem. Cluster-specific differences worth knowing:

Command routings (cluster)

Which node(s) a command goes to, and how fan-out replies are combined, is the driver's job, not redis-rb's. redis-cluster-client routes keyed commands by slot and keyless commands by the server's command tips (request_policy / response_policy from COMMAND INFO), with a built-in routing table for the exceptions (RedisClient::Cluster::Router::RoutingTable). Do not reimplement that in Redis::Cluster by enumerating nodes and calling them one by one — it bypasses the driver's redirection handling, topology refresh and error collection.

When the driver's default routing is wrong for a command, fix it through the driver's command_routings config option rather than in Ruby: Redis::Cluster#initialize_client passes DEFAULT_COMMAND_ROUTINGS (cluster/lib/redis/cluster.rb), a { 'command' => { request_policy:, response_policy: } } hash the driver validates and merges over its table, and caller-supplied command_routings: are merged on top so applications keep the last word. Supported values are the driver's, not the server's full set: request_policy all_shards / all_nodes, and response_policy nil (one reply per node), all_succeeded, one_succeeded, agg_sum (sums Integer replies only). Anything else, e.g. agg_min, is rejected at construction.

Only when no driver response policy produces the documented standalone return value does Redis::Cluster add a thin reply-shaping override (waitaof, himport_*): the driver still does the fan-out and hands back one reply per primary, and the override only collapses that array into the standalone shape. It has to be a method override, not a reply block on the shared command: the driver applies reply blocks per node, before it collects the fan-out, so a block never sees the aggregate. Aggregate the way the server's response_policy tip says (agg_min → per-position minimum, as waitaof does, so thresholds keep their single-node meaning; agg_sum → sum), not the way a sibling command happens to be handled. Guard it with reply.first.is_a?(Array) so a plain reply passes through (a caller who changed the routing, or a later driver that aggregates natively). The missing override does not affect pipelines' reply shape — pipelined/multi route each command to a single node, so the plain standalone shape comes back without any aggregation — but it does not restore the command's meaning there: single-node routing inside a pipeline is chosen by the driver (e.g. any_replica_node_key), not by which node carried the pipeline's writes, so a fan-out command's per-connection guarantee is still lost. Add a cluster test that issues the command inside pipelined to lock in the shape, but don't treat that test as proof the guarantee holds. Prefer contributing the missing policy upstream over growing these overrides. Re-audit DEFAULT_COMMAND_ROUTINGS and the overrides on every redis-cluster-client bump (the gemspec pins an exact version for this reason).

::Redis::Distributed is not Redis Cluster

lib/redis/distributed.rb + lib/redis/hash_ring.rb implement client-side consistent-hash sharding across N independent standalone Redis servers. It is not the Redis Cluster protocol — there are no slot maps, no MOVED/ASK redirects, no automatic resharding. Keys are hashed with CRC32 against an MD5-built ring (160 vnodes/server) and dispatched to one underlying Redis instance. Multi-key commands raise CannotDistribute since co-location isn't guaranteed.

It is separately maintained from Commands — ::Redis::Distributed does not include the Commands mixin; every method is explicitly defined in distributed.rb so it can route to the right node via node_for(key). When adding a new Redis command that should be available here, add a corresponding method to lib/redis/distributed.rb and a test under test/distributed/. The test:distributed Rake task runs as part of the default suite, so missing or broken Distributed implementations will fail CI.

It is supported (recent commits add JSON, arrays (AR*), HEXPIRE/HPTTL, HSCAN, etc.) and not deprecated. For new applications needing horizontal scaling, Redis::Cluster is generally the better choice because the server enforces consistency; ::Redis::Distributed is the right tool when you have N independent standalone Redises and want memcache-style key distribution.

Conventions