Sharding & Replication — Distributed by Design

search1-1 claimed this engine is distributed by design from day one. This chapter delivers the actual mechanics, contrasted directly against two real prior chapters on this site.

Shards — Splitting an Index Across Nodes

A single index is split into multiple shards — and each shard is itself a complete, independent Apache Lucene index, the real payoff of search1-2's own namecheck of Lucene as the underlying engine both Elasticsearch and OpenSearch are built on. The number of primary shards for an index is set at index-creation time, and in most real-world default configurations isn't changeable afterward without reindexing — worth naming honestly, though newer versions have added split/shrink APIs to help with this in some cases.

Each shard can live on a different node in the cluster — this is what lets a single index's own data spread across many machines, and what lets a single search query execute in parallel across multiple shards and nodes simultaneously. This parallelism is part of why search1-7's own aggregations can stay fast even over huge datasets.

Replicas — Redundancy Built the Same Way

Each primary shard can have one or more replica shards — exact copies, kept in sync, living on different nodes than their own primary. If a node holding a primary shard goes down, one of its replicas is automatically promoted to primary, and the cluster keeps serving both reads and writes with no manual intervention required.

This is a real, direct contrast with postgres1-11's own replication material: Postgres's own streaming replication requires deliberate setup — a tool like Patroni or repmgr for automatic failover, per that chapter's own honest material. Here, replica-based redundancy and automatic failover are built into the cluster's own core coordination logic from the start, not something layered on afterward with a separate tool.

Contrasted Against mongodb2-6's Own Sharding

mongodb2-6 covered sharding as a real, mature, but genuinely optional capability, requiring deliberate shard-key design decisions, added specifically when a deployment outgrows a single replica set — a real, substantial engineering decision layered onto an already-complete single-node-capable system.

Here, per search1-1's own throughline, there's no equivalent "un-sharded" mode to fall back to as the default, normal way of running the system. Every index is already conceptually shard-based from the moment it's created — even a tiny single-node dev setup with one shard and no replicas is still, structurally, a one-shard cluster, not an un-sharded system. The same "distributed as the default assumption, not an optional upgrade" pattern search1-1 named abstractly, now shown concretely against a specific, real comparison point.

Cluster Coordination — How Nodes Agree

A cluster needs a way to elect and maintain a "cluster manager" node, responsible for coordinating cluster-wide state — which shards live where, overall cluster health, and so on. Modern versions use a Raft-like consensus protocol for this. The kind of problem being solved here — getting multiple independent nodes to agree on shared state without a single point of failure — is conceptually related to the same class of problem postgres1-11's own replica-promotion material and mongodb2-5's own replica-set election material both deal with.

A Practical Consequence — Cluster Health

A real, checkable operational signal: green (all primary and replica shards allocated), yellow (all primaries allocated, but some replicas aren't — a genuinely single-node dev cluster is permanently yellow, since there's nowhere to place a replica), red (some primary shards themselves are unallocated — real data may be unavailable).

Hands-On Exercises