Exercise 3: The Primary-Shard-Count Capacity-Planning Gotcha — Possible Solution ==================================================================== THE GOTCHA ITSELF ------------------------------ Per this chapter's own warn-box, "since primary shard count is typically fixed at index-creation time, choosing too few shards limits future horizontal scalability, while choosing too many shards for a genuinely small dataset adds real, unnecessary per-shard overhead — each shard has its own real resource cost (file handles, memory, and more)." WHY TOO FEW SHARDS IS A PROBLEM ------------------------------ Per this chapter's own earlier material, "each shard can live on a different node in the cluster — this is what lets a single index's own data spread across many machines." If an index is created with too few shards, there's a hard ceiling on how far that index's own data can ever be spread across the cluster — with, say, only 2 shards, the index's data can never be distributed across more than 2 machines (one shard per machine, at most), regardless of how many additional nodes are later added to the cluster to handle growth. WHY TOO MANY SHARDS IS ALSO A PROBLEM ------------------------------ Per this chapter, each shard "has its own real resource cost (file handles, memory, and more)." Every shard, regardless of how much data it actually holds, consumes some baseline amount of real system resources just by existing. Creating far more shards than a genuinely small dataset actually needs means paying that per-shard overhead repeatedly, for shards that individually hold very little data — wasted resource consumption with no real corresponding benefit. WHY THIS IS A REAL, UPFRONT TRADE-OFF ------------------------------ Per this chapter, "primary shard count is typically fixed at index- creation time" and isn't easily changeable afterward without reindexing. This means the decision has to be made reasonably well BEFORE knowing, with full certainty, how large the dataset will eventually grow — a genuine forecasting problem with real consequences in either direction (too few, hard scaling ceiling; too many, wasted resources), decided at a point when the full picture may not yet be clear. WHY THIS DOESN'T HAVE AS DIRECT AN EQUIVALENT IN POSTGRES'S OWN "ADD A READ REPLICA LATER" FLEXIBILITY ------------------------------ Per this chapter, this is contrasted against "Postgres's own comparatively low-commitment ability to simply add a read replica later." Adding a Postgres read replica (per postgres1-11's own material) is a genuinely incremental, low-risk decision that can be made and revisited at essentially any time as real, observed load demands it — it doesn't require having predicted the exact right number of replicas from the very start, and getting the initial number "wrong" doesn't carry the same structural, hard-to-reverse consequence primary shard count does here. Elasticsearch/OpenSearch's own distributed-by-design architecture, per this chapter, delivers real benefits (parallel search, horizontal scale), but at the cost of this specific upfront, harder-to-revise capacity-planning decision — "a real trade-off of the distributed-by-default design, not a free lunch," per the chapter's own closing line. WHY THIS WORKS AS AN ANSWER ------------------------------ It explains both directions of the gotcha (too few, too many) using the chapter's own reasoning, and explicitly contrasts the fixed, upfront nature of this decision against Postgres's own genuinely more flexible, revisable replica-scaling decision, rather than treating both systems' own scaling decisions as equivalent in kind.