Why Modular Design Matters in Complex Systems

In large-scale distributed systems like Milvus, coordination components often grow complex as they accumulate diverse responsibilities. The QueryView coordinator in Milvus, responsible for managing load, scheduling, schema lifecycle, and readiness, had become a tightly coupled monolith. This complexity made it challenging to maintain, extend, or even reason about individual parts independently — a common pain point in production systems where scalability and agility are critical.

This article dives into how the Milvus team tackled this challenge by extracting distinct, immutable components within the QueryView coordinator. This change not only improved modularity but also paved the way for more robust scheduling and schema management with clear, stable interfaces.

The Problem: A Monolithic Coordinator

Originally, the QueryView coordinator bundled several responsibilities together:

• Load snapshot handling: Capturing the state of data load and replica distribution.
• Replica allocation and scheduling: Deciding where and how data replicas should be placed and balanced.
• Scheduling cost calculations: Using metrics to reconcile the placement of data efficiently.
• Schema lifecycle management: Tracking schema versions and ensuring consistent reads.
• Readiness protocols: Managing component readiness and synchronization progress.

All these concerns lived in one coordinator module, leading to complex interdependencies. For example, load snapshots were mutable and intertwined with scheduling logic, making it hard to isolate bugs or introduce new scheduling strategies without risking side effects. Similarly, schema management was tightly coupled with readiness checks, complicating schema evolution.

The Fix: Extracting Immutable Components and Interfaces

The core idea behind the fix was to split the monolithic coordinator into well-defined, immutable components with clear interfaces. This separation enables each part to evolve independently and be tested in isolation.

Immutable Load Snapshots and Placement Management

Instead of mutable shared state, load information is now captured as immutable snapshots. These snapshots represent a consistent view of the cluster load and replica placements at a given point in time. By treating them as immutable, the system avoids side effects and race conditions during scheduling.

Scheduling with Cost-Based Reconciliation Using LogicalSize Weights

Scheduling was enhanced by introducing cost calculations based on immutable LogicalSize weights. This abstraction allows the scheduler to evaluate placement decisions quantitatively, facilitating more efficient and balanced replica allocation. The cost-based reconciliation process compares current and desired placements, driving incremental adjustments.

Schema Lifecycle Management with Certified Read Schemas

Schema handling was extracted into its own component that manages certified read schemas and snapshot bindings. This ensures that schema versions are tracked consistently and that reads always operate against a known-good schema snapshot. It decouples schema evolution from load and scheduling concerns, improving reliability.

Readiness Protocols with Scoped Loading and Progress Fences

Readiness checks now use scoped loading and progress fences to track component states precisely. This mechanism helps coordinate startup and recovery sequences without conflating readiness with other logic.

New Interfaces to Decouple Dependencies

To glue these components together while maintaining loose coupling, the PR introduces interfaces like LoadInfoCatalog and PrepareDataViewSchema. These interfaces abstract data access and preparation logic, allowing components to interact via contracts rather than concrete implementations.

Maintaining Legacy Support While Removing Unreleased Formats

The change preserves legacy recovery paths to ensure backward compatibility with existing deployments. At the same time, it removes support for unreleased or experimental formats, reducing maintenance overhead and potential confusion.

Lessons Learned: The Power of Clear Boundaries

This refactoring highlights the importance of splitting complex coordination logic into immutable, well-defined components. Clear interfaces enable teams to reason about each part independently, facilitate testing, and reduce the risk of unintended side effects.

For engineers working on distributed systems or large coordination modules, this approach encourages incremental integration of new features without disrupting existing workflows. It also paves the way for future optimizations, such as more advanced scheduling algorithms or schema versioning strategies.

Conclusion

The Milvus team's extraction of QueryView coordinator components exemplifies how thoughtful modularization can tame complexity in distributed systems. By embracing immutability, cost-based scheduling, and interface-driven design, they improved maintainability and extensibility — a valuable blueprint for similar challenges elsewhere.

Credit to the original PR in the milvus-io/milvus repository. Read the full breakdown on nossletter.tech.

Originally published on Nossletter.