F/M Problem as a Distributed System ChallengeThe architectural scenario presented, involving a comprehensive, database-backed forest F and a partial, in-memory forest M, is a classic example of a distributed data system. While it may initially appear as a challenge of managing two coupled data structures, a more robust and scalable perspective is to view them as distinct services with specific roles and responsibilities. This shift in perspective is the first and most critical step toward designing a resilient and maintainable system. It moves the problem from the domain of local data synchronization to the well-understood field of distributed systems, providing access to a rich set of patterns and principles designed to handle the inherent complexities of such architectures.
F and M as Distributed ComponentsTo build a robust system, it is essential to reframe the components F and M not as mere data structures, but as specialized services. In this model:
F is the State Persistence Service. It is the canonical source of truth for the entire forest. Backed by a durable database, its primary responsibilities are to guarantee data integrity, durability, and to provide a complete, historical record of the graph's state. It is the system of record.M is the Work Processing Service. It is a stateful consumer that operates on a high-speed, partial, and potentially ephemeral replica of the data from F. Its purpose is not to be a complete mirror of F, but to maintain a "working set" of data that is relevant for immediate processing tasks. Its state is derived from F and optimized for low-latency computation.
This reframing aligns perfectly with the foundational "database per service" pattern common in microservice architectures. In this pattern, each service is the sole owner of its data store, which other services cannot access directly. While M's data is a derivative of F's, M is solely responsible for managing its own state, its lifecycle, and its internal consistency. The challenge then becomes managing the data flow and consistency guarantees between these two services, which is a core problem in distributed computing. By treating them as separate services, we acknowledge that they operate in different failure domains and are subject to network latency, partitions, and independent failures—the defining characteristics of a distributed system.
The initial impulse to solve the information gap—the fact that M has an incomplete view of F—by having the business logic "tell" M about global state changes (e.g., "this fork is abandoned") is a common but perilous approach. This strategy is a manifestation of a well-known distributed systems anti-pattern called the dual write. A dual write occurs when a single logical operation attempts to write to two separate systems—in this case, persisting a state change in the F database and sending a command to the M service—without a mechanism to guarantee atomicity across both writes.
This approach is brittle because it fails to account for the numerous failure modes inherent in a distributed environment. Consider the following sequence of operations for an action that marks a fork as "abandoned":
The interaction can be visualized as a sequence of steps between three components: the Business Logic F including the database (in Flow’s context e.g. the consensus Follower), and the M (Work Processing Service).
F applies an update, e.g. to finalize block A, to its database. The database commits this change.
setForkAbandoned command to the M (Work Processing Service).The failure points, as described below, occur because there is no atomic guarantee covering both Step 1 and 2.
This seemingly straightforward process can fail in several ways:
setForkAbandoned(A) arrives at M before the events that create vertex A and its descendants have been replicated. M has no context for "vertex A" and cannot process the command. It is forced to either drop the command, leading to inconsistency, or implement a complex secondary system for storing and retrying pending commands, which adds significant fragility and complexity.F commits successfully. However, the business logic service crashes before it can send the command to M. The systems are now in a permanently inconsistent state. F knows the fork is abandoned, but M will never be told and may continue to schedule work for that fork.The root cause of these failures is the attempt to enforce atomicity across service boundaries, a characteristic of monolithic thinking. In a distributed system, we cannot assume that two independent systems can be updated as a single, atomic operation. This realization necessitates a fundamental shift away from direct, imperative commands and toward a more resilient, event-driven model.
The solution to the challenges posed by the F/M distributed system is not to try to eliminate the information gap—which is often impossible—but to embrace it as a fundamental degree of freedom that both components don’t need to be perfectly in sync. This means moving away from the strict guarantees of strong consistency and adopting a model of eventual consistency. This model prioritizes system availability and performance by allowing for temporary inconsistencies, with the mathematical guarantee that all parts of the system will converge to the same state over time, provided no new updates are introduced [1,2].
Strong consistency, often associated with the ACID (Atomicity, Consistency, Isolation, Durability) properties of traditional relational databases, guarantees that once a write is complete, any subsequent read will immediately see that new value [3-5]. While this model is simple to reason about, achieving it in a distributed system requires costly coordination protocols that can severely impact performance and availability [7,2].