Akka.NET Message Delivery Guarantees

The default messaging guarantee of Akka.NET actors is "at most once" message delivery, also known as "fire and forget" messaging; under this guarantee we promise that a message is delivered to an actor between 0 and 1 times. "Fire and forget" is a good default because it's stateless and therefore inexpensive and fast.

However, where this guarantee often falls short is when the question of reliability is introduced, and this usually occurs in the context of sending messages across process boundaries with Akka.Remote and Akka.Cluster. Under these circumstances, what are our options? What can we do to strengthen our Akka.NET actor's resiliency to network partitions, crashes, unexpected actor terminations, and more?

The answer lies in picking a stronger messaging guarantee through the use of more robust actor messaging protocols, such as the AtLeastOnceDeliveryActor base class introduced in Akka.Persistence.

More robust messaging guarantees introduce additional costs and complexity in our Akka.NET actors, and it's extremely unusual to need anything stronger than "fire and forget" messaging for in-memory actors that are all running inside the same local process. Therefore, typically we only use AtLeastOnceDeliveryActors and other actors which implement stronger message delivery guarantees when we're sending messages across the network.

At-Least-Once Delivery Protocols

The AtLeastOnceDeliveryActor in Akka.NET's Akka.Persistence module is pretty straightforward to use, on the surface.

In this model the AtLeastOnceDeliveryActor is the sender of messages that need to be reliably delivered and it accomplishes this goal by assigning a unique correlation ID to each message that is scheduled for delivery. From that point onward, the AtLeastOnceDeliveryActor will reattempt to deliver all outstanding messages once every five seconds by default.

The AtLeastOnceDeliveryActor base classes in Akka.Persistence use what's known as a "acknowledgment protocol," where every message is given its own unique correlation ID and the sender needs to send a "receipt" back which acknowledges that the message with that specific ID was successfully processed.

Once the sender replies with a confirmation message, the AtLeastOnceDeliveryActor will mark that message as "delivered" and never attempt to deliver it again from that point onward.

The syntax for implementing the Deliver side inside an AtLeastOnceDeliveryActor is straightforward:

Command<Write>(write =>
{
    Deliver(_targetActor.Path,
        messageId => new ConfirmableMessageEnvelope(messageId, PersistenceId, write));

    // save the full state of the at least once delivery actor
    // so we don't lose any messages upon crash
    SaveSnapshot(GetDeliverySnapshot());
});

And so is the receive and ConfirmDelivery side too:

Command<ReliableDeliveryAck>(ack => { ConfirmDelivery(ack.MessageId); });

NOTE

Click here to see the full source code for this Akka.Persistence.AtLeastOnceDeliveryReceiverActor sample.

Problems with Akka.Persistence.AtLeastOnceDeliveryActor Implementations

However, while this code is simple to setup, there are some issues with the approach used by the AtLeastOnceDeliveryActor base classes in Akka.Persistence.

These are the issues that we aim to solve through the use of the Akka.Persistence.Extras NuGet package

Duplicate Messages

The biggest issue with all at-least-once delivery protocols is the possibility of duplicates, which can occur for any number of reasons (delays in processing, confirmation message fails to get delivered, back-ups on the sending side, etc...).

We've created the DeDuplicatingReceiveActor as part of Akka.Persistence.Extras and solves this problem through the use of a configurable message de-duplication strategy that is built directly into the DeDuplicatingReceiveActor base class.

Learn more about how to use the DeDuplicatingReceiveActor class to implement "exactly once" message processing strategies in Akka.NET.

Out of Order Messages

At-least-once delivery protocols do not respect message ordering, out of the box. This is because if an AtLeastOnceDeliveryActor receives messages A, B, and C it will attempt to deliver all three messages immediately.

However, if message A fails to get processed and acknowledged by the receiver due to a brief, temporary network partition but B and C are processed successfully, then effectively the receiver was unable to process those messages in order - which can create consistency problems in some cases.

This problem will be solved in the future via the ExactlyOnceStronglyOrderedDeliveryActor inside Akka.Persistence.Extras.

Cumbersome Persistence Strategy

The AtLeastOnceDeliveryActor base classes do not persist their delivery state by default, therefore these actors are vulnerable to "forgetting" their cumulative delivery state in the event of a termination, restart, or crash. So it's incumbent upon end-users to remember to save the actor's delivery state programmatically via SaveSnapshot(GetDeliverySnapshot()); calls periodically and this can result in a rather tedious exercise in writing boilerplate code. Beyond that, however: you have to save the entire delivery state whenever any of it changes, which can be expensive and inefficient.

This problem will be solved in the future via the AtLeastOnceDeliveryActorV2 inside Akka.Persistence.Extras. You can read more about the current AtLeastOnceDeliveryActorV2 proposal here.