The blog of Alexander Langer

JEEventStore - Design Rationale and Feature Wishlist

2015-07-15T00:00:00+00:00

Since no existing event store for Java fits our needs and integrates well with Java EE, I decided to write our own implementation of an event store, which is straight-forward anyways.

As for the architecture of an event store, Jonathan Oliver’s NEventStore is extremely well designed. In principle, it is exactly what we were looking for — except that it is written in C#/.NET and not Java. NEventStore nevertheless served as an architectural template for our own Java event store, which you’ll surely notice when looking at the public API, see e.g. IStoreEvents.cs and EventStore.java.

At the same time, I did not want to reinvent the wheel. In particular, although NEventStore advertises itself as a persistence agnostic Event Store for .NET, there already is a persistence API that is agnostic to the underlying database in Java EE — JPA. While JPA might mean some performance overhead, it is the standard persistence technology provided by Java EE and therefore widely supported on the containers and databases. Additionally, authentication to the database server and other infrastructure concerns such as connection pooling can be configured directly in the container.

Another (minor) problem regards asynchronicity in the Java EE context: Java EE applications must not create or fiddle with their own threads as these interfere with the container’s resources. Anything that shall be run asynchronous must therefore be done with provided Java EE APIs. On the positive side, the container guarantees thread-safety of EJBs and automatically manages a pool of stateless beans, which can be configured by the server operator. In particular, when running multiple nodes in a cluster, the container can automatically route incoming requests between cluster nodes, i.e., we get horizontal scaling almost for free.

Finally, I wanted to keep the event store modular, such that one can easily exchange, say, the underlying persistence layer (e.g., from JPA to JDBC or some NoSQL store), change the serialization format or add caching layers. Implemented properly, the injection of resources and other EJB beans into the services can be configured by the developers or the server operators as needed. This opens the door for extreme modularity. For example, suppose the team would like to add a caching layer to the serialization engine, since they found that too much time is spent in this part of the application. Easy – simply add a caching decorator of the serialization engine and inject this decorator into the event store.

Feature List

This brings us to the feature wish list I had in mind when designing JEEventStore:

Target platform: Java EE.
No additional dependencies except, of course, for the serialization library of choice and/or the persistence engine.
Events shall not be needed to implement an interface provided by the event store to be stored.
Support multiple buckets in the event store (e.g., for multitenancy or multiple bounded context)
Be able to query all events of a bucket or all events of a given event stream; order guarantee per stream
Be able to query all events in a given event stream
Query specific versions of an event stream
Append-only mode: Store events without requiring to read the object from database (saves a database roundtrip)
Event streams support change sets of multiple events with transaction semantics: either store all events in a change set or none; each committed change set bumps the stream version by one
Optimistic Locking
Modularity, i.e., all infrastructure details can be replaced transparent to the clients; support different serialization and persistence engines, to be configured in the deployment descriptor

Modularity

Some of the modularity features I had in mind:

Cache reads from the persistence layer by adding a decorator to the actual persistence engine
Cache de-serialization of events by adding a decorator to the serialization engine
Add encryption by adding a suitable decorator to the serialization engine
Add other persistence engines beside JPA, e.g., direct JDBC access, MongoDB or other NoSQL engines with suitable atomicity semantics (Redis might be a candidate)
Add other serialization engines beside GSON, e.g., XML, Protobuff, …
All event updasting by writing custom type converters
Ease of use: provide drop-in EJB-jars, which developers can simply include in the Maven configuration to add JEEventStore to their application
be able to write own deployment descriptors for more complex setups with multiple decorators
An event notification system to let listeners know about newly committed events (required for CQRS), which runs within the same transaction, to avoid polling of the database. Support polling for persistence layers without transaction semends or when 2-Phase-Commits with the messaging infrastructure shall be avoided.

Literature on Event Sourcing, Event Stores and CQRS

2014-09-02T00:00:00+00:00

A collection of literature I recommend for reading on Event Sourcing/Event Stores and CQRS, to be extended over time. Ranges from high-level beginner material to implementation details and code samples.

Event Sourcing

Event Sourcing: Entry in Martin Fowler’s bliki
Event Sourcing Basics: Introduction to Event Sourcing in the documentation of (Get)EventStore
Aggregates and Event Sourcing: A+ES: Appendix A contributed by Rinat Abdullin in Vaughn Vernon’s IDDD book (see below).
Introducing Event Sourcing: A chapter in the CQRS Journey e-book
Event Sourcing in a CQRS wiki
Events as a Storage Mechanism by Greg Young
Event Sourcing a la Lokad by Rinat Abdullin
Event Sourcing as a strategic advantage by Jimmy Bogard
A better domain events pattern by Jimmy Bogard
Aggregates and their events by Yves Reynhout
Value objects in an event sourced domain model by Yves Reynhout
Event Sourcing Revisited, Event Sourcing applied - the Aggregate, Event Sourcing applied – the application service, Event Sourcing applied – the Repository blog post series by Gabriel Schenker

Code

NEventStore CommonDomain: Originally by Jonathan Oliver and now maintained as part of the NEventStore project, the CommonDomain is a great foundation for C# applications that want to use Event Sourcing and can also serve as a reference for ports into other languages.
IDDD Sample application: The “Collaboration” Bounded Context of Vaughn Vernon’s sample code for this IDDD book (see below) uses event sourcing, see, e.g., the Discussion AR derived from the EventSourcedRootEntity. This sample can get you started with Event Sourcing in Java very quickly.
Lokad IDDD Sample: the source code for Rinat Abdullin’s chapter in the IDDD book (C#)
.NET Event Sourcing: a lightweight framework for .NET, attempts to fill in the gaps between NServiceBus and NEventStore.
AggregateSource: A lightweight infrastructure for doing eventsourcing using aggregates in C# by Yves Reynhout; see also AggregateSource Testing

Testing

Scenario-based Unit Tests for DDD with Event Sourcing
Axon framework - behavior driven testing - we are not using Axon, but the BDD Given–When–Then pattern presented is quite powerful
Trench Talk: Assert.That(We.Understand()); by Yves Reynhout on BDD testing with ES

Event Store

Building an Event Storage by Greg Young
Your Coffee Shop Doesn’t Use Two-Phase Commit by Gregor Hope
How I Avoid Two-Phase Commit and Removing 2PC (Two Phase Commit): Jonathan Oliver on how NEventStore avoids expensive 2PCs
Your EventStream is a linked list by Yves Reynhout

Code

NEventStore: The implementation of the popular event store for .NET is a must-read: It’s very well designed and the code is easy to understand. It can serve as a template for implementation in other languages — in fact, our own JEEventStore was heavily influenced by it.

CQRS (including CQRS+ES)

DDDCQRS mailing list: Worth reading the archive.
CQRS: Entry in Martin Fowler’s bliki
Eventually Consistent - Revisited article by Werner Vogel, CTO of Amazon
CQRS Journey long e-book on CQRS by a Microsoft team, a must-read
Command-Query Responsibility Segregation: video/slides of a talk by Udi Dahan at QCon
CQRS Starter Kit and FAQ
Clarified CQRS by Udi Dahan
Busting some CQRS myths by Jimmy Bogard — definitely worth reading ]
Why you should be using CQRS almost everywhere… by Udi Dahan
When to avoid CQRS by Udi Dahan
CQRS and Event Sourcing by Greg Young
Why I Still Love CQRS (and Messaging and Event Sourcing) by Jonathan Oliver on benefits of CQRS+ES, many links included
CQRS: Out of Sequence Messages and Read Models by Jonathan Oliver
CQRS and user experience by Jimmy Bogard
CQRS – Use Your Common Sense by Steve Bate
CQRS Event Sourcing: Validate UserName uniqueness: Stackoverflow question on a very common use-case: uniqueness constraints
Eventual Consistency and Set Validation: Greg Young on uniqueness constraints
Secondary indexes and constraints on the write side: Yves Reynhout on uniqueness constraints

Code Samples

Greg Young’s SimpleCQRS example: small C# sample project that let’s one quickly grasp the CQRS principle
Lokad.CQRS Sample Project An advanced (and complex) CQRS sample project
CQRS Journey Code: the CQRS Journey sample application for the e-book
SimpleCQRS by Tyrone Groves (C#)

DDD

Domain Driven Design: The blue book by Eric J. Evans
Implementing Domain Driven Design: A book by Vaughn Vernon, which I can recommend even more than Evan’s original book; if you want to read only one book on the topic, read this.
Don’t Create Aggregate Roots by Udi Dahan
FAQ on Aggregates
FAQ on Sagas

Messaging

The LMAX disruptor: Entry in Martin Fowler’s bliki. The LMAX disruptor a high-performance, single-threaded messaging architecture for the JVM that for a financial application handles up to 6 million order per second. Even if such scalability is way out of scope of most projects, you should be aware of it.
NServiceBus: The most popular service bus for .NET. If you are going to use CQRS+ES in a C# project, you should look at this.

Updates to this post

2015-07-15: Added Gabriel Schenker’s blog series on Event Sourcing.

Building an Event Store for Java

2014-09-02T00:00:00+00:00

A common legal requirement in the area of medical software is that the change history of medical health records must be kept available. With traditional relational databases, we used a one-revision–one-row approach, storing each revision of a record in a single row and having an extra field in the table point to the previous revision. This was working fine but put a significant amount of burden onto the development team.

When I learned about the concept of Event Sourcing, I figured that it fits naturally with these requirements, since reliable change history is given for free; in particular, obtaining the state of the whole system at a given time in the past is almost trivial. (Only much later we learned to love the additional benefits of event sourcing.) For a new project, we therefore decided to go the event sourcing path. For that, we were looking for an Event Store that we can integrate with our technology stack, which is based around Java EE (development done on Mac OS X or Linux and the production servers running Linux).

Existing Event Stores

.NET

The .NET-community has quickly adopted Event Sourcing (most often combined with CQRS, i.e., CQRS+ES), and it is therefore not surprising that there are at least two mature open-source event stores for .NET. The two most popular ones are:

NEventStore, originally developed by Jonathan Oliver, now maintained by Andrea Balducci, Damian Hickey, and Jonathan Matheus.
EventStore by Greg Young and his team

If our technology stack was Windows with .NET, we would most definitely use one of these two, in particular since you can buy SLAs for EventStore. However, .NET is simply not an option for us, since we do not want the vendor lock-in that comes with .NET. (Sorry, Mono, we’re not going to use you in production.) Furthermore, due to Germany’s strict data protection laws, we cannot use a cloud solution. For medical software, local deployments are a must.

Java / JVM

I therefore evaluated the event stores that are available for the JVM. In 2013 (the situation hasn’t change much since), I was aware of the following projects that provide an event store, in some form or another:

Axon Framework
Qi4j
JDON
Eligo Eventsourced, now superseded by Akka Persistence
EventStore2

EventStore2

EventStore2 is a standalone event store, but it’s built upon Akka, whose threading model conflicts with JEE containers. Furthermore, its interface depends on its Event class, meaning you’d introduce a technical dependency on the event store library into your domain model, which is a no-go for a clean architecture. We might have been able to fork the project and refactor this, but the dependency on Akka remains.

Axon, JDON, Qi4j, Akka (Persistence)

None of Axon, JDON, Qi4J nor Akka is a standalone event store in the style of the .NET projects EventStore and NEventStore. Rather, these are frameworks that each solve a different problem:

Axon and JDON are CQRS+ES frameworks, where the event store is necessary infrastructure.
Akka Persistence is a library that provides event sourcing for applications using the Actor programming model of Akka.
Qi4j is an implementation of the DCI paradigm for Java. An event sourcing mixin is provided, which, as of today, lacks documentation. However, adding mixins to Java is arguably magic, at the least non-standard and most probably completely unportable — good luck migrating your application later or introducing new team members.

Consequently, because of their broader focus, Axon, JDON, and Qi4j require you to annotate your domain model with framework annotations, which, as above, means introducing a strong technical dependency into your domain model. In particular, let me stress that your domain model is not even testable without these annotations, i.e., without using the framework. Again, this violates the principles of a clean architecture, where your domain model is supposed to have absolutely no dependencies to “outer” layers (particularly not on infrastructure concerns such as persistence).

Furthermore, Axon and Akka apparently require you to derive your domain classes from certain base classes provided by the framework. To quote a very recent blog post by Dariusz Pasciak in the 8th Light blog, entitled ‘Convenient’ Does Not Necessarily Mean ‘Right’:

If you are using a framework in which you are required to derive from some base class and then call methods on that base class in order to accomplish something that you are trying to accomplish, and the framework designers have provided inheritance as the only way of accomplishing this thing that you are trying to accomplish, and you are comfortable with forming such a strong relationship [footnote: Inheritance is one of the strongest forms of coupling in object-oriented code.] with that framework—then deriving your class from some other class that has methods on it that you would like to use may be right. Dariusz Pasciak

Building an Event Store for Java EE

Due to the lack of suitable event stores for Java that can easily be used in a Java EE environment, we decided to build our own, called JEEventStore. In a series of upcoming posts, I’m going to write up the design rationale for our event store, wish-list and feature set, describe its architecture and will tell you how to use it in your own project.