Watching Events

It seems that when thinking about events, we have a tendency to put some of the responsibilities in the wrong place. Of course every time we don't have a proper separation of responsibilities, we get extra complexity. So in this post I will look at some of the issues around the responsibilities and see where they should be allocated.

Short political rant that can safely be ignored now. Why is health care insurance (in the USA) handled essentially through employers and employment contracts? They simply don't belong to each other. The time base is wrong, the administration is wrong, the result is wrong.

End of rant!

I think a little naming context would be helpful here. This will be greatly simplified - just so that the essence is laid bare.

E is an event publisher
E has an output queue onto which itpublishes. This is called EQ
E publishes events of type V1, V2, V3 and V4. Individual events are named v1-1, (the first event of type V1), v1-2 (the second event of type V1), etc.
C1 is a consumer of events - it can consume events of type V1, V2
C2 is a Consumer of events - it can consume events of type V2, V3, V4
C3 is a consumer of events - it can consume all kinds of events.
Each consumer has an input queue - C1 has an input queue C1Q, C2 has C2Q, C3 has C3Q

The event network behaves as follows:

E publishes v1-1. The event network must transfer v1-1 to C1Q and C3Q. C1Q and C3Q. C1 and C3 are now able to process the events by reading their respective queues.

Thinking about the outcome possibilities (assuming E was successfuly able to publish) v1-1.

Both C1 and C3 receive the event v1-1
C1 receives the event v1-1, but C3 does not.
C1 does not receive the event v1-1, but C3 does
neither C1 nor C3 receive the event.

Question for the reader. Where should the responsibility lie in taking action for recognizing that one of these conditions obtains. The possibilities are:
(a) E
(b) C1
(c) C3
(d) none of the above

My answer is (d) - none of the above. It isn't any of these players' responsibility to do this. E's job was to publish the events. C1 and C3's jobs are to handle the events on behalf of their scope. But C1 and C3 are autonomous, so they can't be dealing with each others' failures.

So if it is (d), then there must be some other partcipant - yet to be identified that takes the responsibility. That something else is a proxy for the overall business policy. It needs to exist independently of everything else - and therefore needs to have the proper information fed to it.

Now imagine that C1 in some sense "fails" - it completes, but delivers an exception condition. If that condition is serious enough then something has to know. So it would be sensible C1 were to notify something of its own outcome.

Likewise C2 and C3.

So we have a kind of triplet of notifications (which expand into greater complexity for real sized problems).

E says, "I sent e1-1"
C1 says, "I got e1-1"
C1 says "I handled e1-1 with a normal result"
C3 says, "I got e1-1"
C3 says, "I couldn't do e1-1 - I failed because of a business problem."

So we could then implement a policy rule about what to do under these circumstances. That rule could of course inject new events into the "system." That however is a subject for another time.

Bottom line is that we have a three part system now for handling events. There is the standard pub/sub behavior (and don't get too hung up on the specific technologies). There is the abilty to send out content references using a mechanism that is not part of the event channel (quite RESTful really), and then there is the ability to act on non-receipt or improper handling of the event by one of the event subscribers.

A nice compact model that seperates concerns, so that the individual components can be focussed on their own responsibilities and not prying into each others' business.

Event and Content

This conversation with Nigel started innocently enough. Two people who have very similar views talking about architecture in the large. We described our current problem spaces - they looked very similar, but as is often the case we used the same words to mean different things. Of course once we realized this we got back on track quickly. I'll write this from my perspective, but in reality it would be better written by a dispassionate observer.

I set out to describe the separtion that I am seeing in VPEC-T especially around P-E-C. In my definition, I was thinking of Content as being everything about the Event. I was merrily proceeding down this path, thinking that the view was shared until Nigel spoke. We realized simultaneously that the word covered several concepts. There is the "stuff" about the event itself - the event properties like when it happened, what kind of event it is, what channel it was communicated on - essentially a kind of tag cloud for the event. I suppose in the current vernacular this might be the event meta-data. There is also the "state" that exists as a result of something doing a piece of work and generating the event. That may well need to be made available somehow. So for example, in my sales system the "event" of a sale being made has something useful like when it happened, however there is a whole lot of other information like who the parties to the sale were, what the value of the sale is, who the sales team is, what was sold... - in other word the "business document" representating the state as perceived by the event emitter.

In the Chris world, the tags (event meta data) were part of content. In the Nigel world they aren't part of content per se. Actually that is unfair, they aren't ONLY content. In other words, the event meta data can easily travel with the event. But the state information typically won't.

Reversing that point of view for a second, we get the notion that there is definitely a content "store" somewhere that the processor of the event might go to get the contents, and an event store - somewhere the event data would be stored. The content store doesn't have to be explicitly created by the event creating component, it could equally be some external source (e.g. a weather forecast). The event store's job is to store the events (duh) so that they can be replayed for a variety of reasons:

• An event processor has been out of the loop for a hour and wants to catch up "get me all the sales events for the last hour"
• A simulation wants to be done using some "live" state. What would the impact be if we had decided to close this airport at 2pm yesterday? So inject a new event into the event store and then replay forward with the existing events after that point. Of course there will quickly be diversion from the reality as processed, but that is OK. We have been able to run the simulation with a known starting point

Considering the second bullet above, it is quite convenient to use the same semantics for processing "missed events" as for processing the events in real time. Kind of rule - never pass the content with the event, always pass a reference to content, but make sure that the event tag (meta-) data do come with the event. Make the event store readable so that the event sender isn't responsible for hanging on to the event, and having to remember who has received it and who hasn't

At the end of our time together on this call we were, as expected, in agreement that for a proper separation of concerns, we should treat content as being the state data and the event store as being the place where the events and associated metadat are stored to allow for retries and simulations (among other things).

The Policy part of the call will come next.....

SOA is from Mars EDA is from Venus

It seems that in most Service Oriented Architecture deployments, the predominant paradigm is request/response. Consumer A wants something from Service B. Essentially it is an RPC like model, but with extra overhead. This model has been dominant because of several factors:

• We can think of it like an extension of the earliest programming models we ever dealt with - subroutines and functions
• We don't have to get all snarled up in worrying about parallel processing/multi-tasking and other hard to debug kinds of models
• We can break up problems into conveniently sized chunks by some kind of hierarchic decomposition
• All languages support the model intrinsically.

Does this mean that it is the best way? Perhaps for some problems, but surely not for all. I see a subroutine or function as being a rather surly, uncommunicative programming equivalent of a slob sitting in a chair, swilling beer and only responding when he feels like doing so. Very good at doing one thing at a time, easy to discover (after all who can miss the chair, beer can and TV remote).

This model is so dominant that we have even simulated it on top of asynchronous protocols. So even in a system that doesn't support request/response directly (e.g. an asynchronous, message based system) we still allow for the same semantics, but this time fudging them to use pairs of asynchronous messages. Changing the transport doesn't really change the programming model. We still think in RPC terms.

Enter SOA and we often have the same view. This time instead of an RPC, we sometimes refer to a "Service Call". Immediately we should be on the defensive. Call implies request/response. Should that be what our services do? Well, of course sometimes they should.

Now let's look at things from a different perspective. In an Event Driven Architecture, when something happens, the component that realizes something has happened feels the need to communicate this goodness to anyone who will listen. Some listeners will take action as a result. It is almost a gossiping model, a model where as soon as information is available, it is able to be shared. Not only does sharing take place, but it is absolutely impossible for the originator or broadcaster of the information to control what happens to it. It is out in the aether with all sorts of actions taking place as a result. Some of the actions might be beneficial, some not.

So here, we have a much lighter weight, less surly architecture an approach where the actions are distinguished from the notifications, where the actors are actively listening and making decisions for themselves. An architecture that exhibits autonomy and a loosely coupled style. Of course the trouble with a gossiping style is that a large amount of information is made available, only some of which will be acted on. So it is very "chatty" Chattiness in IT systems correlates well with increased resource consumption. In constrained systems (especially those that are network or disk constrained) increasing resource consumption can have a large, and negative impact on the cost of doing business.

In the EDA model, information "publishers" and "consumers" are constantly doing other things. They are true multi-taskers, ever prioritizing their activities, deciding whether to wait for more information or whether to act immediately. Capable of acting and absorbing information at the same time, making complex decisions largely autonomously. In other words, there is less friction in this style of interaction.

Information, of course, doesn't need to be acted on immediately. A useful tidbit might be recognized a long time after some other tidbit, making a situation clear. The arrival of the final tidbit can trigger a complex set of actions, relying on information disassociated from the original event, but somehow dependent on it.

Bottom line, we need to understand when it is appropriate to use one style or another. Sometimes it is good for a bunch of slobs to hang out, grunt and drink beer. At other times a more nimble, more overlapping, less controlling approach can deliver systems that function more smoothly.