Narcissistic Applications and Architecture

This post comes out of some work that I am doing with a client. Getting to the essence of event processing and what needs to be in place.
As many have observed, the metadata is often as interesting to the enterprise as the actual data. The trouble is that the enterprise doesn't necessarily know ahead of time what may or may not be interesting. perhaps applications that manage the state of domain objects should tell the world when they have changed the state of a domain object that they manage.
It is only when applications start bragging about what they have done that the enterprise has the ability to draw conclusions that range across the domains.
So while the current state of an interesting domain object may well be locked up in a transactional database somewhere, that the state change occurred could (and should) be made available to any/all interested parties.
Let's think in terms of an intelligent (but fictitious) home environment that we will call the IHE.
Our daily activities in the house include:

• Using hot water 
• Turning lights on and off
• Accessing computers
• Watching TV
• Sleeping
• Opening/closing the refrigerator
• Cooking
• Eating
• Managing the trash
• Managing the recycling
• Filling the dishwasher
• Dressing
• Doing (or having done) the laundry 
• Opening/Closing exterior doors
• ...

I have this feeling that my home bills are too high, so it might be interesting to see if any of my activities are inefficient (I leave lights on sometimes - so we have an event, followed by a negative event), if some of my activities can be co-related. Perhaps a change in one suggests an opposite direction change in another.

Now if, hypothetically, all my activities resulted in events being notified and somehow analyzed, then perhaps (and this is a big perhaps) I have the opportunity to look at my patterns and make some changes that result in savings in time, energy or general annoyance.

Of course we do the obvious ones. When we sleep, running the dishwasher is a no-brainer. But what about multiple uses of the oven? What about leaving lights on? What about leaving doors/windows open correlated with when the heating/cooling are running.

The point is that all of these state changes describe the minutiae of my life and I don't have the time, not the energy to capture them. That detail should be captured at the time it happens - if I am truly interested. It shouldn't wait until after the event when my recollection is hopelessly flawed.

Data ambiguity and tolerance of errors

In the current election "season" in the USA there has been much ado about ensuring that only registered voters are allowed to vote. The Republican Party describes this as ensuring that any attempts at fraud are squelched. The Democratic Party describes this as being an attempt to reduce the likelihood that certain groups (largely Democrat voting) will vote. I certainly don't know which view is correct, and that is not the purpose of this post, but it does inform the thinking.

The "perfect" electoral system would ensure that everyone who has the right to vote can indeed do so, do so only once, and that no one who is not entitled to vote does not. Simple, eh? Not so much! Let me itemize some of the complexities that lead to data ambiguity.

• Registration to vote has to be completed ahead of time (in many places).
• The placement of a candidate on the ballot has to be done ahead of time, but write-in candidates are permissable under some circumstances..
• Voters may vote ahead of time.
• Voters vote in the precinct to which they are assigned (at least in some places)
• Voters may mail in their votes (absentee ballots)

Again they don't appear insurmountable except that the time element causes some issues. Here are some to think about:

• What if a person votes ahead of time, and then becomes "ineligible" prior to voting day. Possible causes include death, conviction of a felony, certifiably insane.
• What if a person moves after registration, but before they vote?
• What if a candidate becomes unfit after the ballots are printed and before early voting? (death, conviction of a felony, determination of status - eg not a natural born citizen
• What if a candidate becomes unfit after early votes for that candidate have been cast?
• .....

These are obviously just a few of the issues that might arise, but enough to give pause in thinking about the process. If we really want 100% accuracy we have a significant problem because we can't undo the history. Now if a voter has become ineligible after casting the vote (early voting or absentee ballot or before the closure of the polls if voting on election day), then how could the system determine that? It would be possible, to cross reference people who have voted with the death rolls (except of course if someone voted early so they could take their trip to look at the Angel Falls where they were killed by local tribespeople and no one knew until after the election).

On a more serious note, voting systems deliver inherently ambiguous results. Fortunately that ambiguity is tiny, but in ever closer elections, it gives those of us who think about systems somethings that are very hard to think about. That is, "How do we ensure the integrity of the total process?" and "How good is good enough?"

Actually that thinking should always apply. While we focus on the happy paths (the majority case), we should always be thinking about what the tolerance for error should be. It is, of course politcal suicide to say that there is error in the voting system, but rest assured - even without malice, there is plenty of opportunity for errors to creep in.

CAP Theorem, partitions, ambiguity, data trust

This posting was written in response to Eric Brewer's excellent piece entitled

CAP Twelve Years Later: How the "Rules" Have Changed

I have copied the statement of the theorem here to provide some context:

The CAP theorem states that any networked shared-data system can have at most two of three desirable properties:

• consistency (C) equivalent to having a single up-to-date copy of the data;
• high availability (A) of that data (for updates); and
• tolerance to network partitions (P).

The original article is an excellent read. Eric makes his points with crystal clarity.

Eric,
I have found the CAP theorem and this piece to be very helpful when thinking about tradeoffs in database design - especially of course in distributed systems. It is rather unsettling to trade consistency for anything, but we have of course been doing that for years.

I am interested in your thinking about the topic more broadly - where we don't have partitions that are essentially of the same schema, but cases where we have the "same data" but because of a variety of constraints, we don't necessarily see the same value for it at a moment in time.
An example here. One that we see every day and are quite happy with. That of managing meetings.
Imagine that you and I are trying to meet. We send each other asynchronous messages suggesting times - with neither of us having insight into each other's calendar. Eventually we agree to meet next Wednesday at 11am at a coffee shop. Now there is a shared datum - the meeting. However there are 2 partitions of that datum (at least). Mine and yours. I can tell my system to cancel the meeting. So my knowledge of the state are "canceled", but you don't know that yet. So we definitely don't have atomicity in this case. We also don't have consistency at any arbitrary point in time. If I am ill-mannered enough not to tell you that I don't intend to show, the eventually consistent state is that the meeting never took place - even if you went at the appointed hour.

I would argue that almost all the data we deal with is in some sense ambiguous. There is some probabilty function (usually implicit) that informs one partition about the reliability of the datum. So, if for example I have the reputation for standing you up, you might attach a low likelihood of accuracy to the meeting datum. That low-probability would then offer you the opportunity to check the state of the datum more frequently. So perhaps there is a trust continuum in the data from a high likelihood of it being wrong to a high likelihood of it being right. As we look at shades of probabilty we can make appropriate risk management decisions.

I realize of course that this is broader than the area that you were exploring initially with CAP, but as we see more on the fly analytics, decision making, etc. we will discover the need for some semantics around data synchronization risk. It's not that these issues are new - they assuredly are not. But we have often treated them implicitly, building rules of thumb into our systems, but that approach doesn't really scale.

I would be interested to hear your thoughts.
PS I have cross posted this against the original article as well.