Johnny Cash on Technical Architecure

Yes, that Johnny Cash aka the man in black. He of the deep voice, great songs, San Quentin concert...
I was having a cup of coffee at a local Starbucks yesterday when a friend showed me an "architectural diagram" of the technology components that a customer of his had shown him. Very proudly, all based on open source (because they don't want to pay license fees) they unveiled this masterpiece that they had taken several years to build.

Immediately I was reminded of this terrific song....

http://youtu.be/5GhnV-6lqH8

We architects do need to work on ensuring a few things:

• Don't overdo the technology
• Open source may be the way to go, but joining disparate things up can get expensive fast
• Ensuring that the pieces can connect (bolts and bolt holes anyone 2:05?)

Decisions under uncertainty

What data do we use to make decisions? Do we rely on the most up to date information possible? Do we use rough approximations? How do we reason about our decision thinking. Taking a concrete example first, I will then look for further insights and, of course, solicit comment.
My favorite example of decision under uncertainty is when I am making a purchasing decision and paying for the purchase by check (cheque!). I want to have some degree of certainty that the check will clear, so I have several strategies at my disposal.

1. Call the bank, tell the bank that I have a particular purchase to make, give the intended cheque number and amount and get some kind of a gurantee that the check will clear (even if my expected paycheck deposit bounces).
2. Call the bank/go to an ATM or whatever, get the current (available) balance, find out which payments/deposits have already cleared and satisfy myself that there is enough for the check I am about to write
3. Call the bank/go to the ATM or whatever and get the currrent balance, ensure it is enough to cover the check I am about to write
4. Look in the check ledger (or home accounting system) and satisfy myself that the balance is sufficient.
5. Assume that because I have checks left, I must have money.

Well I was just kidding about #5. These choices all have some degree of risk (from very little risk in #1 to significant risk in #5). But which one do we choose, and why? I suspect it has a great deal to do with how we perceive the risks. So, for example, if I were working for a large company with a great history of making payroll, the likelihood of a pay check bouncing is very low, so I will act confidently and perhaps take any of the choices 2, 3, 4. However working for an underfunded startup, I may well choose 1. There are lots of other variables in here too. How much does the bank charge for bouncing a check - vs the value of the check. How much reputation (aka credit report) damage might I suffer, etc. So I make a decision. But except in case 1, the decision is somewhat uncertain.

In our daily lives, we constantly act on imperfect data, but we use our experiences to guide us. I am pretty sure that my socks will be in the sock drawer, so will go there first. But if they are in the laundry - I might have to take a corrective action, the risk is low, the actual impact is low, so I will go to the drawer first. Oh, and no, I don't consciously do the risk analysis of every small thing - that would be very counter productive. That's why we have experience! We can act without consciously managing trivial risk.

Now moving into information systems, things look a little different - but should they? We attempt to limit risk of doing the wrong things (including being gamed by clever risk expoitation) by using tight centralized control (aka databases of record, single points of truth, etc.) However, these kinds of systems are inherently fragile. They are also more unreliable than we think. The value of any piece of data  is only valid at the time we request it unless we wrap the request with explicit transaction boundaries. But then if we have to serialize access to data every time we want to use any of it, everything would gum up and come to a grinding halt. So we always use approximations

The "trick" is to realize that you are always retrieving an approximation, realize that much decision making can be relegated to a system of reference, but that when making changes you must affect the system of record. And when affecting the system of record we do need transactions! Transactions give us the "truth" at that moment. The decision to change the "truth" is however very often made on approximations. This isn't bad - it's reality. Of course we can contrive situations where we have to have "truth" just as in my example 1 above, but those cases are more rare than we imagine.

It then becomes an architectural and design principle - what source do we we use in order to make decisions? How does the system of record react when attempts are made to update it based on out of date information? That all plays into the Values and Trust axes relevant to that decision making.

Watching Events

It seems that when thinking about events, we have a tendency to put some of the responsibilities in the wrong place. Of course every time we don't have a proper separation of responsibilities, we get extra complexity. So in this post I will look at some of the issues around the responsibilities and see where they should be allocated.

Short political rant that can safely be ignored now. Why is health care insurance (in the USA) handled essentially through employers and employment contracts? They simply don't belong to each other. The time base is wrong, the administration is wrong, the result is wrong.

End of rant!

I think a little naming context would be helpful here. This will be greatly simplified - just so that the essence is laid bare.

E is an event publisher
E has an output queue onto which itpublishes. This is called EQ
E publishes events of type V1, V2, V3 and V4. Individual events are named v1-1, (the first event of type V1), v1-2 (the second event of type V1), etc.
C1 is a consumer of events - it can consume events of type V1, V2
C2 is a Consumer of events - it can consume events of type V2, V3, V4
C3 is a consumer of events - it can consume all kinds of events.
Each consumer has an input queue - C1 has an input queue C1Q, C2 has C2Q, C3 has C3Q

The event network behaves as follows:

E publishes v1-1. The event network must transfer v1-1 to C1Q and C3Q. C1Q and C3Q. C1 and C3 are now able to process the events by reading their respective queues.

Thinking about the outcome possibilities (assuming E was successfuly able to publish) v1-1.

Both C1 and C3 receive the event v1-1
C1 receives the event v1-1, but C3 does not.
C1 does not receive the event v1-1, but C3 does
neither C1 nor C3 receive the event.

Question for the reader. Where should the responsibility lie in taking action for recognizing that one of these conditions obtains. The possibilities are:
(a) E
(b) C1
(c) C3
(d) none of the above

My answer is (d) - none of the above. It isn't any of these players' responsibility to do this. E's job was to publish the events. C1 and C3's jobs are to handle the events on behalf of their scope. But C1 and C3 are autonomous, so they can't be dealing with each others' failures.

So if it is (d), then there must be some other partcipant - yet to be identified that takes the responsibility. That something else is a proxy for the overall business policy. It needs to exist independently of everything else - and therefore needs to have the proper information fed to it.

Now imagine that C1 in some sense "fails" - it completes, but delivers an exception condition. If that condition is serious enough then something has to know. So it would be sensible C1 were to notify something of its own outcome.

Likewise C2 and C3.

So we have a kind of triplet of notifications (which expand into greater complexity for real sized problems).

E says, "I sent e1-1"
C1 says, "I got e1-1"
C1 says "I handled e1-1 with a normal result"
C3 says, "I got e1-1"
C3 says, "I couldn't do e1-1 - I failed because of a business problem."

So we could then implement a policy rule about what to do under these circumstances. That rule could of course inject new events into the "system." That however is a subject for another time.

Bottom line is that we have a three part system now for handling events. There is the standard pub/sub behavior (and don't get too hung up on the specific technologies). There is the abilty to send out content references using a mechanism that is not part of the event channel (quite RESTful really), and then there is the ability to act on non-receipt or improper handling of the event by one of the event subscribers.

A nice compact model that seperates concerns, so that the individual components can be focussed on their own responsibilities and not prying into each others' business.