2011 The year of the event

Todd Biske, on his excellent Blog has offereed the imperative, "Let's make 2011 the year of the event" That is a truly wonderful cause. I am shamelessly taking the idea and adding some of my own thinking here. I urge you to read Todd's original post, though.

I think we have had misgivings about eventing models since the early EAI days. With the advent of EAI we worked with events as integration mechanisms. Without attempting to do anything about the underlying systems, and without a sensible taxonomy.


Now perhaps as we think seriously about events and event models there are 2 big ideas that we need to consider.

The first is that when some business object changes state, it should let "the world" know. Think Business Object trigger - here's a link back to 2008 on that.

The second is situational awareness, Here we really need to have a proper mechanism for understanding the effects of an event across a rabge of "observers" might be. It becomes a question of responsibility. When multiple "observers" see the same event they may all choose to handle it. However they are likely to be inconsistent (not necessarily any harm in that), some might claim ignorance (you never told me that), some might have taken different action knowing how another observer behaved...

Using human communications and human triggered interactions as a model, we need to look at the "situational awareness" aspect and decide what to do about conflicts in understanding, conflicts in behavior

Time Part 2 - Januray 1, 2011

This post follows on from the introduction to thinking about time posted here.

In the previous post, I introduced a few of the nuanaces of time. It isn't about the obvious things like format, timezone, precision, calendar, etc. Those are important and can be the source of errors in coding, but they aren't the worst issues by any means. Of more importance is the meaning of time in a specific context. So to start with. let's look at the kinds of time we might be concerned with. I like to think of time as either a"point in time" or duration. Then associated with times there are various "operations" that are to be applied - dealing with relative time.

So thinking about the common "kinds" of time we see, we might classify this way:

• Points in time
• Real time - coincident (to a fine tolerance) with something else
• Near real time - within the same "transaction" scope as something else
• Actual time - When something happened
• Notification time - When an event was notified. This may or not be the time the event happened
• Observation time - when an observer receives notification that something happened
• Effective time - a point in time when something becomes effective
• Expiry time - a point in time when something is no longer valid
• Cut off time - a point in time when changes are no longer accepted (may behve like an expiry time)
• ?
• Durations
• Validity period - a time duration where something is valid
• Length of time - a time duration where something is happening
• Relative duration - a duration that is defined without there being an actual start time. For example, a half in soccer. A type level concept and not an instance level concept.

Clearly in this intial writing there are some quite nebulous words like "something" or "transaction" or "Event". Perhaps happening would be a more useful word.

Anyhow much of the difficulty in systems comes about because of ambiguity related to time. Something may "happen" causing the place at which it happens to think that the happening has occurred, but it has yet to be observed by other entities. Until it does the other entities cannot know that the state has changed.

This is not just a systems phenomenon, but a physics phenomen too. Light moves at a finite speed, so when we see an object (eg a star) that has emitted light, we are seeing the object as it was at the time the light was emitted, and not as it is "now" - or in the frame of reference of us, the observers. We don't (and cannot) know whether the object even still exists. We make assumptions, of course, but in reality we don't know. In other words there is considerable ambiguity.

The big question then is how to handle such ambiguities/inconsistencies of existence. We could take the "GOD" view. There is only one view of the truth, all questions must be asked of that view and there shall be  no other views.  That's fine, except it does take finite time to ask questions, so even if there is a "GOD" view there is and always will be inconsistency.

We can take a view that allows each observer to have its own "correct" view. But that doens't feel any better. We can never be sure when asking that observer if the same set of data has been included as in another observer's set. So for example, when I am presenting some conclusions, it might be quite reasonable for a questioner to ask if I had included the research that was published in December 2010. The point is that unless we are pretty explicit about what is in and what isn't, it is hard to determine what something actually means.

Ultimately we cop out and attempt to have a GOD view of our own data and then allow for other pockets of data which we "update" by taking extracts from the "GOD" view. That has sort of worked - we batch things together and know that the information was correct at midnight yesterday or other known time.

All changes in systems that work in individual events, where data are not batched, where a system is as up to the moment as possible because we don't know and cannot in today's systems know' what has actually been included and what hasn't.

The taxonomy of time that I presented earlier is really only useful as an analysis set. It helps us as analysts answer questions about the data as the data are purposed and repurposed in the (extended) enterprise. It gives a vocabulary to use when asking questions around a particular happening or duration.

Time Part One - Posted Dec 31 2010

I fear that this is going to be a difficult post. But I feel so passionately about time that I will give it a go. What uses of Time do we need to think about. In some ways it's pretty straightforward, there are instants and durations. But there is often complex interplay between people/systems regarding these.
For example, when I see a program on TV, and it covers something I haven't seen before I make an assumption that it is new. But in reality it probably isn't - others will likely have seen it. I still may tweet or otherwise notify a social group that I saw this "new" programme....... So I find it annoying not to know the time context in which something is published or made available. Of course this allows for jokes like Simon Wardley made at his excellent OSCON presentation.
"...Opens up exciting new prospects for the employment of computers in ways and on a scale that would have seemed pure fantasy only five years ago" Simon was applying the idea to the cloud, but actually the quotation stems from 1966. Without the time context we simply don't know.

And that leads to another area of time complexity - relative time - relative time words. testerday, today, five minutes ago. Of course functions that use relative time are ont idempotent. I get a different answer to "yesterday()" today than I will tomorrow.

As an aside, in synchronous conversation we know what time we are talking about, but in asynch we don't but we still use synch modes. So getting a bit more concrete.

Madame had a tennis game scheduled. I am absent minded. Madame needs reading glasses. On her phone there was a text message from Mary saying, "Are we playing tennis tomorrow?"   Madame asked me to read it to her on Tuesday morning. My assumption was that the tennis game was to be on Wednesday (tomorrow viewed from my contextual frame which was Tuesday morning).  Actually the game was Tuesday because Mary sent the message on Monday. Madame and Mary had had previous communication on the subject and so had further context. So there are lots of different times and interpretations lurking here. There's the contextual time of each participant, there's the actual time of the event, there's the time the message was sent, there's the time Madame's phone received the message... Much scope for ambiguity.

Our command and control/standardization brethren might seek to impose unambiguous temporal semantics onto tennis scheduling, our family life, our friends.... However we can see that's going nowhere. Standardizing the terms - yeah right.

However as thinkers about systems these things do matter. How do we understand the temporal frames of the various participants including those participants that are systems as well as those that are humans. When do we have to worry about these differences? How do we describe them internally (so that the things that care deal properly), but not to attempt to impose an external semantics onto every participant?

These are hard questions and get to the heart of interaction/integration and what happens in architecture.

In subsequent posts I will lay out some starting ideas around thinking about time (something delicious here - time from someone who is often late....)

What do you know and when did you know it?

In my Architect role at Sabre, I presented the following at theIATA Commercial conference in Istanbul.

“The Joined UP Airline”


.....
I am using two major themes today when discussing the notion of the joined up airline. The first harks back to the Watergate Conspiracy in the 1970s in the USA. The President was asked repeatedly, “What do you know and when did you know it?” The second theme is even older – it is a quotation from the American Humorist and author Mark Twain, “A lie is half way around the world before the truth has its boots on”. These two themes coincide suggesting that carriers “Think like your customers”.

So let’s unpack these thoughts – and I will lead with an example here. The case of the missing bag. The passenger knows the bag is missing only once all the baggage has arrived at the carousel. But you carriers knew a lot earlier. What do you know? That the bag isn’t on the proper flight. When did you know it? When it was not loaded or when it was loaded onto a different flight. That is certainly before the customer can know it.

You now have an annoyed customer with a powerful weapon – the weapon of instant communication. The customer will of course send a message like, “Those idiots lost my bag….again.” That message is heard instantly – before you have had a chance to manage the message. That opinion is heard while you are still handling the problem operationally. Your “truth” is still tying its boots while the negative message is already circulating. Wouldn’t it have been better to use the passenger’s contact data to notify him or her as soon as you knew?. No matter that that the passenger is probably out of electronic communication – at least send the message so that s/he doesn’t have to wait at the carousel. Apologize and offer some kind of trip appropriate/status appropriate recompense. In other words act on the information that you have – and do it as soon as you can.

This approach is at the forefront of Sabre’s SabreSonic CSS solution. That is true Customer Sales and Service. Discover the meaningful happenings and act on them immediately. It looks simple from the outside, but joining up the data is hard. It’s easy for the passenger but hard for systems. The transactional systems are simply not designed (and nor should they be) to do the necessary analysis. However they have the raw information. Making sense out of the raw information and then acting on it in an appropriate way turns the negative into a positive. You may still see the angry posting, but through proactive message management you can do something to handle the “flame”.

Of course this looks like just plain good customer relationship management. And so from a practices point of view it is. Where we have been lacking up to now is to have platforms that collect data as it becomes available, organizes and standardizes it, joins it with other relevant data and delivers that joined up data to action oriented systems according to your policies. SabreSonic CSS does just that. It allows you to do things that you were unable to do before. Imagine having the freedom and flexibility to do business the way you want to. Let’s turn the previously impossible into current and probable. And remember data without action may be interesting but it isn’t useful.
....

The role of channels

On the tnooz web site there is a whole lot of discussion about the pricing of travel through channels and what the responsibilities of the various players are. Specifically the link poses the question, "Should airlines be forced to disclose equal pricing and fees in all channels?"

Now pricing is an interesting and really important part of any business/business strategy, and thence business architecture. The fundamental question of "How do we price?" affects how we market, how we sell, to some extent how we produce. So for example in a business with only a direct sales model, the systems (in the broadest sense) that deal with sales don't have to handle other channels. But businesses with multiple channels have to make the role of the channel explicit in all systems that are concerned with sales.

If the channels don't control the pricing, what use are they? Surely the idea (really simplistically) is for the channels to provide the conduits for products from suppliers to consumers and thus to set the price for the product via that conduit.


To be successful as a channel, you presumably have to price items higher than what you get them for. Not all items - you can do loss leading if you wish. A pretty simple market place really.

The airlines are responsible for setting the cost to the channel, not the price to the consumer - except where the airline is the channel. The role of the airline acting as its own channel is different from the airline operating its own business. The travel business is a bit different from other retail businesses where, typically the manufacturer doesn't operate its own channel. Although when we see store within a store concepts (like the cases where a makeup provider sets up shop inside a retailer) the makeup provider is acting as its own channel. The store in which the make-up provider is operating is simply another supplier and associated costs must be factored in.

Purchasing is different again. In a world of competition the channels should be clamoring for the consumers' business. Through all manner of inducements. As a consumer I can choose which channel I purchase through depending on a number of factors. Just as I can choose whether to shop at Harrods (Cool, easily identifiable status symbol bag) or Walmart (Look, I am a thrifty person in the new economy).

So in the travel business we really do have to separate out the responsibilities of the various parties in the transaction. There are more parties than just the airline, channel, and purchaser involved. Hotels, rental car companies, restaurants, golf clubs, credit cards,.... All have parts to play. Each may have multiple parts - the part as the supplier, the part as the purchaser, the part as a channel. If we don't separate the parts, we can create interesting opacity. In some industries that is very desirable. However in a business oriented towards consumers, the consumers want transparency.

The job of the channel is to provide that transparency.

So any player in the relationship that wants to be in the role of the channel (whether it be a traditional travel agent, an airline, a hotel, an online travel agent,...) has to act like a proper channel and take the proper channel responsibilities. Simply put:

* Acquire "inventory" at the proper time/price

* Package acquired "inventory" into offers

* Offer the acquired "inventory" under terms that are attractive

* Accept offers to purchase such inventory
 
*  Fulfill those offers
 
It gets a bit tricky because there are actually multiple channels involved in the end. I can purchase the basic travel through, say, Travelocity or Orbitz. But when I eventually travel, I may end up buying something directly from the airline itself (say a meal). Actually, at that point the airline is acting as a channel for a meal preparation company, so it can choose how to set its prices.
 
In order really to understand a complex, network business, we architects really need to think through the roles in the ecosystem, and pay attention to, and manage business architectures that support the pricing models that matter.
 
When there is turbulence, aim for flexibility. When there is stability, aim for simplicity.

Controls and Trust

I am appalled as I look at systems in various companies with whom I have consulted, or who have employed me at the lack of system controls in key places. If you are in the data delivery business and you have agreements with your customers, wouldn't you want to know that you are meeting your service level agreements? Or better still when you are not going to (for whatever reason) and be able to issue warnings, do something about it, or whatever?

Similarly when looking at flow through from one system to another, can you reasonably be assured that everything that was supposed to be processed was?

Do you count your cash after going to the ATM. Maybe the machine didn't deliver correctly because a couple of notes were stuck together. Maybe a new software version caused a miscount under some weird circumstances. The ATM is a "black box" to me. That means that at its boundaries I have to decide what my trust relationship with it will be.

So when I have systems which are supposed to communicate in some way (e.g. by passing data) what controls should be in place to make sure everything is properly accounted for? Should a sending system keep a count of what it has sent? Should receiving systems similarly keep track? How do we reconcile? Should the reconciliation be in-band? Should it be out-of-band? Is logging adequate? Do we have to account for the "value" of the transmission as well as just counts? What tolerances matter if we are concerned with value (perhaps one system rounds off the value differently from another so at the end of the day the total value has a discrepancy)?

This need for controls is exacerbated by systems that use Events as the primary means of notification. Because at the individual event level we can indeed count, maintain value, etc. But often the controls need to be at an aggregate level. One would think in, for example, an airline boarding system that as long as every boarding event is properly received by the "flight", then the system should be in balance. Try telling that to Easyjet. There is a manual control system whereby the Flight Attendants actually count the number of passengers on the plane and attempt to reconcile that with the "expected" number. How the expected number is derived, I have no idea. It could be simply the number of boarding cards collected - but what about electronic boarding? It could be the "system's" view of how many bums there should be on seats. Whatever it is it doesn't appear to be reliable. Chris Potts (Twitter @chrisdpotts) told me the story of what happens when the count is wrong. they recount, they look for people in bathrooms, they delay the flight. It's all a mess.

In the 1960s when phone phreaking was at its peak, people could make free calls because the control signals (tones) for managing the connection system were on the same band of the infrastructure as the call itself. So when a signal tone was detected (and you could get whistles to generate these tones), the system went into a signalling state. By signalling the correct sequence you could generate the sequence to make free calls. Simple fix - put the controls out of band with what you want to transmit.

In a properly reliable infrastructure, the appropriate controls should be built in from the beginning. Again, you may ask, "What's this got to do with Enterprise Architecture?". I argue that it has a great deal to do with the architecture of the enterprise. Good controls make for good compliance and a high level of confidence in our business practices. Bad controls can make your corporation star in places you don't want to be - the front page of the WSJ, in anecdotes among the social networks, resulting in a loss of confidence in your organization.

Observations on Silos

In this posting Richard Veryard  examines the value of silos and silo thinking in organizations. In some ways silos are necessary organizational mechanisms because they place boundaries on functional activities. However they are also creatures of organizational politics - each silo is headed by "someone important". At the European  Enterprise Architecture Conference 2010, Alec Sharp  observed that silos have negative connotations and that perhaps we should use the term, "Cylinders of Excellence". That at least sounds more empowering.

Often, however, the success metrics of a silo are actually in conflict with those of the organization. Well intentioned success metric/motivational policy in a sales team (e.g. "Orders taken in the last week of the quarter will pay extra commissions") will in all likelihood have exactly the wrong outcome. Commissioned sales force team members are actually incented to do the wrong thing - to sandbag until the last few days of the quarter. This has potential downstream effects. It places extra burden on finance and other back office "systems" (silos?). It doesn't allow the organization to get a true picture of sales performance through the quarter. Somehow the miracle always seems to occur and some huge deal comes in at the end of the quarter to "rescue the quarter", keep the investors happy, commission the sales team...

So as architects we should look at the effects of the silos on organizational goals. Moving the silos around is likely to be a bad idea. Powerful people sit at the tops of silos and the only time to effect change at that scale is when new senior management take over. So rather than being perceived as people who are trying to undermine power, enterprise architects need to be seen as people who facilitate the planning and execution of the enterprise.

A fundamental (inside out) question then is, "How do the goals/reward systems in the current silos ensure that the goals and reward systems of the enterprise? are met" In the example above, the Sales silo would argue that it is helping the enterprise meet its goals by making sure that targets are indeed met. However, there is nothing in the enterprise goals that says "at whatever cost to the rest of the enterprise".

Enterprise architects are fundamentally enterprise level thinkers and thus must be thinking about the enterprise (and possibly the extended enterprise) as a whole. We must be able to understand the implications of conflicts in the enterprise - where Value systems in the silos are either at odds with each other, or at odds with the value systems of the organization as a whole.

Ugliness in the sink

An architectural fundamental is that things that are not related should not cause failure on each others' part. The printer breaking should not stop the processing of orders. It does stop the printing of orders (duh), but should not stop the processing.

Some of the systems I deal with put me in mind of a sewage system. Bear with me here...

So, imagine the following undesirable situation. You are standing at the sink after dinner washing the dishes/pots/pans etc. It had been raining very hard all day, so the drainage systems had been well overtaxed. You are startled to see stuff coming UP the drain (perhaps from your neighbors house - you can tell what they had for dinner too). This is clearly an undesirable effect.

On further analysis you realize that the drainage system couldn't take the runoff fast enough, so your house, being lower than your neighbours, became the lowest point their effluent could reach. Oh dear.

Of course restuarant codes (at least in the USA) have a cure. They make sure there is a buffer zone between the sewage system and any sink where food is prepared. It is an airgap and a sufficiently large gap that any attempt by effluent to rise up will be nullified. It will spread all over the floor instead. Not much better, but definitely not contaminating food in a sink.

So what's the lesson here? Denial of Service through increased load on a system is something to watch out for. make sure you understand the implications and design accordingly.

This is as important at a business level as it is at a technical level. Who hasn't experienced a lack of planning - perhaps an airline offering a fare sale and not scaling enough to have call center representatives. Or the fiasco in Texas where the "business" (the State) offered rebates for trading in old appliances. They underestimated demand horribly.

The enterprise really has to understand its business models and take care to scale appropriately - and communicate that through to all interested and relevant parties

That Can Never Happen

Some of the most ominous words I hear from development teams.

I will illustrate this with a rather contrived example - but one that I hope makes the point easily. No I am not advocating the writing of yet another date handler, but the problem is neatly bounded, well understood and has sufficient complexity to make a good posting.

You might also be wondering what on earth a small coding problem has to do with Enterprise Architecture. I'll get that out of the way up front. It is relevant because we well get to the core of some of the questions around reuse, system development/deployment philosophy, good practices, etc. Not your typical fare in every day EA, but viewing one of the roles of EA as influencer on "development" we have a nice teaching opportunity.

So here's the situation. A team discovers that it needs to handle a variety of date formats and in its environment of choice there isn't a robust date package that has been thoroughly tested. They mostly know the rules (leap years, time zones, Daylight Savings Time, etc.). They also know the source of the data they need to convert/check. It's coming from a system where, "If the date is sent to us wrongly by the source system, then there is a whole lot more wrong than this minor blip. Those issues will have been caught elsewhere." If a statement like like doesn't make you very suspicious, then nothing will. But why is it a problem?

First off, the statement is true - at least in the narrow context. If the source system gets a date "wrong" then indeed this is symptomatic of a larger problem. So far, so good. The developer doesn't do a proper job of checking error possibilities, "Because they can never happen". So if the system is expecting the month to be the three letter abbreviation (e.g. JAN for January, etc.) and it is in English then seeing FEV for February is a problem. There is no English month that starts FEV, but there is a French one. So is the error a typing error (B and V are very close on a standard US keyboard), is it a semantic one - it was supposed to be the French version, or what? Should the developer have to know every language to make sure that all possible month abbreviations are accounted for? Probably not we think. Treat it as an error and move on. But what if the code is badly written and mismatches are not caught because the programmer used some fall through logic and returned DEC for all all invalid months. DEC being the last month and a good candidate for being returned in error, "Because we can never get an invalid month string." OK, perhaps we can't.

And then project #2 comes along. Developer John says to developer Chris, "Didn't you code up that weird date handling routine last year? I want to use it, can you point me to it?" "Sure it is in the project library at...."

So John does some "copy/paste reuse" for a piece of trusted code. After all it has been in production for a good long time. No problems found. Inserts it into his application, all is well and about 6 months later it blows up. Turns out that application 2 was not getting the data from the same source as application 1, so it was possible for invalid data to show up. "That Can Never happen" suddenly became, "How the !@#* DID THAT HAPPEN, I THOUGHT WE HAD TESTED CODE" with recriminations from customers, senior management, Uncle Tom Cobbley and all.

Long story for some short points:

• Just because you reuse something doesn't mean it is tested for your situation
• Copy/paste reuse is often worrying anyway - code handed that way is very context dependent
• Consider the cost of hardening and making the routine a service of some sort.
• Promotion of an item to a reusable artifact puts extra stress on development and testing because the more general corner cases have to be considered.
• Governance and management of reusable components is an important practice
• When something is promoted, make sure its assumptions are known and its tests are included with it - that way at least a potential user of the code can see what conditions have been explicitly tested.

And finally, if it genuinely "Can't happen." you can be sure that someone, somewhere will make it happen! So again, make the assumptions explicit. Obvious isn't it (especially with hindsight)

Openness, opensource, lock in, the downfall of authority

In the last couple of days a few related ideas came floating by. It started when Madame was bemoaning behavior in her profession (she is a University lecturer). Then I got to thinking about Open Source - at the behest of a customer. This led down the path of no longer being able to dictate because you are the authority.
You might wonder where this is going, so let me elaborate.
In school there is a certain amount of concern that the students are tweeting, surfing the net, multitasking, getting dates on Facebook or something like that. The gut reaction of the authoritarians is to find a way to punish the students. Of course in some ways the lecturers have the ultimate authority - dropping a grade - but that is patently unfair. Madame, of course, came up with the practical observation - make the classes interesting enough and they won't be distracted. That does require effort on the part of the lecturer though.
And now to open source. There is a fear that making something open will lower the switching cost and therefore customers will leave. So, the argument goes, increase the lock in by a proprietary method and tie the customers down. The Madame principal, of course, is to provide enough value so that the customers won't want to switch. That of course also requires effort.
We have much anecdotal evidence that lock-in of any sort is despised - sometimes in the software world a reason not to buy. In the current times, where education has become dialog, where software acquisition i also a dialog, we must be prepared to engage directly as equals and not assume positions of authority "because it always worked that way."
There are few excellent companies that actually can get away (at least for a while) with their authoritarian (aka my way or the highway) stance. That happens when the perceived value of the item is so great that the proprietary nature is irrelevant (Apple anyone?), but the advantage can erode quickly when another competitor enters the market (even if that competitor is no more open). The spat in the pricing model between Amazon and Apple comes to mind here. It just took MacMillan to have an alternative and to stand up to the lock in through monopoly bully and the market fractures.

Updates Harmful

I have written about this before, I suspect. So forgive me if this is another representation of that resource.

Hanging out with Nigel Green and John Schlesinger is dangerous, so be warned. There be sacred cows being slaughtered in this post.

It all started innocently enough a year or so when Nigel and I were discussing the role of databases (record keeping) vs the role of databases (transaction scheduling, message management, etc.). Faulty normalization came into the picture too. Then the harm done by data modeling (where we thought we could model the rules in data). Large scale data modeling efforts requiring significant investment and it becoming very hard to see where the return comes from. Then an aha. If we go back a bit in time, the worst office job used to be "filing clerk." An imaginary discussion when someone comes home from work, "Well dear, what did you do at work today?" "We opened a new file for Acme enterprises. That meant that our index cards were all messed up because we had run out of space for companies starting with A, so we had to rewrite some of those cards, but while we were there we saw that some cards pointed to files that no longer existed so we removed those - but only after we had added more space for the letter A companies (which we didn't need right now anyway.)" "That's nice dear, how about a nice cold beer?"

The point is that filing clerks used to be the lowliest members of the office - and yet in their electronic reincarnation they have acquired really expensive care and feeding. Of course the new clerks are the databases, the expensive care and feeding is manifested by having a group of thugs (DBAs) who hold everyone to ransom with their talk of normalization, SGA, proper keys,.. All things which we did pretty easily with clerks. So what's going wrong? What is normalization for?

Taking normalization first - it is simply for ensuring that we don't get update anomalies. That something that is to have the same value regardless of usage actually does have that value. You don't have to have a normalized database to ensure that the update anomalies aren't present. Although it is a bit easier.

What is going wrong, is in many ways a harder question. One fundamental thing going wrong is that we use the "filing cabinet" as a scratch pad. So returning to the physical world for a bit. Let's imagine a filing cabinet in which we store the active accounts (perhaps bank accounts). When someone wishes to open an account, we give them a whole bunch of forms to fill in, they go off and fill them in and hand them back to us. We transcribe those forms and do some checkup on the data contained. Once we are happy with the data, we can now give the stuff to the filing clerk and have the clerk create the new file folder. So where were the forms and the checking? In some kind of "blotter" or case management pile on the clerk's desk. They weren't in the active accounts cabinets. And nor should they be.


No we go to a computerized system. We enter the data from the completed forms into the system and "poof" they create an active account. But actually it is more insidious than that. We go through a series of screens putting in different bits of the account - each leading us to a more perfect account, but we aren't there yet. Eventually they will be in the active accounts database (but probably with an inactive flag) so that they can sometime be transacted. This is nuts. We are using the record keeping database (aka the filing cabinet) to manage work in process. This is not a proper separation of duties.

It gets worse. The company decides to "go online". Expensive consultants are hired, golf outings are scheduled, expensive dinners eaten and the "new account workflow" is eventually unveiled. It, too is a sequence of steps. However, the poor schmuck filling this in has to complete each page of the form before moving on. That means that s/he cannot stop for a break - store a scratchpad version of this, do it out of sequence because they can't remember their spouse's Social Security number or whatever. The people in charge of the design of the system understand that THE SYSTEM needs accurate record keeping, have heard that "it is ALWAYS better to validate the data at the point of capture" and other platitudes, but forget that at the end of the line there is the poor user. For these kinds of data entry systems, (and a whole host of housekeeping systems) we need to store the "process state" separately. Don't use the state of the key entity as a substitute for that. Store where I am in the account opening in the account opening process, not in the entity that represents the account.

So what got this diatribe really going? The notion that updates are unnatural - and probably harmful. I posit that the reason that we do updates is mostly because the common need for retrieval of something is the most recent version of it. So it makes sense to have access to the most recent version and update in place. But that isn't always the most expedient behavior. Certainly the most recent value is often the value you need - especially in an operational system. However more and more systems really need the ability to look back. Even something as simple (looking) as you medical record is not something you want to update. Patient History is key. We don't need to know the current cholesterol level (in isolation), we need its trend. So we don't just update the "cholesterol value" in the patient record. We add a new item for the cholesterol and keep the history. We keep the record sorted in time sequence so we can see the latest. We don't just overwrite the value. Our uses of data are so unpredictable, that simply updating database arbitrarily is going to give us data loss. We don't know in advance how serious that data loss might be. Perhaps it would be better to assume that we will need everything and come up with a scheme that at some backbone level ensures that the current view can be reconstructed by replaying the operational events.

Musings with John Schlesinger

John Schlesinger is an event thinker par excellence. So whenever I get the chance, I visit him in London to validate some thinking - or just to spend time with a terrific guy! So on a recent trip to London the subject turned to the rise of event thinking and the downplaying of the traditional SOA patterns. Of course the SOA traditions are being reborn to encompass the events brigade, but because SOA is so broadly and imprecisely defined that's perfectly OK. The SOA hype is over, long live the SOA hype. But that's perhaps a topic for another time.

The key observation from my lunch with John was one I had suspected, but was not able to frame properly. With a few well chosen sentences John had it for me.

This is all concerned with orchestration and control. So (deep breath), here goes. Where an event is raised and that event is to be processed by some subscriber, any intent to orchestrate the handling of the event by the subscriber results in a massive increase in complexity. (Roger Sessions will love this!). Naively one starts to think you have the "OK/Not OK pair" of possible responses. But then the "Not OK" responses blossom out of control. We have situations where the "Not OK" response must result in the retransmission of the event (and how does that happen?) and other cases where it must not. We have cases where the originator of the event has to interpret the behavior of the recipient. That sounds like some awfully nasty coupling to me. So instead of thinking that one has the "OK/Not OK" duality from the recipients view point actually what you have is the"OK/{set of lots of possible not OKs which the sender has to know about} multi-trality. In short that's just crappy design!

Thanks John

Zachman, Frameworks and EA

This post comes out of a quick, but deep, conversation with @cybersal after the first dinner of the architect irregulars twittergroup at Gopals on 20091125. Other members in attendance were: @richardveryard, @taotwit, @Rsessions, @mattdeacon, and @hstrover. As is often the case, when a bunch of EAs get together, the subject of Frameworks comes up. And whenever we discuss frameworks, the venerable Zachman framework is mentioned. Often with much facial contortion and questions like "How do you actually build it?" or "What are in the interesting bits between the rows?"

And then as @cybersal and I were hoofing it back to Charing Cross - avoiding the crowds where possible, the Framework (at least thinking about the titles of the rows) simply gives a context for discussion. You don't really need the columns. So, for example, when thinking about schemas that business services might use in communication, you are working at "row 3". This tells you as much about what you are NOT supposed to be doing as what you are supposed to be doing. It is a really nice shorthand when one is talking to another EA - since EAs have typically all read or heard John Z. So it isn't about using the Zachman Framework as a "Methodology" (whatever that means) but more of a classification system. If you like a set of membership rules.

Now just because you have a set of membership rules, that doesn't mean you have to have the formal club (and if you are Groucho Marx, "I don't care to be a member of any club that would have me as a member" - but I digress). So, no you don't have to instantiate all the rows of the framework and figure out the mappings between them. However you can say to someone, "Come out of Row 4 and think in Row 3." That is in itself a powerful and useful observation, but doesn't really move EA forward much.

IT Profession? I think not

Recent tweets from @rsessions, @richardveryard, @j4ngis,@cybersal have been looking at how hard various professions are. @richardveryard's observation that "@j4ngis @oscarberg Rocket science isn't even particularly complicated. Goes up, comes down. It is rocket technology that is complicated." in a tweet this morning reminds of a conversation had on the golf course with a very good dr. Let's call him John.
John is, as I have said, a very good doctor. His speciality is anesthesia, but his passion is technology. He is always coming up with schemes to invent solutions to make dr.' s live easier. So much so, that he would probably prefer to do that than what he is trained to do.
So after a particularly inept (actually about normal for us, but inept by anyone else's standards) round of golf, we were trudging wearily back to the 19th. hole when John announces yet another good idea - linking wireless technology, handhelds, voice transcription, remote printing,.... His question to me was, "How hard can this be?".
My response was something along the following lines.
John you are breaking my heart. You are essentially saying that anyone without a modicum of training, experience, expertise, but just with the passion and the idea can bust into my field and take over. Have you no respect? Imagine the situation being reversed. Be me for a day, and I will be you. After all, how hard can it be to administer anesthesia to a patient. You figure out the necessary cocktail, inject it and out they go. I can imagine that there might be a few kinks along the way - like making sure that they wake up - but we can leave that to iteration 2. He was, of course horrified. He asked if I was trying to imply that my chosen line of work was as disciplined as his profession. And for the most part, it probably isn't. The key is I don't work in a profession by any normal definition.
So while we are not in a profession, then any enthusiastic amateur can build "cool stuff". Who cares about the error cases? Who cares about the edge conditions? It is all about the app after all. To take a phrase from the movie industry, "We can fix it in post."
Who cares that the patient lives? Who cares that the patient suffers a quality of life decline? In medicine when we have post - it usually means post mortem. There is no fixing it in post in medicine.

A rant on "SOA Projects"

The appearance of the SOA Manifesto has led me to look closely to the naming of projects and the implications of such names.
Time and time again I see projects titled or described with a technology or architecture in the name. How often do we hear, "The SAP project failed?" It isn't because the software doesn't work, there are a host of other potential reasons - all having to do with the human factors. Likewise with "SOA Projects."
The SOA community (huge generalization here) talks about "SOA Projects." Hogwash, I say. There are very few "SOA Projects." There are and should be many projects where the underlying approach is Service Oriented. There are very good reasons for deploying SOA in the enterprise/division or wherever appropriate. The deployment of SOA governance, technologies, etc. might be considered a SOA project, but creating a business application according to those tenets doesn't make that business application a "SOA Project."
Does this really matter? Isn't calling it a SOA project a convenient shorthand? Isn't calling it a SOA Project a convenient way of getting to the right funding bucket? Well, if that's the way the business operates, I, begrudgingly, guess so.
I think it is more insidious than that, however. By putting the technology or architecture top dead center in the name, it gives us an opportunity to make that the primary goal. Rather like hearing the requirement, "I need a database that..." Well databases are fine things, but a requirement that leads us to a "Database Project" again focuses on the wrong things.
So next time your SAP project failed, ask yourself the question, "Is it SAP that failed? or did the business not realize the anticipated benefits for other reasons?" It's easy to blame the technology.

Technical Debt

There seems to be a theme developing in these posts. I start to see everyday situations and then relate them to everyday happenings - then attempt to do some analysis. Today's is no exception.

Ward Cunningham introduced us to the concept of technical debt in a 1992 experience report. Quoting from wikipedia, "Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite.... The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise. " An excellent follow on blog post from Steve McConnell (at http://blogs.construx.com/blogs/stevemcc/archive/2007/11/01/technical-debt-2.aspx ) expands on the point.

I am going to think a bit more prosaically! I was sitting at the computer minding my own business yesterday afternoon, when from behind me I heard a crash. A picture had fallen off the wall, spreading glass shards everywhere. On closer inspection, I saw that the method of attachment to the wall was inadequate. It was a simple nail, not a proper hanger. What's this to do with technical debt you may ask? Well, presumably when the picture was hung, the hanger (perhaps me) chose a quick and dirty solution - a nail. I could perhaps have used the proper fixture - but that may have entailed considerably more "development effort". I would have had to drive to Lowes or Home Depot to buy the fixture, install it properly... you get the point. Oh and of course there was a deadline. We had a party that night and had to have the picture hung.

That was about 8 years ago. Eventually the hanging approach failed and I now have to pay back the debt for my slapdash method of 8 years ago. The repair cost will far exceed the cost of "doing it right in the first place." However, I made the decision to do it the way I did because I didn't have a good handle on what the longer term implications would be and because I had a deadline. These are both key observations.

When we build systems we often don't actually know what the long term implications might be. We don't for example know what "long term" actually means. We often can't explain why doing it right is more cost effective "eventually". I, for example, wasn't prepared to say to madame, "I'll need to go to the store to get the right fixture, hold the party until I come back and put up the picture." Nor was I prepared to say, "let's not have that feature in the house until after the party."

The point here is that we often make conscious choices about the way we do things. These may be suboptimal in the long run - and they will come back to bite us. However we must suborn the future to the current to make sure things get done.

What is more clear to me is that we understand the operational implications of our choices. We need to build our solutions, "hard enough". We need explicit statements about the "-ilities", so we can apply the right amount of engineering. Let's make sure we get the failure stories as explicit as the success stories when putting our plans together - when deciding what goes into an iteration. Make sure we incur the technical debt for the right reasons.

Watching Events

It seems that when thinking about events, we have a tendency to put some of the responsibilities in the wrong place. Of course every time we don't have a proper separation of responsibilities, we get extra complexity. So in this post I will look at some of the issues around the responsibilities and see where they should be allocated.

Short political rant that can safely be ignored now. Why is health care insurance (in the USA) handled essentially through employers and employment contracts? They simply don't belong to each other. The time base is wrong, the administration is wrong, the result is wrong.

End of rant!

I think a little naming context would be helpful here. This will be greatly simplified - just so that the essence is laid bare.

E is an event publisher
E has an output queue onto which itpublishes. This is called EQ
E publishes events of type V1, V2, V3 and V4. Individual events are named v1-1, (the first event of type V1), v1-2 (the second event of type V1), etc.
C1 is a consumer of events - it can consume events of type V1, V2
C2 is a Consumer of events - it can consume events of type V2, V3, V4
C3 is a consumer of events - it can consume all kinds of events.
Each consumer has an input queue - C1 has an input queue C1Q, C2 has C2Q, C3 has C3Q

The event network behaves as follows:

E publishes v1-1. The event network must transfer v1-1 to C1Q and C3Q. C1Q and C3Q. C1 and C3 are now able to process the events by reading their respective queues.

Thinking about the outcome possibilities (assuming E was successfuly able to publish) v1-1.

Both C1 and C3 receive the event v1-1
C1 receives the event v1-1, but C3 does not.
C1 does not receive the event v1-1, but C3 does
neither C1 nor C3 receive the event.

Question for the reader. Where should the responsibility lie in taking action for recognizing that one of these conditions obtains. The possibilities are:
(a) E
(b) C1
(c) C3
(d) none of the above

My answer is (d) - none of the above. It isn't any of these players' responsibility to do this. E's job was to publish the events. C1 and C3's jobs are to handle the events on behalf of their scope. But C1 and C3 are autonomous, so they can't be dealing with each others' failures.

So if it is (d), then there must be some other partcipant - yet to be identified that takes the responsibility. That something else is a proxy for the overall business policy. It needs to exist independently of everything else - and therefore needs to have the proper information fed to it.

Now imagine that C1 in some sense "fails" - it completes, but delivers an exception condition. If that condition is serious enough then something has to know. So it would be sensible C1 were to notify something of its own outcome.

Likewise C2 and C3.

So we have a kind of triplet of notifications (which expand into greater complexity for real sized problems).

E says, "I sent e1-1"
C1 says, "I got e1-1"
C1 says "I handled e1-1 with a normal result"
C3 says, "I got e1-1"
C3 says, "I couldn't do e1-1 - I failed because of a business problem."

So we could then implement a policy rule about what to do under these circumstances. That rule could of course inject new events into the "system." That however is a subject for another time.

Bottom line is that we have a three part system now for handling events. There is the standard pub/sub behavior (and don't get too hung up on the specific technologies). There is the abilty to send out content references using a mechanism that is not part of the event channel (quite RESTful really), and then there is the ability to act on non-receipt or improper handling of the event by one of the event subscribers.

A nice compact model that seperates concerns, so that the individual components can be focussed on their own responsibilities and not prying into each others' business.

A new twist on the Taj Chaat process...

For those who have seen the previous post on process improvement at my local Indian chaat
restaurant will probably be intrigued on yesterday's twist.

We were buying some appetizers to go. In this case 2 samosas and an order of golgappe. Normally the process (at least for dining) is to write the items on a form, be assigned a number and given the vibrating pager to let us know when ready. Collect food, eat, check out by handing over the pager so they could locate the items to be villed from an accordian file.

For take-out, the process is very different. Again I fill out the little 2-part form. But before handing it over to central oredering, I take both parts of th3e form to the cash register. I pay for the order. Both parts of the form are stamped paid. I take the form back to central ordering where I am given the a number and a vibrator. The order is prepared, vibrator goes off and now what? I go to the station to pick up the order, but since the collection point of the vibrators is the cash register how do I return the vibrator?

This is made harder because the worker at the cash register is a different person from the one I paid. So there is no session state anywhere. I have a vibrator that has vibrated and a cashier who expects to take cash. She can't find the form parts in the accordion file.....

So why is this important?
Again seeing how non-IT organizations think about the processes that run their own businesses makes us aware that we shouldn't be overengineering. There are exceptions - things that don't follow the normal path that we just have to work around. Of course each workaround decreases efficiency, but if they are sufficiently rare, then the overall efficiency is not decreased by having work arounds. However, if the workarounds do become cumbersome, expect them to be worked on without negatively affecting the frequent path activities.

Oh, and by the way yes the golgappe and samosas were worth it! The whole bill was $6.56!

Processes and events

Another day of talking to Nigel Green - thank you Skype! And some thinking around processes and their relationship with events. Again sounds innocent - but it seems as if both of us strongy event-oriented thinkers come to common ground when thinking about processes and orchestration - namely that while you might use low level messaging semantics for implementing processes, event modeling doesn't really help when trying to model processes. However and here the lightbulb began to flicker dimly, the result of executing of a process or process step can become a source of events.

We chose an example from the airline industry - and from our experiences of being travelers. Not from any great insights from the internals of the business. The focus was the check in process at the airport itself.

Clearly we see interesting policies at different places and for different carriers. For example at Mumbai (at least time I went through there), they seal your bag with some kind of security strap. So it can be seen whether the bag has been tampered with. That is less common in the USA. However, at Miami International Airport when I went through a month or so ago, I saw the ability to wrap the bag in a kind of cling wrap. I presume that can be done elsewhere too. That is all by way of background.

Airlines nowadays can and do charge fees for checking baggage. All of the rules require that checked bags undergo a security check. Bags are subject to weight limits. Passengers are subject to bag limits (no more than n per passenger). Ignoring further complexity like whether actually to collect the fees (elite passengers are exempt, for example) there are some quite interesting process decisions to be made.

If the airline chooses to impose the bag fee immediately that the passenger offers the bag for checkin, then there may be some undo logic if the passenger decides not to check the bag after all. It is worse though. Imagine that the bag is too heavy. When is that discovered? For example if the passenger has checked in (and been charged) at a kiosk, then on presentation of the bag it is discovered to be too heavy so an extra collection is made - that could make the passenger decide not to check it after all, so a refund is in order. Or perhaps the passenger may decide to open it and remove some of the heavier items, and get it to the correct weight. That's fine - but what if it has already been secured with tape or wrapped in a cling wrap. Kinda tricky to undo...

Then there is security. Another opportunity for the passenger to open the bag and remove things if they should not be in checked baggage. And so it goes.

Different airlines and different jurisdictions will implement the Policies - "maximize revenue for bags", "make sure the passengers' possessions are safe", and "transport passengers safely" with different process paths. Those paths need to be orchestrated. It isn't clear how an event network will really help that orchestration. In fact I would go so far as to say it complicates it. However at each step of the various process steps (or sub-processes), it would be very useful to spit out an event that provided useful (possibly actionable) information to trigger some other behaviors.

For example, if during the security screen a weapon were found, we would expect an event to be raised to trigger a whole raft of other processes. We would be jumping outside a process domain into another domain. That of airport behaviors to criminal behaviors. So looks like a terrific event.

Even the mundane events may be interesting to somebody. That a passenger decided not to check the bag after a fee was assessed can be helpful when looking at the behavior as a whole for market and planning purposes. Opportunities for process improvement abound.

The weight of the checked luggage is also useful for "weight and balance" on the aircraft. Necessary so proper takeoff parameters can be computed, proper fuel calculations can be performed, etc. So the event raised as a result of successful baggage check-in is quite a handy event to have.

So bottom line, it seems from this (and a whole raft of other possible examples) that we will typically see events being generated as a result of a process step happening - at least when in a process.

Of course there are lots of other ways of generating events, we don't need to formalize processes so that events can be generated. Relatively random behaviors give rise to events too.

Event and Content

This conversation with Nigel started innocently enough. Two people who have very similar views talking about architecture in the large. We described our current problem spaces - they looked very similar, but as is often the case we used the same words to mean different things. Of course once we realized this we got back on track quickly. I'll write this from my perspective, but in reality it would be better written by a dispassionate observer.

I set out to describe the separtion that I am seeing in VPEC-T especially around P-E-C. In my definition, I was thinking of Content as being everything about the Event. I was merrily proceeding down this path, thinking that the view was shared until Nigel spoke. We realized simultaneously that the word covered several concepts. There is the "stuff" about the event itself - the event properties like when it happened, what kind of event it is, what channel it was communicated on - essentially a kind of tag cloud for the event. I suppose in the current vernacular this might be the event meta-data. There is also the "state" that exists as a result of something doing a piece of work and generating the event. That may well need to be made available somehow. So for example, in my sales system the "event" of a sale being made has something useful like when it happened, however there is a whole lot of other information like who the parties to the sale were, what the value of the sale is, who the sales team is, what was sold... - in other word the "business document" representating the state as perceived by the event emitter.

In the Chris world, the tags (event meta data) were part of content. In the Nigel world they aren't part of content per se. Actually that is unfair, they aren't ONLY content. In other words, the event meta data can easily travel with the event. But the state information typically won't.

Reversing that point of view for a second, we get the notion that there is definitely a content "store" somewhere that the processor of the event might go to get the contents, and an event store - somewhere the event data would be stored. The content store doesn't have to be explicitly created by the event creating component, it could equally be some external source (e.g. a weather forecast). The event store's job is to store the events (duh) so that they can be replayed for a variety of reasons:

• An event processor has been out of the loop for a hour and wants to catch up "get me all the sales events for the last hour"
• A simulation wants to be done using some "live" state. What would the impact be if we had decided to close this airport at 2pm yesterday? So inject a new event into the event store and then replay forward with the existing events after that point. Of course there will quickly be diversion from the reality as processed, but that is OK. We have been able to run the simulation with a known starting point

Considering the second bullet above, it is quite convenient to use the same semantics for processing "missed events" as for processing the events in real time. Kind of rule - never pass the content with the event, always pass a reference to content, but make sure that the event tag (meta-) data do come with the event. Make the event store readable so that the event sender isn't responsible for hanging on to the event, and having to remember who has received it and who hasn't

At the end of our time together on this call we were, as expected, in agreement that for a proper separation of concerns, we should treat content as being the state data and the event store as being the place where the events and associated metadat are stored to allow for retries and simulations (among other things).

The Policy part of the call will come next.....