Enterprise Architecture - A Practitioner's View

Clouds and scalability

2011-12-15T07:50:00.000-08:00

This post comes from an online exchange with Roger Sessions (@rsessions on twitter) Leo de Sousa (@leodesousa) and Chris Potts (@chrisdpotts).

Roger makes the point that the various cloud vendors make their case on "scalability" without defining the term sufficiently. As marketing (almost) always does. So he has a point. The question for me then is, "What scales?". It is my firm commitment that when using terms that you intend to quantify, you had better get the dimensions correct. Is scalability a benefit? Of course that depends on what it means. It feels good, hits us in the unthinking (or as Daniel Kahneman calls it "System1") area. It's only when we look more deeply we realize that we have no idea what it means. Yes I'll have 7kg of scalability please.

It all gets to the economics of what you think you want to do. Here are some examples:

I want to be able to increase the workload that my system is capable of without having to buy, provision, manage a bunch of servers - Scaling for workload
I want to be able to add lots of new users without having to.....- Scaling for users
I want my system to be available and priced according to the actual usage. Kinda like electricity. So when all my users are signing in, I want to allocate lots of capacity because that's intensive. But when it is running along smoothly I need less. Scaling for peak demand
I want to empower a demonstration team so they can bring up new instances of a standardized template and demonstrate something to a customer/prospect and then tear it down while incurring as little cost as possible. - Scaling for efficiency of people
I want to be able to add new functionality with less effort/cost than previously. Scaling for functionality
I want to reduce the burden on in house departments (finance, legal, HR or other "overhead" departments) in the deployment of equipment. - Scaling for organizational effectiveness

While I am about it, I wonder what the effective scaling order looks like. For example, maybe I want to scale linearly for workload. In other words as demand increases, supply increases at the same rate. No effective reduction in cost/transaction.

Or I might be prepared for slightly more - the ratio is for each increase in demand, I get a 1.1 increase in cost of supply.

Or I might want to see a reduction - for each increase in demand, my cost goes down .

So as Roger observed, make the vendors of the cloud services be specific about what they are selling when selling scalability

We call that government

2011-09-09T05:11:00.000-07:00

I was reading this post about QANTAS having to stop on the way from Dallas to Brisbane to refuel several times since starting the "nonstop" service. The service is "direct" from DFW to Sydney - which in the strained parlance of the travel industry means it isn't direct, it just means that you have the same flight number all the way, regardless of the number of stops. But I digress...

The posting got me to thinking about the diminishing returns when you add overhead. To fly further, you have to add more fuel. But adding more fuel doesn't give you a linear increase. You have to add more fuel to compensate for the fuel you had added to make you fly further... At the limit, all you do is fly a plane with just fuel and the necessary flight crew. Nothing useful comes of it - except the corner cases where you are testing limits on purpose. The focus is wrong. It isn't about getting the plane there (again some corner cases like getting the plane to a warzone), it's about getting the passengers and freight there.

The mission gets mangled if the focus on the plane not the passengers or freight.
Similarly in many corporations, if we consider the "running of the compnay" to be equivalent to the fuel, then as we add more "running the company" "resources" so we get diminishing returns.

Eventually we have a company that is dedicated to just running itself, does nothing useful, but everyone is busy.

We call that government

Social Media and CRM

2011-08-17T16:28:00.000-07:00

I am pretty late to the blogosphere about the differences between social media and CRM. But in customer meetings I see this kind of confusion all the time. So here goes. Oh, and since I mainly work in the travel industry, my examples come from there.

So first what are the goals of CRM? Knowing the customer. Being able to have enough insight into the customer to persuade the customer to buy more, become more intimate - generally increase their direct value. Maybe to right wrongs - by providing compensation if bad things happen. So CRM tends to foster a set of 1:1 exchanges. Necessary but not sufficient.

In contrast social media is about somehow leveraging an network of relationships. It's one stage on from CRM (or maybe many stages beyond CRM!). So if I have a bad experience on an airline, I need the airline to do what t needs to do for compensation. But the social aspect doesn't stop there. I am heavily armed with well connected devices, a network of acquaintances and friends and time - especially on the flight. So in some ways the Social Network space is a competition between the airline provider, and the customer. Rach trying to get "the story" out to their social network orbits. The magic happens when the stories coincide - when the CRM aspect of looking after me is also told by the airline and by me. The double dip of great publicity.

But when things go badly for a customer (e.g. my luggage misconnected) then the bad story needs to be acknowledged. Through the social channels - posting on the passenger's FB wall for example, and some compensation placed at the same time. Otherwise the annoyed passenger with lots of time will send out a stream of invective to any/all who may listen.

So when thinking about Social Media realize:

It's a conversation
It's about the whole orbit
Don't forget to do CRM blocking and tackling

Microphone ready events

2011-08-17T12:55:00.000-07:00

I was working with some customers this week, and the topic of "how do you generate events?" comes up. Not how as in the dirty mechanics, but how when something is done manually today, getting that systematized. So, please bear with me on the following example.

Most airlines board their passengers in some kinds of groups. So, announcements are made to get the people in the correct groups to board. A common way to do this is for the gate agent to announce that "Group n" is to board now. Perhaps it would be cool to page all the people in the appropriate group and have them rock up to board. Sounds like fun (and remember folks this is just an illustration). So if you were delivering messages to the smartphone or whever, you would simly send the message. Easy in an event based system?

Not so fast. Current systems probably don't recognize a legitimate event for this. It is procedure and policy (+ experience, queue monotoring, temperature on the jet bridge, screaming children,....) that actually triggers announcement - the what I call the "Microphone event". So how would we systematize that?

Change the gate agent system to do this - and think about the complexity involved, and the myriad of other delivered messages?
Automate the system to do something really clever, like start the boarding process clock at t-30 minutes, monitor the number of boarding pases lifted/scanned (as a proxy for queue depth), and when that number approximates the number of people in the group, then release the next group....
Other thoughts?

This brings up the point, that where we have humans sensing and responding to situations, they will raise events and deliver them through a variety of ways. Sometimes shouting throughh the mic is best of all. But if it does become desirable to do these kinds of things automatically, then use what the people do as cues for the kinds of events that the system needs to generate, sense and respond.

Enterprise Architects - With apologies to Buffy Sainte-Marie

2011-07-28T09:27:00.000-07:00

In her great song - Universal Soldier (sung here by Donovan), Buffy Sainte-Marie describes an impossible ideal for a soldier. Sometimes I think we enterprise architects have to have a similar set of impossible and conflicting attributes. We have to be (as discussed in other posts) archivists and activists), but then we also have to be good listeners (not a common trait among activists), understand the business in totality and yet have a reasonable (whatever that is) understanding of the technology that powers it. We have to be agents of change, and stewards of stability. We have to understand the patterns of interactions and their implications. We have to understand systems concepts (systems of record/systems of reference) and we have to understand the financials. We have to understand some of the legal implications of our decisions and we have to understand risk. We have to understand how to negotiate and when to strongarm.

All in all quite a difficult and conflicting set of characteristics

Irregular is the new regular

2011-07-22T10:24:00.000-07:00

That's a bit of a mouthful - but it is the synthesis of a lot of material from the SITA technology conference held near Brussels in June this year.
So, I'll put in context and then see where it leads us. The context is that in the airline industry, cost cutting has become so critical (tiny profit margins flying planes) and the global situation has become so complex that we see more and more disruption. Then the clod-footed governments step in and put excessive fines onto the airlines for delays and the whole schedule becomes a crap shoot. Airlines have rarely actually operated the flifght schedule they publish - something always seems to go wrong, but with the ever decreasing tolerances, the ripple effects of tiny perturbations can be huge. Layer in larger events (weather, volcanoes, strikes, terrorism..) and we have an unholy mess.

Now in many systems development approaches and methods, we are encouraged to do the "happy path" stuff first. Work on the "normal" cases, deliver those and then do the abnormal things on subsequent iterations. That's great if the normal cases truly are normal. And by that I mean deliver most of the value, and absorb the least cost. But what if there is no "normal"? Then how do we prioritize?

Clearly the "customer" - the stakeholder paying for the solution or on whose behalf the system is being developed will have input to that. But therein lies the rub. There's all sorts of stuff that can go wrong that will potentially affect the architecture and design. And no this isn't enterprise architecture - this is much smaller than that. This is solution architecture - and a classic source of tension in development.

The solution architect has the responsibility for ensuring that the solution is fit for purpose - but the main purpose is in understanding the degree to which change will stress the system. If this weren't necessary then every solution would be a simple client/server solution (yeah, tongue firmly in cheek here).

So understanding what's normal/what isn't. What change stressors will be taken into consideration and still not get all crystal-bally is what makes solution architecture hard. It's also what makes it fun!

Don't be seduced by quick wins on the happy path when the need for the system to absorb change stress is paramount. Don't get all wrapped up in clever system integrity and cunning architecture when you are simply managing transactions - e.g. a recipe file!

Intellectual Honesty, Ivory Towers, pragmatism and other emotive phrases

2011-03-21T06:50:00.000-07:00

This isn't just enterprise architecture. It's about architecture at lots of levels.

Like many posts, this all started with an innocent (hah, likely!) tweet and then became richer as people joined in. Here's the original. RT @SLSingh Bad Science by @BenGoldacre asks "Why don’t journalists link to primary sources?" originally posted by @frodedanielsen (someone I have never met). At which point the irrepressible Alkes Buterman and Doug Newdick chimed in.
aleksb6Aleks Buterman

RT @seabird20 RT @SLSingh Bad Science by @BenGoldacre asks "Why don’t journalists link to primary sources?" http://bit.ly/hkcuY4 >>+1 < +42

19 MarFavorite Retweet Reply

dougnewdickDoug Newdick

@aleksb6 @seabird20 how many #entarch are @SLSingh and @BenGoldacre fans?

19 MarFavorite Retweet Reply

But then Aleks posed the hard question... @seabird20 @dougnewdick I'm a big fan of intellectual honesty (#popper) but isn't that easily confused for "academic" #entarch?

That's what got me to this point - and it made me reflect on framing - using language that manages to demean an "opponent" while at the same time boosting your own value. "academic" is just one such word. Often used by teams that are under great pressure to meet a deadline and seeing that deep analysis could extend the time to delivery, demolish some closely held beliefs, make them look stupid because they forgot something. Use in context, "oh you are architects are so academic, that isn't a concern". While others will be thinking ("yet but it will become one..."). Of course the opposite word, used by those not wishing to do deep analysis is "pragmatic". We have a pragmatic solution.... Thus cutting off possible debate/discussion.
Architects can (and should) find themselves in a difficult position here. They (we) must be seen to be taking the side of pragmatism, and yet ensure that the teams are not taking on excessive technical debt. We need to be asking the important questions around operability of the system, its performance and availability envelope, its need for flex when it is used in unanticpated ways. remembering of course that solutions get used in unexpected ways and in unanticipated quantities.
Make sure we use data and data values to describe the system envelope. How many users? How many transactions? How many transactions per second? What is the reliability/bandwidth of the communication path? What does it cost per transaction to use an external service? What if that service is not available? Can the system be unavailable for 1 minute, 5 minutes, 1 hour...? What does it cost to use a "pet" technology? and so on. Otherwise known as the -ilities. Sometimes called non-functional requirements, but I would argue that they are quivalent to functional requirements. If these are not met you don't have a functional system.

In VPEC-T terms this is a classic conflict of values. The EAs are charged with the rather nebulous protection of the future. The project teams are charged with getting value out there fast. Those value systems are fundamentally at odds. So as architects we have the responsibility to be scrupulous in our questioning, demand the data that supports the project's assertions of cost/risk/time to market. Finally those that write the checks have to decide.
What we architects must not do is put up wooly, ivory towered thinking to prove how clever we are, how stupid everyone else is, how the world will come to an end if the solutions are not developed the way we dictate. We must not be sowers of fud, but analysts of relevant data.

Decisions under uncertainty

2011-03-03T05:05:00.000-08:00

What data do we use to make decisions? Do we rely on the most up to date information possible? Do we use rough approximations? How do we reason about our decision thinking. Taking a concrete example first, I will then look for further insights and, of course, solicit comment.
My favorite example of decision under uncertainty is when I am making a purchasing decision and paying for the purchase by check (cheque!). I want to have some degree of certainty that the check will clear, so I have several strategies at my disposal.

Call the bank, tell the bank that I have a particular purchase to make, give the intended cheque number and amount and get some kind of a gurantee that the check will clear (even if my expected paycheck deposit bounces).
Call the bank/go to an ATM or whatever, get the current (available) balance, find out which payments/deposits have already cleared and satisfy myself that there is enough for the check I am about to write
Call the bank/go to the ATM or whatever and get the currrent balance, ensure it is enough to cover the check I am about to write
Look in the check ledger (or home accounting system) and satisfy myself that the balance is sufficient.
Assume that because I have checks left, I must have money.

Well I was just kidding about #5. These choices all have some degree of risk (from very little risk in #1 to significant risk in #5). But which one do we choose, and why? I suspect it has a great deal to do with how we perceive the risks. So, for example, if I were working for a large company with a great history of making payroll, the likelihood of a pay check bouncing is very low, so I will act confidently and perhaps take any of the choices 2, 3, 4. However working for an underfunded startup, I may well choose 1. There are lots of other variables in here too. How much does the bank charge for bouncing a check - vs the value of the check. How much reputation (aka credit report) damage might I suffer, etc. So I make a decision. But except in case 1, the decision is somewhat uncertain.

In our daily lives, we constantly act on imperfect data, but we use our experiences to guide us. I am pretty sure that my socks will be in the sock drawer, so will go there first. But if they are in the laundry - I might have to take a corrective action, the risk is low, the actual impact is low, so I will go to the drawer first. Oh, and no, I don't consciously do the risk analysis of every small thing - that would be very counter productive. That's why we have experience! We can act without consciously managing trivial risk.

Now moving into information systems, things look a little different - but should they? We attempt to limit risk of doing the wrong things (including being gamed by clever risk expoitation) by using tight centralized control (aka databases of record, single points of truth, etc.) However, these kinds of systems are inherently fragile. They are also more unreliable than we think. The value of any piece of data is only valid at the time we request it unless we wrap the request with explicit transaction boundaries. But then if we have to serialize access to data every time we want to use any of it, everything would gum up and come to a grinding halt. So we always use approximations

The "trick" is to realize that you are always retrieving an approximation, realize that much decision making can be relegated to a system of reference, but that when making changes you must affect the system of record. And when affecting the system of record we do need transactions! Transactions give us the "truth" at that moment. The decision to change the "truth" is however very often made on approximations. This isn't bad - it's reality. Of course we can contrive situations where we have to have "truth" just as in my example 1 above, but those cases are more rare than we imagine.

It then becomes an architectural and design principle - what source do we we use in order to make decisions? How does the system of record react when attempts are made to update it based on out of date information? That all plays into the Values and Trust axes relevant to that decision making.

Requirements "gathering"

2011-02-28T08:13:00.000-08:00

This makes it sound as if requirements are like mushrooms in the forest - you take a small basket, go to some secret place and pick all the things that you hope are edible, bring them home, cook them, eat them and, you hope, not die of food poisoning.

But there is a lot that has to go into the creation of a "proper" set of requirements. This post was inspired by some observations from Nigel Green (@taotwit), Alec Sharp (@alecsharp) with Color commentary from Anne Manes (@atmanes), Kristof Dierckxsens (@kdierc) and Doug Newdick (@dougnewdick). All those @ signs give the twitter ids of the people involved. Mine is @seabird20.

The riff all started with this, innocuous little tweet from Alec...

alecsharp @seabird20 My point - I see far too many cases where BAs gather literally 1000s of requirements but never synthesize them into a solution.

This of course led me to thinking - we have people who have analyst in their title, but almost always need to do some level of synthesis. Analysis and synthesis are often presented as opposites, so what gives? Here, the expertchoice team point out that you can't have a solutiion from analysis alone! Well that's kinda obvious, but still well put.

We typically associate analysis and design (with design taking the sysnthesis role in this counterpoint but it really isn't).

Now back to requirements, why do we have thousands of detailed requirements listed on powerpoints, in word docs, in Excel spreadsheets or wherever? Does it really help us to create so much "electronic concrete?"

The Agile movement has been very helpful in helping is do a much lighter weight job of handling the requirements, but I think that many agile practices also miss the boat.
So what matters?

First off, if I have my architect hat on (not necessarily EA hat), I am interested in those requirements that will make a difference to the structure of the functioning system. House analogy time, what it's going to be made of, and how it will be put together not the color of the curtains. And that plays right back into the excellent book, "How buildings learn" by Stewart Brand. Being concerned with the shearing forces, the forces of change.

(Aside):One golden rule, if stuff is going to change a lot don't spend inordinate time documenting it. A mocked up screen is a much quicker/cleaner artifact than a page of documentation. But that also misses the big point.

Counting things might help you to estimate them...Counting things doesn't tell you some of the essential things that will determine the structure. So let's think about some examples:

You are getting together the "requirements" for an accounting system. Do the following make a difference to the structure? (and therefore have got to be right early)?

It has to be multi-currency
We have to support monetary amounts in the 10s of billions and 3 decimal places
Connectivity to remote users is intermittent
The users don't all speak the same language
We have thousands of accounts to manage
We want to consolidate data entry onto a single screen for account master data
We need to be able to drill down from summary data into the details
The system has to be available for transaction posting from 0600 until midnight
...

Of course these aren't real requirements. Some of them make a big difference to our lives. Multi-currency??? Well if you have ever had to convert a single currency system to a multi-currency you understand those implications. So yeah, pay attention. Size of numeric fields? Not so much. It's pretty much a nit. Pain to change in the formal documentation if you have carefully mocked up a screen layout showing (8,2) and you now need (11,3), that's a lot of editing if you have specified every screenshot. So don't!

Drill down? Well that can be huge because it exposes a level of functinality different from just posting transactions.

Do all of the "requirements" appear at the same time? - Definitely not. Looking at a simple mock up showing postings doesn't show that I have multi-currency. We have to do analysis to discover the individual postings need - and then synthesis to put it back together fully in context.

Story telling is vital here - our analysts have to tell stories, they have to understand the business future state that is being proposed, they have to understand the constraints, and then put all the ingredients together at the right time. The switch for analysts to synthesists - and maybe have to all the time. However as Alec said in a tweet:

"alecsharp: @taotwit @seabird20 I often say a better job title would be "Business Synthesist" but a) hard to say after pints and b) terrible acronym "

and another

"alecsharp @seabird20 @taotwit Without synthesis, you end up with reams and reams of disparate "context-free" requirements."

We are often led to incredibly detailed specifications by our large consulting brethren. They do this as means to managing their profitability not the customer's value. "Oh the risk is too great unless we gather these requirements in excruciating detail, so pay us to do that." And then, "Well YOU didn't tell us so now let's have a change order, so pay for that too."

Requirements might be gathered as discrete little pieces of functional capability, But we need the cookery of synthesis to turn them into a non-toxic meal..

More on what event models can learn from database models

2011-01-18T11:44:00.000-08:00

One of my earliest posts was entitled, "What can SOA learn from RDBMS?" In that posting I was arguing for "Event Triggers" that fire when "interesting" business objects change. Of course, "interesting" needs further definition and so for tha matter does "change". However that isn't where I am going in this post.

Historically in the database world, most systems implemented "pessimistic" locking mechanisms. In a pessimistic approach, access (with possible intent to update) to a data resource could not be obtained unless the requestor were able to get exclusive control of the resource. More recent systems employ an "optimistic" approach whereby the data resource is only locked while it is actually being updated. The semantics are quite different, but the intent is the same - makle sure that we don't have conflicting updates.

In general pessimistic locking delivers absolute guarantees, but may cause throughput reductions or excessive resource consumption. Optimistic locking assumes that there will be little contention and therefore doen't consume resources not hold locks "until it has to". In a system with optimistic locks the update requestor may have to deal with the contention behavior (retrying operations or whatever). In the pessimistic scenario, the requesting application can't proceed unless it is safe to do so.

In eventing models, we have similar trade offs that we need to make. No, they aren't lock based, but they do have implications on throughput. These are all focussed around guranteed sequence and guaranteed delivery. Both of these impose throughput limitations on an event based system. Guaranteed sequence isn't really any kind of guarantee however. It is a guarantee that over a certain time frame, messages will arrive in sequence. However this unpacks into not applying events in consuming systems until the guarantee period has expired in some cases.

Why you might ask? Imagine that you have timestamped events coming in, you may not know that an event that comes in does not have an earlier predecessor that has somehow been held up. So you might choose to wait for the appropriate time interval "just to make sure nothing is coming in later" and then apply all events in proper sequence. Set the event sequence window to 5 minutes and in the worst case you are processing 5 minute batches. That doesn't sound very friendly.

So maybe you think about applying sequence numbers to the events. That's fine (well kind of) unless the sequence number calculator is a central shared resource, in which case there may be contention for that. At least however, you can detect that events are out of sequence because if a receiving system perceives a gap in sequence numbers, then it can wait for the "time out period" or the appearance of the missing sequence number(s) whichever happens first and then apply the updates. This is a bit intrusive to the event creators - managing sequence numbers was probably not in their original plan.

In neither of the above scenarios do we get maximal throughput. They are both intrusive and will deliver quite inconsistent throughput. They are rather akin to the pessimistic locking approach taken in database systems.

Looking at it a different way, we can perhaps detect (externally to the applications) that things have happened out of sequence. maybe using an out of band control infrastructure. Thus if a sender identifies when something was sent (using a messaging infrastructure), the receiver's act of receiving will identify when it was received. If the receipts are out of sequence the infrastructure can alert and trigger actions. The assumptions are that:

Sequence errors are few and far between
It is OK to recognize that the sequence has been violated
The infrastructure is capable of re-delivering the events in an appropriate sequence
There have been no irrevocable side effects as a result of the out of sequence receipt.

Attempting to detect post-hoc is likely (and I have not done the work to prove this yet) to deliver mmore consistent (and higher) throughput than attempting to guarantee, with a slight increase in risk.

This looks to be a fruitful area of research (or if the research has already been done, a fruitful area for some best practice patterns) as we treat 2011 as "The Year of the Event" as Todd Biske proposes.

ROI Based Business Cases and Long Range Value

2011-01-13T09:20:00.000-08:00

In a series of interesting twitter posts @taotwit(Nigel Green), @erikproper, @oscarberg, @jdevoo have been discussing modelling Value - other than monetary value. In VPEC-T (modeling based on Values, Policy, Events, Contents and Trust) the abstract concept of Value recognizes things other than monetary value.

In other posts, I have seen reference (and I am sorry that I cannot cite the references) to a primary role of Enterprise Architecture as the team/approach for encouraging project funding across siloes. It was expressed more elegantly in the other posts, I admit!

These various posts and thoughts about Value led me to thinking about entrapreneurial organizations and risk averse organizations. Some of these arguments are kind of old hat, but in companies where innovation is a core driver (eg apple) the introduction of products isn't based (entirely) on ROI. Compare and contrast with very "mature" organizations where business as usual (in the sense that the company delivers the same products, with the same methods, with the same focus on return) because of shareholder or other leadership demands.

Architects are also sometimes thought to be ivory tower/"build it and they will come" kinds of people. This is an accusation that does have merit - we have been guilty of such sins, but it is also used as a weapon for politcal/organizational purposes. The Trust relationships are key here - because architects are likely to be destabilizers (of the status quo and current power bases) means that there will be a failure to Trust the EAs by those who will be affected/disempowered by innovation.

We see this happening frequently in enterprises where there is a process or organization for innovation. "Innovation will be handled only in the xxx organization". Anyone else onnovating outside that organization will be ignored. That's an intersection along the Value/Trust axes.

2011 The year of the event

2011-01-12T15:52:00.000-08:00

Todd Biske, on his excellent Blog has offereed the imperative, "Let's make 2011 the year of the event" That is a truly wonderful cause. I am shamelessly taking the idea and adding some of my own thinking here. I urge you to read Todd's original post, though.

I think we have had misgivings about eventing models since the early EAI days. With the advent of EAI we worked with events as integration mechanisms. Without attempting to do anything about the underlying systems, and without a sensible taxonomy.

Now perhaps as we think seriously about events and event models there are 2 big ideas that we need to consider.

The first is that when some business object changes state, it should let "the world" know. Think Business Object trigger - here's a link back to 2008 on that.

The second is situational awareness, Here we really need to have a proper mechanism for understanding the effects of an event across a rabge of "observers" might be. It becomes a question of responsibility. When multiple "observers" see the same event they may all choose to handle it. However they are likely to be inconsistent (not necessarily any harm in that), some might claim ignorance (you never told me that), some might have taken different action knowing how another observer behaved...

Using human communications and human triggered interactions as a model, we need to look at the "situational awareness" aspect and decide what to do about conflicts in understanding, conflicts in behavior

Time Part 2 - Januray 1, 2011

2011-01-01T15:38:00.000-08:00

This post follows on from the introduction to thinking about time posted here.

In the previous post, I introduced a few of the nuanaces of time. It isn't about the obvious things like format, timezone, precision, calendar, etc. Those are important and can be the source of errors in coding, but they aren't the worst issues by any means. Of more importance is the meaning of time in a specific context. So to start with. let's look at the kinds of time we might be concerned with. I like to think of time as either a"point in time" or duration. Then associated with times there are various "operations" that are to be applied - dealing with relative time.

So thinking about the common "kinds" of time we see, we might classify this way:

Points in time

Real time - coincident (to a fine tolerance) with something else
Near real time - within the same "transaction" scope as something else
Actual time - When something happened
Notification time - When an event was notified. This may or not be the time the event happened
Observation time - when an observer receives notification that something happened
Effective time - a point in time when something becomes effective
Expiry time - a point in time when something is no longer valid
Cut off time - a point in time when changes are no longer accepted (may behve like an expiry time)
?

Durations

Validity period - a time duration where something is valid
Length of time - a time duration where something is happening
Relative duration - a duration that is defined without there being an actual start time. For example, a half in soccer. A type level concept and not an instance level concept.

Clearly in this intial writing there are some quite nebulous words like "something" or "transaction" or "Event". Perhaps happening would be a more useful word.

Anyhow much of the difficulty in systems comes about because of ambiguity related to time. Something may "happen" causing the place at which it happens to think that the happening has occurred, but it has yet to be observed by other entities. Until it does the other entities cannot know that the state has changed.

This is not just a systems phenomenon, but a physics phenomen too. Light moves at a finite speed, so when we see an object (eg a star) that has emitted light, we are seeing the object as it was at the time the light was emitted, and not as it is "now" - or in the frame of reference of us, the observers. We don't (and cannot) know whether the object even still exists. We make assumptions, of course, but in reality we don't know. In other words there is considerable ambiguity.

The big question then is how to handle such ambiguities/inconsistencies of existence. We could take the "GOD" view. There is only one view of the truth, all questions must be asked of that view and there shall be no other views. That's fine, except it does take finite time to ask questions, so even if there is a "GOD" view there is and always will be inconsistency.

We can take a view that allows each observer to have its own "correct" view. But that doens't feel any better. We can never be sure when asking that observer if the same set of data has been included as in another observer's set. So for example, when I am presenting some conclusions, it might be quite reasonable for a questioner to ask if I had included the research that was published in December 2010. The point is that unless we are pretty explicit about what is in and what isn't, it is hard to determine what something actually means.

Ultimately we cop out and attempt to have a GOD view of our own data and then allow for other pockets of data which we "update" by taking extracts from the "GOD" view. That has sort of worked - we batch things together and know that the information was correct at midnight yesterday or other known time.

All changes in systems that work in individual events, where data are not batched, where a system is as up to the moment as possible because we don't know and cannot in today's systems know' what has actually been included and what hasn't.

The taxonomy of time that I presented earlier is really only useful as an analysis set. It helps us as analysts answer questions about the data as the data are purposed and repurposed in the (extended) enterprise. It gives a vocabulary to use when asking questions around a particular happening or duration.

Time Part One - Posted Dec 31 2010

2010-12-31T09:33:00.000-08:00

I fear that this is going to be a difficult post. But I feel so passionately about time that I will give it a go. What uses of Time do we need to think about. In some ways it's pretty straightforward, there are instants and durations. But there is often complex interplay between people/systems regarding these.
For example, when I see a program on TV, and it covers something I haven't seen before I make an assumption that it is new. But in reality it probably isn't - others will likely have seen it. I still may tweet or otherwise notify a social group that I saw this "new" programme....... So I find it annoying not to know the time context in which something is published or made available. Of course this allows for jokes like Simon Wardley made at his excellent OSCON presentation.
"...Opens up exciting new prospects for the employment of computers in ways and on a scale that would have seemed pure fantasy only five years ago" Simon was applying the idea to the cloud, but actually the quotation stems from 1966. Without the time context we simply don't know.

And that leads to another area of time complexity - relative time - relative time words. testerday, today, five minutes ago. Of course functions that use relative time are ont idempotent. I get a different answer to "yesterday()" today than I will tomorrow.

As an aside, in synchronous conversation we know what time we are talking about, but in asynch we don't but we still use synch modes. So getting a bit more concrete.

Madame had a tennis game scheduled. I am absent minded. Madame needs reading glasses. On her phone there was a text message from Mary saying, "Are we playing tennis tomorrow?" Madame asked me to read it to her on Tuesday morning. My assumption was that the tennis game was to be on Wednesday (tomorrow viewed from my contextual frame which was Tuesday morning). Actually the game was Tuesday because Mary sent the message on Monday. Madame and Mary had had previous communication on the subject and so had further context. So there are lots of different times and interpretations lurking here. There's the contextual time of each participant, there's the actual time of the event, there's the time the message was sent, there's the time Madame's phone received the message... Much scope for ambiguity.

Our command and control/standardization brethren might seek to impose unambiguous temporal semantics onto tennis scheduling, our family life, our friends.... However we can see that's going nowhere. Standardizing the terms - yeah right.

However as thinkers about systems these things do matter. How do we understand the temporal frames of the various participants including those participants that are systems as well as those that are humans. When do we have to worry about these differences? How do we describe them internally (so that the things that care deal properly), but not to attempt to impose an external semantics onto every participant?

These are hard questions and get to the heart of interaction/integration and what happens in architecture.

In subsequent posts I will lay out some starting ideas around thinking about time (something delicious here - time from someone who is often late....)

What do you know and when did you know it?

2010-12-31T07:18:00.000-08:00

In my Architect role at Sabre, I presented the following at theIATA Commercial conference in Istanbul.

“The Joined UP Airline”

.....
I am using two major themes today when discussing the notion of the joined up airline. The first harks back to the Watergate Conspiracy in the 1970s in the USA. The President was asked repeatedly, “What do you know and when did you know it?” The second theme is even older – it is a quotation from the American Humorist and author Mark Twain, “A lie is half way around the world before the truth has its boots on”. These two themes coincide suggesting that carriers “Think like your customers”.

So let’s unpack these thoughts – and I will lead with an example here. The case of the missing bag. The passenger knows the bag is missing only once all the baggage has arrived at the carousel. But you carriers knew a lot earlier. What do you know? That the bag isn’t on the proper flight. When did you know it? When it was not loaded or when it was loaded onto a different flight. That is certainly before the customer can know it.

You now have an annoyed customer with a powerful weapon – the weapon of instant communication. The customer will of course send a message like, “Those idiots lost my bag….again.” That message is heard instantly – before you have had a chance to manage the message. That opinion is heard while you are still handling the problem operationally. Your “truth” is still tying its boots while the negative message is already circulating. Wouldn’t it have been better to use the passenger’s contact data to notify him or her as soon as you knew?. No matter that that the passenger is probably out of electronic communication – at least send the message so that s/he doesn’t have to wait at the carousel. Apologize and offer some kind of trip appropriate/status appropriate recompense. In other words act on the information that you have – and do it as soon as you can.

This approach is at the forefront of Sabre’s SabreSonic CSS solution. That is true Customer Sales and Service. Discover the meaningful happenings and act on them immediately. It looks simple from the outside, but joining up the data is hard. It’s easy for the passenger but hard for systems. The transactional systems are simply not designed (and nor should they be) to do the necessary analysis. However they have the raw information. Making sense out of the raw information and then acting on it in an appropriate way turns the negative into a positive. You may still see the angry posting, but through proactive message management you can do something to handle the “flame”.

Of course this looks like just plain good customer relationship management. And so from a practices point of view it is. Where we have been lacking up to now is to have platforms that collect data as it becomes available, organizes and standardizes it, joins it with other relevant data and delivers that joined up data to action oriented systems according to your policies. SabreSonic CSS does just that. It allows you to do things that you were unable to do before. Imagine having the freedom and flexibility to do business the way you want to. Let’s turn the previously impossible into current and probable. And remember data without action may be interesting but it isn’t useful.
....

The role of channels

2010-07-20T05:23:00.000-07:00

On the tnooz web site there is a whole lot of discussion about the pricing of travel through channels and what the responsibilities of the various players are. Specifically the link poses the question, "Should airlines be forced to disclose equal pricing and fees in all channels?"

Now pricing is an interesting and really important part of any business/business strategy, and thence business architecture. The fundamental question of "How do we price?" affects how we market, how we sell, to some extent how we produce. So for example in a business with only a direct sales model, the systems (in the broadest sense) that deal with sales don't have to handle other channels. But businesses with multiple channels have to make the role of the channel explicit in all systems that are concerned with sales.

If the channels don't control the pricing, what use are they? Surely the idea (really simplistically) is for the channels to provide the conduits for products from suppliers to consumers and thus to set the price for the product via that conduit.

To be successful as a channel, you presumably have to price items higher than what you get them for. Not all items - you can do loss leading if you wish. A pretty simple market place really.

The airlines are responsible for setting the cost to the channel, not the price to the consumer - except where the airline is the channel. The role of the airline acting as its own channel is different from the airline operating its own business. The travel business is a bit different from other retail businesses where, typically the manufacturer doesn't operate its own channel. Although when we see store within a store concepts (like the cases where a makeup provider sets up shop inside a retailer) the makeup provider is acting as its own channel. The store in which the make-up provider is operating is simply another supplier and associated costs must be factored in.

Purchasing is different again. In a world of competition the channels should be clamoring for the consumers' business. Through all manner of inducements. As a consumer I can choose which channel I purchase through depending on a number of factors. Just as I can choose whether to shop at Harrods (Cool, easily identifiable status symbol bag) or Walmart (Look, I am a thrifty person in the new economy).

So in the travel business we really do have to separate out the responsibilities of the various parties in the transaction. There are more parties than just the airline, channel, and purchaser involved. Hotels, rental car companies, restaurants, golf clubs, credit cards,.... All have parts to play. Each may have multiple parts - the part as the supplier, the part as the purchaser, the part as a channel. If we don't separate the parts, we can create interesting opacity. In some industries that is very desirable. However in a business oriented towards consumers, the consumers want transparency.

The job of the channel is to provide that transparency.

So any player in the relationship that wants to be in the role of the channel (whether it be a traditional travel agent, an airline, a hotel, an online travel agent,...) has to act like a proper channel and take the proper channel responsibilities. Simply put:

* Acquire "inventory" at the proper time/price

* Package acquired "inventory" into offers

* Offer the acquired "inventory" under terms that are attractive

* Accept offers to purchase such inventory

* Fulfill those offers

It gets a bit tricky because there are actually multiple channels involved in the end. I can purchase the basic travel through, say, Travelocity or Orbitz. But when I eventually travel, I may end up buying something directly from the airline itself (say a meal). Actually, at that point the airline is acting as a channel for a meal preparation company, so it can choose how to set its prices.

In order really to understand a complex, network business, we architects really need to think through the roles in the ecosystem, and pay attention to, and manage business architectures that support the pricing models that matter.

When there is turbulence, aim for flexibility. When there is stability, aim for simplicity.

Controls and Trust

2010-07-06T06:23:00.000-07:00

I am appalled as I look at systems in various companies with whom I have consulted, or who have employed me at the lack of system controls in key places. If you are in the data delivery business and you have agreements with your customers, wouldn't you want to know that you are meeting your service level agreements? Or better still when you are not going to (for whatever reason) and be able to issue warnings, do something about it, or whatever?

Similarly when looking at flow through from one system to another, can you reasonably be assured that everything that was supposed to be processed was?

Do you count your cash after going to the ATM. Maybe the machine didn't deliver correctly because a couple of notes were stuck together. Maybe a new software version caused a miscount under some weird circumstances. The ATM is a "black box" to me. That means that at its boundaries I have to decide what my trust relationship with it will be.

So when I have systems which are supposed to communicate in some way (e.g. by passing data) what controls should be in place to make sure everything is properly accounted for? Should a sending system keep a count of what it has sent? Should receiving systems similarly keep track? How do we reconcile? Should the reconciliation be in-band? Should it be out-of-band? Is logging adequate? Do we have to account for the "value" of the transmission as well as just counts? What tolerances matter if we are concerned with value (perhaps one system rounds off the value differently from another so at the end of the day the total value has a discrepancy)?

This need for controls is exacerbated by systems that use Events as the primary means of notification. Because at the individual event level we can indeed count, maintain value, etc. But often the controls need to be at an aggregate level. One would think in, for example, an airline boarding system that as long as every boarding event is properly received by the "flight", then the system should be in balance. Try telling that to Easyjet. There is a manual control system whereby the Flight Attendants actually count the number of passengers on the plane and attempt to reconcile that with the "expected" number. How the expected number is derived, I have no idea. It could be simply the number of boarding cards collected - but what about electronic boarding? It could be the "system's" view of how many bums there should be on seats. Whatever it is it doesn't appear to be reliable. Chris Potts (Twitter @chrisdpotts) told me the story of what happens when the count is wrong. they recount, they look for people in bathrooms, they delay the flight. It's all a mess.

In the 1960s when phone phreaking was at its peak, people could make free calls because the control signals (tones) for managing the connection system were on the same band of the infrastructure as the call itself. So when a signal tone was detected (and you could get whistles to generate these tones), the system went into a signalling state. By signalling the correct sequence you could generate the sequence to make free calls. Simple fix - put the controls out of band with what you want to transmit.

In a properly reliable infrastructure, the appropriate controls should be built in from the beginning. Again, you may ask, "What's this got to do with Enterprise Architecture?". I argue that it has a great deal to do with the architecture of the enterprise. Good controls make for good compliance and a high level of confidence in our business practices. Bad controls can make your corporation star in places you don't want to be - the front page of the WSJ, in anecdotes among the social networks, resulting in a loss of confidence in your organization.

Observations on Silos

2010-07-01T06:07:00.000-07:00

In this posting Richard Veryard examines the value of silos and silo thinking in organizations. In some ways silos are necessary organizational mechanisms because they place boundaries on functional activities. However they are also creatures of organizational politics - each silo is headed by "someone important". At the European Enterprise Architecture Conference 2010, Alec Sharp observed that silos have negative connotations and that perhaps we should use the term, "Cylinders of Excellence". That at least sounds more empowering.

Often, however, the success metrics of a silo are actually in conflict with those of the organization. Well intentioned success metric/motivational policy in a sales team (e.g. "Orders taken in the last week of the quarter will pay extra commissions") will in all likelihood have exactly the wrong outcome. Commissioned sales force team members are actually incented to do the wrong thing - to sandbag until the last few days of the quarter. This has potential downstream effects. It places extra burden on finance and other back office "systems" (silos?). It doesn't allow the organization to get a true picture of sales performance through the quarter. Somehow the miracle always seems to occur and some huge deal comes in at the end of the quarter to "rescue the quarter", keep the investors happy, commission the sales team...

So as architects we should look at the effects of the silos on organizational goals. Moving the silos around is likely to be a bad idea. Powerful people sit at the tops of silos and the only time to effect change at that scale is when new senior management take over. So rather than being perceived as people who are trying to undermine power, enterprise architects need to be seen as people who facilitate the planning and execution of the enterprise.

A fundamental (inside out) question then is, "How do the goals/reward systems in the current silos ensure that the goals and reward systems of the enterprise? are met" In the example above, the Sales silo would argue that it is helping the enterprise meet its goals by making sure that targets are indeed met. However, there is nothing in the enterprise goals that says "at whatever cost to the rest of the enterprise".

Enterprise architects are fundamentally enterprise level thinkers and thus must be thinking about the enterprise (and possibly the extended enterprise) as a whole. We must be able to understand the implications of conflicts in the enterprise - where Value systems in the silos are either at odds with each other, or at odds with the value systems of the organization as a whole.

Ugliness in the sink

2010-04-12T16:30:00.000-07:00

An architectural fundamental is that things that are not related should not cause failure on each others' part. The printer breaking should not stop the processing of orders. It does stop the printing of orders (duh), but should not stop the processing.

Some of the systems I deal with put me in mind of a sewage system. Bear with me here...

So, imagine the following undesirable situation. You are standing at the sink after dinner washing the dishes/pots/pans etc. It had been raining very hard all day, so the drainage systems had been well overtaxed. You are startled to see stuff coming UP the drain (perhaps from your neighbors house - you can tell what they had for dinner too). This is clearly an undesirable effect.

On further analysis you realize that the drainage system couldn't take the runoff fast enough, so your house, being lower than your neighbours, became the lowest point their effluent could reach. Oh dear.

Of course restuarant codes (at least in the USA) have a cure. They make sure there is a buffer zone between the sewage system and any sink where food is prepared. It is an airgap and a sufficiently large gap that any attempt by effluent to rise up will be nullified. It will spread all over the floor instead. Not much better, but definitely not contaminating food in a sink.

So what's the lesson here? Denial of Service through increased load on a system is something to watch out for. make sure you understand the implications and design accordingly.

This is as important at a business level as it is at a technical level. Who hasn't experienced a lack of planning - perhaps an airline offering a fare sale and not scaling enough to have call center representatives. Or the fiasco in Texas where the "business" (the State) offered rebates for trading in old appliances. They underestimated demand horribly.

The enterprise really has to understand its business models and take care to scale appropriately - and communicate that through to all interested and relevant parties

That Can Never Happen

2010-03-04T06:44:00.000-08:00

Some of the most ominous words I hear from development teams.

I will illustrate this with a rather contrived example - but one that I hope makes the point easily. No I am not advocating the writing of yet another date handler, but the problem is neatly bounded, well understood and has sufficient complexity to make a good posting.

You might also be wondering what on earth a small coding problem has to do with Enterprise Architecture. I'll get that out of the way up front. It is relevant because we well get to the core of some of the questions around reuse, system development/deployment philosophy, good practices, etc. Not your typical fare in every day EA, but viewing one of the roles of EA as influencer on "development" we have a nice teaching opportunity.

So here's the situation. A team discovers that it needs to handle a variety of date formats and in its environment of choice there isn't a robust date package that has been thoroughly tested. They mostly know the rules (leap years, time zones, Daylight Savings Time, etc.). They also know the source of the data they need to convert/check. It's coming from a system where, "If the date is sent to us wrongly by the source system, then there is a whole lot more wrong than this minor blip. Those issues will have been caught elsewhere." If a statement like like doesn't make you very suspicious, then nothing will. But why is it a problem?

First off, the statement is true - at least in the narrow context. If the source system gets a date "wrong" then indeed this is symptomatic of a larger problem. So far, so good. The developer doesn't do a proper job of checking error possibilities, "Because they can never happen". So if the system is expecting the month to be the three letter abbreviation (e.g. JAN for January, etc.) and it is in English then seeing FEV for February is a problem. There is no English month that starts FEV, but there is a French one. So is the error a typing error (B and V are very close on a standard US keyboard), is it a semantic one - it was supposed to be the French version, or what? Should the developer have to know every language to make sure that all possible month abbreviations are accounted for? Probably not we think. Treat it as an error and move on. But what if the code is badly written and mismatches are not caught because the programmer used some fall through logic and returned DEC for all all invalid months. DEC being the last month and a good candidate for being returned in error, "Because we can never get an invalid month string." OK, perhaps we can't.

And then project #2 comes along. Developer John says to developer Chris, "Didn't you code up that weird date handling routine last year? I want to use it, can you point me to it?" "Sure it is in the project library at...."

So John does some "copy/paste reuse" for a piece of trusted code. After all it has been in production for a good long time. No problems found. Inserts it into his application, all is well and about 6 months later it blows up. Turns out that application 2 was not getting the data from the same source as application 1, so it was possible for invalid data to show up. "That Can Never happen" suddenly became, "How the !@#* DID THAT HAPPEN, I THOUGHT WE HAD TESTED CODE" with recriminations from customers, senior management, Uncle Tom Cobbley and all.

Long story for some short points:

Just because you reuse something doesn't mean it is tested for your situation
Copy/paste reuse is often worrying anyway - code handed that way is very context dependent
Consider the cost of hardening and making the routine a service of some sort.
Promotion of an item to a reusable artifact puts extra stress on development and testing because the more general corner cases have to be considered.
Governance and management of reusable components is an important practice
When something is promoted, make sure its assumptions are known and its tests are included with it - that way at least a potential user of the code can see what conditions have been explicitly tested.

And finally, if it genuinely "Can't happen." you can be sure that someone, somewhere will make it happen! So again, make the assumptions explicit. Obvious isn't it (especially with hindsight)

Openness, opensource, lock in, the downfall of authority

2010-02-02T19:28:00.000-08:00

In the last couple of days a few related ideas came floating by. It started when Madame was bemoaning behavior in her profession (she is a University lecturer). Then I got to thinking about Open Source - at the behest of a customer. This led down the path of no longer being able to dictate because you are the authority.
You might wonder where this is going, so let me elaborate.
In school there is a certain amount of concern that the students are tweeting, surfing the net, multitasking, getting dates on Facebook or something like that. The gut reaction of the authoritarians is to find a way to punish the students. Of course in some ways the lecturers have the ultimate authority - dropping a grade - but that is patently unfair. Madame, of course, came up with the practical observation - make the classes interesting enough and they won't be distracted. That does require effort on the part of the lecturer though.
And now to open source. There is a fear that making something open will lower the switching cost and therefore customers will leave. So, the argument goes, increase the lock in by a proprietary method and tie the customers down. The Madame principal, of course, is to provide enough value so that the customers won't want to switch. That of course also requires effort.
We have much anecdotal evidence that lock-in of any sort is despised - sometimes in the software world a reason not to buy. In the current times, where education has become dialog, where software acquisition i also a dialog, we must be prepared to engage directly as equals and not assume positions of authority "because it always worked that way."
There are few excellent companies that actually can get away (at least for a while) with their authoritarian (aka my way or the highway) stance. That happens when the perceived value of the item is so great that the proprietary nature is irrelevant (Apple anyone?), but the advantage can erode quickly when another competitor enters the market (even if that competitor is no more open). The spat in the pricing model between Amazon and Apple comes to mind here. It just took MacMillan to have an alternative and to stand up to the lock in through monopoly bully and the market fractures.

Updates Harmful

2009-11-30T03:40:00.000-08:00

I have written about this before, I suspect. So forgive me if this is another representation of that resource.

Hanging out with Nigel Green and John Schlesinger is dangerous, so be warned. There be sacred cows being slaughtered in this post.

It all started innocently enough a year or so when Nigel and I were discussing the role of databases (record keeping) vs the role of databases (transaction scheduling, message management, etc.). Faulty normalization came into the picture too. Then the harm done by data modeling (where we thought we could model the rules in data). Large scale data modeling efforts requiring significant investment and it becoming very hard to see where the return comes from. Then an aha. If we go back a bit in time, the worst office job used to be "filing clerk." An imaginary discussion when someone comes home from work, "Well dear, what did you do at work today?" "We opened a new file for Acme enterprises. That meant that our index cards were all messed up because we had run out of space for companies starting with A, so we had to rewrite some of those cards, but while we were there we saw that some cards pointed to files that no longer existed so we removed those - but only after we had added more space for the letter A companies (which we didn't need right now anyway.)" "That's nice dear, how about a nice cold beer?"

The point is that filing clerks used to be the lowliest members of the office - and yet in their electronic reincarnation they have acquired really expensive care and feeding. Of course the new clerks are the databases, the expensive care and feeding is manifested by having a group of thugs (DBAs) who hold everyone to ransom with their talk of normalization, SGA, proper keys,.. All things which we did pretty easily with clerks. So what's going wrong? What is normalization for?

Taking normalization first - it is simply for ensuring that we don't get update anomalies. That something that is to have the same value regardless of usage actually does have that value. You don't have to have a normalized database to ensure that the update anomalies aren't present. Although it is a bit easier.

What is going wrong, is in many ways a harder question. One fundamental thing going wrong is that we use the "filing cabinet" as a scratch pad. So returning to the physical world for a bit. Let's imagine a filing cabinet in which we store the active accounts (perhaps bank accounts). When someone wishes to open an account, we give them a whole bunch of forms to fill in, they go off and fill them in and hand them back to us. We transcribe those forms and do some checkup on the data contained. Once we are happy with the data, we can now give the stuff to the filing clerk and have the clerk create the new file folder. So where were the forms and the checking? In some kind of "blotter" or case management pile on the clerk's desk. They weren't in the active accounts cabinets. And nor should they be.

No we go to a computerized system. We enter the data from the completed forms into the system and "poof" they create an active account. But actually it is more insidious than that. We go through a series of screens putting in different bits of the account - each leading us to a more perfect account, but we aren't there yet. Eventually they will be in the active accounts database (but probably with an inactive flag) so that they can sometime be transacted. This is nuts. We are using the record keeping database (aka the filing cabinet) to manage work in process. This is not a proper separation of duties.

It gets worse. The company decides to "go online". Expensive consultants are hired, golf outings are scheduled, expensive dinners eaten and the "new account workflow" is eventually unveiled. It, too is a sequence of steps. However, the poor schmuck filling this in has to complete each page of the form before moving on. That means that s/he cannot stop for a break - store a scratchpad version of this, do it out of sequence because they can't remember their spouse's Social Security number or whatever. The people in charge of the design of the system understand that THE SYSTEM needs accurate record keeping, have heard that "it is ALWAYS better to validate the data at the point of capture" and other platitudes, but forget that at the end of the line there is the poor user. For these kinds of data entry systems, (and a whole host of housekeeping systems) we need to store the "process state" separately. Don't use the state of the key entity as a substitute for that. Store where I am in the account opening in the account opening process, not in the entity that represents the account.

So what got this diatribe really going? The notion that updates are unnatural - and probably harmful. I posit that the reason that we do updates is mostly because the common need for retrieval of something is the most recent version of it. So it makes sense to have access to the most recent version and update in place. But that isn't always the most expedient behavior. Certainly the most recent value is often the value you need - especially in an operational system. However more and more systems really need the ability to look back. Even something as simple (looking) as you medical record is not something you want to update. Patient History is key. We don't need to know the current cholesterol level (in isolation), we need its trend. So we don't just update the "cholesterol value" in the patient record. We add a new item for the cholesterol and keep the history. We keep the record sorted in time sequence so we can see the latest. We don't just overwrite the value. Our uses of data are so unpredictable, that simply updating database arbitrarily is going to give us data loss. We don't know in advance how serious that data loss might be. Perhaps it would be better to assume that we will need everything and come up with a scheme that at some backbone level ensures that the current view can be reconstructed by replaying the operational events.

Musings with John Schlesinger

2009-11-30T03:24:00.001-08:00

John Schlesinger is an event thinker par excellence. So whenever I get the chance, I visit him in London to validate some thinking - or just to spend time with a terrific guy! So on a recent trip to London the subject turned to the rise of event thinking and the downplaying of the traditional SOA patterns. Of course the SOA traditions are being reborn to encompass the events brigade, but because SOA is so broadly and imprecisely defined that's perfectly OK. The SOA hype is over, long live the SOA hype. But that's perhaps a topic for another time.

The key observation from my lunch with John was one I had suspected, but was not able to frame properly. With a few well chosen sentences John had it for me.

This is all concerned with orchestration and control. So (deep breath), here goes. Where an event is raised and that event is to be processed by some subscriber, any intent to orchestrate the handling of the event by the subscriber results in a massive increase in complexity. (Roger Sessions will love this!). Naively one starts to think you have the "OK/Not OK pair" of possible responses. But then the "Not OK" responses blossom out of control. We have situations where the "Not OK" response must result in the retransmission of the event (and how does that happen?) and other cases where it must not. We have cases where the originator of the event has to interpret the behavior of the recipient. That sounds like some awfully nasty coupling to me. So instead of thinking that one has the "OK/Not OK" duality from the recipients view point actually what you have is the"OK/{set of lots of possible not OKs which the sender has to know about} multi-trality. In short that's just crappy design!

Thanks John

Zachman, Frameworks and EA

2009-11-30T03:10:00.001-08:00

This post comes out of a quick, but deep, conversation with @cybersal after the first dinner of the architect irregulars twittergroup at Gopals on 20091125. Other members in attendance were: @richardveryard, @taotwit, @Rsessions, @mattdeacon, and @hstrover. As is often the case, when a bunch of EAs get together, the subject of Frameworks comes up. And whenever we discuss frameworks, the venerable Zachman framework is mentioned. Often with much facial contortion and questions like "How do you actually build it?" or "What are in the interesting bits between the rows?"

And then as @cybersal and I were hoofing it back to Charing Cross - avoiding the crowds where possible, the Framework (at least thinking about the titles of the rows) simply gives a context for discussion. You don't really need the columns. So, for example, when thinking about schemas that business services might use in communication, you are working at "row 3". This tells you as much about what you are NOT supposed to be doing as what you are supposed to be doing. It is a really nice shorthand when one is talking to another EA - since EAs have typically all read or heard John Z. So it isn't about using the Zachman Framework as a "Methodology" (whatever that means) but more of a classification system. If you like a set of membership rules.

Now just because you have a set of membership rules, that doesn't mean you have to have the formal club (and if you are Groucho Marx, "I don't care to be a member of any club that would have me as a member" - but I digress). So, no you don't have to instantiate all the rows of the framework and figure out the mappings between them. However you can say to someone, "Come out of Row 4 and think in Row 3." That is in itself a powerful and useful observation, but doesn't really move EA forward much.

IT Profession? I think not

2009-11-08T07:38:00.000-08:00

Recent tweets from @rsessions, @richardveryard, @j4ngis,@cybersal have been looking at how hard various professions are. @richardveryard's observation that "@j4ngis @oscarberg Rocket science isn't even particularly complicated. Goes up, comes down. It is rocket technology that is complicated." in a tweet this morning reminds of a conversation had on the golf course with a very good dr. Let's call him John.
John is, as I have said, a very good doctor. His speciality is anesthesia, but his passion is technology. He is always coming up with schemes to invent solutions to make dr.' s live easier. So much so, that he would probably prefer to do that than what he is trained to do.
So after a particularly inept (actually about normal for us, but inept by anyone else's standards) round of golf, we were trudging wearily back to the 19th. hole when John announces yet another good idea - linking wireless technology, handhelds, voice transcription, remote printing,.... His question to me was, "How hard can this be?".
My response was something along the following lines.
John you are breaking my heart. You are essentially saying that anyone without a modicum of training, experience, expertise, but just with the passion and the idea can bust into my field and take over. Have you no respect? Imagine the situation being reversed. Be me for a day, and I will be you. After all, how hard can it be to administer anesthesia to a patient. You figure out the necessary cocktail, inject it and out they go. I can imagine that there might be a few kinks along the way - like making sure that they wake up - but we can leave that to iteration 2. He was, of course horrified. He asked if I was trying to imply that my chosen line of work was as disciplined as his profession. And for the most part, it probably isn't. The key is I don't work in a profession by any normal definition.
So while we are not in a profession, then any enthusiastic amateur can build "cool stuff". Who cares about the error cases? Who cares about the edge conditions? It is all about the app after all. To take a phrase from the movie industry, "We can fix it in post."
Who cares that the patient lives? Who cares that the patient suffers a quality of life decline? In medicine when we have post - it usually means post mortem. There is no fixing it in post in medicine.

A rant on "SOA Projects"

2009-11-04T04:39:00.000-08:00

The appearance of the SOA Manifesto has led me to look closely to the naming of projects and the implications of such names.
Time and time again I see projects titled or described with a technology or architecture in the name. How often do we hear, "The SAP project failed?" It isn't because the software doesn't work, there are a host of other potential reasons - all having to do with the human factors. Likewise with "SOA Projects."
The SOA community (huge generalization here) talks about "SOA Projects." Hogwash, I say. There are very few "SOA Projects." There are and should be many projects where the underlying approach is Service Oriented. There are very good reasons for deploying SOA in the enterprise/division or wherever appropriate. The deployment of SOA governance, technologies, etc. might be considered a SOA project, but creating a business application according to those tenets doesn't make that business application a "SOA Project."
Does this really matter? Isn't calling it a SOA project a convenient shorthand? Isn't calling it a SOA Project a convenient way of getting to the right funding bucket? Well, if that's the way the business operates, I, begrudgingly, guess so.
I think it is more insidious than that, however. By putting the technology or architecture top dead center in the name, it gives us an opportunity to make that the primary goal. Rather like hearing the requirement, "I need a database that..." Well databases are fine things, but a requirement that leads us to a "Database Project" again focuses on the wrong things.
So next time your SAP project failed, ask yourself the question, "Is it SAP that failed? or did the business not realize the anticipated benefits for other reasons?" It's easy to blame the technology.

Technical Debt

2009-10-15T05:24:00.000-07:00

There seems to be a theme developing in these posts. I start to see everyday situations and then relate them to everyday happenings - then attempt to do some analysis. Today's is no exception.

Ward Cunningham introduced us to the concept of technical debt in a 1992 experience report. Quoting from wikipedia, "Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite.... The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise. " An excellent follow on blog post from Steve McConnell (at http://blogs.construx.com/blogs/stevemcc/archive/2007/11/01/technical-debt-2.aspx ) expands on the point.

I am going to think a bit more prosaically! I was sitting at the computer minding my own business yesterday afternoon, when from behind me I heard a crash. A picture had fallen off the wall, spreading glass shards everywhere. On closer inspection, I saw that the method of attachment to the wall was inadequate. It was a simple nail, not a proper hanger. What's this to do with technical debt you may ask? Well, presumably when the picture was hung, the hanger (perhaps me) chose a quick and dirty solution - a nail. I could perhaps have used the proper fixture - but that may have entailed considerably more "development effort". I would have had to drive to Lowes or Home Depot to buy the fixture, install it properly... you get the point. Oh and of course there was a deadline. We had a party that night and had to have the picture hung.

That was about 8 years ago. Eventually the hanging approach failed and I now have to pay back the debt for my slapdash method of 8 years ago. The repair cost will far exceed the cost of "doing it right in the first place." However, I made the decision to do it the way I did because I didn't have a good handle on what the longer term implications would be and because I had a deadline. These are both key observations.

When we build systems we often don't actually know what the long term implications might be. We don't for example know what "long term" actually means. We often can't explain why doing it right is more cost effective "eventually". I, for example, wasn't prepared to say to madame, "I'll need to go to the store to get the right fixture, hold the party until I come back and put up the picture." Nor was I prepared to say, "let's not have that feature in the house until after the party."

The point here is that we often make conscious choices about the way we do things. These may be suboptimal in the long run - and they will come back to bite us. However we must suborn the future to the current to make sure things get done.

What is more clear to me is that we understand the operational implications of our choices. We need to build our solutions, "hard enough". We need explicit statements about the "-ilities", so we can apply the right amount of engineering. Let's make sure we get the failure stories as explicit as the success stories when putting our plans together - when deciding what goes into an iteration. Make sure we incur the technical debt for the right reasons.

Watching Events

2009-09-27T15:26:00.000-07:00

It seems that when thinking about events, we have a tendency to put some of the responsibilities in the wrong place. Of course every time we don't have a proper separation of responsibilities, we get extra complexity. So in this post I will look at some of the issues around the responsibilities and see where they should be allocated.

Short political rant that can safely be ignored now. Why is health care insurance (in the USA) handled essentially through employers and employment contracts? They simply don't belong to each other. The time base is wrong, the administration is wrong, the result is wrong.

End of rant!

I think a little naming context would be helpful here. This will be greatly simplified - just so that the essence is laid bare.

E is an event publisher
E has an output queue onto which itpublishes. This is called EQ
E publishes events of type V1, V2, V3 and V4. Individual events are named v1-1, (the first event of type V1), v1-2 (the second event of type V1), etc.
C1 is a consumer of events - it can consume events of type V1, V2
C2 is a Consumer of events - it can consume events of type V2, V3, V4
C3 is a consumer of events - it can consume all kinds of events.
Each consumer has an input queue - C1 has an input queue C1Q, C2 has C2Q, C3 has C3Q

The event network behaves as follows:

E publishes v1-1. The event network must transfer v1-1 to C1Q and C3Q. C1Q and C3Q. C1 and C3 are now able to process the events by reading their respective queues.

Thinking about the outcome possibilities (assuming E was successfuly able to publish) v1-1.

Both C1 and C3 receive the event v1-1
C1 receives the event v1-1, but C3 does not.
C1 does not receive the event v1-1, but C3 does
neither C1 nor C3 receive the event.

Question for the reader. Where should the responsibility lie in taking action for recognizing that one of these conditions obtains. The possibilities are:
(a) E
(b) C1
(c) C3
(d) none of the above

My answer is (d) - none of the above. It isn't any of these players' responsibility to do this. E's job was to publish the events. C1 and C3's jobs are to handle the events on behalf of their scope. But C1 and C3 are autonomous, so they can't be dealing with each others' failures.

So if it is (d), then there must be some other partcipant - yet to be identified that takes the responsibility. That something else is a proxy for the overall business policy. It needs to exist independently of everything else - and therefore needs to have the proper information fed to it.

Now imagine that C1 in some sense "fails" - it completes, but delivers an exception condition. If that condition is serious enough then something has to know. So it would be sensible C1 were to notify something of its own outcome.

Likewise C2 and C3.

So we have a kind of triplet of notifications (which expand into greater complexity for real sized problems).

E says, "I sent e1-1"
C1 says, "I got e1-1"
C1 says "I handled e1-1 with a normal result"
C3 says, "I got e1-1"
C3 says, "I couldn't do e1-1 - I failed because of a business problem."

So we could then implement a policy rule about what to do under these circumstances. That rule could of course inject new events into the "system." That however is a subject for another time.

Bottom line is that we have a three part system now for handling events. There is the standard pub/sub behavior (and don't get too hung up on the specific technologies). There is the abilty to send out content references using a mechanism that is not part of the event channel (quite RESTful really), and then there is the ability to act on non-receipt or improper handling of the event by one of the event subscribers.

A nice compact model that seperates concerns, so that the individual components can be focussed on their own responsibilities and not prying into each others' business.

A new twist on the Taj Chaat process...

2009-09-27T07:46:00.000-07:00

For those who have seen the previous post on process improvement at my local Indian chaat
restaurant will probably be intrigued on yesterday's twist.

We were buying some appetizers to go. In this case 2 samosas and an order of golgappe. Normally the process (at least for dining) is to write the items on a form, be assigned a number and given the vibrating pager to let us know when ready. Collect food, eat, check out by handing over the pager so they could locate the items to be villed from an accordian file.

For take-out, the process is very different. Again I fill out the little 2-part form. But before handing it over to central oredering, I take both parts of th3e form to the cash register. I pay for the order. Both parts of the form are stamped paid. I take the form back to central ordering where I am given the a number and a vibrator. The order is prepared, vibrator goes off and now what? I go to the station to pick up the order, but since the collection point of the vibrators is the cash register how do I return the vibrator?

This is made harder because the worker at the cash register is a different person from the one I paid. So there is no session state anywhere. I have a vibrator that has vibrated and a cashier who expects to take cash. She can't find the form parts in the accordion file.....

So why is this important?
Again seeing how non-IT organizations think about the processes that run their own businesses makes us aware that we shouldn't be overengineering. There are exceptions - things that don't follow the normal path that we just have to work around. Of course each workaround decreases efficiency, but if they are sufficiently rare, then the overall efficiency is not decreased by having work arounds. However, if the workarounds do become cumbersome, expect them to be worked on without negatively affecting the frequent path activities.

Oh, and by the way yes the golgappe and samosas were worth it! The whole bill was $6.56!

Processes and events

2009-09-27T07:21:00.001-07:00

Another day of talking to Nigel Green - thank you Skype! And some thinking around processes and their relationship with events. Again sounds innocent - but it seems as if both of us strongy event-oriented thinkers come to common ground when thinking about processes and orchestration - namely that while you might use low level messaging semantics for implementing processes, event modeling doesn't really help when trying to model processes. However and here the lightbulb began to flicker dimly, the result of executing of a process or process step can become a source of events.

We chose an example from the airline industry - and from our experiences of being travelers. Not from any great insights from the internals of the business. The focus was the check in process at the airport itself.

Clearly we see interesting policies at different places and for different carriers. For example at Mumbai (at least time I went through there), they seal your bag with some kind of security strap. So it can be seen whether the bag has been tampered with. That is less common in the USA. However, at Miami International Airport when I went through a month or so ago, I saw the ability to wrap the bag in a kind of cling wrap. I presume that can be done elsewhere too. That is all by way of background.

Airlines nowadays can and do charge fees for checking baggage. All of the rules require that checked bags undergo a security check. Bags are subject to weight limits. Passengers are subject to bag limits (no more than n per passenger). Ignoring further complexity like whether actually to collect the fees (elite passengers are exempt, for example) there are some quite interesting process decisions to be made.

If the airline chooses to impose the bag fee immediately that the passenger offers the bag for checkin, then there may be some undo logic if the passenger decides not to check the bag after all. It is worse though. Imagine that the bag is too heavy. When is that discovered? For example if the passenger has checked in (and been charged) at a kiosk, then on presentation of the bag it is discovered to be too heavy so an extra collection is made - that could make the passenger decide not to check it after all, so a refund is in order. Or perhaps the passenger may decide to open it and remove some of the heavier items, and get it to the correct weight. That's fine - but what if it has already been secured with tape or wrapped in a cling wrap. Kinda tricky to undo...

Then there is security. Another opportunity for the passenger to open the bag and remove things if they should not be in checked baggage. And so it goes.

Different airlines and different jurisdictions will implement the Policies - "maximize revenue for bags", "make sure the passengers' possessions are safe", and "transport passengers safely" with different process paths. Those paths need to be orchestrated. It isn't clear how an event network will really help that orchestration. In fact I would go so far as to say it complicates it. However at each step of the various process steps (or sub-processes), it would be very useful to spit out an event that provided useful (possibly actionable) information to trigger some other behaviors.

For example, if during the security screen a weapon were found, we would expect an event to be raised to trigger a whole raft of other processes. We would be jumping outside a process domain into another domain. That of airport behaviors to criminal behaviors. So looks like a terrific event.

Even the mundane events may be interesting to somebody. That a passenger decided not to check the bag after a fee was assessed can be helpful when looking at the behavior as a whole for market and planning purposes. Opportunities for process improvement abound.

The weight of the checked luggage is also useful for "weight and balance" on the aircraft. Necessary so proper takeoff parameters can be computed, proper fuel calculations can be performed, etc. So the event raised as a result of successful baggage check-in is quite a handy event to have.

So bottom line, it seems from this (and a whole raft of other possible examples) that we will typically see events being generated as a result of a process step happening - at least when in a process.

Of course there are lots of other ways of generating events, we don't need to formalize processes so that events can be generated. Relatively random behaviors give rise to events too.

Event and Content

2009-09-13T07:31:00.001-07:00

This conversation with Nigel started innocently enough. Two people who have very similar views talking about architecture in the large. We described our current problem spaces - they looked very similar, but as is often the case we used the same words to mean different things. Of course once we realized this we got back on track quickly. I'll write this from my perspective, but in reality it would be better written by a dispassionate observer.

I set out to describe the separtion that I am seeing in VPEC-T especially around P-E-C. In my definition, I was thinking of Content as being everything about the Event. I was merrily proceeding down this path, thinking that the view was shared until Nigel spoke. We realized simultaneously that the word covered several concepts. There is the "stuff" about the event itself - the event properties like when it happened, what kind of event it is, what channel it was communicated on - essentially a kind of tag cloud for the event. I suppose in the current vernacular this might be the event meta-data. There is also the "state" that exists as a result of something doing a piece of work and generating the event. That may well need to be made available somehow. So for example, in my sales system the "event" of a sale being made has something useful like when it happened, however there is a whole lot of other information like who the parties to the sale were, what the value of the sale is, who the sales team is, what was sold... - in other word the "business document" representating the state as perceived by the event emitter.

In the Chris world, the tags (event meta data) were part of content. In the Nigel world they aren't part of content per se. Actually that is unfair, they aren't ONLY content. In other words, the event meta data can easily travel with the event. But the state information typically won't.

Reversing that point of view for a second, we get the notion that there is definitely a content "store" somewhere that the processor of the event might go to get the contents, and an event store - somewhere the event data would be stored. The content store doesn't have to be explicitly created by the event creating component, it could equally be some external source (e.g. a weather forecast). The event store's job is to store the events (duh) so that they can be replayed for a variety of reasons:

An event processor has been out of the loop for a hour and wants to catch up "get me all the sales events for the last hour"
A simulation wants to be done using some "live" state. What would the impact be if we had decided to close this airport at 2pm yesterday? So inject a new event into the event store and then replay forward with the existing events after that point. Of course there will quickly be diversion from the reality as processed, but that is OK. We have been able to run the simulation with a known starting point

Considering the second bullet above, it is quite convenient to use the same semantics for processing "missed events" as for processing the events in real time. Kind of rule - never pass the content with the event, always pass a reference to content, but make sure that the event tag (meta-) data do come with the event. Make the event store readable so that the event sender isn't responsible for hanging on to the event, and having to remember who has received it and who hasn't

At the end of our time together on this call we were, as expected, in agreement that for a proper separation of concerns, we should treat content as being the state data and the event store as being the place where the events and associated metadat are stored to allow for retries and simulations (among other things).

The Policy part of the call will come next.....

Signal to Noise

2009-07-10T03:30:00.000-07:00

This posting is sparked by a conversation between Nigel Green and me. We were having a spirited discussion about choices in following people in Twitter. But the concept generally applies in almost all event based architectures.

The fundamental principle is that it has to be somehow worth it to sort through the whole mess of communication to get what you need or want. Thinking in interface architecture terms, a point to point interface has a very high signal with very little noise. However, tailoring the communication channel is very expensive - at least for the originator of a message. Similarly a total mind dump of everything you are thinking about at a particular moment can have a very poor signal to noise ratio when viewed from the perspective of someone trying to decode the message.

I have used the example of the Christmas Letter before - the letter that contains all the family news and is sent to every acquaintance. For the originator of the message, it is a very efficient way of communicating stuff. No thinking about what each individual subscriber cares about - let the subscribers figure it out. If there is enough signal in the letter, then filtering through the noise is worth it. If not, it goes quickly to the recycling bin.

Likewise in social media. As a twitterer and as a blogger I don't know what individual followers are going to want. As a blogger, I don't even know who the followers are. And I don't want to. If people find that the signal is rich enough they will follow, if they don't they won't. A very efficient way of information delivery - and a very handy simplification technique in event driven thinking. The basic question is, "Is the signal to noise ratio high enough so that it is worth me continuing to listen on this channel, or are there are other channels with better ratios?" As a subscriber I can often make that choice.

Twitter is such a great example, because the space is flat. When I tweet, it could be on any topic that I find interesting and relevant. To me, the twitterer, it is all signal. To you the follower it is probably mostly noise. If there are occasional useful signals you will continue to follow, if not you won't. Attempting to put some kind of ontology over twitter defeats the purpose because the publisher has no clue which permutations of tags are relevant to any given listener. I regualraly unfollow twitterers when the signal to noise ratio degrades too much.

Real Life Caching

2009-06-21T06:41:00.001-07:00

The thought behind this post comes from the rich pageant of daily life with Madame. Yesterday morning we were discussing when she should visit her mother (a sprightly 96 year old who lives about 1,000 miles from us). So it was a matter of looking for decent airfares, timing the visit to coincide with my absence, minimizing the flight time (it is always requires a change in either Chicago or Detroit).. What you may wonder is relevant about this in architecture. It all has to deal with the location of the process, and the location of some of the data required to complete the process.

The computer was in her home office – up a flight of stairs from the main living areas of the house. Her purse was down in the kitchen. She had dutifully retrieved her credit card from her purse prior to starting the transaction (pre-cached the data), and carried it upstairs. Because of where I work, I almost always try to ensure that bookings are made on Travelocity.com – keep all possible revenue in the family.

So the process goes fairly smoothly – Madame is using my Travelocity account to make the booking (easier than creating a new one at that moment). Of course this would be a security violation, so let's say, I was acting as the operator for this purpose! We get all the way through the process (well almost) when we realize that her frequent flier number is not in the account anywhere. Also that data has not been cached anywhere (not in her head, not by bringing the physical card to the point where the transaction is being executed.) So an IO request is issued – resulting in the slow IO processor (me), walking from her office, downstairs to her purse, finding it, and returning the whole purse. Why the whole purse you ask? One reason is privacy – it is hers and I don't access its contents without permission (PII rules). Second is that I don't know what other esoterica might be needed, so I want to save my weary legs – and not have to make another trip to the purse data store (which by the way is a heap storage model).

The IO request is completed. Madame parses the purse data structure until she finds the relevant card, she then types in the number and completes the transaction.

As always with these stories there are a couple of questions:

Would it have been more sensible for her to bring the whole purse data structure instead of just caching the credit card? In hindsight, yes. But how would she make that determination. The stairs are steep and it is a long way. The purse could be heavy (too much payload) and unwieldy.
Should I have brought the whole purse or just the needed card, or better still a lightweight interpretation of the number (she didn't need all of the information on the frequent flier card)?
Does it make sense to cache the whole content of her purse in any room where she might execute transactions? After all with wi-fi that could be anywhere, including outside the firewall (i.e. in the garden or yard)? What about security? The gardeners and the maids (would that we were so lucky!) would then have access to the cached data. That could be troublesome.
Is the network transport reliable? Could I have been distracted in the bringing of the data back upstairs – through answering the telephone, needing some refreshment prior to attacking the stairs, pure absent-mindedness or what?

Of course the point isn't to think about home processes much – we will do things that seem expedient without a whole lot of thought. But when designing systems, we have to think quite carefully about these kinds of things. By putting them in this sort of context, it can sometimes help to think things through.

And, oh yes, this is not really architecture – it is design. However, there are patterns that emerge, and they become important aspects of architecture.

Architecture on the Product Side

2009-06-05T06:29:00.001-07:00

In an previous post, I wrote about the difference between the application of technology to the building of products, vs the application of technology in running the business. They are pretty different animals, if nothing else because of the way they are deployed.

So, I get to wonder what the head of architecture for the product side of one of these companies does. I can take one of the Architecture frameworks and apply it – but to what? Individual products (each with own P&L, development teams, practices, degrees of product maturity)? Across the whole product organization itself, so that we can begin to standardize/use same services across the product suites. When the products can be deployed in several ways (in house operated and sold as ASP), on-premise at the customer, on=premise hosted by one customer and used by another,…So as the products are deployed we have to think about the how the platform affinities work.

The customers want to customize the way the work is performed using the products/services that they have bought. So somehow we have to think of ripping the process logic out of fixed applications.

There are cross product dependencies as well. So sometimes we have to rip code out of one product and insert it into another.

Oh and the up time for these products must be in the 99.999 range – the products handle very time/life critical activities.

And finally a legacy that dates back a long way – pretty much bulletproof, but definitely has its own way of presenting itself to the world.

This Architect's world feels very different to me than the "traditional" EA world where the need to build for wide deployment and customization is less important.

Any thoughts anyone?

Offbeat Integration Approaches

2009-05-21T09:30:00.000-07:00

I am collecting humerous terms for different kinds of integration. I will seed the discussion with a few, but would be interested in having others added, with descriptions in the comments to this posts. It would be delightful to have a large glossary of terms.

I'll start the ball rolling with some oldies but goodies.

SneakerNet (n,vt) Delivering of (usually) files from one place to another on some form of removable media. The sneakers, of course refer to the need to get it there quickly. Often used to bypass security or other limits. E.g. email attachment size limitations, one system unable to connect to the same network as another system. File type limitations. Usage. I can't email this to you, have you got a thumb drive, I'll sneakernet it to you.

FunnelCast Delivering of information to a large number of people by shouting through a megaphone. The most common means of getting data from the page of a professor's notebook to the page of a student's notebook.

Swivel Chair I first heard this from my colleague Adrian Apthorp. The need to take data from one system and enter it into another can be solved by sitting on a swivel chair so you can switch between the screens with minimal effort.

Have at it and exercise your creativity.

Chria

A gedanken on identity

2009-05-08T05:20:00.001-07:00

This post arises from conversations (over many years) with John Hall, the late Keith Robinson, Bob Brown, Keri Healy, Nigel Green, Richard Veryard and Fred Fickling. It has to do with how we identify things, immutability and most recently REST. So first the story(originally related to me by John Hall):

Jason has just commissioned a wonderful boat (the Argo) so he could have adventures in the Mediterranean. He commissioned a crew of likely volunteers and set sail in search of (among other things) the Golden Fleece. This was not a short voyage – in fact it lasted many years and they had many adventures – none of which are relevant to this gedanken. Like all boats (holes in the water in which you throw money), Argo needed repairing quite often. So every winter , Argo would be taken to a boatyard and refitted. Old parts were replaced, holes patched, etc.

After several years of this, there were no original parts left on the Argo at all. Everything had been replaced, even down to the smallest dowels. Question 1. Is this boat the "same" Argo as the one originally built? Certainly the crew, the local registrar of ships and taxation authorities would think so. After all, there was no need to reregister it (her, for lovers of boats). However, we know for sure it isn't the same, in fact there isn't a single original component on the boat. So, in an information system, what is the "identity" of the Argo? Does it depend on which information system (registrar/taxation/crew sign-up, repair management for example) we are thinking about?

What we didn't know, is that the wily repair shop had kept all the old parts and had been secretly reassembling them into a boat. When the final wraps came off this new boat, he announced that this was the real Argo, and that he knew the other to be a fake. Question 2. Which is the "real" Argo? Certainly this depends on who wants to know and why? The registrar of ships might well take a pragmatic point of view and decide that the repaired Argo was the "real one" and that the rebuilt one is "something else". Of course the fine piece of woodwork on which the boat's name was carefully inscribed says Argo in each case.

Question 3 (a-f and beyond). Which of our various lenses can/should we be looking at this through? The Checkland CATWOE approach becomes an important set here because the Weltanschauung (loosely translated to be the worldview) partially determines that. I suspect that the Weltanschauung is the overarching concept that allows us to choose our lenses. So what does VPEC-T say here? What does Cynefin say here? What does Systems Thinking have to say? What do Value Networks have to say?...

Question 4. Why bother? This gedanken on its own is a fun mental exercise, but there are some underlying issues that crop up time again in our real world systems. As we start to look at the world through RESTful lenses, we come head-on into representations and the state transitions of representations, and making a clear distinction between the thing and a representation of it. So we have the Argo – and then some systematic representations of Argo. When an event (a repair perhaps) happens to Argo, we may decide to change the state of the Argo representation. Or some system may decide to, while another may not. For example crew scheduling for Argo may care less about repair events than registration. So is there just one representation of Argo (with one URI) or are there many representations of Argo?

As we wrestle with the knotty problems of identification we are forced to combat our views of history – what has happened in the past to the things we care about. How we can or cannot change other people's views of the same thing (the event streams we care about when changes are made are different from other people's). We have to worry about placing ourselves at a point in time and asking questions like, "If I were asking the question last January what would your answer have been? (very useful in market research, courts of law, etc.)"

At the bottom of this is really the "who cares about what?" question and how if we attempt to create universal data models and databases of things we are doomed. Perhaps the best we can do is to keep track of the Event and Content streams as they apply to Policy, manage our own representations of things and broadcast our state changes against those representations pushing the responsibility onto the "subscribers" to decide whether they care.

Services, SOA and Web Services

2009-05-06T04:45:00.001-07:00

This article
(Can SOA Give You Good Service) identifies some of the troubles with
words when dealing with all the Service terms.
I found it a very helpful article indeed because it helps keep Service and SOA on the straight and narrow. Nice call out of the Web Service vs Service.

Interesting observation that "A service is a logical representation of a repeatable business activity that has a specified outcome (e.g. check customer credit; provide weather data; consolidate drilling reports). It is self-contained, may be composed of other services, and is a "black box" to its consumers."

Nowhere in this definition is there any requirement to get an answer back, except
maybe a status of "yup, I've done that" or "No couldn't do it, sorry". For sure
a result could come back. For sure some system state may be changed (perhaps because
of a side effect), but the key is that it is a black box. Where we do see some confusion is in granularity. I occasionally here the word operation being applied to Services.

So in your example "Check Customer Credit" above, is that really a Service invocation
or is an invocation of an operation on another Service (perhaps a Customer Management service)?

Just dealing with the granularity would help me in understanding the kinds of numbers
that are bandied about at conferences. "We have over 3000 services" vs "We have
40 services (and 3000 operations). Is that simply a packaging issue? Coming back
to the whole WS-something discussion, you make a telling point that "A Web service
is a software system designed to support interoperable machine-to-machine interaction
over a network. It has an interface described in a machine-processable format (specifically WSDL). Other systems interact with the Web service in a manner prescribed by its description using SOAP messages, typically conveyed using HTTP with an XML serialization in conjunction with other Web-related standards".

First up this says machine-to-machine. And while the computer at which a user is sitting, operating the browser is most definitely a machine, I am not convinced that WS-something is the most appropriate way of managing that interaction. It may be, but the point is not proven. Secondly, it is not completely clear to me when services actually need to be exposed (assuming that we have defined service correctly). So how much WSDL wrapping do I do? And why? In some ways - especially for internal code, WSDL wrapped service invocation is just really expensive function or subroutine calling. So even though we can define what a service is, it is harder to define what a service isn't, and harder still to choose the right architectural approach to allow for machine/machine interoperability, and human/system interoperability.

More Fun(?) with words

2009-04-30T08:58:00.001-07:00

Sometimes the TwitterSphere is just too constraining to get a thought across fully. I naively posted a question about the definition of "Application" – to see what would come back. It turned into a delightfully healthy discussion with some twists and turns along the way. It isn't every day that Duns Scotus and Humpty Dumpty show up in the same post – at least not unless Richard Veryard (@richardveryard) is involved!

So a good question is "Why do you want to define application?" My answer is actually that I don't want to DEFINE it, I just want to know what people might mean when they bandy the term about. As Nigel Green (@taotwit) and I chatted this morning at length on this and other topics, the recurring theme was, "I'd like a quick way to parse a conversation." In other words, when I am talking to someone and trying to understand what it is they want (requirements anyone?) I would like to know their frame of reference so that we can communicate.

How often have we heard requirements that say things like, "I want a database that …." Actually, I suspect it isn't usually the database that the requestor wants, it is some way of manipulating the data with a purpose in mind. Maybe, even, an application (gasp).

So just as in a previous post where I was wondering about type/instance nomenclature, so here I am wondering about other opportunities for miscommunication.

Richard makes an interesting point, "@seabird20 If something persistently escapes or evades precision, then maybe it was the wrong concept in the first place." That may well be true, but we can't unbreak the egg. The words are out there, their meanings are many, we can't (and shouldn't) attempt to unify the vocabulary (even the Académie francaise has stopped trying to keep French completely pure – le weekend anyone?)

So this is not a cry for definition and ontology for its own sake. It is a means to collect lots of definitions so we can understand each other better. Of course, the more we share context, the more "shorthand" we can use to express ourselves – because we have either tacitly or explicitly agreed to the vocabulary. It is in the "getting to know you" stages where shared context and trust are established. It is at those early stages where slight misunderstandings can blossom into full-fledged disagreements and a loss of opportunity to trust.

Process Improvement and the Chaat Cafe

2009-04-19T14:47:00.001-07:00

My favorite Indian restaurant is a 20 minute walk from my house. It has the most wonderful Indian snacks/appetizers and of course kulfi. It is also a thoroughly confusing place. I will try to describe the changes in process that I have seen over the last 18 months – since it opened.

First, some background. It is set up with three major "stations" where things are cooked. The flat top where the dosas, parathas and other flatbread dishes are prepared. The drink area (lassis, etc.) and the chaat area (samosa, golgappe, etc.)

Version 1.

You wander into the restaurant, decide what you are going to have and fill out a 2 part form in front of the station. Hand it to the cook. The cook makes the dish and when ready calls out over the speakers. You go and collect it from the counter. When time to pay, you take the white parts (tops) of the 2 part form to the register. They total up the bill, you pay and leave. Problem – this didn't scale well because we could never hear the loudspeaker announcements. Well we could hear that there was an announcement, but not what was being said.

First Process Improvement

Numbers were attached to each table. Your table number was required to be placed on the bottom of the form. Problem then was you had to select your table prior to placing an order. How do you keep it while ordering? Can someone else grab it? Much chaos ensued. Advantage was that the loudspeaker announcer just had to call the table number. You still handed your forms to the cooks, etc. There is still the possibility that unscrupulous diners would forget the white chits or (as I have to admit to having done) throwing one away by mistake when clearing away the dishes. So my guess is that auditing showed some yellow chits (what the cooks did) which didn't have matching pair white chits. And no way of running down the offenders until much too late. Everyone seems to pay cash there!

Second Process Improvement

Technology was introduced! Central order taking was introduced. So now, the restaurant has done away with the need to grab a table before creating an order. It has introduced a central ordering location. The forms, however are essentially the same and are not at the central location. The forms are still in front of the cooking stations. So you fill the form out at the cooking station and carry it to the central ordering facility. There (if you have not already been given one) you are given a vibrating pager. Presumably local in-restaurant network only. This is similar to the pagers often used in restaurants to tell you when your table is ready. When you place the order (on the 2 part form) you enter the number of the pager device. Now when each part of your order is ready, the pager device goes off. As you collect the dish, you need to stop by central ordering, so they can turn the thing off. If you fail to do that, it keeps vibrating and you have no indication when another part of your order is ready. Also, unknown to the first time user, the device will be used for computing the bill, so if you are looking for individual receipts, you need one vibrator each.

When you go to pay, you hand your vibrator to the checkout clerk. This automatically brings up the bill. I haven't seen the behind the scenes magic that does this yet. They also have an accordion file into which they put all the yellow copies into the slot numbered with the same number as the vibrating pager. You pay and leave…

So what you may ask?

This is an interesting exercise in changing processes and system state knowledge to adapt to conditions. In the first 2 iterations all the state was pushed to the diners (white copies) and that state information was used for "billing" and "payments". In the final case we see an audit database (the accordion file) and a shared key (vibrator number and slot number).

So by looking at what was causing trouble (what Policies and Values were not being dealt with correctly), the store restaurant owner instituted quite significant changes that make the process of ordering/getting the food/eating it/paying for it (the order to burp process) a great deal more streamlined. Very little automation needed, almost no effect on the primary Value of the place – "Somewhere that serves high quality, tasty, vegetarian chaat with a flavor of home."

It has been fun to watch this humble restaurant put so much thought and effort into helping itself run smoothly and maintain a great connection with its customers. Would that we in IT be as effective.

The trials of pedantry – aka type/instance confusion and other adventures in meaning

2009-04-19T09:44:00.001-07:00

"I had Taco Bell for lunch today". I keep hearing that usage from co-workers, students, random people in passing. At some level it makes complete sense – I get it that the person speaking ate at food obtained at a Taco Bell "restaurant" (why anyone would do that is beyond the scope of this posting). However, compare and contrast that with, "I had salad for lunch today." The second describes the food (pretty generically), the first describes the place where the food came from.

Now imagine you don't know that Taco Bell is a food chain, and maybe you are not doing analysis in your primary language). What assumption would you make about the statement, "I had Taco Bell for lunch today?" I think the first reaction would be that "Taco Bell" would be classified as a kind of food, not a kind of place. And Because that is a first impression, it would set a context that could be hard to shake.

As an aside, while working in France a couple of years back, the nearby Pizza Joint was called "Speedy Rabbit". When a project team member suggested we have "Speedy rabbit" for a working dinner – and that I, being the project manager and therefore an impediment to progress should go and fetch it. Since in that part of France, rabbit is often served, and often really tasty, I had visions of having to chase this thing down in my underpowered rental car. Turns out it was a pizza joint and that it was a simple pickup. Not a rabbit in sight. Oh well.

As architects/analysts we often find words or phrases that are used in surprising ways (idiomatically one might say if one were being charitable). The insiders to the group get it, understand the meaning, know when the statement refers to Taco Bell the company HQ, the Taco Bell on the corner, the specific lunch (without knowing the actual details of the meal). So we have to be careful when we are faced with words like "office" or "transaction". We need to know the context that the speaker is coming from and to be able to decouple our understanding of the word from the understanding that the speaker means.

How often have you heard statements like "we have 7 transactions in our banking system"? My first reaction is, "wow those must be huge transactions" or this bank is going under soon. And then I realize that of course it is transaction types or kinds of transactions, not the instances. Once again context rules. If you ask the question, "How many transactions did you do today?" You may well get an answer of 100 million. If you ask the question, "How many transactions can be executed?" You might get the answer 7.

So, how do we know when we are getting a "type" answer? How do we know when we are getting an "instances" answer? How do we make sure handle the precision that we need, without pushing all the annoying explicit packet context onto the people we are striving to understand. We have to learn the context, apply the context properly, and hide the distinctions unless it becomes absolutely necessary. One way of doing this nicely is to have stash of instance models handy. So when talking about concepts we should talk about them extensionally when we mean to describe the instances, and intensionally when talking about the type.

EA and Finance

2009-04-15T07:19:00.000-07:00

This post come out an initial simple twitter question, "Should EAs understand finance?" The conversation went all over the place. In 144 character bursts. It is hard to get the essence of what I was thinking in these short bursts.

Some of the slices:

How does EA justify its own costs? Interesting, important, but not where I had hoped to take things.
General discussion on values - again interesting but not what I was looking for.
Simplification - again, doesn't get to the meat.
IT should understand spend on everything (not just IT) - this is getting closer. It isn't necessarily about spend, though.

I guess what I should have asked is broader and better categorized than what I did ask.

When I say finance, my first thought was should the EA be able to understand (at a deep level) the balance sheet OF THE BUSINESS?

Second when it comes time to look at an annual report, should the EA be able to interpret the financials? Including the footnotes?

When it comes to investment decisions - should the EA bee in a position to understand the implications of different kinds of investments and tradeoffs? A new airplane hangar vs increase in parts inventory vs a new data center?

Note these are ALL: business decisions.

Now, since the spend on "IT" is so relatively large (1.3% to 5% of turnover or revenue) - depending on what's counted and how you count it, should the EA be an active participant in the allocations of this spend (budgetary vs actual)?

As technologies evolve - the shift from inhouse data centers to outsource (back to insource - outsource costing, in house operated), cloud, the economics become driving factors. How well should the EA understand these economics? How much should the EA be involved in these discussions? Is the EA involvement any different in these IT kinds of decisions than in other technology shifting decisions. For example changing the delivery fleet from gas (petrol) burning to biodiesel/electric/hybrid/CNG, LPG, LNG. All of these have huge effects on the business and its bottom line.

When a business has the opportunity to allocate "money" across its various operations, should the EAs be in that discussion? Normally it is an hierarchical, line of business, roll up approach with little input from EA except the CIO might ask, "What's your budget?" This is nonsense. The EA must have a budget, for sure, but to properly assist the business, it must be an active participant in budgeting decisions.

So as the EA has the opportunity to influence the business (used loosely) that s/he supports, where are the lines around finance?

Innovation and incremental improvement

2009-04-05T20:30:00.001-07:00

I was watching the Formula 1 race from Malaysia this morning. It was on live and early, so I had the TV to myself. For the past several years F1 has been dominated by McLaren and Ferrari. For the past few years the changes to the formula have been relatively small.

This year, however, the FIA have made some pretty dramatic changes – resulting in a major shake-up. The old factory Honda team is no more, but has become reborn as the Brawn F1 team (Ross Brawn, the man behind the rise of Michael Schumacher being the team principal). Toyota have also done well, with the red bull/toro rosso teams also having good outings.

So, what changed. I contend that this is a great example of the difference between steady, linear improvement as managed in lean/six sigma kinds of processes, and the need to be radically different as is the case when major innovation is necessary.

The FIA changed the rules dramatically. Back to slick tires, much reduced rear wing area, the Kinetic Energy Recovery System (KERS) which charges batteries under braking so the power can be deployed at times to suit the driver. Adding an extra 80 hp for about 6 seconds a lap. Engine revs reduced to 18,000, etc.

The teams that appeared to treat the rule changes as evolutionary are doing poorly/ The teams that recognized the radical nature of the changes – and with little to lose, are doing much better.

So, perhaps where innovation is concerned we should not try to put it into well defined, six sigma, DMAIC based processes. Let the creative juices flow, make changes in a non linear fashion until the platform has become relatively stable and then shift to six sigma type approaches.

Entropy in Systems

2009-03-28T15:18:00.001-07:00

The concept of entropy in thermodynamics is well known. This article covers the landscape well. In systems, especially systems that involve human and computer interactions, we have a similar notion.

Entropy, historically, has often been associated with the amount of order, disorder, and/or chaos in a thermodynamic system. The traditional definition of entropy is that it refers to changes in the status quo of the system and is a measure of "molecular disorder" and the amount of wasted energy in a dynamical energy transformation from one state or form to another.

So too in our information systems. The amount of effort that we have to undertake to move a system from one state to another (essentially to make a modification to it) is in 2 parts. The effort expended towards making the change useful, and all the rest of the effort.

This applies both to the change to the system itself and any changes that we make as a result of executing the system. So, for example, if we consider some system fragment, "Visit the doctor because of pain in the shoulder", then any time/effort spent in doing something other than getting the diagnosis/treatment is wasted and increases the system entropy. This time/effort may include (but is not limited to)

Getting to the Dr.
Filling in new patient paperwork
Having payer status checked
Explaining symptoms to receptionist
Waiting in waiting room
Waiting in treatment room
Waiting for X-Ray results
Driving to radiology lab
Filling in radiology lab paperwork
Waiting at radiology lab
…

The point of this is that this "system" is unbelievably wasteful of a precious resource (at least precious to me), my time. So from my perspective all of the above steps indicate great inefficiency. Perhaps because of complexity, perhaps because of a lack of cohesive thinking.

Considering another example, perhaps closer to work for many – the HR portal. That is often the one information system that has been designed to almost entirely ignore the majority of its users. There is often a huge learning curve for the majority of the employees. Of course the users who specify the system, the HR department have it well designed for their own convenience – and what they believe is the convenience of the employees. I leave you to draw your own conclusions!

So at one level, we have the idea of the system in use with every use increasing the entropy of the system.

Now think about attempting to make a change to a system. A whole new dynamic sets in. The need to understand the system in place so it can be changed. This can involve very detailed analysis – deep understanding. The more pieces there are – and the more interconnections, the more understanding there has to be. So depending on the design of the system in place there can be a greater or lesser effect on its entropy. If the system is very involved/convoluted then the understanding as a percentage of the useful work done will be high. If the system is relatively straightforward then the understanding as a percentage of the useful work done will be less.

So systems entropy might be thought of as ( _i=0ⁿ∑(W_i- V_i)) where W is the work performed at any state change I and V is the Valuable work performed at any state change i.

Defining and normalizing what we mean by work – and considering some normalized work value equation is, of course complex. For example in the getting shoulder diagnosed and treated, the system implicitly values the Dr.'s time as being more valuable than mine – so the system is optimized to make sure that the Dr.'s change entropy is least. What should be happening is that the total change in entropy should be minimized.

Typically systems that are overly complex, overly bureaucratic or optimized to support a minimal number of stakeholders will exhibit the greatest increases in entropy under a given state change.

As architects we have a responsibility to be looking out across the landscape of a system as a whole and finding ways of minimizing the increases in entropy across common state changes.

Prototyping in rails

2009-03-04T13:12:00.001-08:00

Now this is a truly bizarre thought. Many business apps and the underlying databases are a bit more complicated than the standard web app. Occasionally it is even necessary to populate the model that you plan to implement and let the users test drive the model through the UI.

This approach used to be called rapid-prototyping, but has rather fallen out of favor.

However, as the browser is becoming (has become) the dominant UI now, there is perhaps an opportunity to do a rapid prototype of some system functionality so the users can be assured that you have the relationships right and that you can quickly demonstrate some scenarios.

This tends to be a royal pain, and then it hit me. Rails is such a quick to develop framework that getting some gnarly prototypes out quickly is very little problem. Of course we don't want to be too convincing because the users might think we have done the hard bits. But really what we will have done is:

Create a simple database model with the interesting relationships (keys/fks, etc.)
Populated that model with some made up data - careful cases
Allowed the users to manipulate/navigate the data through a familiar approach. For example in an auction site, you may have many bids. So a 1:m relationship there. We could show an auction with its many bids easily, then click on a bid to go deeper, ensuring we have a link back to the auction.

using RESTful principles and RAILS quite a lot of this is generated with the scaffold. That's a lot of power.

Evaluating enterprise software

2009-03-04T12:50:00.000-08:00

Two things have happened at pretty much the same time. The first is an evaluation of some enterprise class software in conjunction with and on behalf of a client. The second, the appearance of this article in my email - from Alex Rosen. http://www.pentaho.com/docs/a_new_business_model_for_bi.pdf

While the problem to be solved for the client is not BI, there are a lot of behaviors in common between the paper and the experience with the vendor at this client.

We had gone through many of the due diligence stages, spoken to various vendors, created shortlists, evaluated Proofs of Concept until we were down to a vendor that we wished to trial more deeply.

There were some key things we wanted to find out, so we devised tests, questionnaires, scheduled calls, read as much documentation as the vendor would let us see, but we really felt that the vendor was being very stingy with information.

Often direct questions would be answered with, "It doesn't work that way" or "no we can't do that". In the vendor's mind this might have meant, "This is not in scope for the pilot." That isn't how it came across.

It took a huge amount of probing before the vendor finally started to be more forthcoming with information - at which point things went a lot more smoothly.

So, since this is a pattern oft repeated, why do many vendors behave this way?

I can't answer for this specific case, but some ideas include:

The process is still sales controlled and the sales team wants to control everything.
The process is still sales controlled, so it is important to marginalize the people who can't say yes (but who can say no).
The process is still sales controlled and the teams aren't used to the more implementation oriented lines of questioning
There are opportunities for consulting dollars, so by keeping the prospect in the dark there is more revenue.
The vendor believes that the solution takes some getting used to, so is rationing the information to prevent the customer from being overwhelmed.
The vendor doesn't have adequate documentation and doesn't wish to expose that fact just yet.
The vendor wants to see the possibilities of hard questions coming so that they can prepare for answers.

There may be other reasons that I am not aware of - and the ones I mention are pure speculation. On the current project, I can't say that any of these reasons are correct.

The key takeaway for me is to learn as much as possible about the solution before pilots. Get any FAQ documentation, get the configuration documentation so you can what's possible from the configuration. Ask the hard questions, don't take the answer, "You'll see it in the Pilot."

You are judged by the company you keep

2009-03-02T07:36:00.000-08:00

In the past week or so I have been "followed" by several quite unsavory characters offering services in which I have not the least interest. However, if someone were to plot my social graph, they might see these characters "following me", and draw a conclusion that I might have an interest.

Now if that someone were a prospective employer, or someone trying to establish that I was an unfit parent, I can imagine that my Twitter social graph could be of interest - and possibly grounds for non-hiring or withdrawal of parental rights.

Is that right? No I don't think it is - since I didn't do anything except exist - its the online equivalent of someone putting a flyer under my windshield wipers while parked at the airport. The trouble is that I at least know the flyer is there, the person who put it there (kinda) knows that s/he put it there, but no one else does unless it is a deliberate campaign of mis-information (putting kiddie porn magazines under an ex's wipers and then alerting authorities anonymously).

So, bottom line the associations you have are knowable to anyone with a mind to discover them, and thus open to all manner of misuse - of which blackmail is one of the less benign.

Agile methods and Design-Build

2009-03-01T07:03:00.001-08:00

I was intrigued by a term in my Sunday paper this morning. There is a project underway to extend the light rail line from Dallas to the DFW airport. It passes within a few miles of my house, so I am quite interested in seeing how it is going to look. The Mayor of Irving (Herber gears) wrote an opinion piece in the paper. In it he stressed the Design-Build approach to the project.

Design-Build sounds to me (at least on the face of it) an awful lot like Agile Development for software solutions. Perhaps it is the concrete version. Off to Wikipedia - http://en.wikipedia.org/wiki/Design-build (among other places) where I found these gems...

"Design-build focuses on combining the design, permit, and construction schedules in order to streamline the traditional design-bid-build environment. This does not shorten the time it takes to complete the individual tasks of creating construction documents (working drawings and specifications), acquiring building and other permits, or actually constructing the building. Instead, a design-build firm will strive to bring together design and construction professionals in a collaborative environment to complete these tasks at the same time."

"Potential problems of the design-build process include:
Premature cost estimating,
A short-cut design process,
Decreased accountability by the service provider, and
Correction of work. "

"It is important to note that the design-build method, while not focused on saving the owner construction costs, nonetheless often saves the owner money on the overall project. The combined effects of carrying a construction loan (which typically carries a higher interest rate than permanent financing) and an earlier useful on-line date usually yields considerable overall profitability to the project and may make seemingly unfeasible projects into genuine opportunities.
The compression is an important aspect of the implementation of this system. Other attributes include:
Enhanced communication between the service provider and the client,
Increased accountability by the service provider,
Single source project delivery, and
A value based project feedback system"

Sound familiar? - It sure does to me.

This is in contrast (as the rather opinionated Wikipedia artice states) to

"For nearly the entire twentieth century, the concept of Design-Build was classified as a non-traditional construction method in the United States, which is the last country to still embrace the old standard of Design-Bid-Build"

So if we think this through, we see the following important ideas:

The documents that are produced during the project are for the benefit of moving the project along, not for preparing bid-packages. That alone would appear to have tremendous potential to save time and money
There had better be some kind of a major plan in place first (City Plan anyone?) to ensure that the design-build doesn't go off the rails.
There had better be standards/codes in place so that we don't end up with shanty towns - the guage of the railway lines should be the same throughout, the standards for connection to standard services should be standard (electricity voltage, connectors, phases, frequency)
The approach is "owner driven" not architect or contractor driven
There is opportunity to adjust for changing requirements - new materials/material standards, unexpected terrain or landscape needs, changes in aesthetics,...

Of course Design-Build as an approach is not an excuse for no requirements - just as Agile Software Development does not mean, "We will come into the project with a bunch of good ideas and figure out the real requirements as we go along."

The Fat Controller

2009-02-26T08:17:00.000-08:00

In the wonderful "Railway Series" of children's books by the Rev. W.V. Awdry there is an officious character named "The Fat Controller". He is in charge of the railway and is often involved in activities that perhaps would have been better delegated. So, moving on many years and I have embraced the MVC architectural pattern (from my early Smalltalk times) and am actively involved in building a web application that uses Rails.

My own history is that I tend to work outwards from the model - after all the domain is where the interesting rules are enforced, and I can figure out how to implement those without stretching my lousy artistic design sense. After all who needs more than a command line to test a domain model?

Taking this inside out approach certainly isn't typical in many of the Rails books that I have been reading of late. I see crazy statements from "experts" like "the only things that are models are wrappers for ActiveRecords" If it doesn't inherit from ActiveRecord it should not be in the Models directory. So where do these soi-disant experts recommend putting such logic? Ahhh, in the Controller of course. How they reconcile that with DRY (Don't Repeat Yourself) principles defeats me. Oh, and what if we want several different ways of receiving and managing the same data (maybe a b2b feed, an email message, a web form, a tweet, a txt message, an iphone app...), where should that logic reside.

The cognoscenti imply that there should be a single controller for this, so we now have a soup of concerns not a separation of concerns. Putting all the above into a single controller results in creating a Fat Controller of whom the Rev. W.V. Awdry would be proud.

Just say no to fat controllers

Controllers don't have to be activated by views.
Models don't just wrap active records
We don't want fat controllers gumming up the works with their officiousness.

Sir Topham Hatt, please don't teach my controllers to become overweight!

Hijacking of Architecture

2009-02-09T09:39:00.000-08:00

In the beginning there was Zachman, three columns and the "Framework for Information Systems Architecture." This was originally published in 1987 in an IBM Systems Review, I believe.

There was much wailing and gnashing of teeth as it was realized that the three columns weren't enough. It needed to embody the 6 questions, "Who, What, Where, When, Why, How" at the 6 levels, so we have a 36 cell matrix. This matrix described what the "Information Systems Architecture" was all about - but didn't offer any prescriptions. How do you create a Business Logical "What?" (data) model? And how is that different from an Application Logical "What" (data) model? And while you are about it, how do you get traceability from one to the other?

Meanwhile the technologists got hold of the word architecture and immediately subverted it into the technology layers - essentially taking on the dimensions of design. So we have architects instead of designers, so those of us at the overall business and IT dimensions were left without a word to categorize ourselves. Job titles such as Java Architect start to cop up - and we immediately get pollution of the name space - or if you prefer overloading of the architect term. So when I would rock up and say to the business, "I am an architect and I am here to help you", I would get about the same reception as someone from the IRS - not very popular. The sorts of things I would hear are, "We aren't ready for programming yet." or "Has osmeone designed how this system is going to be put together?". In other words architecture became tactical, project based and confined to the technology realms.

So we look to other terms/phrases and 2 jump out. One is "City Planning" and the other is "Enterprise Architecture". Well it is hard to sell city planning to a company that makes shoes - not the sexiest term there is. And Enterprise Architecture? Well, that's another term that has been taken over by IT. Even frameworks like TOGAF (The Open Group Architecture Framework) have been more heavily focussed on the technology realm - but that does appear to be changing with version 9 (released Feb, 2009).

Enterprise Architecture Forums on Linkedin and Google seem also to be focussed on keeping track of physical artifacts and dealing (again) with the technology realm.

So it appears to me that the arcchitects are out of luck again. The technologists have coopted architecture at all levels.

So we perhaps should not be using the term architecture at all when trying to have sensible dialog with our business colleagues. We have done such a good job (as an industry) of confusing the term that it is time for a new one. No sooner have we coined it than it will become another victim of grandiloquence. Maybe we should use a term universally despised by the technology community (Analyst anyone?) because then it won't be coopted.

My friend Nigel Green talks about some of these issues in his blog http://bit.ly/1xrMTL. What those of us who straddle the Business/IT divide have to do is facilitate the communication across that divide. Using the language of business - using a framework like VPEC-T

Architecture and web companies

2009-01-28T08:27:00.001-08:00

I have been doing consulting work for a couple of companies whose products are entirely informational. Essentially companies that provide information services over the web. I have been struck at both of them about the mixing of the technology that delivers the "product" (often really a service, but it helps my head to think in terms of a product!) and the technology that runs the business.

An example from the genuine product world will help illustrate what I am thinking about. When making and selling hamburgers, there is a clear separation of what is delivered and how it is accounted for, tracked - essentially how the back office runs. The selling of hamburgers is a sufficiently different proposition than the creation of the back office business systems. I wouldn't attempt to combine the 2. Yes it is important that the sales information flows... (see earlier post on flow of goods, flow of money, flow of information). However, I don't have my fryer installation crew and my cooks building the systems.

Where the product is informational, companies often think that the product and the back office rely on IT, so they must be the same. So we have the people who think in terms of product, features, etc. responsible for the potentially more mundane chores of installing and managing the back end systems - giving the internal business the data it needs to run and manage the business, and the sales/support and other staff the tools they need to do the job.

In reality these are entirely different groups - and should be. Yes they might share common technology needs/data/platforms (although there is little guarantee even of that). Yes they may share communications infrastructure and communication methods/platforms. But in reality the activities required to deliver a world class product and the activities to provide robust "run/manage the business" systems are as different as flipping burgers and accounting for the flipped burgers. Mixing the teams (and thus not getting a proper separation of "IT" responsibilities) leads to some very brittle systems - often because the value of the "run/manage the business" applications is almost always subjugated to the "develop and operate the product" systems.

327x and web scalability

2009-01-27T10:45:00.001-08:00

These are strange bedfellows at first blush. But the more I think about them the more parallels I see.

Every book I read (and every web solution I build) talks at length about statelessness - especially session statelessness. Obviously data state is important, I really would like my bank account to know its balance and not derive it by applying all the transactions since the day I opened it every time I want to know my balance.

But I digress.

When I was a neophyte developer, the IBM 3270 family of "green screens" were just becoming mainstream. I had the enviable task of writing a series of macros in PL/I to emulate the behavior of the assembler macros for "basic mapping support." Fun project...

Anyhow in doing said project, I learned more about the 3270 than any human should have to. The key lesson of the device was that the hardware was directly addressable, had the concept of the "field" and would send back only fields that had the "Modified Data Tag" bit turned on. That meant that if a field were modified by a user, that field would be sent, but unchanged data wasn't. If nothing else that cut down the amount of data transmitted compared with approaches that refreshed the whole screen.

One much exploited approach was that the serving application could send the data out with the modified data tag already turned on. This of course meant that the device would send that data back regardless of whether the user actually modified the data or not. Immediately there was an opportunity to manage session state. Just send the stuff you needed next time back in a field with the modified data tag on. That way you have enough context for the next invocation.

The next leap was the ability to use "invisible" fields. Fields that were mapped to the screen but marked invisible (so you couldn't see their contents). Handy for passwords etc. However, if you set a field to invisible + modified data tag on, you could send suitable session data back, but the user at the screen didn't have to deal with it. You got the best of both worlds. Context information sent with the request and visible impact to the user.

Does this all sound familiar? f course nowadays it comes in in the header instead of the data, but it is the same general idea. If an architectural approach demands context data with every call, have the server send it back as part of the resource, so it automatically comes in on the transmission.

Plus ca change, plus c'est la meme chose!

Data ambiguity Part 1

2009-01-19T10:29:00.000-08:00

I was having breakfast on Saturday with an old friend. He pointed me to this article by Werner Vogels (Amazon.com CTO). http://www.allthingsdistributed.com/2008/12/eventually_consistent.html

This posting by Mr. Vogels is very insightful - along the data replica dimension of distributed data management. This is clearly an important dimension, but it isn't the only one. There is a more general problem of data ambiguity. This isn't necessarily just a database problem, but an overall systems problem.

The basic thought is that when you have two representations of a piece of data that are supposed to have the same value, when do they actually have the same value (and who cares?).

We can imagine the following cases.

The 2 values must always have identical values (to an outside observer). Inside the "transaction" that sets the state of a datum, the values can be mutually inconsistent, but that inconsistency is not manifested to an observer outside of the transaction.

The 2 values will need to be "eventually consistent" - this case is admirably covered by Mr. Vogels.

The 2 values will rarely have identical values, but there are mechanisms for "explaining" the discrepancies.

The first case is almost a default case - yes we would like that please. The second case is a good perspective from a data replication perspective - essentially dealing with a common schema. The third case is the tricky case.

The first case is unattainable at large scale using ACID transactions for replicas of data at Internet scale is simply impractical for performance.

The third case is interesting because of situations where "transactions" can occur against either copy of the data independently and in arbitrary sequences. The communication mechanism between the systems that can update copies of the data may be reliable, or they may be intermittent. That isn't completely the issue.

So, to illustrate this kind of system, let's take a popular application - Quicken. Many people use Quicken to manage their household accounts. The idea is to be able to use Quicken as a kind of front end to bank accounts - but it is only intermittently connected.

At any moment, the balance that Quicken reports and the balance that the bank reports are very likely to be different values. Of course from a data management perspective they are actually different fields, however that subtlety will be lost on the majority of users. Why will the 2 have different values for the balance field? There are lots of reasons, e.g.

Transactions have arrived at the bank without being notified to Quicken yet. For example, in an interest bearing account, the interest payment will be automatically added to the balance on the bank's view of the account. Or possibly a paid in check has bounced - the bank will have debited the check amount and (possibly) added a penalty.
Transactions are processed in a different sequence in general. When a user writes the checks, there is no guarantee that they will be processed by the bank in the order in which they were written (in fact, the policy varies, e.g. process the biggest checks first if there are many to be processed because that maximizes overdraft charges in the event that an account goes overdrawn).

These reasons boil down to the need to have system autonomy for throughput (imagine having to wait at the bank to process check 101 until check 100 had been processed).

Of course it doesn't matter to us that the systems are rarely fully synchronized, that the "balance" doesn't agree across them - we have accounting methods to help us reconcile. In other words we can accept that everything is OK without caring whether the systems have the same value of the balance.

Which communication method and when?

2009-01-16T11:25:00.000-08:00

While this posting doesn't just deal with Enterprise Architecture it does begin to explore how we choose the tool (from quite a wide array) that we might choose for a communication at any given moment.

Just thinking of my own case, I have an unseemly number of communication mechanisms/paths.

For work - 3 email accounts (my employer and 2 clients)
For my business 1 email account
For other purposes 5 email accounts

2 Instant Messenger accounts
2 Groove Ids (for secure file sharing and messaging)
Twitter
Posterous
2 blogs (cooking and this one)
Facebook

Telephone/voicemail (4 numbers) - 1 mobile 1 home, 2 clients
Several RSS feeds on news, technology, etc.
Text messaging
Corporate sharepoint
Client wiki

These obviously aren't all 2-way, but having 25 major channels - and following several news sources, a few Twitter folks that I follow(about 50) it is clear that I have oo much time on my hands!

So why do it? It really comes down to personae and convenience. Taking just the corporate emails - each company (including my employer) has its own email infrastructure. Each client uses its own email addressing scheme to send stuff around. I can't get from one client's system into another's (and nor should I be able to).

If I am doing frivolous things, I tend to use my hotmail account. If I am doing semi-serious, but still relatively public things, I use my gmail account. For my own business and when I know the person at the other end, I tpically use my own business email.

Twitter is a great source of interesting updates. Admittedly of the 500 or so Tweets/day that I receive, about 50 are interesting to me and about 30 really interesting. So my filters are not as good as they could be.

I use the phone, but not a lot. Most of my communication is asynchronous. I text a lot, contribute to my own blogs, read a bunch of news sources. The only things I don't seem to do are listen to/download music or video.

So why is this important from a business perspective? Because we each make our own choices about which media to use. The enterprise needs to enable many different channels for the various purposes.

Is Twitter a corporate tool? Absolutely - especially for corporate travel departments. It's the easiest way to get information out quickly.

Is Facebook a corporate tool? Absolutely - keping track of alumni, enabling corporate communities (extending the ecosystem).

Are blogs corporate tools? Absolutely - a great way for the corporation to provide an authentic experience to the community.

Is Groove a corporate tool? Absolutely - secure internal and external file and message sharing.

Is IM a corporate tool? Absolutely - again enabling community.

Is email a corporate tool - sadly yes. But as we have observed many times, it is very heavyweight. Sometimes the only way to get information in and out of corporations.

Phone/Voicemail? Absolutely.

I would argue that every form of communication that I use has its place in my daily corporate life. Even hotmail and gmail have helped when the corporate network is down and I have to get a response out.

Enterprise are really going to have to rethink communication - recognizing that critical information is going to leak over many channels. Draconian security groups will simply be bypassed since information will continue to flow.

Then we have the symmetry/asymmetry question. How much of what I do is simply reading other people's stuff (following them personally, subscribing to their publications or what?) vs engaging in dialog.

When dialog of some kind is needed, which of the many tools I have at my disposal do I use? My rule of thumb is whatever the person I am communicating with last used when talking to me. Of course it depends on whether it is a single short thought (twitter), a complex large file (Groove/email/SharePoint) or something in between....

If everything is moving to the network why do I want applications?

2009-01-06T09:36:00.000-08:00

There's an odd dichotomy happening. We are seeing pretty massive shifts to services obtained over the network for all sorts oif things (buying stuff being the most obvious, but there are so many). Yet we also insist on having our "applications" local too?

By local application, I mean a chunk of functionality that runs on a client device and must be installed independently of the rest of the functionality on the device. So the browser is an application, but things that run inside it aren't (at least not by this definition).

Much of what we can do with our smartphones, etc. can be done using a browser (possibly using the mobile version of the web site), but with some very specific look and feel needs, we tend to download and install specific applications. This is of course, especially true on the iPhone where the apple applications are legion and well liked by the applerati.

So what drives this?
First and foremost, I believe is the desire to be in control of one's own destiny. The networks are not ubiquitous enough yet that we can rely on them to have what we need available whenever we might need it.

Second network cost - that can be expensive after a while.

Third pure preference - we like the look and feel of the apps we install and not of those we don't

Fourth capability. Organizaing/classifying is a core "client side" requirement. Rich experiences for doing that preclude us from using totally network based approaches - although tagging really assist with this.

There are probably lots of other reasons, but with capability moving into the network, it seems strange that i-apps are growing in strength and popularity

Asymmetry - the word for 2009

2008-12-28T06:29:00.000-08:00

Almst all the dealings we have are asymmetrical, but somehow that hasn't been anything to worry about. We are, however, seeing asymmetry beginning to bite in many unpleasant ways.

The first is the power that organizations have over individuals. Random/arbitrary increases of interest rates on charge cards with little recourse. Banks making errors, taking a long time to pay what's owed and then when they pay it twice by mistake demand that the error be corrected, "Immediately or else...". The petty bureaucracies of home owners' associations - you can't park a Ford F150 in your driveway, but a Cadillac Escalade is OK (Thanks Frisco, Texas).

The second is in the everyday communcation between parties. There is the kind of power asymmetry described above, but then there is what I call "interest asymmetry". Where one party in a conversation say something - which is of no interest to the other party. We have all been involved in conversations with spouses, other family members, children, where something that is riveting to them is really dull to us. In the interests of harmony, I will not cite specific examples here....

At a wider level, we this interest asymmetry shoing up when we use social neyworking sites. We have the opportunity to converse with many people using these tools, but these conversations have inherent asymmetry too. What we choose to say is, at least, interesting to us. What we choose to "listen to" has variable degrees of utility. I am interested in family postings about the kids, but not teribly interested in the ins and outs of Cpmmercial Property Law in England (something my brother in law is an expert in). For non-family/non-friends I am typically interested in work related stuff, or special interests (food, sailing...). So when I see the jumb;ed stream of messages, I put filters on, e.g. "Oh this is Paul talking about LIBOR again, I think I will ignore it." or "This is Robin talking about the kids, Christmas trees, presents, etc." I will ignore that." The atter case because I follow Robin's inciteful postings on technology, but not on his children.

People who are followed by a large crowd (because of celebrity, interests, self-promotion) have an even more asymmetric communication approach using the media of social networking because they have so much to say, and limited opportuinty to listen if all their followers were to respond. While they will often have set themselves up with expectations of symmetric communication, the style quickly becomes asymmetric.

Pub-sub

2008-12-23T11:28:00.000-08:00

We often casually describe event driven architectural patterns as being "pub/sub". This is, of course, an over simplification and misses the point.

The key is to think of this kind of architecture is subscription driven or subscription dominated. This has been brought home. big time, in the social networking frameworks (like Twitter). People who post on Twitter essentially say whatever comes into their heads. We follow individuals or groups because, on balance, we get more out of following them than not. However, we will need to filter. For example, there are Twitterers who post about the industries that I am interested in, the beer they like to drink, their cooking interests, their children, their other hobbies,.....

I typically don't want to see all of that, but the poster shouldn't be deterred and stop. The poster's responsibility is to post. The listener's responsibility is to filter the dull stuff - or the stuff that is dull to the specific listener. That is often hard because the signal to noise ratio for any specific listener will be different than the signal to noise ratio for any other.

The4 same is true in any kind of event dominated system - human or otherwise. The listener is in a position to make decisions about what it is interested in, what it may respond to. The "teller" must continue to deliver the narrative.

Filing...

2008-12-23T11:19:00.000-08:00

In the old - pre-computer days, there used to be a job performed in every office, namely that of the filing clerk. The filing clerk was usually pretty low in the organization - often a school leaver with few qualifications, and with luck and many years experience could become a filing manager. In other words not a great value creating job for the corporation, but one where messing up could add a lot of cost.

Legion were the companies that I worked at where filing clerks had messed up - the most extreme being the travel industry person who didn't know what to do with the "audit coupons" on a paper airline ticket. She filed them in a shoe box under her desk...

Fast forward to the database world and guess what we have. A super fast filing system. So feeding the database is like feeding the filing cabinets. Stuff is put away so you can find it again, but it isn't the operational life blood of the company. The operational life blood is the interactions between humans, the interactions between systems - in reality the events that cause value to be created for the organization. Our systems are event driven and data-filed - not database driven, at least not if they are to be truly valuable and truly model the way that value is created in the information systems.

Auto insurance

2008-09-01T05:53:00.000-07:00

This posting is a reprise of an idea I put out in 2000. We were beginning to see the rise of "exchanges", Component models, interface based programming and other aids to distributed systems. I had 2 major interests at the time. One was in "edge" computing, the other in facades across business models.

I wrote a thought piece about auto insurance and a direct link between insurance and the vehicle. I was reminded of that piece by a TV commercial that I saw today. In the commercial, the car knew which insurance company was covering it, and if it didn't like the company, then it would find reasons not to go. In one case it ejected the keys from the ignition, in another it refused to unlock its doors and in the third case, the tires deflated.

The piece I wrote years ago also had knowledge between the car and the insurer, but had some more active features. For example, if we could envisage a car with multiple settings (the boring, around town go to work setting or the "boy racer" weekend setting, or others), then we could imagine having variable rate insurance - essentially the insurance premium is selected based on the car mode that is selected. If you never engage the "boy racer" mode, you never get charged that premium. So essentially we are looking at dynamic premium pricing based on a number of conditions all known about from the edge.

Lots of implications here. We could have a different key for each family member - or a key/thumbprint combo. If the thumbprint isn't a licensed (i.e. known to the insurance company) person, then the car doesn't start. So you could lock out unauthorized (friends of your children) drivers. What about speeding? The car knows that you have been speeding :-(. A car rental company (Acme) in Connecticut attempted to fine a renter for excessive speed. This was struck down in February 2002 by the Department of Consumer Protection. However, it may well be possible for insurance companies to use this kind of monitoring to assess premiums and to assist with accident "fault finding."

Lots of issues of course, but the key is to be thinking in these kinds of terms - where real time (or nearly real time information) can be used to guide decision making - especially pricing.

An interface or an implementation?

2008-07-15T11:38:00.000-07:00

I have been having conversations over the last few weeks about whether a service (in the SOA sense) is an interface or an implementation. This is a surprisingly tricky path to navigate.

At one client the following question (or a variant) comes up quite frequently, "We want to service enable our C++ back end code, but the services framework is all Java how can we get them to co-exist?"

So we have an implementation, we can't use it directly but we do want to leverage it. Clearly the implementation itself isn't the service. If it were there would be no discussion. Also the interfaces it already provides may not be the same as the operations we would want to offer.

So we take the time-honored approach and wrap the code. The wrapper then exposes the service's operations and deals with the complexity of mapping the operations to whatever the original code supported. Now what is the Service in this scenario? Perhaps it is the wrapper code - after all it is the wrapper that has the signature, the wrapper that is directly invoked, the wrapper that will have the nice QOS measures, the wrapper I will look up in my registry.... However the wrapper doesn't "do" anything. Of course if the wrapper is really generic and abstract then it doesn't help to describe the service as being the wrapper. No one will have a clue when they go to the registry (white pages) what the service does. A description like "Wraps existing C++ code so it can be made available in the services framework" is hardly a confidence booster.

So what to do? My general favorite is to use a special purpose framework (auto generated if possible) to wrap the C++ (or other legacy) code. Make sure that the service is the wrapper and not the C++ implementation and manage that in the registry.

After all, one of the SOA principles is autonomy. The actual implementation doesn't have to be constrained as long as the interface supports the appropriate operations.

Introduction to the GIM model

2008-07-15T10:27:00.000-07:00

At some primitive level there are three sets of things that flow in an organization. These are Goods, Information and Money. Not a shattering insight, but the interactions between these three categories give us some really good insights into how a business works. This isn't a typical customer focussed, outside in kind of view, but deals with what actually happens inside the business.

Why is this view of the business important?
It is holistic, actions that occur in one of these flows has impact on the others. Shipping the software to a customer (Physical Goods) can tell the money flow that revenue recognition can occur and can deliver information that the shipment has taken place.

Looking at the interactions of these flows gives us clues into:

Adherence to accounting practice (when do we recognize revenue, for example?)
Inefficiences in process (oh, we recalculate the revenue in many different places, even though nothing has changed)
Opportunities to improve quality of service (let's move the credit check - an information flow request) to earlier in the cycle so we don't incur costs before we know if the customer can/will pay)
Business modeling - "what if" (What happens if we attempt to move the credit check earlier?)
Real time information delivery from both Goods and Money (How many units did we ship today?)
Realization that the information need to control a Goods flow isn't available identifies a process breakdown

Ultimately, of course, what we deal with in information systems are informational abstractions of the Goods and Money. For us in IT it is all information. However to the business it isn't so, although we use the lens of the information system to help us look at the underlying realities. Our information system lenses are so distorted, however, that we often don't or can't know what is properly in or out of focus.

There are some real nuances to worry about here too. For example, in a content delivery system (e.g. a newspaper), the information (content) is treated as the Goods. Likewise in banking much of the money that flows through the bank is actually treated as Goods - but with a significant impact on Money and Information as well.

It is not trivial to create a GIM model, but the effect is enormous:

Silo thinking is reduced - all streams can see the effects on each other of actions taken. Moving Credit Check later in the Information Stream affects the Goods stream because it will curtail process actions.
Sub-optimal decision making is highlighted - "If I take this action, what breaks?
It provides a common "grammar" for talking about the business between the business and IT - a goal that has not been reached in many years of trying.
An end to end trace of a business process - with all the stakeholders shown can be built and optimized.

Why doesn't a process model work?

This is a form of process model - one where we specifically illustrate what happens to all of the major "data" sets (Goods, Information, Money). So instead of being a step by step way through the process, it is a way to handle the information exchanges more effectively. So it is an enriched process model.

What other models help?

We can leverage many of the standard models (e.g. in UML), but nothing gives us a complete enough view.

How do we do it?

It revolves around a fundamental understanding of business modeling - starting with the operational flow of goods. It is after all the operational flow of goods from start to finish that defines what the business delivers. So drive from flow of goods - starting anywhere. That depends on the scope and depends on the business. The key is that it is an operational goods flow that matters. That operational goods flow means starting with operational process.

In the operational goods flow, swim lanes and swimming pools are appropriate, but often it is a good idea to use business iconography and not boxes/lines to show things. That way it illustrates that we are firmly in the business domain, not IT.

For each step in the handling of the flow ask:

Has this step had any effect on the Money side - e.g. recognized revenue, incurred measureable cost, disposed of an asset...
What information might be available as a result of this action being taken. Is the information interesting "real time", is the information needed for subsequent analysis, is the information in any sense privileged?

Yes, the questions really are that simple. The answers aren't and the implications aren't, but the questions are. What you do with the wealth of information is another matter.

We will notice over time that the timing needs of the Money flow and the Information flow don't exactly match the Goods flow. This is to be expected. We will also notice that the goods flow will continue regardless of whether the Money and Information flows are keeping up. In many cases we will see that the other flows play "catchup" and deal with the implications of Goods flow later. This is a primary cause for inconsistencies between different parts of the business and difficulties in rationalization.

It is of course impossible to fully serialize the business so that the flow of Goods is interrupted until all the Money and Information flows have properly completed. Thta's one reason why we look at the physical Goods flow - that simply doesn't wait.

In subsequent articles, I will talk about some of the interesting patterns and interactions that we see when doing this kind of modeling.

What can SOA learn from RDBMS?

2008-06-13T07:29:00.000-07:00

At first blush, there doesn't look to be a lot of synergy between Service Oriented Architecture and relational database management systems (RDBMS). The primary function of a database management system is, after all, to manage the storage, retrieval and integrity of the data it is called upon to manage. It's that last word, integrity, that provides insight into commonality.

In the late 1980s Sybase introduced the notion of triggers into their flagship RDBMS product. A trigger was a piece of code that was executed when "something interesting" happened to the data under management. Triggers provided a very flexible way of reacting to changes like, for example, deleting a row from the database. That deletion could then cause a series of other actions (usually embedded in stored procedures), for example cascading the delete of one row to others - and thus enforcing referential integrity. Quite an elegant solution at the database level.

Of course, being about the only capability available for guaranteeing that the events would be detected, triggers started to get abused, whole rafts of business logic were embedded in stored procedures, and the database became the logic engine and the storage engine. No real separation of concerns there.

Now let's forward to 2008 and look for parallels and opportunities. For transaction processing, we are beginning to see much lighter weight data management engines (look at Google's bigtable implementation or Amazon's simpledb). Business logic is being pushed into services - probably where it should be. That leaves us with a bit of a hole. The value that triggers provided is still needed, but now it should be at the same level of data abstraction as the data managed by the services. That level is typically at the business object level.

So taking the simple concept of a trigger, we can ask ourselves if there is value to the enterprise in knowing when "interesting" things happen to business objects. (Of course this is still too broad, but bear with me here). There are some business events that are quite interesting to know about. For example a big customer win (resulting in the creation of a new customer in business terms) is likely to be very interesting to the organization as a whole. Can be notified as a morale booster/internal PR operation, can add some heft to other sales activities,... The list is endless. It isn't sensible to notify this at the database level, that is too low level, to proprietary. It makes a whole lot more sense to publish the "event" on the corporate internal nervous system.

It is impossible for the customer management service to know who or what might be interested in the information. Just as in the database world, the table with the "add trigger" doesn't actually know which other tables might be affected. All the customer management service knows is "that something just happened and there may be others interested". It then becomes the job of the corporate nervous system to let this event be "broadcast". Anything interested can then pick it up and make its own decisions.

It is therefore the responsibility of the ESB to provide this kind of capability in the containers that surround and manage the business objects, just as it was the DBMS's responsibility a generation earlier. It is the enterprise business architecture's job to determine which of these events should be triggered, and how the interested parties might react.

No there isn't much new here, we have had pub/sub architectures before - usually to aid with application integration. What is new here is that we can use the standards based capabilities that our SOA frameworks give us, together with a firm understanding of the need to drive change, to guide us to delivering a more flexible, sustainable architecture.

SOA is from Mars EDA is from Venus

2008-05-07T18:01:00.000-07:00

It seems that in most Service Oriented Architecture deployments, the predominant paradigm is request/response. Consumer A wants something from Service B. Essentially it is an RPC like model, but with extra overhead. This model has been dominant because of several factors:

We can think of it like an extension of the earliest programming models we ever dealt with - subroutines and functions
We don't have to get all snarled up in worrying about parallel processing/multi-tasking and other hard to debug kinds of models
We can break up problems into conveniently sized chunks by some kind of hierarchic decomposition
All languages support the model intrinsically.

Does this mean that it is the best way? Perhaps for some problems, but surely not for all. I see a subroutine or function as being a rather surly, uncommunicative programming equivalent of a slob sitting in a chair, swilling beer and only responding when he feels like doing so. Very good at doing one thing at a time, easy to discover (after all who can miss the chair, beer can and TV remote).

This model is so dominant that we have even simulated it on top of asynchronous protocols. So even in a system that doesn't support request/response directly (e.g. an asynchronous, message based system) we still allow for the same semantics, but this time fudging them to use pairs of asynchronous messages. Changing the transport doesn't really change the programming model. We still think in RPC terms.

Enter SOA and we often have the same view. This time instead of an RPC, we sometimes refer to a "Service Call". Immediately we should be on the defensive. Call implies request/response. Should that be what our services do? Well, of course sometimes they should.

Now let's look at things from a different perspective. In an Event Driven Architecture, when something happens, the component that realizes something has happened feels the need to communicate this goodness to anyone who will listen. Some listeners will take action as a result. It is almost a gossiping model, a model where as soon as information is available, it is able to be shared. Not only does sharing take place, but it is absolutely impossible for the originator or broadcaster of the information to control what happens to it. It is out in the aether with all sorts of actions taking place as a result. Some of the actions might be beneficial, some not.

So here, we have a much lighter weight, less surly architecture an approach where the actions are distinguished from the notifications, where the actors are actively listening and making decisions for themselves. An architecture that exhibits autonomy and a loosely coupled style. Of course the trouble with a gossiping style is that a large amount of information is made available, only some of which will be acted on. So it is very "chatty" Chattiness in IT systems correlates well with increased resource consumption. In constrained systems (especially those that are network or disk constrained) increasing resource consumption can have a large, and negative impact on the cost of doing business.

In the EDA model, information "publishers" and "consumers" are constantly doing other things. They are true multi-taskers, ever prioritizing their activities, deciding whether to wait for more information or whether to act immediately. Capable of acting and absorbing information at the same time, making complex decisions largely autonomously. In other words, there is less friction in this style of interaction.

Information, of course, doesn't need to be acted on immediately. A useful tidbit might be recognized a long time after some other tidbit, making a situation clear. The arrival of the final tidbit can trigger a complex set of actions, relying on information disassociated from the original event, but somehow dependent on it.

Bottom line, we need to understand when it is appropriate to use one style or another. Sometimes it is good for a bunch of slobs to hang out, grunt and drink beer. At other times a more nimble, more overlapping, less controlling approach can deliver systems that function more smoothly.