Innovation and incremental improvement

I was watching the Formula 1 race from Malaysia this morning. It was on live and early, so I had the TV to myself. For the past several years F1 has been dominated by McLaren and Ferrari. For the past few years the changes to the formula have been relatively small.

This year, however, the FIA have made some pretty dramatic changes – resulting in a major shake-up. The old factory Honda team is no more, but has become reborn as the Brawn F1 team (Ross Brawn, the man behind the rise of Michael Schumacher being the team principal). Toyota have also done well, with the red bull/toro rosso teams also having good outings.

So, what changed. I contend that this is a great example of the difference between steady, linear improvement as managed in lean/six sigma kinds of processes, and the need to be radically different as is the case when major innovation is necessary.

The FIA changed the rules dramatically. Back to slick tires, much reduced rear wing area, the Kinetic Energy Recovery System (KERS) which charges batteries under braking so the power can be deployed at times to suit the driver. Adding an extra 80 hp for about 6 seconds a lap. Engine revs reduced to 18,000, etc.

The teams that appeared to treat the rule changes as evolutionary are doing poorly/ The teams that recognized the radical nature of the changes – and with little to lose, are doing much better.

So, perhaps where innovation is concerned we should not try to put it into well defined, six sigma, DMAIC based processes. Let the creative juices flow, make changes in a non linear fashion until the platform has become relatively stable and then shift to six sigma type approaches.

Entropy in Systems

The concept of entropy in thermodynamics is well known. This article covers the landscape well. In systems, especially systems that involve human and computer interactions, we have a similar notion.

Entropy, historically, has often been associated with the amount of order, disorder, and/or chaos in a thermodynamic system. The traditional definition of entropy is that it refers to changes in the status quo of the system and is a measure of "molecular disorder" and the amount of wasted energy in a dynamical energy transformation from one state or form to another.

So too in our information systems. The amount of effort that we have to undertake to move a system from one state to another (essentially to make a modification to it) is in 2 parts. The effort expended towards making the change useful, and all the rest of the effort.

This applies both to the change to the system itself and any changes that we make as a result of executing the system. So, for example, if we consider some system fragment, "Visit the doctor because of pain in the shoulder", then any time/effort spent in doing something other than getting the diagnosis/treatment is wasted and increases the system entropy. This time/effort may include (but is not limited to)

• Getting to the Dr.
  
• Filling in new patient paperwork
  
• Having payer status checked
  
• Explaining symptoms to receptionist
  
• Waiting in waiting room
  
• Waiting in treatment room
  
• Waiting for X-Ray results
  
• Driving to radiology lab
  
• Filling in radiology lab paperwork
  
• Waiting at radiology lab
  
• …
  

The point of this is that this "system" is unbelievably wasteful of a precious resource (at least precious to me), my time. So from my perspective all of the above steps indicate great inefficiency. Perhaps because of complexity, perhaps because of a lack of cohesive thinking.

Considering another example, perhaps closer to work for many – the HR portal. That is often the one information system that has been designed to almost entirely ignore the majority of its users. There is often a huge learning curve for the majority of the employees. Of course the users who specify the system, the HR department have it well designed for their own convenience – and what they believe is the convenience of the employees. I leave you to draw your own conclusions!

So at one level, we have the idea of the system in use with every use increasing the entropy of the system.

Now think about attempting to make a change to a system. A whole new dynamic sets in. The need to understand the system in place so it can be changed. This can involve very detailed analysis – deep understanding. The more pieces there are – and the more interconnections, the more understanding there has to be. So depending on the design of the system in place there can be a greater or lesser effect on its entropy. If the system is very involved/convoluted then the understanding as a percentage of the useful work done will be high. If the system is relatively straightforward then the understanding as a percentage of the useful work done will be less.

So systems entropy might be thought of as ( _i=0 ⁿ∑(W_i - V_i)) where W is the work performed at any state change I and V is the Valuable work performed at any state change i.

Defining and normalizing what we mean by work – and considering some normalized work value equation is, of course complex. For example in the getting shoulder diagnosed and treated, the system implicitly values the Dr.'s time as being more valuable than mine – so the system is optimized to make sure that the Dr.'s change entropy is least. What should be happening is that the total change in entropy should be minimized.

Typically systems that are overly complex, overly bureaucratic or optimized to support a minimal number of stakeholders will exhibit the greatest increases in entropy under a given state change.

As architects we have a responsibility to be looking out across the landscape of a system as a whole and finding ways of minimizing the increases in entropy across common state changes.

Prototyping in rails

Now this is a truly bizarre thought. Many business apps and the underlying databases are a bit more complicated than the standard web app. Occasionally it is even necessary to populate the model that you plan to implement and let the users test drive the model through the UI.

This approach used to be called rapid-prototyping, but has rather fallen out of favor.

However, as the browser is becoming (has become) the dominant UI now, there is perhaps an opportunity to do a rapid prototype of some system functionality so the users can be assured that you have the relationships right and that you can quickly demonstrate some scenarios.

This tends to be a royal pain, and then it hit me. Rails is such a quick to develop framework that getting some gnarly prototypes out quickly is very little problem. Of course we don't want to be too convincing because the users might think we have done the hard bits. But really what we will have done is:

Create a simple database model with the interesting relationships (keys/fks, etc.)
Populated that model with some made up data - careful cases
Allowed the users to manipulate/navigate the data through a familiar approach. For example in an auction site, you may have many bids. So a 1:m relationship there. We could show an auction with its many bids easily, then click on a bid to go deeper, ensuring we have a link back to the auction.

using RESTful principles and RAILS quite a lot of this is generated with the scaffold. That's a lot of power.

Evaluating enterprise software

Two things have happened at pretty much the same time. The first is an evaluation of some enterprise class software in conjunction with and on behalf of a client. The second, the appearance of this article in my email - from Alex Rosen. http://www.pentaho.com/docs/a_new_business_model_for_bi.pdf

While the problem to be solved for the client is not BI, there are a lot of behaviors in common between the paper and the experience with the vendor at this client.

We had gone through many of the due diligence stages, spoken to various vendors, created shortlists, evaluated Proofs of Concept until we were down to a vendor that we wished to trial more deeply.

There were some key things we wanted to find out, so we devised tests, questionnaires, scheduled calls, read as much documentation as the vendor would let us see, but we really felt that the vendor was being very stingy with information.

Often direct questions would be answered with, "It doesn't work that way" or "no we can't do that". In the vendor's mind this might have meant, "This is not in scope for the pilot." That isn't how it came across.

It took a huge amount of probing before the vendor finally started to be more forthcoming with information - at which point things went a lot more smoothly.

So, since this is a pattern oft repeated, why do many vendors behave this way?

I can't answer for this specific case, but some ideas include:

• The process is still sales controlled and the sales team wants to control everything.
• The process is still sales controlled, so it is important to marginalize the people who can't say yes (but who can say no).
• The process is still sales controlled and the teams aren't used to the more implementation oriented lines of questioning
• There are opportunities for consulting dollars, so by keeping the prospect in the dark there is more revenue.
• The vendor believes that the solution takes some getting used to, so is rationing the information to prevent the customer from being overwhelmed.
• The vendor doesn't have adequate documentation and doesn't wish to expose that fact just yet.
  
• The vendor wants to see the possibilities of hard questions coming so that they can prepare for answers.

There may be other reasons that I am not aware of - and the ones I mention are pure speculation. On the current project, I can't say that any of these reasons are correct.

The key takeaway for me is to learn as much as possible about the solution before pilots. Get any FAQ documentation, get the configuration documentation so you can what's possible from the configuration. Ask the hard questions, don't take the answer, "You'll see it in the Pilot."

You are judged by the company you keep

In the past week or so I have been "followed" by several quite unsavory characters offering services in which I have not the least interest. However, if someone were to plot my social graph, they might see these characters "following me", and draw a conclusion that I might have an interest.

Now if that someone were a prospective employer, or someone trying to establish that I was an unfit parent, I can imagine that my Twitter social graph could be of interest - and possibly grounds for non-hiring or withdrawal of parental rights.

Is that right? No I don't think it is - since I didn't do anything except exist - its the online equivalent of someone putting a flyer under my windshield wipers while parked at the airport. The trouble is that I at least know the flyer is there, the person who put it there (kinda) knows that s/he put it there, but no one else does unless it is a deliberate campaign of mis-information (putting kiddie porn magazines under an ex's wipers and then alerting authorities anonymously).

So, bottom line the associations you have are knowable to anyone with a mind to discover them, and thus open to all manner of misuse - of which blackmail is one of the less benign.

Agile methods and Design-Build

I was intrigued by a term in my Sunday paper this morning. There is a project underway to extend the light rail line from Dallas to the DFW airport. It passes within a few miles of my house, so I am quite interested in seeing how it is going to look. The Mayor of Irving (Herber gears) wrote an opinion piece in the paper. In it he stressed the Design-Build approach to the project.

Design-Build sounds to me (at least on the face of it) an awful lot like Agile Development for software solutions. Perhaps it is the concrete version. Off to Wikipedia - http://en.wikipedia.org/wiki/Design-build (among other places) where I found these gems...

"Design-build focuses on combining the design, permit, and construction schedules in order to streamline the traditional design-bid-build environment. This does not shorten the time it takes to complete the individual tasks of creating construction documents (working drawings and specifications), acquiring building and other permits, or actually constructing the building. Instead, a design-build firm will strive to bring together design and construction professionals in a collaborative environment to complete these tasks at the same time."

"Potential problems of the design-build process include:
Premature cost estimating,
A short-cut design process,
Decreased accountability by the service provider, and
Correction of work. "

"It is important to note that the design-build method, while not focused on saving the owner construction costs, nonetheless often saves the owner money on the overall project. The combined effects of carrying a construction loan (which typically carries a higher interest rate than permanent financing) and an earlier useful on-line date usually yields considerable overall profitability to the project and may make seemingly unfeasible projects into genuine opportunities.
The compression is an important aspect of the implementation of this system. Other attributes include:
Enhanced communication between the service provider and the client,
Increased accountability by the service provider,
Single source project delivery, and
A value based project feedback system"

Sound familiar? - It sure does to me.

This is in contrast (as the rather opinionated Wikipedia artice states) to

"For nearly the entire twentieth century, the concept of Design-Build was classified as a non-traditional construction method in the United States, which is the last country to still embrace the old standard of Design-Bid-Build"

So if we think this through, we see the following important ideas:

• The documents that are produced during the project are for the benefit of moving the project along, not for preparing bid-packages. That alone would appear to have tremendous potential to save time and money
• There had better be some kind of a major plan in place first (City Plan anyone?) to ensure that the design-build doesn't go off the rails.
• There had better be standards/codes in place so that we don't end up with shanty towns - the guage of the railway lines should be the same throughout, the standards for connection to standard services should be standard (electricity voltage, connectors, phases, frequency)
• The approach is "owner driven" not architect or contractor driven
• There is opportunity to adjust for changing requirements - new materials/material standards, unexpected terrain or landscape needs, changes in aesthetics,...

Of course Design-Build as an approach is not an excuse for no requirements - just as Agile Software Development does not mean, "We will come into the project with a bunch of good ideas and figure out the real requirements as we go along."

The Fat Controller

In the wonderful "Railway Series" of children's books by the Rev. W.V. Awdry there is an officious character named "The Fat Controller". He is in charge of the railway and is often involved in activities that perhaps would have been better delegated. So, moving on many years and I have embraced the MVC architectural pattern (from my early Smalltalk times) and am actively involved in building a web application that uses Rails.

My own history is that I tend to work outwards from the model - after all the domain is where the interesting rules are enforced, and I can figure out how to implement those without stretching my lousy artistic design sense. After all who needs more than a command line to test a domain model?

Taking this inside out approach certainly isn't typical in many of the Rails books that I have been reading of late. I see crazy statements from "experts" like "the only things that are models are wrappers for ActiveRecords" If it doesn't inherit from ActiveRecord it should not be in the Models directory. So where do these soi-disant experts recommend putting such logic? Ahhh, in the Controller of course. How they reconcile that with DRY (Don't Repeat Yourself) principles defeats me. Oh, and what if we want several different ways of receiving and managing the same data (maybe a b2b feed, an email message, a web form, a tweet, a txt message, an iphone app...), where should that logic reside.

The cognoscenti imply that there should be a single controller for this, so we now have a soup of concerns not a separation of concerns. Putting all the above into a single controller results in creating a Fat Controller of whom the Rev. W.V. Awdry would be proud.

Just say no to fat controllers

Controllers don't have to be activated by views.
Models don't just wrap active records
We don't want fat controllers gumming up the works with their officiousness.

Sir Topham Hatt, please don't teach my controllers to become overweight!

Hijacking of Architecture

In the beginning there was Zachman, three columns and the "Framework for Information Systems Architecture." This was originally published in 1987 in an IBM Systems Review, I believe.

There was much wailing and gnashing of teeth as it was realized that the three columns weren't enough. It needed to embody the 6 questions, "Who, What, Where, When, Why, How" at the 6 levels, so we have a 36 cell matrix. This matrix described what the "Information Systems Architecture" was all about - but didn't offer any prescriptions. How do you create a Business Logical "What?" (data) model? And how is that different from an Application Logical "What" (data) model? And while you are about it, how do you get traceability from one to the other?

Meanwhile the technologists got hold of the word architecture and immediately subverted it into the technology layers - essentially taking on the dimensions of design. So we have architects instead of designers, so those of us at the overall business and IT dimensions were left without a word to categorize ourselves. Job titles such as Java Architect start to cop up - and we immediately get pollution of the name space - or if you prefer overloading of the architect term. So when I would rock up and say to the business, "I am an architect and I am here to help you", I would get about the same reception as someone from the IRS - not very popular. The sorts of things I would hear are, "We aren't ready for programming yet." or "Has osmeone designed how this system is going to be put together?". In other words architecture became tactical, project based and confined to the technology realms.

So we look to other terms/phrases and 2 jump out. One is "City Planning" and the other is "Enterprise Architecture". Well it is hard to sell city planning to a company that makes shoes - not the sexiest term there is. And Enterprise Architecture? Well, that's another term that has been taken over by IT. Even frameworks like TOGAF (The Open Group Architecture Framework) have been more heavily focussed on the technology realm - but that does appear to be changing with version 9 (released Feb, 2009).

Enterprise Architecture Forums on Linkedin and Google seem also to be focussed on keeping track of physical artifacts and dealing (again) with the technology realm.

So it appears to me that the arcchitects are out of luck again. The technologists have coopted architecture at all levels.

So we perhaps should not be using the term architecture at all when trying to have sensible dialog with our business colleagues. We have done such a good job (as an industry) of confusing the term that it is time for a new one. No sooner have we coined it than it will become another victim of grandiloquence. Maybe we should use a term universally despised by the technology community (Analyst anyone?) because then it won't be coopted.

My friend Nigel Green talks about some of these issues in his blog http://bit.ly/1xrMTL. What those of us who straddle the Business/IT divide have to do is facilitate the communication across that divide. Using the language of business - using a framework like VPEC-T

Architecture and web companies

I have been doing consulting work for a couple of companies whose products are entirely informational. Essentially companies that provide information services over the web. I have been struck at both of them about the mixing of the technology that delivers the "product" (often really a service, but it helps my head to think in terms of a product!) and the technology that runs the business.

An example from the genuine product world will help illustrate what I am thinking about. When making and selling hamburgers, there is a clear separation of what is delivered and how it is accounted for, tracked - essentially how the back office runs. The selling of hamburgers is a sufficiently different proposition than the creation of the back office business systems. I wouldn't attempt to combine the 2. Yes it is important that the sales information flows... (see earlier post on flow of goods, flow of money, flow of information). However, I don't have my fryer installation crew and my cooks building the systems.

Where the product is informational, companies often think that the product and the back office rely on IT, so they must be the same. So we have the people who think in terms of product, features, etc. responsible for the potentially more mundane chores of installing and managing the back end systems - giving the internal business the data it needs to run and manage the business, and the sales/support and other staff the tools they need to do the job.

In reality these are entirely different groups - and should be. Yes they might share common technology needs/data/platforms (although there is little guarantee even of that). Yes they may share communications infrastructure and communication methods/platforms. But in reality the activities required to deliver a world class product and the activities to provide robust "run/manage the business" systems are as different as flipping burgers and accounting for the flipped burgers. Mixing the teams (and thus not getting a proper separation of "IT" responsibilities) leads to some very brittle systems - often because the value of the "run/manage the business" applications is almost always subjugated to the "develop and operate the product" systems.

327x and web scalability

These are strange bedfellows at first blush. But the more I think about them the more parallels I see.

Every book I read (and every web solution I build) talks at length about statelessness - especially session statelessness. Obviously data state is important, I really would like my bank account to know its balance and not derive it by applying all the transactions since the day I opened it every time I want to know my balance.

But I digress.

When I was a neophyte developer, the IBM 3270 family of "green screens" were just becoming mainstream. I had the enviable task of writing a series of macros in PL/I to emulate the behavior of the assembler macros for "basic mapping support." Fun project...

Anyhow in doing said project, I learned more about the 3270 than any human should have to. The key lesson of the device was that the hardware was directly addressable, had the concept of the "field" and would send back only fields that had the "Modified Data Tag" bit turned on. That meant that if a field were modified by a user, that field would be sent, but unchanged data wasn't. If nothing else that cut down the amount of data transmitted compared with approaches that refreshed the whole screen.

One much exploited approach was that the serving application could send the data out with the modified data tag already turned on. This of course meant that the device would send that data back regardless of whether the user actually modified the data or not. Immediately there was an opportunity to manage session state. Just send the stuff you needed next time back in a field with the modified data tag on. That way you have enough context for the next invocation.

The next leap was the ability to use "invisible" fields. Fields that were mapped to the screen but marked invisible (so you couldn't see their contents). Handy for passwords etc. However, if you set a field to invisible + modified data tag on, you could send suitable session data back, but the user at the screen didn't have to deal with it. You got the best of both worlds. Context information sent with the request and visible impact to the user.

Does this all sound familiar? f course nowadays it comes in in the header instead of the data, but it is the same general idea. If an architectural approach demands context data with every call, have the server send it back as part of the resource, so it automatically comes in on the transmission.

Plus ca change, plus c'est la meme chose!

Data ambiguity Part 1

I was having breakfast on Saturday with an old friend. He pointed me to this article by Werner Vogels (Amazon.com CTO). http://www.allthingsdistributed.com/2008/12/eventually_consistent.html



This posting by Mr. Vogels is very insightful - along the data replica dimension of distributed data management. This is clearly an important dimension, but it isn't the only one. There is a more general problem of data ambiguity. This isn't necessarily just a database problem, but an overall systems problem.



The basic thought is that when you have two representations of a piece of data that are supposed to have the same value, when do they actually have the same value (and who cares?).



We can imagine the following cases.

1. The 2 values must always have identical values (to an outside observer). Inside the "transaction" that sets the state of a datum, the values can be mutually inconsistent, but that inconsistency is not manifested to an observer outside of the transaction.


2. The 2 values will need to be "eventually consistent" - this case is admirably covered by Mr. Vogels.


3. The 2 values will rarely have identical values, but there are mechanisms for "explaining" the discrepancies.

The first case is almost a default case - yes we would like that please. The second case is a good perspective from a data replication perspective - essentially dealing with a common schema. The third case is the tricky case.

The first case is unattainable at large scale using ACID transactions for replicas of data at Internet scale is simply impractical for performance.

The third case is interesting because of situations where "transactions" can occur against either copy of the data independently and in arbitrary sequences. The communication mechanism between the systems that can update copies of the data may be reliable, or they may be intermittent. That isn't completely the issue.

So, to illustrate this kind of system, let's take a popular application - Quicken. Many people use Quicken to manage their household accounts. The idea is to be able to use Quicken as a kind of front end to bank accounts - but it is only intermittently connected.

At any moment, the balance that Quicken reports and the balance that the bank reports are very likely to be different values. Of course from a data management perspective they are actually different fields, however that subtlety will be lost on the majority of users. Why will the 2 have different values for the balance field? There are lots of reasons, e.g.

• Transactions have arrived at the bank without being notified to Quicken yet. For example, in an interest bearing account, the interest payment will be automatically added to the balance on the bank's view of the account. Or possibly a paid in check has bounced - the bank will have debited the check amount and (possibly) added a penalty.
• Transactions are processed in a different sequence in general. When a user writes the checks, there is no guarantee that they will be processed by the bank in the order in which they were written (in fact, the policy varies, e.g. process the biggest checks first if there are many to be processed because that maximizes overdraft charges in the event that an account goes overdrawn).

These reasons boil down to the need to have system autonomy for throughput (imagine having to wait at the bank to process check 101 until check 100 had been processed).

Of course it doesn't matter to us that the systems are rarely fully synchronized, that the "balance" doesn't agree across them - we have accounting methods to help us reconcile. In other words we can accept that everything is OK without caring whether the systems have the same value of the balance.

Which communication method and when?

While this posting doesn't just deal with Enterprise Architecture it does begin to explore how we choose the tool (from quite a wide array) that we might choose for a communication at any given moment.

Just thinking of my own case, I have an unseemly number of communication mechanisms/paths.

For work - 3 email accounts (my employer and 2 clients)
For my business 1 email account
For other purposes 5 email accounts

2 Instant Messenger accounts
2 Groove Ids (for secure file sharing and messaging)
Twitter
Posterous
2 blogs (cooking and this one)
Facebook

Telephone/voicemail (4 numbers) - 1 mobile 1 home, 2 clients
Several RSS feeds on news, technology, etc.
Text messaging
Corporate sharepoint
Client wiki

These obviously aren't all 2-way, but having 25 major channels - and following several news sources, a few Twitter folks that I follow(about 50) it is clear that I have oo much time on my hands!

So why do it? It really comes down to personae and convenience. Taking just the corporate emails - each company (including my employer) has its own email infrastructure. Each client uses its own email addressing scheme to send stuff around. I can't get from one client's system into another's (and nor should I be able to).

If I am doing frivolous things, I tend to use my hotmail account. If I am doing semi-serious, but still relatively public things, I use my gmail account. For my own business and when I know the person at the other end, I tpically use my own business email.

Twitter is a great source of interesting updates. Admittedly of the 500 or so Tweets/day that I receive, about 50 are interesting to me and about 30 really interesting. So my filters are not as good as they could be.

I use the phone, but not a lot. Most of my communication is asynchronous. I text a lot, contribute to my own blogs, read a bunch of news sources. The only things I don't seem to do are listen to/download music or video.

So why is this important from a business perspective? Because we each make our own choices about which media to use. The enterprise needs to enable many different channels for the various purposes.

Is Twitter a corporate tool? Absolutely - especially for corporate travel departments. It's the easiest way to get information out quickly.

Is Facebook a corporate tool? Absolutely - keping track of alumni, enabling corporate communities (extending the ecosystem).

Are blogs corporate tools? Absolutely - a great way for the corporation to provide an authentic experience to the community.

Is Groove a corporate tool? Absolutely - secure internal and external file and message sharing.

Is IM a corporate tool? Absolutely - again enabling community.

Is email a corporate tool - sadly yes. But as we have observed many times, it is very heavyweight. Sometimes the only way to get information in and out of corporations.

Phone/Voicemail? Absolutely.

I would argue that every form of communication that I use has its place in my daily corporate life. Even hotmail and gmail have helped when the corporate network is down and I have to get a response out.

Enterprise are really going to have to rethink communication - recognizing that critical information is going to leak over many channels. Draconian security groups will simply be bypassed since information will continue to flow.

Then we have the symmetry/asymmetry question. How much of what I do is simply reading other people's stuff (following them personally, subscribing to their publications or what?) vs engaging in dialog.

When dialog of some kind is needed, which of the many tools I have at my disposal do I use? My rule of thumb is whatever the person I am communicating with last used when talking to me. Of course it depends on whether it is a single short thought (twitter), a complex large file (Groove/email/SharePoint) or something in between....

If everything is moving to the network why do I want applications?

There's an odd dichotomy happening. We are seeing pretty massive shifts to services obtained over the network for all sorts oif things (buying stuff being the most obvious, but there are so many). Yet we also insist on having our "applications" local too?

By local application, I mean a chunk of functionality that runs on a client device and must be installed independently of the rest of the functionality on the device. So the browser is an application, but things that run inside it aren't (at least not by this definition).

Much of what we can do with our smartphones, etc. can be done using a browser (possibly using the mobile version of the web site), but with some very specific look and feel needs, we tend to download and install specific applications. This is of course, especially true on the iPhone where the apple applications are legion and well liked by the applerati.

So what drives this?
First and foremost, I believe is the desire to be in control of one's own destiny. The networks are not ubiquitous enough yet that we can rely on them to have what we need available whenever we might need it.

Second network cost - that can be expensive after a while.

Third pure preference - we like the look and feel of the apps we install and not of those we don't

Fourth capability. Organizaing/classifying is a core "client side" requirement. Rich experiences for doing that preclude us from using totally network based approaches - although tagging really assist with this.

There are probably lots of other reasons, but with capability moving into the network, it seems strange that i-apps are growing in strength and popularity

Asymmetry - the word for 2009

Almst all the dealings we have are asymmetrical, but somehow that hasn't been anything to worry about. We are, however, seeing asymmetry beginning to bite in many unpleasant ways.

The first is the power that organizations have over individuals. Random/arbitrary increases of interest rates on charge cards with little recourse. Banks making errors, taking a long time to pay what's owed and then when they pay it twice by mistake demand that the error be corrected, "Immediately or else...". The petty bureaucracies of home owners' associations - you can't park a Ford F150 in your driveway, but a Cadillac Escalade is OK (Thanks Frisco, Texas).

The second is in the everyday communcation between parties. There is the kind of power asymmetry described above, but then there is what I call "interest asymmetry". Where one party in a conversation say something - which is of no interest to the other party. We have all been involved in conversations with spouses, other family members, children, where something that is riveting to them is really dull to us. In the interests of harmony, I will not cite specific examples here....

At a wider level, we this interest asymmetry shoing up when we use social neyworking sites. We have the opportunity to converse with many people using these tools, but these conversations have inherent asymmetry too. What we choose to say is, at least, interesting to us. What we choose to "listen to" has variable degrees of utility. I am interested in family postings about the kids, but not teribly interested in the ins and outs of Cpmmercial Property Law in England (something my brother in law is an expert in). For non-family/non-friends I am typically interested in work related stuff, or special interests (food, sailing...). So when I see the jumb;ed stream of messages, I put filters on, e.g. "Oh this is Paul talking about LIBOR again, I think I will ignore it." or "This is Robin talking about the kids, Christmas trees, presents, etc." I will ignore that." The atter case because I follow Robin's inciteful postings on technology, but not on his children.

People who are followed by a large crowd (because of celebrity, interests, self-promotion) have an even more asymmetric communication approach using the media of social networking because they have so much to say, and limited opportuinty to listen if all their followers were to respond. While they will often have set themselves up with expectations of symmetric communication, the style quickly becomes asymmetric.

Pub-sub

We often casually describe event driven architectural patterns as being "pub/sub". This is, of course, an over simplification and misses the point.

The key is to think of this kind of architecture is subscription driven or subscription dominated. This has been brought home. big time, in the social networking frameworks (like Twitter). People who post on Twitter essentially say whatever comes into their heads. We follow individuals or groups because, on balance, we get more out of following them than not. However, we will need to filter. For example, there are Twitterers who post about the industries that I am interested in, the beer they like to drink, their cooking interests, their children, their other hobbies,.....

I typically don't want to see all of that, but the poster shouldn't be deterred and stop. The poster's responsibility is to post. The listener's responsibility is to filter the dull stuff - or the stuff that is dull to the specific listener. That is often hard because the signal to noise ratio for any specific listener will be different than the signal to noise ratio for any other.

The4 same is true in any kind of event dominated system - human or otherwise. The listener is in a position to make decisions about what it is interested in, what it may respond to. The "teller" must continue to deliver the narrative.

Filing...

In the old - pre-computer days, there used to be a job performed in every office, namely that of the filing clerk. The filing clerk was usually pretty low in the organization - often a school leaver with few qualifications, and with luck and many years experience could become a filing manager. In other words not a great value creating job for the corporation, but one where messing up could add a lot of cost.

Legion were the companies that I worked at where filing clerks had messed up - the most extreme being the travel industry person who didn't know what to do with the "audit coupons" on a paper airline ticket. She filed them in a shoe box under her desk...

Fast forward to the database world and guess what we have. A super fast filing system. So feeding the database is like feeding the filing cabinets. Stuff is put away so you can find it again, but it isn't the operational life blood of the company. The operational life blood is the interactions between humans, the interactions between systems - in reality the events that cause value to be created for the organization. Our systems are event driven and data-filed - not database driven, at least not if they are to be truly valuable and truly model the way that value is created in the information systems.

Auto insurance

This posting is a reprise of an idea I put out in 2000. We were beginning to see the rise of "exchanges", Component models, interface based programming and other aids to distributed systems. I had 2 major interests at the time. One was in "edge" computing, the other in facades across business models.

I wrote a thought piece about auto insurance and a direct link between insurance and the vehicle. I was reminded of that piece by a TV commercial that I saw today. In the commercial, the car knew which insurance company was covering it, and if it didn't like the company, then it would find reasons not to go. In one case it ejected the keys from the ignition, in another it refused to unlock its doors and in the third case, the tires deflated.

The piece I wrote years ago also had knowledge between the car and the insurer, but had some more active features. For example, if we could envisage a car with multiple settings (the boring, around town go to work setting or the "boy racer" weekend setting, or others), then we could imagine having variable rate insurance - essentially the insurance premium is selected based on the car mode that is selected. If you never engage the "boy racer" mode, you never get charged that premium. So essentially we are looking at dynamic premium pricing based on a number of conditions all known about from the edge.

Lots of implications here. We could have a different key for each family member - or a key/thumbprint combo. If the thumbprint isn't a licensed (i.e. known to the insurance company) person, then the car doesn't start. So you could lock out unauthorized (friends of your children) drivers. What about speeding? The car knows that you have been speeding :-(. A car rental company (Acme) in Connecticut attempted to fine a renter for excessive speed. This was struck down in February 2002 by the Department of Consumer Protection. However, it may well be possible for insurance companies to use this kind of monitoring to assess premiums and to assist with accident "fault finding."

Lots of issues of course, but the key is to be thinking in these kinds of terms - where real time (or nearly real time information) can be used to guide decision making - especially pricing.

An interface or an implementation?

I have been having conversations over the last few weeks about whether a service (in the SOA sense) is an interface or an implementation. This is a surprisingly tricky path to navigate.

At one client the following question (or a variant) comes up quite frequently, "We want to service enable our C++ back end code, but the services framework is all Java how can we get them to co-exist?"

So we have an implementation, we can't use it directly but we do want to leverage it. Clearly the implementation itself isn't the service. If it were there would be no discussion. Also the interfaces it already provides may not be the same as the operations we would want to offer.

So we take the time-honored approach and wrap the code. The wrapper then exposes the service's operations and deals with the complexity of mapping the operations to whatever the original code supported. Now what is the Service in this scenario? Perhaps it is the wrapper code - after all it is the wrapper that has the signature, the wrapper that is directly invoked, the wrapper that will have the nice QOS measures, the wrapper I will look up in my registry.... However the wrapper doesn't "do" anything. Of course if the wrapper is really generic and abstract then it doesn't help to describe the service as being the wrapper. No one will have a clue when they go to the registry (white pages) what the service does. A description like "Wraps existing C++ code so it can be made available in the services framework" is hardly a confidence booster.

So what to do? My general favorite is to use a special purpose framework (auto generated if possible) to wrap the C++ (or other legacy) code. Make sure that the service is the wrapper and not the C++ implementation and manage that in the registry.

After all, one of the SOA principles is autonomy. The actual implementation doesn't have to be constrained as long as the interface supports the appropriate operations.

Introduction to the GIM model

At some primitive level there are three sets of things that flow in an organization. These are Goods, Information and Money. Not a shattering insight, but the interactions between these three categories give us some really good insights into how a business works. This isn't a typical customer focussed, outside in kind of view, but deals with what actually happens inside the business.

Why is this view of the business important?
It is holistic, actions that occur in one of these flows has impact on the others. Shipping the software to a customer (Physical Goods) can tell the money flow that revenue recognition can occur and can deliver information that the shipment has taken place.

Looking at the interactions of these flows gives us clues into:

• Adherence to accounting practice (when do we recognize revenue, for example?)
• Inefficiences in process (oh, we recalculate the revenue in many different places, even though nothing has changed)
• Opportunities to improve quality of service (let's move the credit check - an information flow request) to earlier in the cycle so we don't incur costs before we know if the customer can/will pay)
• Business modeling - "what if" (What happens if we attempt to move the credit check earlier?)
• Real time information delivery from both Goods and Money (How many units did we ship today?)
• Realization that the information need to control a Goods flow isn't available identifies a process breakdown

Ultimately, of course, what we deal with in information systems are informational abstractions of the Goods and Money. For us in IT it is all information. However to the business it isn't so, although we use the lens of the information system to help us look at the underlying realities. Our information system lenses are so distorted, however, that we often don't or can't know what is properly in or out of focus.

There are some real nuances to worry about here too. For example, in a content delivery system (e.g. a newspaper), the information (content) is treated as the Goods. Likewise in banking much of the money that flows through the bank is actually treated as Goods - but with a significant impact on Money and Information as well.

It is not trivial to create a GIM model, but the effect is enormous:

• Silo thinking is reduced - all streams can see the effects on each other of actions taken. Moving Credit Check later in the Information Stream affects the Goods stream because it will curtail process actions.
• Sub-optimal decision making is highlighted - "If I take this action, what breaks?
• It provides a common "grammar" for talking about the business between the business and IT - a goal that has not been reached in many years of trying.
• An end to end trace of a business process - with all the stakeholders shown can be built and optimized.

Why doesn't a process model work?

This is a form of process model - one where we specifically illustrate what happens to all of the major "data" sets (Goods, Information, Money). So instead of being a step by step way through the process, it is a way to handle the information exchanges more effectively. So it is an enriched process model.

What other models help?

We can leverage many of the standard models (e.g. in UML), but nothing gives us a complete enough view.

How do we do it?

It revolves around a fundamental understanding of business modeling - starting with the operational flow of goods. It is after all the operational flow of goods from start to finish that defines what the business delivers. So drive from flow of goods - starting anywhere. That depends on the scope and depends on the business. The key is that it is an operational goods flow that matters. That operational goods flow means starting with operational process.

In the operational goods flow, swim lanes and swimming pools are appropriate, but often it is a good idea to use business iconography and not boxes/lines to show things. That way it illustrates that we are firmly in the business domain, not IT.

For each step in the handling of the flow ask:

1. Has this step had any effect on the Money side - e.g. recognized revenue, incurred measureable cost, disposed of an asset...
2. What information might be available as a result of this action being taken. Is the information interesting "real time", is the information needed for subsequent analysis, is the information in any sense privileged?

Yes, the questions really are that simple. The answers aren't and the implications aren't, but the questions are. What you do with the wealth of information is another matter.

We will notice over time that the timing needs of the Money flow and the Information flow don't exactly match the Goods flow. This is to be expected. We will also notice that the goods flow will continue regardless of whether the Money and Information flows are keeping up. In many cases we will see that the other flows play "catchup" and deal with the implications of Goods flow later. This is a primary cause for inconsistencies between different parts of the business and difficulties in rationalization.

It is of course impossible to fully serialize the business so that the flow of Goods is interrupted until all the Money and Information flows have properly completed. Thta's one reason why we look at the physical Goods flow - that simply doesn't wait.

In subsequent articles, I will talk about some of the interesting patterns and interactions that we see when doing this kind of modeling.

What can SOA learn from RDBMS?

At first blush, there doesn't look to be a lot of synergy between Service Oriented Architecture and relational database management systems (RDBMS). The primary function of a database management system is, after all, to manage the storage, retrieval and integrity of the data it is called upon to manage. It's that last word, integrity, that provides insight into commonality.

In the late 1980s Sybase introduced the notion of triggers into their flagship RDBMS product. A trigger was a piece of code that was executed when "something interesting" happened to the data under management. Triggers provided a very flexible way of reacting to changes like, for example, deleting a row from the database. That deletion could then cause a series of other actions (usually embedded in stored procedures), for example cascading the delete of one row to others - and thus enforcing referential integrity. Quite an elegant solution at the database level.

Of course, being about the only capability available for guaranteeing that the events would be detected, triggers started to get abused, whole rafts of business logic were embedded in stored procedures, and the database became the logic engine and the storage engine. No real separation of concerns there.

Now let's forward to 2008 and look for parallels and opportunities. For transaction processing, we are beginning to see much lighter weight data management engines (look at Google's bigtable implementation or Amazon's simpledb). Business logic is being pushed into services - probably where it should be. That leaves us with a bit of a hole. The value that triggers provided is still needed, but now it should be at the same level of data abstraction as the data managed by the services. That level is typically at the business object level.

So taking the simple concept of a trigger, we can ask ourselves if there is value to the enterprise in knowing when "interesting" things happen to business objects. (Of course this is still too broad, but bear with me here). There are some business events that are quite interesting to know about. For example a big customer win (resulting in the creation of a new customer in business terms) is likely to be very interesting to the organization as a whole. Can be notified as a morale booster/internal PR operation, can add some heft to other sales activities,... The list is endless. It isn't sensible to notify this at the database level, that is too low level, to proprietary. It makes a whole lot more sense to publish the "event" on the corporate internal nervous system.

It is impossible for the customer management service to know who or what might be interested in the information. Just as in the database world, the table with the "add trigger" doesn't actually know which other tables might be affected. All the customer management service knows is "that something just happened and there may be others interested". It then becomes the job of the corporate nervous system to let this event be "broadcast". Anything interested can then pick it up and make its own decisions.

It is therefore the responsibility of the ESB to provide this kind of capability in the containers that surround and manage the business objects, just as it was the DBMS's responsibility a generation earlier. It is the enterprise business architecture's job to determine which of these events should be triggered, and how the interested parties might react.

No there isn't much new here, we have had pub/sub architectures before - usually to aid with application integration. What is new here is that we can use the standards based capabilities that our SOA frameworks give us, together with a firm understanding of the need to drive change, to guide us to delivering a more flexible, sustainable architecture.