Entropy in Systems

The concept of entropy in thermodynamics is well known. This article covers the landscape well. In systems, especially systems that involve human and computer interactions, we have a similar notion.

Entropy, historically, has often been associated with the amount of order, disorder, and/or chaos in a thermodynamic system. The traditional definition of entropy is that it refers to changes in the status quo of the system and is a measure of "molecular disorder" and the amount of wasted energy in a dynamical energy transformation from one state or form to another.

So too in our information systems. The amount of effort that we have to undertake to move a system from one state to another (essentially to make a modification to it) is in 2 parts. The effort expended towards making the change useful, and all the rest of the effort.

This applies both to the change to the system itself and any changes that we make as a result of executing the system. So, for example, if we consider some system fragment, "Visit the doctor because of pain in the shoulder", then any time/effort spent in doing something other than getting the diagnosis/treatment is wasted and increases the system entropy. This time/effort may include (but is not limited to)

• Getting to the Dr.
  
• Filling in new patient paperwork
  
• Having payer status checked
  
• Explaining symptoms to receptionist
  
• Waiting in waiting room
  
• Waiting in treatment room
  
• Waiting for X-Ray results
  
• Driving to radiology lab
  
• Filling in radiology lab paperwork
  
• Waiting at radiology lab
  
• …
  

The point of this is that this "system" is unbelievably wasteful of a precious resource (at least precious to me), my time. So from my perspective all of the above steps indicate great inefficiency. Perhaps because of complexity, perhaps because of a lack of cohesive thinking.

Considering another example, perhaps closer to work for many – the HR portal. That is often the one information system that has been designed to almost entirely ignore the majority of its users. There is often a huge learning curve for the majority of the employees. Of course the users who specify the system, the HR department have it well designed for their own convenience – and what they believe is the convenience of the employees. I leave you to draw your own conclusions!

So at one level, we have the idea of the system in use with every use increasing the entropy of the system.

Now think about attempting to make a change to a system. A whole new dynamic sets in. The need to understand the system in place so it can be changed. This can involve very detailed analysis – deep understanding. The more pieces there are – and the more interconnections, the more understanding there has to be. So depending on the design of the system in place there can be a greater or lesser effect on its entropy. If the system is very involved/convoluted then the understanding as a percentage of the useful work done will be high. If the system is relatively straightforward then the understanding as a percentage of the useful work done will be less.

So systems entropy might be thought of as ( _i=0 ⁿ∑(W_i - V_i)) where W is the work performed at any state change I and V is the Valuable work performed at any state change i.

Defining and normalizing what we mean by work – and considering some normalized work value equation is, of course complex. For example in the getting shoulder diagnosed and treated, the system implicitly values the Dr.'s time as being more valuable than mine – so the system is optimized to make sure that the Dr.'s change entropy is least. What should be happening is that the total change in entropy should be minimized.

Typically systems that are overly complex, overly bureaucratic or optimized to support a minimal number of stakeholders will exhibit the greatest increases in entropy under a given state change.

As architects we have a responsibility to be looking out across the landscape of a system as a whole and finding ways of minimizing the increases in entropy across common state changes.

Prototyping in rails

Now this is a truly bizarre thought. Many business apps and the underlying databases are a bit more complicated than the standard web app. Occasionally it is even necessary to populate the model that you plan to implement and let the users test drive the model through the UI.

This approach used to be called rapid-prototyping, but has rather fallen out of favor.

However, as the browser is becoming (has become) the dominant UI now, there is perhaps an opportunity to do a rapid prototype of some system functionality so the users can be assured that you have the relationships right and that you can quickly demonstrate some scenarios.

This tends to be a royal pain, and then it hit me. Rails is such a quick to develop framework that getting some gnarly prototypes out quickly is very little problem. Of course we don't want to be too convincing because the users might think we have done the hard bits. But really what we will have done is:

Create a simple database model with the interesting relationships (keys/fks, etc.)
Populated that model with some made up data - careful cases
Allowed the users to manipulate/navigate the data through a familiar approach. For example in an auction site, you may have many bids. So a 1:m relationship there. We could show an auction with its many bids easily, then click on a bid to go deeper, ensuring we have a link back to the auction.

using RESTful principles and RAILS quite a lot of this is generated with the scaffold. That's a lot of power.

Evaluating enterprise software

Two things have happened at pretty much the same time. The first is an evaluation of some enterprise class software in conjunction with and on behalf of a client. The second, the appearance of this article in my email - from Alex Rosen. http://www.pentaho.com/docs/a_new_business_model_for_bi.pdf

While the problem to be solved for the client is not BI, there are a lot of behaviors in common between the paper and the experience with the vendor at this client.

We had gone through many of the due diligence stages, spoken to various vendors, created shortlists, evaluated Proofs of Concept until we were down to a vendor that we wished to trial more deeply.

There were some key things we wanted to find out, so we devised tests, questionnaires, scheduled calls, read as much documentation as the vendor would let us see, but we really felt that the vendor was being very stingy with information.

Often direct questions would be answered with, "It doesn't work that way" or "no we can't do that". In the vendor's mind this might have meant, "This is not in scope for the pilot." That isn't how it came across.

It took a huge amount of probing before the vendor finally started to be more forthcoming with information - at which point things went a lot more smoothly.

So, since this is a pattern oft repeated, why do many vendors behave this way?

I can't answer for this specific case, but some ideas include:

• The process is still sales controlled and the sales team wants to control everything.
• The process is still sales controlled, so it is important to marginalize the people who can't say yes (but who can say no).
• The process is still sales controlled and the teams aren't used to the more implementation oriented lines of questioning
• There are opportunities for consulting dollars, so by keeping the prospect in the dark there is more revenue.
• The vendor believes that the solution takes some getting used to, so is rationing the information to prevent the customer from being overwhelmed.
• The vendor doesn't have adequate documentation and doesn't wish to expose that fact just yet.
  
• The vendor wants to see the possibilities of hard questions coming so that they can prepare for answers.

There may be other reasons that I am not aware of - and the ones I mention are pure speculation. On the current project, I can't say that any of these reasons are correct.

The key takeaway for me is to learn as much as possible about the solution before pilots. Get any FAQ documentation, get the configuration documentation so you can what's possible from the configuration. Ask the hard questions, don't take the answer, "You'll see it in the Pilot."

You are judged by the company you keep

In the past week or so I have been "followed" by several quite unsavory characters offering services in which I have not the least interest. However, if someone were to plot my social graph, they might see these characters "following me", and draw a conclusion that I might have an interest.

Now if that someone were a prospective employer, or someone trying to establish that I was an unfit parent, I can imagine that my Twitter social graph could be of interest - and possibly grounds for non-hiring or withdrawal of parental rights.

Is that right? No I don't think it is - since I didn't do anything except exist - its the online equivalent of someone putting a flyer under my windshield wipers while parked at the airport. The trouble is that I at least know the flyer is there, the person who put it there (kinda) knows that s/he put it there, but no one else does unless it is a deliberate campaign of mis-information (putting kiddie porn magazines under an ex's wipers and then alerting authorities anonymously).

So, bottom line the associations you have are knowable to anyone with a mind to discover them, and thus open to all manner of misuse - of which blackmail is one of the less benign.

Agile methods and Design-Build

I was intrigued by a term in my Sunday paper this morning. There is a project underway to extend the light rail line from Dallas to the DFW airport. It passes within a few miles of my house, so I am quite interested in seeing how it is going to look. The Mayor of Irving (Herber gears) wrote an opinion piece in the paper. In it he stressed the Design-Build approach to the project.

Design-Build sounds to me (at least on the face of it) an awful lot like Agile Development for software solutions. Perhaps it is the concrete version. Off to Wikipedia - http://en.wikipedia.org/wiki/Design-build (among other places) where I found these gems...

"Design-build focuses on combining the design, permit, and construction schedules in order to streamline the traditional design-bid-build environment. This does not shorten the time it takes to complete the individual tasks of creating construction documents (working drawings and specifications), acquiring building and other permits, or actually constructing the building. Instead, a design-build firm will strive to bring together design and construction professionals in a collaborative environment to complete these tasks at the same time."

"Potential problems of the design-build process include:
Premature cost estimating,
A short-cut design process,
Decreased accountability by the service provider, and
Correction of work. "

"It is important to note that the design-build method, while not focused on saving the owner construction costs, nonetheless often saves the owner money on the overall project. The combined effects of carrying a construction loan (which typically carries a higher interest rate than permanent financing) and an earlier useful on-line date usually yields considerable overall profitability to the project and may make seemingly unfeasible projects into genuine opportunities.
The compression is an important aspect of the implementation of this system. Other attributes include:
Enhanced communication between the service provider and the client,
Increased accountability by the service provider,
Single source project delivery, and
A value based project feedback system"

Sound familiar? - It sure does to me.

This is in contrast (as the rather opinionated Wikipedia artice states) to

"For nearly the entire twentieth century, the concept of Design-Build was classified as a non-traditional construction method in the United States, which is the last country to still embrace the old standard of Design-Bid-Build"

So if we think this through, we see the following important ideas:

• The documents that are produced during the project are for the benefit of moving the project along, not for preparing bid-packages. That alone would appear to have tremendous potential to save time and money
• There had better be some kind of a major plan in place first (City Plan anyone?) to ensure that the design-build doesn't go off the rails.
• There had better be standards/codes in place so that we don't end up with shanty towns - the guage of the railway lines should be the same throughout, the standards for connection to standard services should be standard (electricity voltage, connectors, phases, frequency)
• The approach is "owner driven" not architect or contractor driven
• There is opportunity to adjust for changing requirements - new materials/material standards, unexpected terrain or landscape needs, changes in aesthetics,...

Of course Design-Build as an approach is not an excuse for no requirements - just as Agile Software Development does not mean, "We will come into the project with a bunch of good ideas and figure out the real requirements as we go along."