Sundown On Microsoft

If you read this blog regularly you'll know that I've mentioned Writely a few times in it—in a very positive way. Writely is a free Internet based wordprocessor that provides an awesomely good service. It allows you to do what neither Microsoft not Open Office have succeeded in allowing you to do—to collaboratively work on a document. I use it all the time. I use it for this Blog even though it's not a collaborative piece of writing.

Microsoft really did need to acquire Writely to forestall the revolution that it portends. But instead Google acquired it. If you want to know who will win the commercial contest between Google and Microsoft, you have just been given the answer.

SOA for Dummies

Judith Hurwitz, Carol Baroudi and I have been commissioned to write “SOA for Dummies”. We were already writing chapters, but last week we finalized the publishing contract and consequently we are now allowed to mention what we are doing.

It was our idea rather than Wiley's (Wiley is the publisher of the Dummies series)—we didn't wait to be approached. We'd being doing a good deal of work on SOA and we were thinking of writing a book anyway, because there is so much confusion about what SOA actually is. Carol came up with the idea of doing a Dummies book and promptly contacted Wiley, who agreed to a deal. For them it was a no-brainer. They would probably have come up with the idea some time this year, but we stepped in before that.

So, from now on I'll be hitting the keys furiously. Don't be surprised if some Dummies talk leaks into this blog.

More ESP Rules

Last week I promised to continue with proposing rules for ESP. As I noted, the first four rules were about performance. The next three are about what ESP actually is:

Rule 5. ESP is the processing of a well-defined “window” of events as a batch of information drawn from an arriving stream of event records.

This actually counts as a broad definition of ESP so perhaps it should be rule number one. Anyway the point here is that ESP involves defining a window (or segment) of records within the stream and processing them as a batch of records as they arrive. You could define the window by number, for example; the most recent 30 event records or by time, for example; all the records that arrive in each 900 milliseconds. The event records in the window are processed as a batch.

The point is that some processing of data streams, a series of transactions say, or even a video stream, involves serial processing of records one by one. That is not ESP. It is serial processing. ESP is not serial processing.

Rule 6. ESP products must provide a set processing capability for the events in the window.

One way of doing this, which some ESP vendors have chosen, is to implement a version of SQL, which provides the usual SQL set operations on the event stream. With ESP you are selecting groups of records from the stream window looking for specific relationships and values and acting upon them when you find them.

Examples help. In financial systems you may be looking for a group of records that define and arbitrage opportunity; a price in one market and a price in a different market, with other information such as exchange rate, with which to tie the two together. In a RFID data stream you may simply be looking for all event records of a particular kind with approximately the same read time (as this might indicate a pallet of goods passing an RFID reader. In a click stream we may simply want to pluck out all the clicks from a given IP address as this would indicate a path taken through a web site. The point is that we need to be able to apply set oriented processing.

Rule 7. ESP products must be able to process a time series of events including catering for events out of order.

There are two types of data streams, those where order is important and those where it is not. For example; imagine that we are simply counting votes in an election of some kind and the voting data is arriving in a stream. We do not care about its order, if all we are interested in is the vote totals. For most event streams there is some real world order, usually based on time, which we do care about. For price information, for example, we care about the time at which a price was agreed. Sometimes we might not care about time per se, but we do care about the order in which events occurred.

In any event the arrival order of event information may not be the time order. We may want to define an event window based on actual event time not arrival time and the software must be able to cope with that.

More rules next week.

In The UK

I spent last week in the UK, mostly in a hotel, doing some consultancy and meeting up with one or two old friends. In the course of it I ran across some facts that I found interesting. Here they are:

• The diagnosis, treatment and care for cancer patients accounts for 2% of the world's economy. In the advanced economies the figure tends to be higher, accounting for about 30% of the health sector. You could talk in terms of there being a cancer industry - because there is.
• A high proportion of prisoners in the UK are illiterate. There's a high correlation of illiteracy and criminality. Correlation of criminality and having had a single parent or having been raised in care is also high.

So now, let's tie these two disparate facts together. Prevention is better than cure. Few people would dispute that. The prevention of cancer would significantly reduce the costs, because many cancers are less serious and hence less expensive if caught earlier. Prostate cancer is famously so. Reduce illiteracy and unwanted children and you'll reduce the level of crime. In fact the most effective crime reducing legislation introduced in the past 40 years was the legalization of abortion (read Freakonomics by Steven D. Levitt and Stephen J. Dubner for details of this in the US). Such legislation, of course, never had that as a target—it was an unexpected side-effect.

Now you would think that, given the seriousness and cost of crime and cancer, there would be a high investment in prevention—but there isn't. And that boils down to a general lack of knowledge, which leads in turn to a general lack of political support for investment in prevention. It's surprising, for example, that the public is happy to pay for criminals to go to prison when the individual cost of keeping them there (in the advanced economies) is more than the cost of keeping someone in a 5 star hotel.

Now lets introduce a third strand to this: the investment in IT. There is a great deal of investment in IT and a great deal of what I have seen recently is expensive investment in trying to cure problems (such as in email management, IT security investments, system management, etc.) Much of this is necessary at the time that the investment is made but might not have been necessary if the right investment had been made earlier. And when the IT user cannot be blamed, the IT vendor usually can—because the IT user depends on the vendor to behave strategically.

Enough said. We're done here.