...on a mote of dust suspended in a sunbeam

The famous "Pale Blue Dot" photo showing earth as seen from the edge of the solar system:

"Look again at that dot. That’s here. That’s home. That’s us. On it everyone you love, everyone you know, everyone you ever heard of, every human being who ever was, lived out their lives. The aggregate of our joy and suffering, thousands of confident religions, ideologies, and economic doctrines, every hunter and forager, every hero and coward, every creator and destroyer of civilization, every king and peasant, every young couple in love, every mother and father, hopeful child, inventor and explorer, every teacher of morals, every corrupt politician, every ‘superstar,’ every ‘supreme leader,’ every saint and sinner in the history of our species lived there — on a mote of dust suspended in a sunbeam." Carl Sagan 


Also illustrated by the photo is the superiority of imagination over knowledge ... knowledge sent the Voyager to the edge of the solar system and beyond, but imagination (by Sagan) turned its camera toward earth to take this photo - perhaps one the most impact-full photos of all time.

To OSGi or not to OSGi ... that is NOT the question

The topic of OSGi is attracting some attention these days, at least in my neck of the woods. The short of it is that a lot of web application developers are asking whether they should use OSGi or not. My answer: this is not the question you should be asking!

Yes, I know that the popular narrative is that OSGi makes your system more modular, makes dependency management a thing of the past, certainly solves all CLASSPATH issues, allows you to have multiple version of the same bundle running at the same time and enables you to start, stop, and deploy new bundles without restarting your framework (or server).

The reality though is that OSGi is a component technology (much like SOA or EJB) and a tool, it perhaps can make a well design system be implemented with more ease and fidelity to its original design, but it can NOT do anything for a poorly designed or organically grown system and in fact makes it more complex. So the right question to consider is what qualities my system has to have so that OSGi can actually help me?

To me, the answer is similar to SOA system, or any granularly componentized system, 

1- Granularity and boundaries of components: This is perhaps the most important aspect of a distributed system. OSGi unit of component is a "bundle", physically bundle is a jar file, it can logically be a Java class or a full subsystem - such as Jetty or a large web application - or anything in between, OSGi does not offer any hint here - nor should it. Granularity of modules is an architectural matter. For existing system, this (breaking the system apart into logical modules) is almost always the most difficult step toward any modularization. If your system already does not clear module with defined boundaries, and if it is organically grown, there is no easy or automated way to decide what modules are, needless to say that simply creating one massive OSGi module does not help you at all and simply add one more layer of useless abstraction on top of everything else.  Your best bet here is to use a dependency graph tool and try to isolate and bundle packages/jar files based on some topological sort method. This often requires refactoring of existing code to remove bad dependencies and transform the graph into architecture layering you intend. This brings me to the second aspect of a well design modular system.

2- Layering and velocity: In order to define your logical modules correctly, you need to define some form of layering that informs your dependency management i.e. a lowest layer (let's call it Kernel), with no dependencies but the standard runtime, one layer above (say Core), with dependency on Kernel. Core is a layer and may include multiple logical module (jars/packages), then you may have Service and Application layer etc. You need to decompose and map your entire code based to your own pre-defined layer and in addition decide a velocity (release cycle) for each layer, as well as whether your release of lower layer would be forever backward compatible or it would impact higher layers. (more on this in 4)

Again OSGi does not offer help here - and nor should it - it simple is a technology.

3- Dependency Management: defining layers does not guarantee that dependency schema will be enforced, still you need to manage it (preferably using tools and automatically) to make sure that for example your Kernel does not depend on your Core layer or there is no cyclical dependencies among your logical modules. More tricky yet is the nature of dependencies. If dependencies are not managed you may notice that there is a very large module in say , Core, layer with a large number of services and applications depending on it. At first glance it may look like a useful module! but the large size should make you suspicious, often time the upper layers simply leak what should be located in application or service layer down to lower layer - lack of engineering disciplines, knowledge, time of all above.

In this case, OSGi would help you capture and discover the dependency, but does not tell you that they should not be there to begin with.

4- Version-ing policy: As I said in (2), well designed systems has layers, from “lower” layer to “upper” layers – based on topological sort of dependency graph. Typically each layer has a version number visible to other layers (some may choose to have each module in a layer to have a version number visible to all other module in upper layer, this makes life a bit more difficult for module in upper layers). One should decide how many active version of each layer (or module) to be active at any given time. This, seemingly straightforward, decision have significant implications, options are

-        If at any given time you maintain only one version, everything is a bit easier, but you either have to maintain perpetual backward compatibility or force all the upper layer change at the same pace with lower layers.

-        If you maintain multiple version, you don’t need to be backward compatible and you may transition upper layer gradually – very desirable. But you have to deal with two version at  the same time (not only at runtime, but development branches, testing …)

For most web application, people maintain one version and deal with the downsides – often in form of a backward compatible changes. OSGi can help with maintain multiple version at the same time - something that is certainly useful for client side application, for web application most people I talked to are not planning to use this feature.

5- Testing strategy: Distributed systems are tough to test. A monolithic system is one large binary, you can build, deploy it and test it. For distributed system, test environment has to be setup, one would build only his module, the other module you depend on must be ready (either as out of process services in SOA, or bundles in OSGi) and have the right version. If you are using, say, five modules and each of them has two active version, there are 32 possible combinations you need to test (to be exhastive) – one reason having only one version at the time is often preferable. Again OSGi does not help you with designing your test strategy, you should have one regardless of technology you use to modularize you system.

6- Deployment: Last but not least is deployment of your system/web application. You need to decide whether to deploy OSGi framework as a web application under you Servlet container, or deploy your servlet container as a bundle in your OSGi framework. If you are using SOA, you need to decide where to deploy each service and how to bundle service stubs with your application (if any stubs are needed), or you may use a combination by deciding that each service stub is an OSGi bundle. In any case, there should be a clear design for correct deployment of a distributed/modular system.

If you design a system in a way that these aspects are taken into accounts, then OSGi probably helps you implement it easier - although for web applications the issue to "two runtimes" is a bit too much for my taste - but then again if all the above aspects are taken into account, you may not have an urgent need to OSGi anyway (banks offer credit of people with good credit but probably the don't need it anyway....). Often times engineers and managers who work on poorly designed or organically grown systems, and in an effort to reduce complexity and increase productivity of people working on them, stumble upon OSGi...if you fit into this category, my recommendation is to focus on fixing the underlying issues that makes your system complex, inter-dependent and coupled. Until you do that, OSGi (or any other alphabet soup of technology) will not help you. 

2011 OpenID Retail Summit - March 8 @ PayPal Offices

Our collegues at PayPal are hosting the OpenID Retail Summit tomorrow from noo to 5:00pm. It sounds like it will be an interesting exchange of ideas between identity providers and retailer (as relying parties). I will be presenting our experiences (and wishes) as large relying party tomorrow @ 3:00pm.

I will post the presentation and a short post tomorrow.

CardSpace II : "Change of Authentication Behavior"

After I wrote a short post on announcement of end of life for CardSpace, Kim Cameron (a great visionary in the field of digital identity and someone I learned a LOT from but never had the opportunity to meet) reflected on it here.

A few people (independently) emailed me and asked me whether Kim is implying that Card Space lack of adoption, at least partially, was “eBay’s fault”.

I read his post, partially quoted below, a few times and although to me it more sounds like a reflection, but I see how one may think that way. So, I decided to add a bit more details to my original post on “change of behavior”.  

“In the history of computing there have actually been plenty of cases where users DID change their behavior - even though at first only a few people could understand or use the new alternatives.  But those “early adopters” were able to try the new inventions on their own.  They didn’t need anyone else to approve something or decide they would like it first.  Once convinced, they could show the new ideas to others.

When Visicalc appeared, I don’t know how many people in IT would have bet that every accountant in the world would soon be throwing out his pencils and starting to use spreadsheets for things no one can even now believe are possible!  The same is true for a thousand other applications people came to love. 

But because authentication doesn’t stand on its own, users never got the chance to start using Information Cards “just because they felt like it”.  They needed web sites to make the same bet they did by implementing Information Card support as an option.  

Web sites didn’t want to bet.  They wanted to keep to “the matter at hand” and prevent their users from getting lost or distracted.  The result: a preemptive chill settled over the technology, and we never really got to see what users would make of it.

My conclusion:  regardless of what new features they support, user centric identity solutions need to be built so they work with as many existing web sites as possible.  They can’t require buy-in from the all the big web sites in order to be useful.” 

I agree that users, and people in general, do change their behavior, and sometimes relatively quickly – if there is a direct and tangible benefit for them to do so. 

In case of CardSpace, our end users simply did not perceive the benefits, besides they had to download a large binary and deal with unfamiliar experience. Ashih Jain's post capture the sentiment around the time we were experimenting with CardSpace – 2008 -  (One seller kept asking me why I was asking him to “do all this just to login” ).

eBay is in the commerce business,  we do take reasonable risks in changing people’s behavior around commerce related activities such as listing, classification, payment, feedback, finding, shipping, trust etc. but for login and authentication we rely largely on users' already learned behavior (or as Andrew Nash refers to it the established “steady state” of authentication).

Perhaps one way to get a large number of users familiar with Card Space, and generate consumer demand for it, was full adoption within MSFT (including Windows login panel). Microsoft owns one of the largest (if not the largest) user-facing authentication experiences in the world, if they had fully supported Card Space everywhere a login panel appears today, maybe adoption story for Card Space would have been different. That would have made it much easier for company like eBay to adopt it as well. 

It is Finally Official: The End of MSFT CardSpace

There were a few things I wanted to write about recently ranging from all the discussion around OSGi in the Java community (and eBay) to what I would expect from an industrial grade Identity Provider (hint: it is not about which protocol it uses) and from when not to use No SQL to the emergence of a little known art and science called Entity Resolution, but let me resume my posts after a month with this:  Microsoft announced that it would not ship CardSpace 2.0.

Having worked on a authentication concept with MSFT for eBay sellers, I had mixed feelings about this. On one hand I was on the record not supporting the use of CardSpace for eBay sellers (or buyer). On the other hand I am concerned that technical community discounts the significance of Claim Based identity altogether and concludes that "FaceBook Conncet" is all we'll ever need.

There is a good reflection (from an insider's point of view) on Card Space here. (courtesy Gunnar Peterson) My personal view (and the reason I didn't support the adoption of Card Space at eBay) though centers around the challenges of "Change of Behavior" required by Card Space.

Basically, CardSpace failed b/c it requied uses to change their behavior. See, the "User name and password" protocol (a simple challenge and response) IS a protocol, one where a human being (a normal user) is a participant in. It has taken about 20-30 years (depending on how you count) to train users what to do when they see a "login panel" , the "login panel" contract is so widely understood that despite all of its short coming is the most viable remote authentication protocol we have today. It is flawed, it is costly, it is not secure, but it is a widely understood by users on the other end of the protocol. CardSpace, despite all its advantages, was not understood, would (and did) make people confused, they did not know what to do when the CardSpace screen popped up ... a technology whose adoption depends on change of a strongly learned behavior is unlikely to succeed (or at least I didn't think eBay sellers - not the early adopters of technology - would learn and accept it).

It also didn't help that a lot of browsers didn't support it (installing a plug-in does not count), and the fact that developers didn't know how to issue cards (or validate, update or revoke them).

Having said that, I did like the idea of decentralized identity provider and not having any one identity provider to be THE identity provider that everyone else had to rely on (putting user in control of their own identity). Compare this with a world where one identity provider (be it facebook or Google or twitter or anyone else) is the dominant identity provider because it is easy for RPs to embed a simple button  and for users to click on it.

Kingdom of Nouns ...

The other day I took advantage of a rare couple of hours of peace and decided to catch up on my blog readings, I came across a nice post by Joel Spolsky from the Joel on Software fame on Map/Reduce. The post is probably one of the clearest writing on origin of Map/Reduce and what it does - if you haven't read it I highly recommend it, even if you are a Map/Reduce pro - in the course of explaining Map/Reduce he talks about how thinking in terms of functional languages made inventing Map/Reduce possible and implying that if one only think in term of OO languages where functions (or verbs) are not first class citizens coming up with an abstraction such as map() or reduce() function is tough. He includes a link to another interesting post title "Execution in Kingdom of Nouns" that contends that Java is a kingdom of nouns and verbs are "owned" by nouns. I like this post too, especially the witty narration. However I disagree with conclusion (or implication) that one can not effectively or elegantly model certain class of thoughts and abstraction with a statically typed language like Java.

Look at any natural language, they ALL seems to be kingdom of nouns. Look at how babies start to talk, they almost excursively uses nouns for a while and THEN add verbs to form sentences (or form thoughts). The core example of the post for taking garbage out that I copied here:

get the garbage bag from under the sink
  carry it out to the garage
  dump it in the garbage can
  walk back inside
  wash your hands
  plop back down on the couch
  resume playing your video game (or whatever you were doing) 

(italic emphasis is from original post)

This example is there to show that a normal function is a sequence of verbs and 

can be easily expressed without needing a "noun", but the noun here is hidden, 

it is the "subject" that is actually doing it. In a function language like JavaScript

the replacement would be the "global" scope, or some form of function that author would

create called "takeOutTheGarbage()" and call it from different "context"s, those context

would be the nouns.

I admit,I am not a language guy, I am more of a modeling guy but i do think that 

type spaces formed around collaborating nouns (subject, objects etc.) are capable of modeling and asbtracting

any concept/thoughts. I do agree that some folks in the "Javaland" overuse classes and have too many

factories and adapters, mediators, visitors etc. but this is more of a "fashion" issue than a basic

OO modeling issue. 

I also agree (and have seen) that functional languages are better for certain tasks

but I wishing for live in a "kingdom of verbs" is a bit extreme in my view, after all 

verbs are all performed by somebody/soemthing unless you prefer to live in the land of anonymous. 

Best description of what architecture is ... by Charles Darwin (yes, THE Charles Darwin)

Here is a quote attributed to Darwin:

"It is not the strongest of the species that survives, nor the most intelligent that survives. It is the one that is the most adaptable to change." 

Right there, for me, is the best definition of software architecture I have seen. Software architecture is whatever you do that allows your system to be adaptable to whatever changes its environments throws at it (scale, hardware failure, legal changes, change of business model, new forms of distribution etc.), better yet Darwin goes to to say

"In the long history of humankind (and animal kind, too) those who learned to collaborate and improvise most effectively have prevailed."

So in his view the keys to adaptability are collaboration and improvisation, which I'd translate it into modern systems as "Communication" and "Experimentation". Communication is THE key to collaboration and a quality lacking in a large number of leading engineer/architects. The core communication skill here is not about communicating what one's particular solution to a problem is, but more a style that is about inquiry, a style that illicit input and views from all significant parties and synthesize them into models and hypothesis and communicate them back and form common understanding (does your architect do that?)
 
 Experimentation is both a personal quality and something that must built into any system as a first class architecture requirement. Experimentation is generally under-valued in architecture discussion. Systems in general have multiple views, the most popular being the 4+1-views, I would add an "Experimentation View", a view of architecture that describes which sub-systems can be experimented with and in what ways, and how would an experiment success be measured. More on "Experiment View" later, but for, I am even more convinced that analogies with and inspiration from biological systems is worthy guide for designing software systems.