CAP Therom and Digital Identity

If you read this blog chances are you are familier with the CAP theorem, it basically states that any distributed system operating at scale can choose at most two of the followings three:
- Consistency
- Availability
- Partition Tolerance

There other examples of pick any "2 out of 3" in life are:

- The management rule of thumb: Good, Cheap, Fast
- Graduate student dilemma: Fun, Grades, Sleep (replace fun with your own idea of it)
- Investment advice: Low Risk, High Return, Legality - if you pick low risk and high return chances are you are compromising legality :-)

They way I look at all these "rules" is that the space of each of these domains offer only two degrees are freedom and once you choose two points (that effectively determine or fix your degree of freedom, the third point will be chosen for you)

For example in "Good, Cheap, Fast", your degrees of freedom are basically time and money, once you choose how much time and money you want to spend all three qualities are determined. So now, if you choose time and money not directly, but indirectly via the choice of say good and fast, you automatically also chosen "not cheap".

Interestingly digital identity offers the same 2 out of 3 dynamics among the three main attributes of

- Quality of Identity
- Usability
- Cost

"Quality of Identity" is a measure of how uniquely a set of data represents a real world person and how strongly an IDP stands behind such assertion (for example whether IDP guarantees up to a certain amount damages resulting from inaccurate data), usability is how easy it is for IDP to provision such identities.

It is clear that if an IDP chooses to provide high quality identity and also wants to makes its provisioning easy to use (or scalable for that matter), it has to spend a lot of money.

In practice though, IDPs segment the user base and only provide high quality identity for users to whom maximum credit are extended (e.g. users who can sell the most on eBay).

Enole - A New Identity Linking Startup

The other day, via Venture Beat, I learn about a new start up called Enole. Since the company is in Identity domain I was naturally interested and start looking around. The idea is as interesting as it is old: to unify all identities online and offline and to carry it with you with your cell phone. (use your fav search engine to look for products, ideas and patent on storing all sorts of identity information on cell phones ranging from credit card to login name and passwords to codes that unlock cars and doors etc.)
However it is often not the novelty of an idea that sets a company apart but the strength in execution and market strategy. So with a bit of healthy skepticism I looked at the site.
There is the usual marketing and hype in the web site – which I completely understand. Beyond the hype Enole is a service that allows a developer/application (identified by a pre-issued token) to link up a user and a device. A user simply means an identifier string (email or any other string). So effectively there is a link among three entities Application that creates a link, user and device.
The rest of the API is about the basic create, update, query of link, users and devices. However there a few key questions that are not answered in the documentation:
- Can an application that has not created a user or device query them?
- How does the system deals with duplicate email addresses and identifiers across multiple applications?
- If another application picks up my devices unique id, can they query Enole DB? If they can and get my information, do I need to authorize (OAuth style) the
information before application can use it?
- How would a user remove his/her account.
- How the "sign-up" tokens are stored and what happens if they are compromised

In essence, the API document describes the CRUD operation of user, device and link entity but does not describe a formal authentication protocol between users, devices etc. As such it can not really be evaluated in details.

Additionally, it is not very clear how end users (not developers) are in control of their own identity and information. This is really big for any company that has ambitions in the identity area.


Also, if you have been reading my blog regularly you know that I give a LOT of significance to the language and terminology one uses to describe either her problem or solution. The language used in Enole site is not that of identity, authentication or communication protocol domains. This, to me, is not the most positive sign.

Having said that, I think Enole is an early stage start up and like any other early stage start up their original idea will evolve, I will keep an eye them to see how they evolve and how their technology be used.

Recall and Precision: It is not how many bad guys you caught, it is how many good guys suffered

The measurement of "Recall and Precision" is front and center in all of our fraud prevention measures and algorithms, you can read a general description of this concept and the mathematical definition in Wikipedia, but I have had better luck explaining the concept with this example:

Imagine there is a band of armed rubbers (say 5 guys) in your town and you sent your best cops to round them up. After a day they come back arresting a group of men. How do know if they did a good job?

The obvious answer is whether they have arrested ALL the gang members. So the measurement is "how many gang members have they arrested?" in this regard 5 is better than 4 and 4 is better than 3. Simple

But is that enough? Let's imagine three out comes

1 - The cops came back having arrested 5 guys, all of them gang member. This is perfect, they arrested ALL the RIGHT people, and ZERO WRONG person, recall = Precision = 100%.

2 - The cops came back having arrested 10 guys, 5 gang members of 5 random and innocent guys. In this case recall=100% but precision is 50% - which in this case is clearly not acceptable (even worse they could have arrested all men in the community, recall still would be 100% but precision would be near zero - that is called Carpet Bombing)

3- The cops came back with 3 guys, all gang member, no innocent guys was arrested. In this case recall= 60% but precision=100% - this is called Proof Beyond the Reasonable Doubt i.e. a philosophy of design whereby it is better to let a bad guy go free then to harm a good guy.

In modeling risk and fraud and designing algorithms to prevent them, we always have to measure the algorithm based on their recall and precision. Low precision methods typically cost a lot in term of customer support and friction in user experience, low recall algorithms and method result in higher losses for the company.

in designing a Risk Management strategy, I tend to side with lower recall then lower precision and then manage the ratio of loss/revenue with the higher revenue generated by higher precision - or right customer who were let in.




What if the cops came back with 10 guys, 5 are gang members and 5 are innocent? in this case they arrested all gang members (100% catch rate or 100% recall) but

Special Forces, Regular Army and Start ups

From time to time I advise start ups on mostly technical matters, inevitably they ask me for referrals to hire engineers or help with editing their job postings.

Typically they email me a very detailed and specific job description for a, say, Data Architect or Ajax messaging programmer or Java Engineer complete with an obscure list of libraries and technologies that the candidate expected to be master in (probably those technologies were suggested by their VC to them in their last meeting)

Start ups should be built like special forces, with a few brave and smart generalists with a lot of heart and "get it done" attitude, not by deep specialists in any certain area - I am talking about the first few engineers here - a statistical modeler with no programming skills or a Data Architect with focus on data replication makes a fine contractor for a short time to advise your core engineering team, but should not be one the first 5 engineers you hire.

Wanna Make Your Life Hard? There is an app for that, or maybe 100,000

These days it is not uncommon to find some one with 100+ apps installed on his/her phone (maybe you are one of those?). I imagine most of those apps are used once or twice and will never be used, the only time you see them (or their icon) is when you scroll past by them to get to your 2 or 3 (or five) apps you actually use frequently.

This app mania clutters UIs and waste a lot time, and I can't help but thinking "do we simply forget all the lessons and progress of last 10-15 years simply because a device came along with attractive aesthetics?"

What happened to "browser is the operating system?" in the days that most desktop apps are moving to cloud and being delivered thru browsers, why is it that most (if not all) mobile apps are native code? Yes native apps still have certain capability that browsers do not offer YET, but with HTML 5 and Web Kit (and maybe with a little bit of industry support to close the remaining gaps) browser based apps would be sufficient for large majority of mobile apps.

Don't get me wrong, there will always be need for native apps, but much like with desktop, over time only a few apps will be privileged to be native, most other will and should be browser based standard, write one, run on any browser apps.

This makes development of mobile apps simpler and more cost effective, instead of maintaining three apps for iPhone, Andriod and Blackberry, most developer only maintain one version, and you only install a few apps that truly need to be native apps.

"My Dad is 50 years old, weighs 30 lbs and is 40 ft tall...I like him b/c he is silly"

A very competent and respected colleague of mine, in his early 30s, 5’9” and reasonably proportional I might add, has the following written by his 5 year old son posted in his cube:

“My dad is 50 years old, 30lbs and 40 feet tall and I like him because he is silly”

Besides it being funny and making me think what my daughters really think about me, it reminds me of how our products can be perceived and used by other engineers: They way they see it not they way “they are”.

If you are developing systems, services and products for other developers, it is important to understand that your will be used they way others understand it and not the way you planned it to be used. I have seen a session service used as a general purpose cache, ESB riddled with use-case specific routing logic, authentication service used as data pre-fetch mechanism  and presentation logic (in XSL) saddled with complex and critical business logic – of course in XSL. All examples of how we all use tools to solve our problems regardless of whether or not the tool is meant for it (remember eating jelly with knife?)

The first solution of course is documentation, but here is what that documentation should include:

- Real world, working and end to end examples of the right way to use the services/product

- Examples should be cut-and-paste able (to the extent possible)

- Realistic examples of wrong way of using systems, updated periodically as new misuses surface

- Direction and pointers for how to address the original need leading to the misuse.

- Address of a user (or expert monitored) forum where users can get help

Presentation for my eBay DevCon 2010 Talk

Here is the slide deck for my talk @ eBay DevCon 2010" The User Who Came In From the Cloud.


If you ever wanted to consume identity from external identity providers (Facebook, Google, Yahoo, PayPal etc.) you may want to check it out.

The userwhocameinfromthecloud devcon2010-v_5

View more presentations from farhangkassaei.