The other day, via Venture Beat, I learn about a new start up called Enole. Since the company is in Identity domain I was naturally interested and start looking around. The idea is as interesting as it is old: to unify all identities online and offline and to carry it with you with your cell phone. (use your fav search engine to look for products, ideas and patent on storing all sorts of identity information on cell phones ranging from credit card to login name and passwords to codes that unlock cars and doors etc.)
However it is often not the novelty of an idea that sets a company apart but the strength in execution and market strategy. So with a bit of healthy skepticism I looked at the site.
There is the usual marketing and hype in the web site – which I completely understand. Beyond the hype Enole is a service that allows a developer/application (identified by a pre-issued token) to link up a user and a device. A user simply means an identifier string (email or any other string). So effectively there is a link among three entities Application that creates a link, user and device.
The rest of the API is about the basic create, update, query of link, users and devices. However there a few key questions that are not answered in the documentation:
- Can an application that has not created a user or device query them?
- How does the system deals with duplicate email addresses and identifiers across multiple applications?
- If another application picks up my devices unique id, can they query Enole DB? If they can and get my information, do I need to authorize (OAuth style) the
information before application can use it?
- How would a user remove his/her account.
- How the "sign-up" tokens are stored and what happens if they are compromised
In essence, the API document describes the CRUD operation of user, device and link entity but does not describe a formal authentication protocol between users, devices etc. As such it can not really be evaluated in details.
Additionally, it is not very clear how end users (not developers) are in control of their own identity and information. This is really big for any company that has ambitions in the identity area.
Also, if you have been reading my blog regularly you know that I give a LOT of significance to the language and terminology one uses to describe either her problem or solution. The language used in Enole site is not that of identity, authentication or communication protocol domains. This, to me, is not the most positive sign.
Having said that, I think Enole is an early stage start up and like any other early stage start up their original idea will evolve, I will keep an eye them to see how they evolve and how their technology be used.
Monday, July 12, 2010
Saturday, July 3, 2010
Recall and Precision: It is not how many bad guys you caught, it is how many good guys suffered
The measurement of "Recall and Precision" is front and center in all of our fraud prevention measures and algorithms, you can read a general description of this concept and the mathematical definition in Wikipedia, but I have had better luck explaining the concept with this example:
Imagine there is a band of armed rubbers (say 5 guys) in your town and you sent your best cops to round them up. After a day they come back arresting a group of men. How do know if they did a good job?
The obvious answer is whether they have arrested ALL the gang members. So the measurement is "how many gang members have they arrested?" in this regard 5 is better than 4 and 4 is better than 3. Simple
But is that enough? Let's imagine three out comes
1 - The cops came back having arrested 5 guys, all of them gang member. This is perfect, they arrested ALL the RIGHT people, and ZERO WRONG person, recall = Precision = 100%.
2 - The cops came back having arrested 10 guys, 5 gang members of 5 random and innocent guys. In this case recall=100% but precision is 50% - which in this case is clearly not acceptable (even worse they could have arrested all men in the community, recall still would be 100% but precision would be near zero - that is called Carpet Bombing)
3- The cops came back with 3 guys, all gang member, no innocent guys was arrested. In this case recall= 60% but precision=100% - this is called Proof Beyond the Reasonable Doubt i.e. a philosophy of design whereby it is better to let a bad guy go free then to harm a good guy.
In modeling risk and fraud and designing algorithms to prevent them, we always have to measure the algorithm based on their recall and precision. Low precision methods typically cost a lot in term of customer support and friction in user experience, low recall algorithms and method result in higher losses for the company.
in designing a Risk Management strategy, I tend to side with lower recall then lower precision and then manage the ratio of loss/revenue with the higher revenue generated by higher precision - or right customer who were let in.
What if the cops came back with 10 guys, 5 are gang members and 5 are innocent? in this case they arrested all gang members (100% catch rate or 100% recall) but
Imagine there is a band of armed rubbers (say 5 guys) in your town and you sent your best cops to round them up. After a day they come back arresting a group of men. How do know if they did a good job?
The obvious answer is whether they have arrested ALL the gang members. So the measurement is "how many gang members have they arrested?" in this regard 5 is better than 4 and 4 is better than 3. Simple
But is that enough? Let's imagine three out comes
1 - The cops came back having arrested 5 guys, all of them gang member. This is perfect, they arrested ALL the RIGHT people, and ZERO WRONG person, recall = Precision = 100%.
2 - The cops came back having arrested 10 guys, 5 gang members of 5 random and innocent guys. In this case recall=100% but precision is 50% - which in this case is clearly not acceptable (even worse they could have arrested all men in the community, recall still would be 100% but precision would be near zero - that is called Carpet Bombing)
3- The cops came back with 3 guys, all gang member, no innocent guys was arrested. In this case recall= 60% but precision=100% - this is called Proof Beyond the Reasonable Doubt i.e. a philosophy of design whereby it is better to let a bad guy go free then to harm a good guy.
In modeling risk and fraud and designing algorithms to prevent them, we always have to measure the algorithm based on their recall and precision. Low precision methods typically cost a lot in term of customer support and friction in user experience, low recall algorithms and method result in higher losses for the company.
in designing a Risk Management strategy, I tend to side with lower recall then lower precision and then manage the ratio of loss/revenue with the higher revenue generated by higher precision - or right customer who were let in.
What if the cops came back with 10 guys, 5 are gang members and 5 are innocent? in this case they arrested all gang members (100% catch rate or 100% recall) but
