Saturday, November 24, 2012

Via Quora: How has eBay's review system evolved in the past few years?



The feedback system and "feedback score" is one of the first things associated with eBay (maybe second only to auction). The main purpose of any feedback system is to harness the "signals" community members send to "feedback processor" to encourage and "enforce" desirable behavior. Anyone who ever thought about or design a rating or feedback system knows that the task is not as easy as it may first seem. And the first step is to understand what the different attribute of a feedback system is.

A member of Quora community asked me to answer the question "How has eBay's review system evolved in the past few years?". The question provided me with an opportunity (and a little push) to talk about six attributes of any feedback system. See here for the full answer and if you are not using Quora, I strongly recommend you consider using it. It is addictive. 

Friday, September 14, 2012

Primary Goals and Strategic Metrics vs. Operational Metrics



Measurement is key to any successful engineering effort, the key is what metrics you choose to measure and how to interpret the measurement.  In general there are two types of metrics associated with two different types of goals

- Primary (or strategic) goals and related metrics
- Operational Goals and related operational metrics

When goals are defined, It is very important to define which category a goal belongs to so that progress (or lack thereof can be measured accurately).

An example makes the difference between the two of metrics clear. Imagine you want to drive from San Francisco to Los Angeles. Your goal is very clear: getting to L.A. and the metric associated with it is also clear, how far are you from L.A. (or how far have you traveled so far). You drive a car, so you measure your car's engine temperature and fuel level - these are your operational metrics.

You can keep your engine temperature within a reasonable range and maintain proper fuel level, but these are not the goals of your trip. If you are happy simply b/c your engine does not overheat, and you are not concerned with how far from L.A you are, you are not measuring the right metrics.

Distinction between primary goals and operational goals is not always as easy to spot. Imagine you are designing an Order Management service (or any other service) - it is easy - maybe even common - to measure number of calls to the service per hour, day etc. and assuming that the goal is being achieved. However, if you service can only be called by one type of client (b/c it is the only one that can supply an input parameter for example), are you really achieving one of the most important (and common) goal of service design which is re-use of a capability by all consumers? Here, number of calls is an operational metric, it has to be measured, it may be necessary but not sufficient. The measure the primary goal - you need to measure the diversity of consumers (across languages, device, platforms)

For another example assume you deploy a distributed cache service - the goal of any cache service is to improve performance (and scalability) - again,it is easy to measure metrics such as number of calls to the service or even more specific metrics such as "hit rate" -but the primary goal here is to measure - from consumer point of view - performance and scale improvement.

One main reason, operational metrics are often measured, and used for decision making, instead of primary goal or strategic metrics is that they are generally easier to measure. It is certainly easier to measure "number of calls" to a service than whether a service increases its consumer productivity. "How to Measure Anything",  by Douglas Hubbard is a great resource for techniques to measure hard to quantify goals and metrics.

Wednesday, March 14, 2012

What really is this "Managed Market Place" anyway?

I work for a division within eBay Inc. called "Managed Market Places". The name is a bit curious. I was asked, more than once and by range of people, what really "managed marketplace" is? Is it a new type of marketplace by eBay (no it is not!), is it a vertical/niche marketplace within eBay (no it is not!), some one on Quora even interpreted it as it means that eBay simply "manages" the marketplace as oppose to growing it! ( if this was the case why would eBay announce that to the whole word by labeling it as such?)

So then what exactly is MMP (as it is known internally) and why is it important?

the nature of the Internet lends itself perfectly to the basic concept of a "marketplace": a mechanism for buyer and seller to find each other. Marketplaces were and still are an important and growing part of the internet. The growing list of niche marketplaces include etsy, zarrly, odesk, airbnb, taskrabbit, yardsellr, zimride and many many more. (not to mention marketplaces from facebook, google, yahoo and other major players)

At the first glance, it looks simple enough: create a site that brings the parties to a transaction together (from buyer and seller of antique to two people who want to share a ride or a room), and either take a cut of the transaction or make money by advertising. This is indeed the basic concept behind a marketplace - or an unmanaged marketplace. Marketplace itself is not a party to any transaction. Buyer and seller deal with each other directly and take the risk (or bulk of the risk) of direct transaction. EBay operated, more or less, as an un-managed marketplace for a while too.

In managed marketplace on the other hand, neither party to a transaction takes a risk, in other word marketplace guarantees the success of transaction, no risk (at least ideally). Of course a managed marketplace can "manage" other aspect of interaction such as inventory, quantity, price, promotions etc. as well but for now we only focus on risk as it is the focus of eBay MMP as well.

The evolution of simple internet marketplaces to managed marketplaces is an important trend, as the Internet users become more sophisticated and demand more from services they use online. The AirBnB incident back in July of 2011 is a perfect illustration of how "unmanaged" marketplaces will be forced to offer a higher level of assurance/risk mitigation and become managed marketplaces.

What does it mean from systems and architecture point of view? Here are five main aspects that is particularly different in dealing with managed marketplaces

1- The first significant change is that of people's mind set: You have to see yourself in risk management business, or at least assume that risk management is a major part of your operations. What this changes first, and foremost, is that you now have to identify, assess, prioritize, mitigated (or plan to) and measure risk. In all likelihood all of these activities (and the tools and systems you need to perform them) are new to you if you are dealing with a simple/un-managed marketplace.
2- Central to any consumer risk management scheme is "Identity", and I don't mean OpenID or OAuth or SSO... I meant attribute, assurances, verification, accuracy, uniqueness or mapping a real world entity to a digital identity (Entity Resolution)
3- Data is the core to efficient risk management, and big data and your ability to collect and analysis them becomes central to your ability to operate the marketplace at a reasonable cost (minimum losses)
4- Coherent Architecture become even more important. Simply because your systems becomes more complex and more integrated. A simple marketplace is just that, a marketplace site/application. A managed marketplace would include identity provisioning and verification, risk definition, measurement, management at user and at transaction level, a system for filing claims and disputes, systems dealing with ever changing legal and business landscape that enforces what you can and can not do with data you collect and finally integrating all these system in a productive way (seamless but without coupling them)
5- Even Driven and Complex Event processing: This already has a big role in distributed system, but it plays more and more important role in distributed risk management. Real time assessment of risk becomes critical and due the cost/performance of risk assessment, incremental assessment or risk based on primitive and complex event generated over entire session (or even life time of a user) will be the only practical solution.