API vs. Service - Terminology Matters

Those who know me or worked with me know that, in technical context, I am a stickler for accurate terminology and non-ambiguous usage of words. I owe it to my high school algebra teacher who tried valiantly to teach us (a group of 11 grader geeks in Alborz high school in mid 80s)  Field Theory. As it is the classic methodology, he would spend so much time on setting up the problem, introducing terms, definitions, axioms, propositions, properties, lemma ..that by the time he got to the main theorem, he had lost most of us. When we objected, he would say

If your words don't have accurate meanings, you sentences are ambiguous, when you utter ambiguous sentences, your arguments are confusing and when arguments are confusing, you start a war. 

We were way too young to understand the wisdom in these words then, but I am a believer now.

In the past few years, I have seen very frequently the two terms API and Service used interchangeably. They are not the same! Granted, there is no official definition but I would like to offer one that helps in all discussions regarding both APIs and Services.

Service is a software component, one with well defined contract and boundaries (both code and data), services are build by Service Providers (or Producer). Service orientation is a style of architecture for building distributed systems.

API on the other hand, describes consumers view of the world. API is what is exposed to consumers and determine the programming model by which consumers interact with producers.

Services are often exposed as APIs.  APIs MAY (or may not) be implemented by a service. Figure below illustrates this concept:

API endpoints are denoted by eN. In this picture, API e1 is an aggregation of three operations each belonging to a service. API e2 on the other hand is a simple delegation (or routing) to one operation of Service 1 (S1.O1)

This picture defines what API and a Service is clearly and allows us to separate consumer concerns from producers concerns, for example API consumer (and API type space) may include a definition of a User that includes a unique identifier, email, a brand name and a reputation indicator. Consumers expect an endpoint to supply this type - they call this endpoint an API - however no sane domain model would encapsulate and implement the notions of Identity, Branding and Reputation in one service. The User API end point aggregate (and transform) the output of the three services, but more importantly allows producers to build services with right boundary and granularity independent of how they need to be consumed.

As the last point, I have to emphasis that our STRONG preferences is to keep the API intermediation layer "dumb" and simple, actually we often encourage simple delegation, and rarely aggregate or transform outputs and almost never do anything remotely resembling business process orchestration or anything involving business logic (aggregation and transformation are strict mathematical operations not business logic). All API ecosystems need to have an intermediation layer, but it has to be kept as dumb as possible (but not dumber).

Now that I defined APIs and Services, in my next post, I will focus on Portfolio of Service - a collection of coherent services that power APIs (usually at a scope of a company or large domain). I specifically list the lessons we learned from designing, implementing and operating CommerceOS.

CommerceOS Part V - Processes, Metrics and Measurement

Part I of this series of posts, described the what CommerceOS, Part II explained the structure and overview, in part III we focused on the standards that make a Microservice portfolio work. Part IV was an overview of Shared Services, in this post (the last part) - I write about processes and metrics.

Processes

Platforms at their core are abstractions of capabilities. Their value comes from encapsulating frequently and commonly use capabilities, normalize their variations and expose them for use - freeing up applications/consumer to focus on less common (and therefore more value add) capabilities.

Platform are built based on standards, and maintained and enhanced via processes - without processes platforms simply deteriorate over time. This is akin to the second laws of thermodynamics - entropy of systems always increases UNLESS you spend energy - and energy in this context translates to processes.  It is very unlikely that a portfolio of services, especially with higher level business domain abstractions and their nuances survive for a long time without some form of processes. The key though is to have meaningful process. We have three criteria for designing "meaningful" processes

- Process should have a clear and measurable goal.
- Processes should be transparent with clear decision making process
- Processes should have a SLA (time bounded)

CommerceOS defines only three processes, all at the portfolio level, they are
- Process for adding of a new service to the portfolio
- Process of adding a new standard to the portfolio
- Process for cross domain, portfolio-wide decision making

1- Service Life Cycle: Ensures the followings

•  A given service is neither duplicating (doing the same thing with different name) nor diluting (doing different thing with the same name) existing capabilities. If Order/Cancel and Order/Refund do "the same thing" - we have a duplication. It leads to confusion and more than likely major bugs - after all the intend to do the same thing, but over time functionality diverges and depending on which API is called Order is either not properly canceled or refunded. If User/Order and /Order return types that are drastically different (especially semantically), that is an example of "Dilution" - in this case of Order type. 
•  Extracting potential new types for the common type space
• Ensuring an audit opportunity for security, legal and regulatory functions.
• Automated assessment of a service (once implemented) against a set of standards

2- Portfolio Standards and Type Space: Process controls adopting a proposal for standardizing some aspect of system. The standard process ensure that

• Standard is really needed - Since all service providers and consumer must comply with standards (and they are implemented in service and application run time) the bar for adopting a standard is high. The goal is to standardized only what must be standardized - no more. 
• Standard define behavior not implementation choices: Also we try limit the standards to aspects of interaction between services and applications or services and operational tools//system and not service implementation. This is an important distinction. We have no standard on language, technology stack, data technology etc. 
• Standard is automatically testable - Otherwise it is not scalable to test all service for standard behavior manually.
• Standard is properly documented and implemented - For all standards we build support in services, application run time and libraries, or in portfolio tools

3- Decision Making (Huddle): Service and application teams are autonomous, they own all the decisions made regarding the code and systems they own (and accountability that comes with it) - however, there are always decisions at the portfolio level i.e. decisions that impact multiple teams. These decisions are often tactical and made in context of large projects (strategic decision are lframed in form of standards) and the often relate to either orchestrations of complex business processes or  migrations related decisions that impact other services or applications. The decision making process calls for
- Clear documentation of question(s) in hand
- Clear documentation of options and proposal
- Identification of decision maker

Debate and transparency are encouraged, the discussions are time-bounded and produces one of four Outcomes:

• Agree, Implement: Everyone implements the decision.
• Agree, do not implement: Tech debt is captured for the team(s) that don't implement the decision.
• Disagree, Imprement: Dissenting view is acknowledged and recorded. 
• Disagree, do not implement: One time escalation to a team of 3 senior technologist to make a final decision  

Measurement 

In general, there are two types of metrics, strategic metrics and operational metrics. In this context, strategic goals are the main reasons a platform is built (a car is built to transport people), operational metrics show the platform health (a car engine temperature must be within a certain range).

CommerceOS has the following the following primary goal

- Improving the productivity of application developers who develop apps to facilitate all form of commerce using any technology stack, on any device/screen or platform globally.

Of course to achieve this goal, there are plenty of operational goals, the major groups of operational metrics are:

1. Service Production Metrics: Encapsulating all eBay Marketplace capabilities in form of RESTful services 
2. Service Consumption and Adoption Metrics: Exposing capabilities to all marketplace participants on all device, platform and geographies 
3. Non-functional Aspects: Improving all non-functional aspects such as scale, quality, availability, security, cost etc.

The primary goal is not easy to measure. One way we experiment with measuring the goal is by Application developer NPS assuming that a diverse and wide range of application developers are included.

The operational metrics categories above are measured by a large set of metrics such as
- Coverage (% of resources/noun and verbs in marketplace dictionary that are exposed as services)
- Adoption (% of traffic enabled by services)
- Uniformity (% of device/mobile and web traffic using THE SAME set of service)
- Security (wide range of metrics here, require its own post - but primarily # of security issues reported per period)
- Availability (large set of metrics here per service)
- Quality: Number of P1,P2 bugs reported for a service
- Compliance - Number of standard each service is compliant with (with a core set as mandatory)

What is CommerceOS - Part IV - Foundations and Shared Services

Part I of this series of posts, described the what CommerceOS, Part II explained the structure and overview, in part III we focused on the standards that make a Microservice portfolio work. In this post I will focus on shared services. Let's start with the natural question

What is a Shared Service:

Of course most services are written to be re-usable and to be "shared" among multiple applications and services. But naturally the scope of use varies for each service. At the lower levels of any dependency graphs, in any service portfolio, there are services with much wider scope of use, a lot more services depend on them and their function is less domain specific and is more generic and "platformy" functions.

Although there are no fast, easy and mathematical rules to define a "shared service" - we have defined a few criteria:

•  Shared service can not depend on any business services (basic sanity of dependency graph)
•  Shared services are deployed and are accessible to all other services - they are part of the services runtime.
•  Shared service have more stable interfaces and slower release cycle - this is normally the case since the functionality is not business specific, so it does not change with business requirements.
• Shared service is, conceptually, something that is useful/applicable to any company (not just eBay marketplace), so at least in theory, they can be either open sourced or made available to different business/company.

The following services are CommerceOS list of shared services (at the time of this writing)

1. Identity and Access Management: Provisioning Identity for all Apps and Services. Plus an OpenID Connect based authentication protocol and a SAML based JWT based security token services. 
2. Billing: Multi-Tenant billing and invoicing based on billing event stream. 
3. Messaging: Few internal message buses, primary message bus is an internally developed light-weight and highly scale-able Business Event Stream (BES) - it is a publish-subscribe only messaging system. Transitioning to AMQP based messaging system to support wider set of messaging semantics - some use of Kafka.
4. Logging: Distributed logging, uses TIBCO message buses, stores in HBase and integrated with Hadoop. 
5. Monitoring: A real time version of Map-Reduce (Red Lemur) - similar to Apache Storm. 
6. Tracking: of API calls and user activity events. Custom event API, types and structure, over TIBCO buses, 
7. Config and Metadata:  Custom key/value pair service transitioning to Zookeeper based config management 
8. Content Management: A template language for defining content and its variations, translation workflow and a run time distributed content repository backed by MongoDB and managed by ZooKeeper
9. Discovery and Registry: A build time registry tool based on Google Discovery Doc (GDD) and backed by a Casandra cluster.  
10. Routing and Inter-mediation: WSO2 based ESB, kept simple. Only provides routing and location transparency. Most composition and transformation of lower services are done by higher level application services, intermediary layer is kept "dumb" as much as possible. 
11. Crypto Services: HSM based hashing and encryption service, based on each tenant security policy.
12. Object Storage: HBase-based storage for images (or any BLOB)
13. Caching: Memecahed based API/Protocol implemented by Couchbase  

Shared services are multi-tenant, and can be consumed either as a service (majority of cases) or as a separate private instance (a few cases). Shared services comply with the rest of CommerceOS standards. The access to shared services are controlled by tokens provisioned by CommerceOS STS (secure token service) based on the provisioning rules for each application or service.

What is CommerceOS Part III - Standards

In the first part of this series I described the goal and motivation of eBay CommerceOS initiative and that CommerceOS is eBay version of Microservices. In the second part I described the five major components of CommerceOS (technology, processes and org). In this post, I will focus on one of those components: The standards and patterns.

Let me start with an example: Service/API authentication:

Most eBay applications use between 20-50 services, lots of them require application as well user's identity (for security and functional reasons) - imagine if each service (or domain of services) accepted a different type of token with different issuer, different syntax and validation semantics and different binding of that token to protocol (in the body, in the header with different names, combined with other headers etc.). An application would have learn and then write a lot of boiler plate code to obtain token, store and then submit via request to different services, then it would have to parse and learn the error semantics for all types of authentication done by each service. All these activities make the code more complex to write, test and operate and they do not add any value to the main function of the app. Now extend this across all types of horizontal concerns and you get an idea why standardization backed by run time libraries is a must for any service portfolio at scale.

CommerceOS defines a set of standards and implement them in our framework for developing services (called Raptor) - this does not mean that eBay MP does not allow or discourage services to be built using any other technology stack, but if a service is built using the standard libraries and run-time - it gets the support of all standards.

Pattern and standards include about ~30 different aspects of service design, including

1.  Identity & Access Management, 
2.  Base Request & Response standard and extended headers
3. Compact Header Encoding (more efficient use of headers)
4.  Tracking 
5.  Internationalization 
6.  Error Handling 
7.  Version Management 
8.  Service Descriptor 
9.  Service Life Cycle, Registry and Discovery  
10.  Addressing and End Points
11.  Sorting, Pagination, Filters and Views 
12.  Instrumentation of Services
13.  Messaging and Events
14. Security
15. Migration 
16. Fail-over and Recovery  
17. Multi-tenancy 
18. Integration (with 2nd and 3rd parties)
19. Configuration and Metadata Management 
20. Content and Translation 
21. Persistent Storage, Replication
22. Failure and Recovery 
23. Service Modeling & Interface Development Model (IDM)
24. Multi-Tenancy 
25. Base-API Operation (operations that all APIs must answer)
26. Escape Response 
27. Asynchronous Service Design 

One significant CommerceOS activity stream centers around design and then implementation of pattern and standards. We focus a lot on correct and accurate documentation followed by implementation in run-time libraries and or as shared services. CommerceOS also defines a process for developing and adopting a new standard. This process is modeled after and is very similar to Internet standard development process (working groups, open discussions, editors) - with the exception of it has a solid timeline to time bound the process. This way all service providers and application developer can participate and/or comment and influence the standard.

In the rest of this series of post, I will go into a bit more details on some of the more significant or interesting standards (if you want to know more about any standard I didn't explain just contact me). In the next post (part III) I will focus on the two sets of principles/patterns that formed our thinking around portfolio design and individual service design and how we measure goals and operational metrics.

Service Descriptor and Interface Contract

CommerceOS emphasis on a formal contract. We use Google Discovery Document (GDD) as the basis of our service descriptor and we extend it to include aspect of service contract we need to manage the service life cycle during build time or run-time. In Java environment, Service interfaces are annotated using a standard annotation library, our discovery tool then generate JSON based discovery document that is used for interoperability with the rest of our tool set used by application teams, product manager in other product teams, tech writers etc.
COS service contract has four main parts

1. Service meta data, this include the basic meta data as well as attributes for financial, regulatory and legal needs such as whether a service handle financial data (what types) - whether a service handles personal data, location of personal data etc.
2. Service interface and types as describe in Google Discovery Documents
3. Service instrumentation contract - this is the contract service has with its operational environment and defines events service generates and consumes (including events required for technical and business health monitoring)
4. Service admin contract, loosely JMX based API for adminstartor to set or get certain attributes and influence service behaviors  

Service teams own the contract and its maintenance, but syntax and semantics are standardized.

Service Versionning

CommerceOS services must be versioned, we use Major.Minor.Maintenance format. Service team need to declare/decide how many back versions they support. No team is allowed to support zero back version and break backward compatibility - since this forces all applications to migrate.
We allow multiple versions to be alive at the same time, API Router will ensure a given request goes to the right end point. Data and entities are designed to be backward compatible.
Services are not allowed to be perpetually backward compatible since this practice erode code quality and accumulate significant "dead code" that leads to drop in agility and complexity of test.


Service Life Cycle and Registry

One of the most significant decision for CommerceOS was to standardized and establish a widely understood set of mile stones (called life cycle) for service development. This may sound like the dreaded "G" word (Center Governance) - but in practice a large org can not plan an optimal and rapid release cycle without it.

Before the life cycle standardization, the only defacto mile stone for service team was "live to site" i.e. when the service end points were available and functional in production. Application developers (web and mobile) would then start their serious development, effectively serializing the timeline i.e. Delivery time = Max(Services Delivery) + App Delivery.

Without a wide understood and supported mile stones, service team often change their service implementation and interface till the very end of a project timeline, forcing application developer to wait till the "dust settles". CommerceOS establishes a set of mile stone, the first of which is "interface published" this mean the service descriptor is ready, and an end point is exposed that can respond based on the service descriptor - this end-point, in concept, is similar to Java Proxy API - in that it can produce a "fake" response to the request based on the contract - no real implementation required. Application development can practically starts at this point, to a large degree decoupling app development time from service development time.

Service teams can change the interface - but often the thoughts and consideration that went to interface design leads to more or less stable interfaces, the implementation can change freely at any time. This align with one of our portfolio principles of "Stable interface, agile implementation".

Base Request and Response

CommerceOS services and application talk over http, but the exchange has to happen with a common dialect i.e. certain semantics has to be expressed and binded to the underlying HTTP transport in a common way -  this saves individual service provider and app team time to re-invent the wheel also prevent a lot of bugs and issues. Base request and response define a set of common headers and encoding that all COS service and apps understand, a few examples are
Syntax to express compact headers, Authorization headers, Identification of request and request chain, serialization and encoding of request and response, session identification, location, locale and cultural preferences bindings to the protocol, the proper use to HTTP header v.s body and alternative binding to HTTP body.


Migration

One of the practical and most important aspects of establishing service or micro service architecture for large companies with "legacy" code is migration. By migration, I specifically mean migrating either monolithic applications with direct data access or application the use older, legacy services to application that consume contract based micro services. We have established a pattern, called "Bay Bridge" for service migration. It has four major steps

- Smoke Test: Turn on new service (with new data storage), only use it for a very small number of traffic for a few consumers, dual write into (and read from) both new service/storage and old/legacy storage. primary source of truth still is the legacy.
- Load/Sync: Copy/transform data from legacy storage to the new service storage as appropriate. This phase itself may include smaller phases depending on data. The more long lasting data/entity is the more critical this phase is e.g. User is a very long lasting entity while an Auction listing may last only 7 days or a session may be stored only for few hours. The main goal of this phase is to bring new and legacy storage to parity.
- Fly with Safety Net: Dual read/write continues, but the primary is the new storage/service now
- Clean up: Old storage is cleaned up and deprecated




Persistence
CommerceOS allows services to choose their own persistence storage and technology, depending on types of data a service handles (preferences vs. financial data or blog post vs. password and credential) there are pattern for whether systems should prefer CA (financial), PA (most anything)

From logical point of view, services team are required to have isolated storage i.e. no other service or application should read/write directly from primary database of other services - sometimes (especially with bulk data) it is not efficient to consume a classic service interface (serializing and deserializing is too much over head) - in these cases service must expose a "data feed" - push style, and still should not allow other services to directly read its primary storage.

Services are required to register they database and structure of logical entities stored (there is no "governance" of such entities just registration for discovery process)

Fail Over and Recovery 

There two types of fail over and recovery in CommerceOS, Transparent and Degraded.

Transparent failures are the failure of stateless application server/service or database hosts. application servers are all running behind load balancers with virtual IPs (and sometimes load balancing is done using run-time discovery ZooKeeper style) - database failures are handled by partitioning data and replication (Casandra style) the typical failures of services and databases are handled without service code realizing the failure, the system continue to operate with no impact.

The other types of fail-over is "degraded" - in this case failure is not transparent to a service for example when an Pricing service (that calculate total order price) calls an Incentive calculation and receives a failure - say due to yet another system failure in Incentive subsystem that could not be handled (e.g. it uses a non-partition, non-replicated DB that failed) - the Pricing service now has to "degrade" its function in a way that it still calculate the total order price. This is a higher level and more domain specific handling of failure yet a few aspect of it can be abstracted and implemented in run-time. In particular, we define light-weight processing framework. It is pipeline based programming model, each pipeline has a series of phases, phases can be assembled dynamically at run-time. each pipeline is executed by an Executor that is the main run time for pipelines. each phase can be annotated as required, optional, alternative. Each phase has a few life cycle state, the two most important ones are up and down. If a required phase is down, and if no alternative is designated, the pipeline fails, if an optional phase fails executor executes an alternative as designated, if no alternative is designated process continues.
This simple framework provides an abstraction for degraded functionality.

Escape Response

Escape response is a small, yet important, aspect that illustrates the need for standards in a give portfolio. An example illustrates the concept, imagine that due to a security breach, you need all users to change their passwords. You can change 100s of applications to message the user for password change. What do you do?

Applications make service calls all the time, a particular response header is called "Escape Response" and it include an end point and a unique number. All application know (and it is implemented in the service invocation library as well) that if they see the escape header, they must re-direct (device and platform specific) user to the given end point. This "escape" path allows the system to take over from any compliant service using a standard syntax, semantics and protocol binding.

What is CommerceOS - Part II - Basic Structure

In the first post of this series, I described the motivations and goals of CommerceOS. I stated that CommerceOS is eBay MP version of Micro Services, It is a portfolio of services (RESTful and deployed in the cloud) that enable developers to build applications that facilitates commerce among people and/or entities, on any device, any language, rapidly, economically and securely.

In this post I will start describing some technical details, best practices and what we learned from building a service portfolio at scale.

The easiest way to start describing CommerceOS  is with this basic picture illustrating the over all topology of the system

This diagram shows a high level logical architecture of the ecosystem and its main components

- Applications: this includes applications on mobile, and all other devices, web apps as well as 3rd party applications, merchant and partners back ends. etc. Basically all consumers of CommerceOS portfolio. The number of applications using CommerceOS services are in the range of 10s of thousands and those include eBay applications (1st party) as well as partners and 3rd party apps

- Services Portfolio: These are services and business capabilities that enables marketplaces operations as well as Commerce primitives used to build any commercial app. Examples are Listing, Checkout, Order Management, Listing Details, Watch List, Cart etc. These services are often (but not always) multi-tenants. Collection of these services is called "The Portfolio", The number of services in CommerceOS portfolio is the range of 100s built by team across the globe.

- Shared Services: core platform capabilities such as Identity & AM, Security Token Services, Session, Tracking, Messaging (of all types) etc.

- API Middle Tier : Part of "shared services" is a middle tier  that mainly functions as simple router connecting API request to internal services implementing them.


CommerceOS is structured into five major activity streams (and similar organization structure), each essential to build and operate a Microservice ecosystem at scale. The tracks are

1- Standard and Patterns
2- Services Portfolio
3- Foundations and Shared Services
4- Evangelism and Advocacy
5- Governance, Metrics and Measurements

Standards, Pattern

The first observation about an ecosystem with 1000s of apps and 100s of services is that if key aspects of service interface and implementation - such as error handling, tracking, identity internationalization, version policy, monitoring, billing etc. - are left to each service team to decide, developing an application that consumes on average 20-50 services will be extremely difficult and unproductive task. This approach may also creates legal liability (dealing with financial or personal data). A decision we made early on, was to formally define (and carefully document) standards that make all components work together. This may not be what you expect from a Micro service portfolio, but we learned that formal documenting of standards (and versioning them) is extremely valuable both for system developers who would be implementing those standards in form of run time libraries and shared services as well as service and application developers to learn about the portfolio in general. See part III for more detailed on standards

Foundations and Shared Services

A portfolio of services requires three major foundation pieces
- Run time libraries for services (in supported tech stacks)
- Shared Services that are used by the entire portfolio (and you really don't want replication of any of its functionalists), services such as Identity and Access Management, Caching, Messaging, Crypto Services, config, registry and discovery, content management)
- Productivity Tools: Vital for operation of portfolio are tools (and dash boards) for monitoring, recovery, synchronization, life cycle management, learning and exploration tools etc.

See part IV for more information

Service Portfolio 

This is the actual portfolio of business capabilities, the services are designed and built by de-centralized (and global) services teams - they are often between 3-10 people, and they are free to use any technology stack they see fit - of course if they use the eBay standard Java/Spring based framework (called Raptor) they get a lot of functionality for free - but they don't have to. Services must publish their descriptor formally and some elements of their type spaces (such as User, Listing, Order, Cart, Claim, Product etc.) are standardized (i.e. they are part of a Common Type Space) - services also need to declare events they produce.

Evangelism and Advocacy (E&A)

This may be one of the most significant, yet poorly understood, area of any service portfolio management. People often view this as "soft skill" area. However, they are vital to BOTH adoption and evolution of service portfolio. Let's see what they are

- Evangelism: Main goal is to make sure application developer community (especially internal one) know all capabilities in the portfolio and how to use them. It includes documentation, sample codes, sample requests sets (especially for edge and error cases) as well as basic performance characteristics of a service. Without proper evangelism, developers will not know what capabilities exist, and how to use them properly, as the result, they will re-build the same capability (often with slightly different name, semantics and implementation).

Advocacy can be thought of as the reverse of evangelism - its primary goal is to make sure services cover all the right capabilities application developers need. Through advocacy (and collecting requirements) - we learn whether we need to introduce (or promote) a new type to our common type space, or whether we need a new service (such as integration with certain data provider - such as phone IMEI or business address provider). 

Both E&A are critical in re-use at scale - re-use (especially at the domain capability layer) is not developers' natural reaction. The higher you are on the stack, the tougher the re-use become. It is less likely that a team (or engineers) rebuild JVM, or a messaging protocol, but business abstraction such as Identity, Cart, Order, Search Result, Listing, Invoice etc. as well as processes that operate on the, are duplicated all the times. E&A is the key to get some level of re-use at the business domain.

Governance, Metrics and Measurements

The last major area of activity for CommerceOS is Governance, Metrics and Measurements. A lot of people believe that no governance should be required in developing a Micro service portfolio, this only leads to chaos - particularly for application developers that often consume a large number of services. A more sensible suggestion is "decentralized governance". This sounds like a good idea, but the devil is in the details. CommerceOS adopted a light - but well defined - governance model and processes that enables it. I fully realize that this may not be a popular concept in today's environment driven by buzzwords such as "autonomy" and "agility", however my personal experience is that

- Right level of governance makes everyone job mush faster and less painful

- Process has to be transparent, objective and driven by an explicit and measured SLA - what people hate is arbitrary process run by "people who think they know more than you do".

Having said that, CommerceOS defines processes to govern three aspects

1. Adding a service into CommerceOS portfolio (or deprecating of a service)
2. CommerceOS Common Type Space
3. Adoption of an standard or pattern into CommerceOS

Each process is well defined and has largely objective criteria.

See part V for more details.

In part III, I will go over the standards and patterns that make all services and apps interoperate successfully, Part IV focus on Foundations and Core Services and part V talks a bit about processes and measurements.

What is CommerceOS - Part I - Introduction

I am often ask what is CommerceOS, equally as often I answer based on what a successful end state for CommerceOS looks like: It is a set of capabilities exposed largely as RESTful APIs that allows anyone to rapidly, cost effectively and securely develop applications that facilitate commerce between or among real or legal entities, on any platform, device or language, globally.

This means if you want to develop an application for a consignment store to sell its merchandize online (on or off eBay) CommerceOs makes it easy for you. If you work for a large multi-national corporation and want to list your products on eBay and on your site, then let local, regional and international buyers purchase your product, CommerceOS is your APIs. If you want to write a pop-up app on iPhone and Android that showcases adventure riding gears and products by mashing up content and pictures with commerce and community, CommerceOS should make your job easier and cheaper, and of course, if you are an eBay mobile developer, CommerceOS APIs enables you to build your applications ...and countless other scenarios like these, you got the picture.

This answer, although accurate from utility and use point of view, does not describe CommerceOS from technical and architectural point of view. A casual observer may not appreciate the technical, process and organizational complexities that should be overcome to achieve CommerceOS vision. In this series of posts I try to explain CommerceOS from technical aspect.

The rest of this post dive a bit deeper into motivations and problem definitions, in the second part,  I will focus on how we have approached the problem and architectural details of CommerceOS. Part III explains set of technical standards and patterns used by all CommerceOS services and consumers, and are the basis of interoperability of the entire system. Part IV is a review of a few core services of CommerceOS called "Shared Services". Finally Part V deals with process we use to maintain and enhance the platform.

First a bit of a context. Back in 2009-2010 timeframe (where CommerceOS transformation started) eBay had a global engineering workforce in the range of thousands (as it does today). The collective job of this engineering team was to work on an essentially web-based architecture (like so many other web era companies). This architecture assumed (perhaps more implicitly than explicitly) that

• Buyer interacted with marketplace largely via a web browser, running on a desktop/laptop.
• Sellers were small(er) and interacted either via web UI or APIs designed to replicate Web UI - essentially APIs were designed for sellers.
• API code base and the business logic powering web UI were largely two independent code basis.
• Data and table were shared among all applications and services. Systems were single tenant and monolithic from identity and tenancy point of view. 
• Both data and code for all subsystems/capability (such Identity, Checkout or Payment services, Search) was hard to isolate and package independently.
• Operation technology was fixed capacity, manually provisioned, under-utilized.

The platform was stable and scalable (albit with scale-cost characteristic by, then, current state of technology - fixed capacity, built for burst) with its own set of processes that made it possible for global teams to get their job done. However, in 2009-2010, it was very clear that we are a few years into development of trends that would re-shaped (or ram through, depending on your point of view) traditional web-based architecture and require a different scale-cost and productivity trajectories. These trends were:

1- The ever present mandate for efficiency and cost control driven by the need to expand or at least preserve margins or reallocate resource. This is natural trend for any market and segment as they become more mature and scale.
2- The mobile and connected device revolution which was shifting more and more of traffic to non-web, non-PC devices.
3- The advances in cloud computing that drove the operation cost down, but demanded functionality to be packaged cleanly and independently to be deployed wherever capacity was available.
4- The shift of eBay merchant mix from smaller, more casual, lower volume sellers/merchant to larger, more architectural and operationally formal and higher volume sellers.

(I will talk about how technologies known as Big Data changed and impacted eBay architecture in a different post)

It was very clear that the forces of the four factors above require a different architectural approach from what, then, current web-oriented architecture was. We needed to bring down the cost of both infrastructure and operation as well as increase the productivity of our developers to address #1 above.

To address the multi-screen revolution (#2), we needed more than just a set of APIs (which eBay has had since 2003) - eBay marketplaces needed all screens (included desktop) to consume one set of APIs not two sets of code-bases one power the "main" web UI and the other to power all API/devices. This is a much harder problem than simply developing APIs. Additionally, we needed all APIs, regardless of which one of nearly 100 teams across the globe develop it, to look like a coherent portfolio of APIs (given the laws of thermodynamics, you can guess how tough this is).

To take advantage of promises of cloud computing (elasticity of capacity, better resource utilization, related cost saving etc.) we needed our business logic and data to be encapsulated in well-defined and isolated modules, with clear dependencies that lend themselves to packaging suitable for cloud environment. Refactoring code to isolate systems and services is tough enough, but separating shared data and isolating storages and tables are an order of magnitude more challenging.

And finally, the larger, more sophisticated our merchants and partners became, the more we needed to use formal integration methodologies and feed-based interactions built on de-facto industry standard models of Order, Product, Inventory, Cart, Returns and less on auction-centered models such as Item, transaction etc. This represented a deep change in entity and domain models and re-architecture of code and, perhaps more importantly, migration from old models to new models.

Given where we were and where we wanted to evolve marketplace architecture, we came up with a definition of a vision for CommerceOS as the basis for alignment and execution, knowing full well that no one paragraph accurately covers all aspect of this initiative. Having said that, here is the definition we came up with:

CommerceOS is eBay MP initiative to transform its architecture in such a way that large majority of its platform capabilities and business processes are exposed as RESTful APIs in the cloud in such a way that all marketplace participants can consume them, uniformly, securely, effectively and with high quality.

This may sound a bit abstract, but let me parse it:

The primary goal is to "transform architecture" - this means both technology stack (for services and applications) and processes not just technology (it never works that way).

Platform capabilities and business processes: That basically means everything that MP does from identity, verification of attributes, and caching to listing, pricing, order management and search need to be re-designed as RESTFul services and exposed to correct consumers. The modifier "Majority" indicates that some interactions are not strictly RESTful for legacy or integration requirements. We are pragmatic about it.

Our services are RESTful, it is a de-facto standard and we developed a few internal spec to uniformly covers aspects such as use of non-http verbs, security, tracking, internationalization, filtering, constraints etc.

The deployment target for services is "Cloud". This is not a feel good or casual term, it strictly mean that a service must be produced in a way that it can be packaged fully isolated from other services with well known dependency, clear version number. This allow us to deal with a unit of capability as a building block and use technologies such as Docker effectively (we had a few de-tour here with OSGi but course corrected).

"All participants" indicates that the consumers will be applications written not only for buyers and small sellers who interact with marketplace via a browser, but everyone else on all devices including large sellers who never interact via ebay.com, CS and their internal tools, partners such as logistical service providers as well as internal eBay staff and, of course, eBay application on all devices.

Then there is a set of non-functional requirements, each key to how APIs are built, they include

Uniform: basically one code base supporting all consumer, with no "primary" web consumer.
Secure: self-explanatory, platform MUST be secure with well defined identity, access management and auditing. Secure also include availability.
Efficient: APIs must be efficient to consume, i.e. application developer must spent min amount of time and effort to consume it (docs, samples and sandboxes), APIs also should be cost effective to operate (see cloud requirement)
High Quality: An umbrella term covering functional quality (no-bugs) as well as performance.

CommerceOS started with this vision definition and context, in the next post, I will describe the architecture and structure of CommerceOS, both as a technology platform and from organization/process point of view.

Via Quora: How has eBay's review system evolved in the past few years?

The feedback system and "feedback score" is one of the first things associated with eBay (maybe second only to auction). The main purpose of any feedback system is to harness the "signals" community members send to "feedback processor" to encourage and "enforce" desirable behavior. Anyone who ever thought about or design a rating or feedback system knows that the task is not as easy as it may first seem. And the first step is to understand what the different attribute of a feedback system is.

A member of Quora community asked me to answer the question "How has eBay's review system evolved in the past few years?". The question provided me with an opportunity (and a little push) to talk about six attributes of any feedback system. See here for the full answer and if you are not using Quora, I strongly recommend you consider using it. It is addictive. 

Primary Goals and Strategic Metrics vs. Operational Metrics

Measurement is key to any successful engineering effort, the key is what metrics you choose to measure and how to interpret the measurement.  In general there are two types of metrics associated with two different types of goals

- Primary (or strategic) goals and related metrics
- Operational Goals and related operational metrics

When goals are defined, It is very important to define which category a goal belongs to so that progress (or lack thereof can be measured accurately).

An example makes the difference between the two of metrics clear. Imagine you want to drive from San Francisco to Los Angeles. Your goal is very clear: getting to L.A. and the metric associated with it is also clear, how far are you from L.A. (or how far have you traveled so far). You drive a car, so you measure your car's engine temperature and fuel level - these are your operational metrics.

You can keep your engine temperature within a reasonable range and maintain proper fuel level, but these are not the goals of your trip. If you are happy simply b/c your engine does not overheat, and you are not concerned with how far from L.A you are, you are not measuring the right metrics.

Distinction between primary goals and operational goals is not always as easy to spot. Imagine you are designing an Order Management service (or any other service) - it is easy - maybe even common - to measure number of calls to the service per hour, day etc. and assuming that the goal is being achieved. However, if you service can only be called by one type of client (b/c it is the only one that can supply an input parameter for example), are you really achieving one of the most important (and common) goal of service design which is re-use of a capability by all consumers? Here, number of calls is an operational metric, it has to be measured, it may be necessary but not sufficient. The measure the primary goal - you need to measure the diversity of consumers (across languages, device, platforms)

For another example assume you deploy a distributed cache service - the goal of any cache service is to improve performance (and scalability) - again,it is easy to measure metrics such as number of calls to the service or even more specific metrics such as "hit rate" -but the primary goal here is to measure - from consumer point of view - performance and scale improvement.

One main reason, operational metrics are often measured, and used for decision making, instead of primary goal or strategic metrics is that they are generally easier to measure. It is certainly easier to measure "number of calls" to a service than whether a service increases its consumer productivity. "How to Measure Anything",  by Douglas Hubbard is a great resource for techniques to measure hard to quantify goals and metrics.

What really is this "Managed Market Place" anyway?

I work for a division within eBay Inc. called "Managed Market Places". The name is a bit curious. I was asked, more than once and by range of people, what really "managed marketplace" is? Is it a new type of marketplace by eBay (no it is not!), is it a vertical/niche marketplace within eBay (no it is not!), some one on Quora even interpreted it as it means that eBay simply "manages" the marketplace as oppose to growing it! ( if this was the case why would eBay announce that to the whole word by labeling it as such?)

So then what exactly is MMP (as it is known internally) and why is it important?

the nature of the Internet lends itself perfectly to the basic concept of a "marketplace": a mechanism for buyer and seller to find each other. Marketplaces were and still are an important and growing part of the internet. The growing list of niche marketplaces include etsy, zarrly, odesk, airbnb, taskrabbit, yardsellr, zimride and many many more. (not to mention marketplaces from facebook, google, yahoo and other major players)

At the first glance, it looks simple enough: create a site that brings the parties to a transaction together (from buyer and seller of antique to two people who want to share a ride or a room), and either take a cut of the transaction or make money by advertising. This is indeed the basic concept behind a marketplace - or an unmanaged marketplace. Marketplace itself is not a party to any transaction. Buyer and seller deal with each other directly and take the risk (or bulk of the risk) of direct transaction. EBay operated, more or less, as an un-managed marketplace for a while too.

In managed marketplace on the other hand, neither party to a transaction takes a risk, in other word marketplace guarantees the success of transaction, no risk (at least ideally). Of course a managed marketplace can "manage" other aspect of interaction such as inventory, quantity, price, promotions etc. as well but for now we only focus on risk as it is the focus of eBay MMP as well.

The evolution of simple internet marketplaces to managed marketplaces is an important trend, as the Internet users become more sophisticated and demand more from services they use online. The AirBnB incident back in July of 2011 is a perfect illustration of how "unmanaged" marketplaces will be forced to offer a higher level of assurance/risk mitigation and become managed marketplaces.

What does it mean from systems and architecture point of view? Here are five main aspects that is particularly different in dealing with managed marketplaces

1- The first significant change is that of people's mind set: You have to see yourself in risk management business, or at least assume that risk management is a major part of your operations. What this changes first, and foremost, is that you now have to identify, assess, prioritize, mitigated (or plan to) and measure risk. In all likelihood all of these activities (and the tools and systems you need to perform them) are new to you if you are dealing with a simple/un-managed marketplace.
2- Central to any consumer risk management scheme is "Identity", and I don't mean OpenID or OAuth or SSO... I meant attribute, assurances, verification, accuracy, uniqueness or mapping a real world entity to a digital identity (Entity Resolution)
3- Data is the core to efficient risk management, and big data and your ability to collect and analysis them becomes central to your ability to operate the marketplace at a reasonable cost (minimum losses)
4- Coherent Architecture become even more important. Simply because your systems becomes more complex and more integrated. A simple marketplace is just that, a marketplace site/application. A managed marketplace would include identity provisioning and verification, risk definition, measurement, management at user and at transaction level, a system for filing claims and disputes, systems dealing with ever changing legal and business landscape that enforces what you can and can not do with data you collect and finally integrating all these system in a productive way (seamless but without coupling them)
5- Even Driven and Complex Event processing: This already has a big role in distributed system, but it plays more and more important role in distributed risk management. Real time assessment of risk becomes critical and due the cost/performance of risk assessment, incremental assessment or risk based on primitive and complex event generated over entire session (or even life time of a user) will be the only practical solution.

The Uncommon Security Common Sense

I can not claim that I actually counted or classified all the reasons peoples cite for not taking security (or for that matter sound and well thought through system design) seriously from the start, but the three following lines seems to be the most common ones:

1- The "it is too contained" line: So what is the big deal? at worst it may affect a very small percentage of my users.
2- The "it is too early" line: Oh my system/site/project is too small and we only have a few users, we really don't have time/resources for this.
3- The "it is too small" line: My project is too small or too obscure for anyone to care.

By the way, I have heard these lines or their equivalent not only when it comes to security engineering (or re-engineering) but also in designing business policies or risk management measure to prevent fraud, or in general negative user experiences as well as general system design.

Now to be fair, these reasons all sound like "common sense", after all why would you take on additional cost and time for your project or accept the expense and risk of re-engineering your code to fix an issue that may only affect 1% or 0.01% of your users? or why should you spend two weeks to fortify a system that takes you 3 days to design and it is "just an experiment"? and finally who really cares about a small project some where with some obscure URLs that takes an email address as one of its inputs and shows some useful error message if the email is not registered? does ANYONE really care?

Well, as it turns out, security common sense (like many other form of common sense) is actually quite uncommon ! Let's look at these frequently cited common sense logic a bit closer.

To demonstrate the fallacy behind the first logic (it is too contained, it only affect 0.01% of users) I cannot think of any better illustration than the words of presidential candidate Herman Cain where he said that "for each woman who has accused him of harassment there are probably thousands who haven't" and he is 100% accurate and right! But does that make any difference? In all likelihood his presidential bid is all but over. Or could the Washington D.C police chief during the "D.C Sniper Attacks" have possibly argued that the whole thing was not a big deal b/c only 0.001% of D.C metro population were actually killed and therefore there is no need for massive mobilization of police, FBI, ATF and even secret service !?

The same math is thru for security, it does not matter if only 1000 users out of 10MM become victim of a
poorly secured or design system. What matters is how many people hear and learn about it - and you can be
sure that at least in this day and age that number is a few order of magnitude larger than the actual number of
victims. The sense of insecurity that this causes in the rest of the user community and its economic cost is the real math that matters not the fact that only 1000/10MM=0.01% users were affected.

The second line "it is too early" or its equivalents "we don't have enough time or resources" is the most commons line not only in security matters but also system design and architecture aspects as well. What is interesting here is that the exact premise cited for not focusing on security (or sound design for that matter), is why security should be taken seriously i.e. "I am too new to afford not to be secure", if you are releasing a new product (or brand or a site) you REALLY DO NOT HAVE A SECOND CHANCE TO MAKE A FIRST IMPRESSION. If you are not secure, or if your first few user gets taken advantage off (think of AirBnB incident) you are doomed. To further demonstrate the risk in this argument I submit the following picture of one of the more famous car design mistakes : Honda Odyssey 1998

Honda designed this in a hurry to get into the growing minivan market dominated by Dodge/Chrysler. They decided to differentiate by replacing a convenient power sliding door with a traditional door! Imagine what would have happened of this was a new no-name company without Honda's established brand? Of course Honda corrected the mistake in 1999 model and beyond and went on to have one of the most successful Minivans. But if you are not Honda, you better spend time and money on designers and marketers to tell you,  in the first try, that whoever buys a minivan *needs* a sliding door.

Now we get to the third line "Who really cares about me?" I have to admit that I have the most sympathy with people who resort to this logic. After all it is tough to imagine how capable and resourceful the modern fraudester/hacker community is without actually having a brush with them. I do not get into the details -  if you are interested you can briefly scan Rick Howard's excellent book "Cyber Fraud, tactics, Techniques and Procedure" - for the purpose of this writing I'd suggest you assume the following is true:

In the game of "Who wants to break into my system" your adversary is more motivated (financially or politically) than you are, more experienced than you are, is more innovative than you are, is more nimble than you are, wants it worst than you do, has a smaller cost base than you do (and therefore) all he needs is 0.01% (or smaller) of your users - the ONLY advantage that you have is that you right the rule of the game. Do not give up that advantage easily. You WILL lose the game.

Btw, the end point URL that takes an email and very nicely checks and display an error if email does not belong to a valid user - was actually found (although it was a little obscure URL not linked to from anywhere -  and used to extract valid company X user emails (cost $5000+) from a large list of non-verified harvested emails (cost $50) - a vital part of phishing industry value chain.

OIX Attribute Exchange Summit - Washington DC

Open Identity Exchange (OIX) is holding this year's Attribute Exchange Summit in Washington, DC.

Identity attributes are core of the concept of digital identity. As federated identity ecosystem getting more mature and adoption grows among more sophisticated RPs  - with more consequential use cases such as government, health, education, commerce ... - so does the need for wider sets of attributes with more accurate and fresh values. This presents both tough challenges and opportunities for IDPs.

The challenges center, as one may expect:, around aggregating, correlating, transform and maintaining fresh copy of attributes in a cost effective manner and in a way that it does not compromise the privacy (and other rights) of the principle owner. IDPs can differentiate based on the range of attributes they provide in this way and there in lies the opportunities.

I will be talking more about identity attributes, their life cycle, uses cases and how they help establish and elevate trust among parties to commercial transactions (online and off-line) as part of a panel with Don Thibeau, OIX/OIDF chairman and Abbie Barbir, VP BoA.

If you are planning to attend, I'd be happy to hear from you.

OAuth vs. OpenID Connect ?

OpenID Connet 1.0 Spec is finally released (actually it was release back in Aug). Its release was accompanied by two predictable categories of questions/sentiments, one not very well informed and the other one a legitimate question:

-        OpenID is dead

-        OpenID Connect is really OAuth so why do we need a new protocol?

Granted, this is normally coming from software engineers and social application programmer community and not from identity community, but I feel they are significant enough to be addressed, especially at the time that more and more entities contemplating to become identity providers and they need to decide which protocol they should implement.

First, on the demise of “OpenId”:  It is true that the earlier versions of Open ID (version 1 and version 2) are, for all intent and purposes, depreciated and will not gain a whole lot of traction. But the general idea of “Open” standards for communicating between RPs and IDPs that enables users to provision fewer accounts and have a portable identity while still maintaining control over their privacy and data is alive and well and actually is even more vital than before.

Second, on relationship between OAuth and OpenID Connect, OAuth is a general protocol for authorizing an agent to access a resource on behalf of resource’s owner. OAuth does not assume any particular knowledge about the resource itself. What does this mean? Let’s go back to the canonical OAuth use case of a user who would like to authorize a printing services to access her photos from a Photo service provider. Now imagine that the photo service is slightly sophisticated and recognizes a few properties associated with photos e.g. resolution, size, whether they are shots with no humans, and if there shots with humans, who appears in the photos – basically let’s assume the resource served by SP has more semantics that simple “access”.

Now imagine that the user wants to grant access to only JPEG photos of himself and not a full access to all photos. How would the IDP encode this semantics in the authorization request and response? How would the SP know that they should only provide access to a subset of images?

To be sure, this is doable using OAuth, but the implementer has to add additional parameters to request and response or possibly constraint the input values of some other parameters.

A protocol that is built this way to access a specialized resource, would be a photo access protocol built on top of OAuth.

In essence this is exactly what OpenID Connect it: It is a protocol built on top of OAuth that supports features that are often desired and used when the resources being delegated is “identity” and attribute about an identity.

To illustrate the point, here are what we, at eBay, had to do for an internal authentication protocol on top of OAuth:

-  

      Force Authentication: adding parameters to authorization request to force users to authenticate no matter what the authentication state with IDP is

-        Authorization Behavior: adding parameters to authorization request to indicate to IDPs whether it should display the consent page and how to display the login page (overlay, full page)  

-        Standard Claim Set: Defining the default set of attributes returned by IDP

-        Requested Attributes: adding mechanism to allow RPs to ask for additional attributes, and annotating them to indicate whether explicit user consent is required.

-        Authentication Context: adding a fragment to response to communicated authentication context (single v.s multi-factor, PIN vs. Password, number of retires etc.)

-        Protection: adding parameters to indicate how access tokens should be protected (encryption, signature and order of operations)

-        Token Validation end point: adding an endpoint to introspect access tokens on demand.

These are all features and facets that OpenID Connect enables in a standard and interoperable fashion. In absence of a standard such as OpenID Connect though, any RPs integrating with our IDP had to implement basically a proprietary protocol, be it on top of OAuth.

The point is that if you want to operate an IDP and you want to use just OAuth, you have to add a few things to OAuth, depend on the depth of your requirements, to make it work for “Identity” resource. This is exactly what Facebook did with FB Connect – and they also did a good job of wrapping it with JavaScript plug-ins. The goal of OpenID Connect is to use OAuth as the basic access authorization protocol and add identity specific features to it so that it becomes a standard “identity protocol” that can enable seamless interoperability. 

PayPal Access & Commercial Identity

Today eBay Inc. announced an identity and attribute provider product called PayPal Access. Some described as a "Facebook Connect for Commerce", others described it as an easy registration tool for mobile site. Today at the X,Commerce Innovate Conference someone suggested to me that this is the first step for eBay Inc. to offer full cloud based user management for e-commerce sites and merchants. You can also see the official press release from eBay Inc. here.

Most of the press and coverage today focused on "Consumer Identity" - or more accurately Consumer Commercial Identity - and the benefit of PayPal Access for consumers and online merchants visited by those consumer. Consumer identity is indeed one facet of "Commercial identity" - but there is another side to commercial identity, a less understood - and arguably less sexy - side and that is Merchant Identity. What do I mean by this? Let's look at a scenario:

Merchants themselves are consumers of so many online and offline services (think of it as B2B services) - a company that sells on eBay - or any other online channel - has an eBay account, an account with a shipping company (FedEx), a Facebook account, perhaps another account with a email marketing service, bank account etc. Clearly merchants suffer from the same "account and password hell" that consumers do - but this hell is a lot deeper and hotter for merchants, consider these facts

- Most merchants have employees/contractors who create these accounts on behalf of the merchant,
- A lot of these employees (for smaller merchants) are part time or temps
- Employee turn over is high

Here in addition to the usual forgetting one's password - which for merchant leads to loss of productivity and money - sometimes the person who created the account simply leaves - if you are lucky and s/he good terms, you end up having to chase the employee and restore your access, if not, you are exposed to unauthorized access by the employee or down right "account take over".

You might say, what is the difference between this and consumer identity, these employee are consumers to and technically there is no difference. But look closer. Merchant use cases are fundamentally different. In consumer identity use cases, a consumer is a principle and gives consent on his/her own behalf to a agent (another site or application), the IDP itself recognizes the consumer is the principle and allows her (and ONLY her) to change or revoke this access. In Merchant cases, what appear to be the consumer is really not a principle binded to the merchant identity but an employee. In this case IDP must recognize this "hierarchical relationship" and allow and "admin" employee of merchant to monitor and manage the life cycle of tokens (and identities) of employees.

In the use case above, merchant X would not reveal its primary eBay user name and password to any employee, the would provision an account for each employee. Employee then logs into eBay using her own account - and via PayPalAccess - All the while PayPal Access monitors and manage all the tokens issued to all employees of merchant X. Should an employee leave or changes function, the token can be revoked by merchant X admin regardless of employee's decision.

If this sounds familiar to LDAP or ActiveDirectory, b/c it really serves the same function: Enterprise Identity, in this case enterprise is really a merchant. This is not unexpected in the world where enterprise identity, consumer identity (a.k.a social identity) are converging - and there is a need for a cloud based enterprise user management.

Please note that this is NOT an annoucement (or leak) for PayPal Access Cloud-Base user directory. IT IS NOT, REALLY. I just wanted to point out the there is two sides to commercial identity, a sexy side (consumer) and a side that can make you money (merchants).

In the next post, I will write a bit about Consumer Commercial Identity and how it may be different that social identity.

Magician vs. Engineer

Steve Jobs, the man, died a few days ago. Steve Jobs the symbol and the icon in all likelihood lives on for a long time. In this status he is joined perhaps only by one other man: Bill Gates (whether people agree or disagree with his business tactics, feel that MSFT produced low quality or hard to use software... no one can deny the fact that he was one of the first few who realized that software would be the key to pervasive computing, co-founded the first real software company and put software engineering as a profession on the map).

That is why I was so excited to watch them being interviewed on the same stage and at the same time @ D5 in 2007. When the news of Steve Jobs passing came out, I went back and watched it again and this time I found it simply fascinating - the session is about 1.5 hours, an hour the interview and 30 minutes Q&A.



It is long, but it is well worth the time.

After about 4.5 years, and with hindsight, it is so amazing to see Steve Jobs explaining his vision about "Post-PC" era and superiority of native applications, merits of integration of hardware and software, and in general, what we can only now recognize as a general description of  iPad. (with the notable absence of any reference to App Store)

Interestingly, Bill Gates, in response to what he sees as post PC era, talks, spot on, about tablet computing, significance of touch and the "convergence device".  Keep in mind that MSFT had been doing basic research in this field for a long time.

In my view, both men shared the same over all knowledge of trends and technologies in 2007, both knew that a device that is basically touch enabled and connected will dominate the future. Then why Apple was able to come up with iPad and MSFT ended up with a few tablet from bunch of manufacturers that only a few ever saw live in action, let alone use.

Amazingly, Bill Gates answers this question himself - in what I think is the most interesting exchange of the interview: a member of audience (a woman @ about 1:22 into the video) asks both Jobs and Gates "...What did you learn about running your own business that you wished you had thought of sooner or first by watching the other guy". Bill Gates volunteers an answer first and says:

"...he (Steve) has an intuitive taste for product and people, we sat in Mac product reviews and question would come up that I would view it as engineering question b/c that is how my mind works and I'd see Steve make a decision based on a sense of people and product that is even hard for me to explain, the way he does things are just different and I think it is magical and in that case WOW" (he never mentions what that case was)

There, ladies and gentlemen, you have it! This is why MSFT and Bill Gates, as great as he is, with all the knowledge of trends, technologies could never quiet come up with that "convergence device". They (or any one else) did not have that magical "intuitive taste for product and technology". As Bill Gates said it, it is even hard for him to explain let alone replicate that intuition.

A lot of things have been said and written about Steve Jobs greatness, but none captures and express the essence of what Jobs did better than the description given by the only other icon of modern computing Bill Gates: intuition vs.solution , soul vs. mechanics, empathy vs. sympathy ..Magic vs. Engineering.

Interviewing @ eBay Part V - Dir. and VP of Engineering

My base line for interviewing senior engineering management  is the following, non-scientific, completely made up, definition:

Job of a manager is to develop & allocate resources and manage execution of projects to satisfy time, budget and quality constraints and to minimize risks and promote efficiency (repeatability of the whole process). Leadership (as in the ability to inspire and influence change) is desirable, and indeed necessary the higher one goes in the management chain.

Based on this, there are five types of questions I’d normally ask,

·        General/ice breaker: One or two questions based on resume or general questions such as Why eBay? Why now

·        Technical Management : Ability to manage large teams, projects, timelines, budgets and plans

·        Leadership: ability to conceive , evangelize and cause positive change

·        Personal: I mean to assess personal integrity, awareness, reflection

·        Field Specific: Needless to say, a director of DB engineering gets specific DB questions, director of commerce must know details of order management and payment processing VP of personalization get collaborative filtering and VP of applications get “how you build a large web app” question.

 Here are the current (and expanding) bank of question I’d normally draw from, please email me or comment if you have other suggestions...

 -        General

o   First 90 days at eBay?

o   Why eBay, Why Now?
o   Talk about the most defining event in your professional life. 

o   What are the 2,3 interesting and promising trend you see?

o   What is leadership to you? What is management to you?

There are safe and conservative answers for these questions, however answers that reflect
measured risk taking and authenticity are always preferable. 

-         

 -        Tech Management

o   How would you structure an engineering org?

o   How do you measure the progress and success of a project?

o   How do you decide the allocation of engineering resources in multiple locations?

o   How do you manage promotion process?

o   How do you allocate bonus budget among your team members?

o   How do you see your relationship with product and architecture function?

o   How do you manage your hiring process? Who would you hire?

o   A new technology for part of your stack is emerging (e.g. a new presentation technology or a better and open sourced database, new JVM or cache …) would you replace your existing technology stack with the new one, why or why not?

o   How do make sure knowledge sharing is effective among your team members?

o   How do you ensure the quality of your delivery?

o   What is the most important job of a technology manager (pick one!), why?

o   How do you monitor the day to day tasks and assignment of your team?

o   What is your view about innovation? How do you practically manage an “innovative” team?

o   How do you deal with “NIH” – Not Invented Here – issue?

o   What is your talent development philosophy?

o   How do you chose and prepare your successor?

o   Would you direct your team to execute on a course of action you do not support?

o   How do you increase productivity? How do you measure it?
o   How do you empower your team to do "the right thing"? even when there is no time or budget for it?

    

-        Technology

o   How do you manage development life cycle? What development life cycle do you use?

o   What is the current technology stack you are using? What are the benefits? What are the drawbacks? Why it was chosen? (please don’t say “I don’t know, it was chosen before I join”)

o   How do you plan for migration from an old, stable large system to a newer version of the same system?

o   How do you feel about redundant work? is there an occasion that It may be useful?

o   When do you use open source? What are the challenges? When do you use vendors?

o   What is Agile to you? What are the benefits and challenges?

o   What 2,3 question you would like to ask about eBay technology?

o   Explain the CAP theorem.

o   What are the measures/steps you take when your system is in operation?

      The Person

o   How do you stay current of state of technology? What part of stack are you interested in the most?

o   How have you improved over the years as a manager?

o   What is your proudest moment as a manager?

o   Tell me about your biggest mistake, how did you realize it was a mistake? What did you do afterwards?

o   What is one criticism that your subordinates make about you?      

o   How can twitter be used to improve eBay?

o   How do you improve eBay? Now, I just told you to do something else – that you feel is not right - , how do you react?

o   What is your dream company to work for? Imagine now that you have an offer from that company, what should eBay do so that you work for eBay instead?

-         

  -        Leadership

o   How do you deal with a “failing” project? Or a project in crisis?

o   Your plan requires the cooperation of another team, but that team has its own priorities and plans, how do you convince them to allocate time and resource to your project?

o   How do you influence and convince a group of people over whom you have no authority? Give an example

o    How do you mentor/coach your team member?

o   You receive a call at 2am telling you that the entire search (or checkout) is down, what do you do?

o   Two senior technical leaders (or teams) escalate a technical difference to you (MemCache v.s NoSQL or doing it now vs. doing it in the future …) how do you settle the matter?