What is CommerceOS - Part I - Introduction

I am often ask what is CommerceOS, equally as often I answer based on what a successful end state for CommerceOS looks like: It is a set of capabilities exposed largely as RESTful APIs that allows anyone to rapidly, cost effectively and securely develop applications that facilitate commerce between or among real or legal entities, on any platform, device or language, globally.

This means if you want to develop an application for a consignment store to sell its merchandize online (on or off eBay) CommerceOs makes it easy for you. If you work for a large multi-national corporation and want to list your products on eBay and on your site, then let local, regional and international buyers purchase your product, CommerceOS is your APIs. If you want to write a pop-up app on iPhone and Android that showcases adventure riding gears and products by mashing up content and pictures with commerce and community, CommerceOS should make your job easier and cheaper, and of course, if you are an eBay mobile developer, CommerceOS APIs enables you to build your applications ...and countless other scenarios like these, you got the picture.

This answer, although accurate from utility and use point of view, does not describe CommerceOS from technical and architectural point of view. A casual observer may not appreciate the technical, process and organizational complexities that should be overcome to achieve CommerceOS vision. In this series of posts I try to explain CommerceOS from technical aspect.

The rest of this post dive a bit deeper into motivations and problem definitions, in the second part,  I will focus on how we have approached the problem and architectural details of CommerceOS. Part III explains set of technical standards and patterns used by all CommerceOS services and consumers, and are the basis of interoperability of the entire system. Part IV is a review of a few core services of CommerceOS called "Shared Services". Finally Part V deals with process we use to maintain and enhance the platform.

First a bit of a context. Back in 2009-2010 timeframe (where CommerceOS transformation started) eBay had a global engineering workforce in the range of thousands (as it does today). The collective job of this engineering team was to work on an essentially web-based architecture (like so many other web era companies). This architecture assumed (perhaps more implicitly than explicitly) that

• Buyer interacted with marketplace largely via a web browser, running on a desktop/laptop.
• Sellers were small(er) and interacted either via web UI or APIs designed to replicate Web UI - essentially APIs were designed for sellers.
• API code base and the business logic powering web UI were largely two independent code basis.
• Data and table were shared among all applications and services. Systems were single tenant and monolithic from identity and tenancy point of view. 
• Both data and code for all subsystems/capability (such Identity, Checkout or Payment services, Search) was hard to isolate and package independently.
• Operation technology was fixed capacity, manually provisioned, under-utilized.

The platform was stable and scalable (albit with scale-cost characteristic by, then, current state of technology - fixed capacity, built for burst) with its own set of processes that made it possible for global teams to get their job done. However, in 2009-2010, it was very clear that we are a few years into development of trends that would re-shaped (or ram through, depending on your point of view) traditional web-based architecture and require a different scale-cost and productivity trajectories. These trends were:

1- The ever present mandate for efficiency and cost control driven by the need to expand or at least preserve margins or reallocate resource. This is natural trend for any market and segment as they become more mature and scale.
2- The mobile and connected device revolution which was shifting more and more of traffic to non-web, non-PC devices.
3- The advances in cloud computing that drove the operation cost down, but demanded functionality to be packaged cleanly and independently to be deployed wherever capacity was available.
4- The shift of eBay merchant mix from smaller, more casual, lower volume sellers/merchant to larger, more architectural and operationally formal and higher volume sellers.

(I will talk about how technologies known as Big Data changed and impacted eBay architecture in a different post)

It was very clear that the forces of the four factors above require a different architectural approach from what, then, current web-oriented architecture was. We needed to bring down the cost of both infrastructure and operation as well as increase the productivity of our developers to address #1 above.

To address the multi-screen revolution (#2), we needed more than just a set of APIs (which eBay has had since 2003) - eBay marketplaces needed all screens (included desktop) to consume one set of APIs not two sets of code-bases one power the "main" web UI and the other to power all API/devices. This is a much harder problem than simply developing APIs. Additionally, we needed all APIs, regardless of which one of nearly 100 teams across the globe develop it, to look like a coherent portfolio of APIs (given the laws of thermodynamics, you can guess how tough this is).

To take advantage of promises of cloud computing (elasticity of capacity, better resource utilization, related cost saving etc.) we needed our business logic and data to be encapsulated in well-defined and isolated modules, with clear dependencies that lend themselves to packaging suitable for cloud environment. Refactoring code to isolate systems and services is tough enough, but separating shared data and isolating storages and tables are an order of magnitude more challenging.

And finally, the larger, more sophisticated our merchants and partners became, the more we needed to use formal integration methodologies and feed-based interactions built on de-facto industry standard models of Order, Product, Inventory, Cart, Returns and less on auction-centered models such as Item, transaction etc. This represented a deep change in entity and domain models and re-architecture of code and, perhaps more importantly, migration from old models to new models.

Given where we were and where we wanted to evolve marketplace architecture, we came up with a definition of a vision for CommerceOS as the basis for alignment and execution, knowing full well that no one paragraph accurately covers all aspect of this initiative. Having said that, here is the definition we came up with:

CommerceOS is eBay MP initiative to transform its architecture in such a way that large majority of its platform capabilities and business processes are exposed as RESTful APIs in the cloud in such a way that all marketplace participants can consume them, uniformly, securely, effectively and with high quality.

This may sound a bit abstract, but let me parse it:

The primary goal is to "transform architecture" - this means both technology stack (for services and applications) and processes not just technology (it never works that way).

Platform capabilities and business processes: That basically means everything that MP does from identity, verification of attributes, and caching to listing, pricing, order management and search need to be re-designed as RESTFul services and exposed to correct consumers. The modifier "Majority" indicates that some interactions are not strictly RESTful for legacy or integration requirements. We are pragmatic about it.

Our services are RESTful, it is a de-facto standard and we developed a few internal spec to uniformly covers aspects such as use of non-http verbs, security, tracking, internationalization, filtering, constraints etc.

The deployment target for services is "Cloud". This is not a feel good or casual term, it strictly mean that a service must be produced in a way that it can be packaged fully isolated from other services with well known dependency, clear version number. This allow us to deal with a unit of capability as a building block and use technologies such as Docker effectively (we had a few de-tour here with OSGi but course corrected).

"All participants" indicates that the consumers will be applications written not only for buyers and small sellers who interact with marketplace via a browser, but everyone else on all devices including large sellers who never interact via ebay.com, CS and their internal tools, partners such as logistical service providers as well as internal eBay staff and, of course, eBay application on all devices.

Then there is a set of non-functional requirements, each key to how APIs are built, they include

Uniform: basically one code base supporting all consumer, with no "primary" web consumer.
Secure: self-explanatory, platform MUST be secure with well defined identity, access management and auditing. Secure also include availability.
Efficient: APIs must be efficient to consume, i.e. application developer must spent min amount of time and effort to consume it (docs, samples and sandboxes), APIs also should be cost effective to operate (see cloud requirement)
High Quality: An umbrella term covering functional quality (no-bugs) as well as performance.

CommerceOS started with this vision definition and context, in the next post, I will describe the architecture and structure of CommerceOS, both as a technology platform and from organization/process point of view.