The Uncommon Security Common Sense

I can not claim that I actually counted or classified all the reasons peoples cite for not taking security (or for that matter sound and well thought through system design) seriously from the start, but the three following lines seems to be the most common ones:

1- The "it is too contained" line: So what is the big deal? at worst it may affect a very small percentage of my users.
2- The "it is too early" line: Oh my system/site/project is too small and we only have a few users, we really don't have time/resources for this.
3- The "it is too small" line: My project is too small or too obscure for anyone to care.

By the way, I have heard these lines or their equivalent not only when it comes to security engineering (or re-engineering) but also in designing business policies or risk management measure to prevent fraud, or in general negative user experiences as well as general system design.

Now to be fair, these reasons all sound like "common sense", after all why would you take on additional cost and time for your project or accept the expense and risk of re-engineering your code to fix an issue that may only affect 1% or 0.01% of your users? or why should you spend two weeks to fortify a system that takes you 3 days to design and it is "just an experiment"? and finally who really cares about a small project some where with some obscure URLs that takes an email address as one of its inputs and shows some useful error message if the email is not registered? does ANYONE really care?

Well, as it turns out, security common sense (like many other form of common sense) is actually quite uncommon ! Let's look at these frequently cited common sense logic a bit closer.

To demonstrate the fallacy behind the first logic (it is too contained, it only affect 0.01% of users) I cannot think of any better illustration than the words of presidential candidate Herman Cain where he said that "for each woman who has accused him of harassment there are probably thousands who haven't" and he is 100% accurate and right! But does that make any difference? In all likelihood his presidential bid is all but over. Or could the Washington D.C police chief during the "D.C Sniper Attacks" have possibly argued that the whole thing was not a big deal b/c only 0.001% of D.C metro population were actually killed and therefore there is no need for massive mobilization of police, FBI, ATF and even secret service !?

The same math is thru for security, it does not matter if only 1000 users out of 10MM become victim of a
poorly secured or design system. What matters is how many people hear and learn about it - and you can be
sure that at least in this day and age that number is a few order of magnitude larger than the actual number of
victims. The sense of insecurity that this causes in the rest of the user community and its economic cost is the real math that matters not the fact that only 1000/10MM=0.01% users were affected.

The second line "it is too early" or its equivalents "we don't have enough time or resources" is the most commons line not only in security matters but also system design and architecture aspects as well. What is interesting here is that the exact premise cited for not focusing on security (or sound design for that matter), is why security should be taken seriously i.e. "I am too new to afford not to be secure", if you are releasing a new product (or brand or a site) you REALLY DO NOT HAVE A SECOND CHANCE TO MAKE A FIRST IMPRESSION. If you are not secure, or if your first few user gets taken advantage off (think of AirBnB incident) you are doomed. To further demonstrate the risk in this argument I submit the following picture of one of the more famous car design mistakes : Honda Odyssey 1998

Honda designed this in a hurry to get into the growing minivan market dominated by Dodge/Chrysler. They decided to differentiate by replacing a convenient power sliding door with a traditional door! Imagine what would have happened of this was a new no-name company without Honda's established brand? Of course Honda corrected the mistake in 1999 model and beyond and went on to have one of the most successful Minivans. But if you are not Honda, you better spend time and money on designers and marketers to tell you,  in the first try, that whoever buys a minivan *needs* a sliding door.

Now we get to the third line "Who really cares about me?" I have to admit that I have the most sympathy with people who resort to this logic. After all it is tough to imagine how capable and resourceful the modern fraudester/hacker community is without actually having a brush with them. I do not get into the details -  if you are interested you can briefly scan Rick Howard's excellent book "Cyber Fraud, tactics, Techniques and Procedure" - for the purpose of this writing I'd suggest you assume the following is true:

In the game of "Who wants to break into my system" your adversary is more motivated (financially or politically) than you are, more experienced than you are, is more innovative than you are, is more nimble than you are, wants it worst than you do, has a smaller cost base than you do (and therefore) all he needs is 0.01% (or smaller) of your users - the ONLY advantage that you have is that you right the rule of the game. Do not give up that advantage easily. You WILL lose the game.

Btw, the end point URL that takes an email and very nicely checks and display an error if email does not belong to a valid user - was actually found (although it was a little obscure URL not linked to from anywhere -  and used to extract valid company X user emails (cost $5000+) from a large list of non-verified harvested emails (cost $50) - a vital part of phishing industry value chain.

OIX Attribute Exchange Summit - Washington DC

Open Identity Exchange (OIX) is holding this year's Attribute Exchange Summit in Washington, DC.

Identity attributes are core of the concept of digital identity. As federated identity ecosystem getting more mature and adoption grows among more sophisticated RPs  - with more consequential use cases such as government, health, education, commerce ... - so does the need for wider sets of attributes with more accurate and fresh values. This presents both tough challenges and opportunities for IDPs.

The challenges center, as one may expect:, around aggregating, correlating, transform and maintaining fresh copy of attributes in a cost effective manner and in a way that it does not compromise the privacy (and other rights) of the principle owner. IDPs can differentiate based on the range of attributes they provide in this way and there in lies the opportunities.

I will be talking more about identity attributes, their life cycle, uses cases and how they help establish and elevate trust among parties to commercial transactions (online and off-line) as part of a panel with Don Thibeau, OIX/OIDF chairman and Abbie Barbir, VP BoA.

If you are planning to attend, I'd be happy to hear from you.

OAuth vs. OpenID Connect ?

OpenID Connet 1.0 Spec is finally released (actually it was release back in Aug). Its release was accompanied by two predictable categories of questions/sentiments, one not very well informed and the other one a legitimate question:

-        OpenID is dead

-        OpenID Connect is really OAuth so why do we need a new protocol?

Granted, this is normally coming from software engineers and social application programmer community and not from identity community, but I feel they are significant enough to be addressed, especially at the time that more and more entities contemplating to become identity providers and they need to decide which protocol they should implement.

First, on the demise of “OpenId”:  It is true that the earlier versions of Open ID (version 1 and version 2) are, for all intent and purposes, depreciated and will not gain a whole lot of traction. But the general idea of “Open” standards for communicating between RPs and IDPs that enables users to provision fewer accounts and have a portable identity while still maintaining control over their privacy and data is alive and well and actually is even more vital than before.

Second, on relationship between OAuth and OpenID Connect, OAuth is a general protocol for authorizing an agent to access a resource on behalf of resource’s owner. OAuth does not assume any particular knowledge about the resource itself. What does this mean? Let’s go back to the canonical OAuth use case of a user who would like to authorize a printing services to access her photos from a Photo service provider. Now imagine that the photo service is slightly sophisticated and recognizes a few properties associated with photos e.g. resolution, size, whether they are shots with no humans, and if there shots with humans, who appears in the photos – basically let’s assume the resource served by SP has more semantics that simple “access”.

Now imagine that the user wants to grant access to only JPEG photos of himself and not a full access to all photos. How would the IDP encode this semantics in the authorization request and response? How would the SP know that they should only provide access to a subset of images?

To be sure, this is doable using OAuth, but the implementer has to add additional parameters to request and response or possibly constraint the input values of some other parameters.

A protocol that is built this way to access a specialized resource, would be a photo access protocol built on top of OAuth.

In essence this is exactly what OpenID Connect it: It is a protocol built on top of OAuth that supports features that are often desired and used when the resources being delegated is “identity” and attribute about an identity.

To illustrate the point, here are what we, at eBay, had to do for an internal authentication protocol on top of OAuth:

-  

      Force Authentication: adding parameters to authorization request to force users to authenticate no matter what the authentication state with IDP is

-        Authorization Behavior: adding parameters to authorization request to indicate to IDPs whether it should display the consent page and how to display the login page (overlay, full page)  

-        Standard Claim Set: Defining the default set of attributes returned by IDP

-        Requested Attributes: adding mechanism to allow RPs to ask for additional attributes, and annotating them to indicate whether explicit user consent is required.

-        Authentication Context: adding a fragment to response to communicated authentication context (single v.s multi-factor, PIN vs. Password, number of retires etc.)

-        Protection: adding parameters to indicate how access tokens should be protected (encryption, signature and order of operations)

-        Token Validation end point: adding an endpoint to introspect access tokens on demand.

These are all features and facets that OpenID Connect enables in a standard and interoperable fashion. In absence of a standard such as OpenID Connect though, any RPs integrating with our IDP had to implement basically a proprietary protocol, be it on top of OAuth.

The point is that if you want to operate an IDP and you want to use just OAuth, you have to add a few things to OAuth, depend on the depth of your requirements, to make it work for “Identity” resource. This is exactly what Facebook did with FB Connect – and they also did a good job of wrapping it with JavaScript plug-ins. The goal of OpenID Connect is to use OAuth as the basic access authorization protocol and add identity specific features to it so that it becomes a standard “identity protocol” that can enable seamless interoperability. 

PayPal Access & Commercial Identity

Today eBay Inc. announced an identity and attribute provider product called PayPal Access. Some described as a "Facebook Connect for Commerce", others described it as an easy registration tool for mobile site. Today at the X,Commerce Innovate Conference someone suggested to me that this is the first step for eBay Inc. to offer full cloud based user management for e-commerce sites and merchants. You can also see the official press release from eBay Inc. here.

Most of the press and coverage today focused on "Consumer Identity" - or more accurately Consumer Commercial Identity - and the benefit of PayPal Access for consumers and online merchants visited by those consumer. Consumer identity is indeed one facet of "Commercial identity" - but there is another side to commercial identity, a less understood - and arguably less sexy - side and that is Merchant Identity. What do I mean by this? Let's look at a scenario:

Merchants themselves are consumers of so many online and offline services (think of it as B2B services) - a company that sells on eBay - or any other online channel - has an eBay account, an account with a shipping company (FedEx), a Facebook account, perhaps another account with a email marketing service, bank account etc. Clearly merchants suffer from the same "account and password hell" that consumers do - but this hell is a lot deeper and hotter for merchants, consider these facts

- Most merchants have employees/contractors who create these accounts on behalf of the merchant,
- A lot of these employees (for smaller merchants) are part time or temps
- Employee turn over is high

Here in addition to the usual forgetting one's password - which for merchant leads to loss of productivity and money - sometimes the person who created the account simply leaves - if you are lucky and s/he good terms, you end up having to chase the employee and restore your access, if not, you are exposed to unauthorized access by the employee or down right "account take over".

You might say, what is the difference between this and consumer identity, these employee are consumers to and technically there is no difference. But look closer. Merchant use cases are fundamentally different. In consumer identity use cases, a consumer is a principle and gives consent on his/her own behalf to a agent (another site or application), the IDP itself recognizes the consumer is the principle and allows her (and ONLY her) to change or revoke this access. In Merchant cases, what appear to be the consumer is really not a principle binded to the merchant identity but an employee. In this case IDP must recognize this "hierarchical relationship" and allow and "admin" employee of merchant to monitor and manage the life cycle of tokens (and identities) of employees.

In the use case above, merchant X would not reveal its primary eBay user name and password to any employee, the would provision an account for each employee. Employee then logs into eBay using her own account - and via PayPalAccess - All the while PayPal Access monitors and manage all the tokens issued to all employees of merchant X. Should an employee leave or changes function, the token can be revoked by merchant X admin regardless of employee's decision.

If this sounds familiar to LDAP or ActiveDirectory, b/c it really serves the same function: Enterprise Identity, in this case enterprise is really a merchant. This is not unexpected in the world where enterprise identity, consumer identity (a.k.a social identity) are converging - and there is a need for a cloud based enterprise user management.

Please note that this is NOT an annoucement (or leak) for PayPal Access Cloud-Base user directory. IT IS NOT, REALLY. I just wanted to point out the there is two sides to commercial identity, a sexy side (consumer) and a side that can make you money (merchants).

In the next post, I will write a bit about Consumer Commercial Identity and how it may be different that social identity.

Magician vs. Engineer

Steve Jobs, the man, died a few days ago. Steve Jobs the symbol and the icon in all likelihood lives on for a long time. In this status he is joined perhaps only by one other man: Bill Gates (whether people agree or disagree with his business tactics, feel that MSFT produced low quality or hard to use software... no one can deny the fact that he was one of the first few who realized that software would be the key to pervasive computing, co-founded the first real software company and put software engineering as a profession on the map).

That is why I was so excited to watch them being interviewed on the same stage and at the same time @ D5 in 2007. When the news of Steve Jobs passing came out, I went back and watched it again and this time I found it simply fascinating - the session is about 1.5 hours, an hour the interview and 30 minutes Q&A.



It is long, but it is well worth the time.

After about 4.5 years, and with hindsight, it is so amazing to see Steve Jobs explaining his vision about "Post-PC" era and superiority of native applications, merits of integration of hardware and software, and in general, what we can only now recognize as a general description of  iPad. (with the notable absence of any reference to App Store)

Interestingly, Bill Gates, in response to what he sees as post PC era, talks, spot on, about tablet computing, significance of touch and the "convergence device".  Keep in mind that MSFT had been doing basic research in this field for a long time.

In my view, both men shared the same over all knowledge of trends and technologies in 2007, both knew that a device that is basically touch enabled and connected will dominate the future. Then why Apple was able to come up with iPad and MSFT ended up with a few tablet from bunch of manufacturers that only a few ever saw live in action, let alone use.

Amazingly, Bill Gates answers this question himself - in what I think is the most interesting exchange of the interview: a member of audience (a woman @ about 1:22 into the video) asks both Jobs and Gates "...What did you learn about running your own business that you wished you had thought of sooner or first by watching the other guy". Bill Gates volunteers an answer first and says:

"...he (Steve) has an intuitive taste for product and people, we sat in Mac product reviews and question would come up that I would view it as engineering question b/c that is how my mind works and I'd see Steve make a decision based on a sense of people and product that is even hard for me to explain, the way he does things are just different and I think it is magical and in that case WOW" (he never mentions what that case was)

There, ladies and gentlemen, you have it! This is why MSFT and Bill Gates, as great as he is, with all the knowledge of trends, technologies could never quiet come up with that "convergence device". They (or any one else) did not have that magical "intuitive taste for product and technology". As Bill Gates said it, it is even hard for him to explain let alone replicate that intuition.

A lot of things have been said and written about Steve Jobs greatness, but none captures and express the essence of what Jobs did better than the description given by the only other icon of modern computing Bill Gates: intuition vs.solution , soul vs. mechanics, empathy vs. sympathy ..Magic vs. Engineering.

Interviewing @ eBay Part V - Dir. and VP of Engineering

My base line for interviewing senior engineering management  is the following, non-scientific, completely made up, definition:

Job of a manager is to develop & allocate resources and manage execution of projects to satisfy time, budget and quality constraints and to minimize risks and promote efficiency (repeatability of the whole process). Leadership (as in the ability to inspire and influence change) is desirable, and indeed necessary the higher one goes in the management chain.

Based on this, there are five types of questions I’d normally ask,

·        General/ice breaker: One or two questions based on resume or general questions such as Why eBay? Why now

·        Technical Management : Ability to manage large teams, projects, timelines, budgets and plans

·        Leadership: ability to conceive , evangelize and cause positive change

·        Personal: I mean to assess personal integrity, awareness, reflection

·        Field Specific: Needless to say, a director of DB engineering gets specific DB questions, director of commerce must know details of order management and payment processing VP of personalization get collaborative filtering and VP of applications get “how you build a large web app” question.

 Here are the current (and expanding) bank of question I’d normally draw from, please email me or comment if you have other suggestions...

 -        General

o   First 90 days at eBay?

o   Why eBay, Why Now?
o   Talk about the most defining event in your professional life. 

o   What are the 2,3 interesting and promising trend you see?

o   What is leadership to you? What is management to you?

There are safe and conservative answers for these questions, however answers that reflect
measured risk taking and authenticity are always preferable. 

-         

 -        Tech Management

o   How would you structure an engineering org?

o   How do you measure the progress and success of a project?

o   How do you decide the allocation of engineering resources in multiple locations?

o   How do you manage promotion process?

o   How do you allocate bonus budget among your team members?

o   How do you see your relationship with product and architecture function?

o   How do you manage your hiring process? Who would you hire?

o   A new technology for part of your stack is emerging (e.g. a new presentation technology or a better and open sourced database, new JVM or cache …) would you replace your existing technology stack with the new one, why or why not?

o   How do make sure knowledge sharing is effective among your team members?

o   How do you ensure the quality of your delivery?

o   What is the most important job of a technology manager (pick one!), why?

o   How do you monitor the day to day tasks and assignment of your team?

o   What is your view about innovation? How do you practically manage an “innovative” team?

o   How do you deal with “NIH” – Not Invented Here – issue?

o   What is your talent development philosophy?

o   How do you chose and prepare your successor?

o   Would you direct your team to execute on a course of action you do not support?

o   How do you increase productivity? How do you measure it?
o   How do you empower your team to do "the right thing"? even when there is no time or budget for it?

    

-        Technology

o   How do you manage development life cycle? What development life cycle do you use?

o   What is the current technology stack you are using? What are the benefits? What are the drawbacks? Why it was chosen? (please don’t say “I don’t know, it was chosen before I join”)

o   How do you plan for migration from an old, stable large system to a newer version of the same system?

o   How do you feel about redundant work? is there an occasion that It may be useful?

o   When do you use open source? What are the challenges? When do you use vendors?

o   What is Agile to you? What are the benefits and challenges?

o   What 2,3 question you would like to ask about eBay technology?

o   Explain the CAP theorem.

o   What are the measures/steps you take when your system is in operation?

      The Person

o   How do you stay current of state of technology? What part of stack are you interested in the most?

o   How have you improved over the years as a manager?

o   What is your proudest moment as a manager?

o   Tell me about your biggest mistake, how did you realize it was a mistake? What did you do afterwards?

o   What is one criticism that your subordinates make about you?      

o   How can twitter be used to improve eBay?

o   How do you improve eBay? Now, I just told you to do something else – that you feel is not right - , how do you react?

o   What is your dream company to work for? Imagine now that you have an offer from that company, what should eBay do so that you work for eBay instead?

-         

  -        Leadership

o   How do you deal with a “failing” project? Or a project in crisis?

o   Your plan requires the cooperation of another team, but that team has its own priorities and plans, how do you convince them to allocate time and resource to your project?

o   How do you influence and convince a group of people over whom you have no authority? Give an example

o    How do you mentor/coach your team member?

o   You receive a call at 2am telling you that the entire search (or checkout) is down, what do you do?

o   Two senior technical leaders (or teams) escalate a technical difference to you (MemCache v.s NoSQL or doing it now vs. doing it in the future …) how do you settle the matter?

OpenID Tech Summit - Mountain view, CA - 9/12-13

I am attending the OpenID Tech summit tomorrow (Monday) and Tuesday at the MSFT Silicon Valley campus.

There are two main topics, first the official announcement of  OpenID Connect - a standard built on top of OAuth 2.0 to that allows RPs obtain connect and obtain extensible profile information about an identity -  and second is the introduction of a concept called Account Chooser - a UX pattern for federated login pages proposed based the experience of Google in dealing with federated authentication scenarios.

I am also part of a panel discussion on "Identity Schizophrenia - How users want to apply their online identities" moderated by Allen Tom, OIDF Board Member. It is scheduled for Tuesday September 13 @ 1:40pm. For a full schedule of the summit see here.


It should be interesting ... If you there tomorrow, please do stop by and say hi ...

Interviewing @ eBay Part IV - Product Management Interview

First let’s review, very briefly, what product managers are expected to do – I also highly recommend you read the following Q&A from Quora contributors:

What are your top 3-5 books or resources for tech product managers?

What are the best books for Product Managers

In my view the core of product management role is to understand firm’s resources and capabilities, existing products and markets, customer needs and wants, existing and adjacent market dynamics and economics and use the intersection of these four factors to conceive of and design new products or improve and evolve the existing ones in a manner that is profitable for the firm - i.e. reduces costs or increases revenue. 

That is indeed a tall order, and rarely can be performed by one person – it is a role – but an individual product manager should perform any part of this role.

(Notice that we are not talking about product marketing manager, project manager or program manager, the focus is only product management as it is defined above)

In addition to – or to accomplish - this core function– product managers at eBay work closely with business to understand the markets and trends, participate in conception of ideas, communicate and get buy-in from all stake holder (formal, actual or both), help marshal resources and come up with execution planning and ensure the success roll out – and post roll and operations out activities.

To me, the core traits of a product manager are clear and analytical thinking ability, communication and influence and discipline (in capturing assumptions, solutions, exemptions, follow ups, coordination as required by the breadth of activities above etc.)

In a typical interview (45 min) you can expect 4,5 questions from below list:  

-        Technical questions:

Most eBay products are either technology-based or have a strong technology component to them, so you have to understand technology (as in software engineering, operations, statistics …) also you need to build credibility with engineers, for these two reasons expect a few technical questions, I don’t personally ask you to code, unless you volunteer to (a plus) or you state on your resume that you are “fluent in Java”, then I consider it a fair game. By the way if you are applying for a “Technical PM” please read the “architect” interview post. These two jobs are almost the same at eBay.

o   What is the general architecture of a web application, how about a mobile application?

o   Your product has to use service of a service provider – the service is available online – what are the list of question you would like to know about this service provider?

-        Analytical ability

o   What is BMW’s revenue (do not look it up, I actually change the company randomly)

o   In a marketplace the actual instance of fraud is decreasing, but the perception of fraud is increasing, what is going on?

o   What data/information you’d like to know in order to estimate eBay revenue

-        Business and Strategy

o   Typical management consultant questions on strategy, competition, profitability, new markets etc.      

o   How do you grow eBay revenue by 20% in one year?

o   What adjacent markets eBay should consider entering?

o   Should eBay expand into Japan?

o   Should eBay buy ETSY?

o   Should eBay buy Yelp?

-        Product Design

o   How do you improve eBay buying experience, how about selling experience?

o   Should eBay accept Facebook Identity, if so, what are the considerations?

o   What do you think about “Social Commerce”, hype or real?

o   How do you incentivize excellent selling behavior on eBay?

o   How would you plan the launch of a product? Say fashion vault in Germany or integration of a new shipping carrier into the system.

o   How should eBay verify and confirms the identities of all sellers and buyers?

o   How best you think eBay can combine e-commerce and offline commerce?

o   What are the risks an electronic marketplace faces?

o   How should eBay implement “calendar of event” feature for sellers?

o   How do you improve eBay feedback system?

o    You are asked to improve eBay registration performance, what would you do?

o    What set of metrics would you use to measure the health of a marketplace?

-        Awareness of markets and trends 

o   Which companies eBay marketplaces should acquire

o   What are the eBay main competitors and why?

o   What trends (technology, consumers, economical, social etc.) will impact eBay business and how

o   Describe the economy of electronic payment industry 

o     What are your favorite products and why (please be prepared to mention something other than iPad or iPod)

o   What is the biggest product blunder in your mind and why

o   Which web sites do you visit regularly?

-        E-Commerce and Payment

o   How do you design a multi-merchant shopping cart

o   What is the “e-commerce funnel” – how do you optimize it.

o   How should “best match” algorithm be designed?

o   What are the risks an electronic marketplace faces?

o   You meet an eBay seller that complains about low sales volume, what would recommend him to do?

o   How do you measure the success of a shopping cart?

o   How can you use one's FB and Twitter accounts to improve searches on eBay?

o  How would you design an effective refund experience, how do you measure its effectiveness?

-        Personal qualities and fit  

o   How do you influence people?

o   What is leadership to you? Give me an example where you demonstrated leadership

o   Tell me about the most interesting project you worked on in your career

o   Suppose a technical leader is telling you that your product requirements is not implementable, what do you do?

o   How do you ensure that your product idea/project get priority over competing ideas/products?

Of course, the list of question changes from time to time and you may not get the same exact
question, but this is the general flavor of your interview. Again, if you happen to see this post
before you interview, please let me know.

Interviewing @ eBay Part III - Software Architecture Interview

I don’t know of any job title/role in technology that is more controversial, and evoke more emotional reaction, than that of an “architect”. Engineer, engineering manager, product manager, accountant, business developer etc. all have almost the same definition/responsibilities from company to company,  architects role though vary widely: in some firms one cannot do anything without an architect permission and in some others the role is completely eliminated.

You should first know that architecture is a role with a wide definition (TOGAF alone defines five types of architect - enterprise, business, data, application, IT). EBay architects play a combination of tech lead, internal evangelist, tech management and product management, and role is often the agent of change for eBay technical direction, tech stack, technology choices, process and methodologies …

Interviewing and selecting an architect is especially challenging. In addition to core skills of a software engineer (yes if you are interviewing for an architect position, you should be comfortable coding – no Java guru, but be able to code), the main attributes I am looking for are:

-        Integrity: Change in technology often brings about change in organization and power structure, people currently in power know this and may not be enthusiastic about  it, architect should have the integrity and courage to call for change when  it is not popular.

-        Leadership: integrity and courage is necessary but not sufficient, in this role you should have leadership i.e. the ability to influence, inspire and induce change in direction (often major changes) in a way that people want to make the change, not forced to (you will have no formal power anyway)

-        Clarity : last but not least, architects MUST bring clarity to situations where goals are unclear, definition of problem is fuzzy, needs are uncertain, data is incomplete, assumptions are inaccurate, yet delivery is urgent and pressure is high …bringing clarity to all aspects of such situations are often the most important function of an architect at eBay.

So for interview, expect some of the core software engineering questions, with much more emphasis on modeling and problem solving plus few of the followings:

• -        When you are asked to “architect” a system – say photo album app – what does that mean to you? What tasks do you perform? What would be your deliverables? How would you interact with engineers?
• -        How do you ensure the delivered system conforms to your architecture?
• -        Model and Design eBay
• -        From the time you type in www.ebay.com , to when you see eBay home page, explain what happens under the hood, at all layers
• -        How does Ajax-style interaction impact a traditional/classical page-oriented architecture? What are the changes it would force to the classic architecture.
• -        How would proliferation of Mobile application impact the classical web based architecture?
• -        Explain Map/Reduce in simple but reasonably accurate term, in a way a marketing person can appreciate it.
• -        Describe challenges and best practices in developing a distributed system – such as SOA based system.
• -        Describe the qualities of a well-designed API or service interface .
• -        Describe your favorite application development framework or design, explain its benefits and shortcomings  (e.g. Spring or Struts, or your own framework)
• -        Compare and contrast SQL and NoSQL DBs, when do you use each?
• -        How do you store a social graph like LinkedIn or Facebook?
• -        How do you decide to buy or build a piece of technology?
• -        eBay, as other online merchants and markets, has a policy against sale of fire arms, how do you design a system to enforce this policy?
• -        How do you design an application – such as a cart or check out flow - in a way that product and UI folks can experiment with and optimized different aspect of it?
• -        At any given time, eBay support a set of widely used browsers, for the rest, it display a warning message and asks users to upgrade to another browser. How so you design this system?
• In a large and distributed system, how do you ensure data-consistency for critical functions such as  authentication/login 
• Discuss a few significant technology trends, why do you think they are important? How would you anticipate their impact on current architecture/system?
• What would you do in your first month of working for eBay

If some of the questions sound vague, it is because they are! (btw, they are a lot clearer than what you'd face with in reality). Remember that you need ask questions, seek and bring clarity to the problem definition before you jump into the solution.

Again if you are interviewing for a particular specialty such as Security, I18N, Messaging, Operations etc. you should expect particular question in those areas (I will post a list of question for my security and identity architecture interview later), but for system and application architecture, be prepared for at least 3 or 4 questions from the list above. 

Interviewing @ eBay Part II - Software Engineering Interview

I am writing this hoping that candidate interviewing with eBay find this BEFORE their interview (if you did please let me or your interviewer know), but if you are not interviewing with eBay you may still find it useful.

Be prepared for the following category of questions:

-        

Explain a project or a problem you worked on

Be prepared to talk in some detail about an interesting, or challenging, important and otherwise mentionable project in your career. Even if you are fresh out of school, there must have been some special class or final project. You should clearly talk about the problem, and describe how you arrived at the solution, your implementation and result …what you learned, where/if you failed, how you fixed it etc. The way you communicate, what you choose to communicate is almost as important as what the project actually was. So be direct, clear and to the point. Be prepared to defend your choices. Do not play it “safe” by saying “..well this was not my decision”, or “Oh..I didn’t like this approach, my boss asked me to do this” etc.

-        Data Structure and Algorithm

Brush up on basic computer science, know data structure. You will be asked a few questions about graphs, hash maps, trees and complexity. Familiarize yourself with how graphs are represented in memory and in persistent storage, how maps and hash maps are implemented, how to traverse trees – basic stuff. Please DO NOT trivialize the questions by saying “oh, there is APIs for this in Java, I never need this in real life”. This, in addition to demonstrating bad judgment and wrong situation assessment, does not get you off the hook!  You should demonstrate that you are an engineer not a technician.  

-        Programming Language

Well, you are applying for a software engineering position, so you must know one or more modern programming languages very well (at least much better than I do) – eBay is a Java shop, so knowing Java really helps.  You should be proficient in basics (variables storage classes, access modifiers, memory management, basic object orientation) as well as advanced features such as multi-threading and concurrency, generics, network programming etc.

You will be asked to write code or code snippets on the white board. I am amazed how many people are surprised when they are asked, for example, to write a simple singleton class on the board. It shows how comfortable you are with coding, syntax does not matter at all, so don’t be shy.

-        Basic Modeling  

You are expected to know how to model basic stuff for example a simple book store, or an email client. You should be able to break it up to basic entities and their relationships. Sometimes I ask candidates to model “eBay Marketplace”, do not be overwhelmed, you are not expected to model entire eBay, do as much as you can and do “out loud” thinking. You approach is as important as final design.

-        Problem Solving

We are engineers, we solve problems. So you have to be able to frame and analyze problems, recognize tradeoffs in different solution and pick one. So prepared for questions such as “Estimate eBay marketplaces revenue” or “What happens if minimum wage is raised to $100/hr” or “Is hybrid cars more economical or full electric cars?”, “how do you prevent a corrupt DBA from stealing eBay data”, it goes without saying that the actual answer is not really as important as how you think thru the problem.

-        “Soft Stuff” - General attitude, fit, personal qualities

These may or may not be actual questions, but I’d like to see evidence of several personal qualities for example that you have passion, you care about the stuff that you are tenacious and don't give up easily, that you recognize your mistake you learn from it, that you know what a reasonable compromise is and are willing to reach one, that you can deal with conflict in a constructive way – and for more senior candidate - that you can influence people around you especially on why and what and not simply on how. Also I ask our candidates to tell me what they think a few major trends in technology are and why

-        And finally, please know the company

Last but not least, please familiarize yourself with eBay if you are not familiar with it, nothing gets your rejected faster than saying “eBay? Who uses eBay? I am really not familiar with it” – OK, this is an extreme case but hey it is real. It is  

Of course if you are being interviewed for a particular area, e.g. security, Hadoop or Eclipse tools development, you should expect a few questions in addition to the above in each particular areas, for example in security, you should be very comfortable with authentication protocols and practices (zero proof, Diffie-Hellman algorithm, Kerberos etc.) authorization techniques, RBAC, ABAC, XACML , basic cryptography etc.

You may ask, how about other stuff, such as JavaScript, CSS, SQL, Unit Test etc. etc. Yes there are a lot of technologies and techniques used in building large internet apps, but my beleife is that if you know the basics and possess the basic qualities, even if you don’t know the rest, you will be a successful productive engineer.

Interviewing @ eBay, Part I - The basics

When someone interviews with eBay, s/he is given an interview schedule with the name and title of all interviewers, the natural expectation (at least mine) is that s/he searches for the name of all those people as part of the pre-interview preparation. I view this as minimum due diligence that a candidate should do 11 years into the 21^st century. So I hope whoever interview with me at eBay finds and reads this post (if you do, please let me know)

Now that you found this, I will give you a leg up over other candidates:  in the series of four posts, I tell you what questions I would be asking in my interviews for four positions:

• -        Software engineer
• -        Product Managers
• -        Software Architects
• -        Engineering Managers (Sr. Managers, Director, Sr. Director and VPs)

Before we start with specific position, let me first cover the common questions and aspects for all interviews.

I look for the following “necessary” – but not sufficient - qualities that make a candidate productive. In a nutshell, person should be smart, know his field, willing to work hard, willing to compromise and get things done and get along with people under a range of circumstances.

Smart: I am not talking about genius, or someone that can solve puzzles in 10 seconds, but some one that is generally sharp, can think on his/her feet and is solve problems. One of the clearest indication of it is whether someone listens to question, asks follow up questions to clarify what is being asked and then clearly and directly answers that question and then stop. No rambling, no answering other questions and no circular, perpendicular or random answers!

Knowledgeable: Candidate must have proficient level of knowledge in his/her domain, this is separate from being smart, each field requires certain level of experience and formal education – I expand on this with specific question in each of the fields above.

Work ethics: Regardless of how smart and knowledgeable one may be, s/he has to be focused and will to work hard. Real engineering tasks are 10-20% about great ideas, and 80% about grunt work, boring details, dealing with plumbing,  debug, re-build, fine tune etc. If you are not willing to do that, you won’t be successful.

Pragmatic: You must be willing to compromise, change course, give up credit, change your familiar and favorite terminology etc. to get things done. All the smarts, knowledge, hard work often is wasted if you cannot get it done and out at the end. I ask what you are willing and what you are not willing to compromise on for a given project and why, what you would do if you feel a wrong decision was made…

Culture fit: The last of the “necessary qualities” is the ability to get along with others under all sorts of circumstances: uncertain and insufficient data, deadline pressures, failures, inter personal and inter group rivalries … under all those conditions, you should be able to maintain your relationships and get along with others. One of the greatest indicators of whether someone can do it by the way, is sense of humor.

Next post: my list of questions for Software Engineering positions.

On Bullshit...

As some one who

1- Deals with his fair share of a BS
2- Needs to have definitions for everything

I have been missing a formal definition for BS. Courtesy philosopher Harry G. Frankfurt book "On BullShit"  - via Ian Bogost post - that issue has been remedied now.

"It is impossible for someone to lie unless he thinks he knows the truth. Producing bullshit requires no such conviction. A person who lies is thereby responding to the truth, and he is to that extent respectful of it. When an honest man speaks, he says only what he believes to be true; and for the liar, it is correspondingly indispensable that he considers his statements to be false. For the bullshitter, however, all these bets are off: he is neither on the side of the true nor on the side of the false. His eye is not on the facts at all, as the eyes of the honest man and of the liar are, except insofar as they may be pertinent to his interest in getting away with what he says. He does not care whether the things he says describe reality correctly. He just picks them out, or makes them up, to suit his purpose."


WOW ..didn't know there were so much academic work done on BS.

Social Norms, Market Norms and IDPs

Last week LinkedIn caused a backlash and lost a lot of good will – including mine – by opting everyone into their social advertising program. Before that there was (and still is) a LOT of discussion – and disagreement – about Google+ strict real name policy and the words facebook and privacy can almost be used as antonyms…

But why users react so strongly to use of their identity, relationship and data by these networks, after all most people, I presume, understand that companies like Google, FB, LinkedIn etc. are for profit firms with the goal to make money, and that is a the core of most of their policies (and not that anything is wrong with that!).

As someone once said “if you are not paying them, you are not their customer, you are their product”, and, as harsh as it sounds, they do (and have to) sell their product one way or another.

I am not claiming that one unified theory explains the users’ strong reactions to them - and possible solution - but Dan Airely in “Predictable irrationality” may come close. If you have not read “Predictable irrationality” you should. It is an easy read and a very enlightening book – not to mention entertaining – by behavioral economist Dan Airely.

Among many interesting observations he makes is the notion of “Market and Social Norms”. It is basically a very simple and common sense notion: All of us live in two worlds simultaneously, a social world where exchanges of good and services are regulated by social norms, the need for being part of a community and a delayed- reciprocity, and a market world where exchange of goods and services are regulated by a cold, sharp edge rules of the market, prices, interest rates, cost and benefits. Life is good as long as these world are kept separated (as George Costonza famously pronounced) – but when you mix the two the real trouble starts and “it blows up”.

Airely gives a few examples: You go to your mother-in-law for thanksgiving and offer to pay her a large sum for the sumptuous spread she put on the table for you … and next year you’d be sitting in front your TV with a frozen dinner. Or more vividly, the example of a guy who takes out a girl to three or four expensive dates and finally brings up the subject of money and how much the romance is costing him! …and of course suffers the dire consequences. Next time he will sure remember Woody Allen’s word of wisdom: “The most expensive sex is a free one”.

The “Social Networking” sites and identity seems to be a classic example of crossing social and market norms. The use of social networking sites is free and there is no other signal that the relationship between user and network operator is a market or commercial relationship. Users may feel/perceive that they are dealing with a “host” one that allows them to interact with their friends, family and catch up on “social” stuff …you know, everyday life that is so outside the “market norms”. I don’t have data but I feel users do understand that these sites have to make money, and are OK with some ad poping up from time to time or being displayed alongside their content, but I doubt that most people understand how those ads relates to their data (posts, searches, conversations etc.) . Every now and then someone (say WSJ or some tech blog) reminds them of how their data is being shared with others – or why the need to “report” accurate names – and that is the moment where “worlds collide”.

Maybe (and only maybe), if an identity provider (or social network) actually started an informed conversation with its users about the data sharing and gave them meaningful control, can it bring the relationship to the “market world”. In that world users are not the network “product”, but informed partners. For example, if an identity provider (or network operator) makes it clear to me that they can get me a good deal on a camera lens with free shipping if I give them my shipping address and the consent for them to obtain the shipping history to that address from a few large merchants, then I may be happy to share my address – that is firmly a relationship in market domain – and I am happy, but when I start to see ads from drunk-driving lawyers in Denver area b/c I searched for “maximum legal blood alcohol level”  - I was taking an online traffic school exam to clear a ticket – while attending a conference in Denver area, that is crossing  the line, and that is when I might have come close to understanding how that girl felt on the fourth date in Dan Airely’s example!