Question: Tell me about a time you drove a change based on metrics.
Answer: The company the candidate is working for provides a market place to buy used items. The market place is classified in the sense that items are grouped into categories. The target is to help buyers to find the most relevant items. Using text as information is useful in a few cases (e.g. buying a phone where the buyer can specify features in text form) and problematic in other categories like fashion. In this case, visual features are more helpful. The platform provided by the company allows the user to take a picture of an item that is similar to the one which the user is looking for. Computer Vision is used to extract the features which are used for search. In this context, the click-through rate and conversion rate were used to determine the visual search’s success.
Q: How is the click-through rate defined?
A: Given the list of search results, it is the number of clicks on items in this list.
Q: How is the conversion rate defined?
A: Proportion of the number of clicks which resulted in action and the total number of clicks.
The candidate realized that the click-through rate and conversion rate were low for shoes in the fashion area. He talked to UX designers and initiated a CS questionnaire. The result was that customers thought that the search results contained too many irrelevant items! Sneakers were classified as high heels and vice versa. The candidate’s initiative lead to a re-adjustment of the prediction model. In turn, this increased the conversion rate from 5% to 12% for all items and all categories. The click-through rate increased for shoes only.
Competency asserted: Dive Deep
Job title: Software Development Engineer II, SDE II
Interviewer role: Software Development Manager
In the retail context of an online market place, the candidate he used existing metrics – conversion rate and click-through rate – to identify an issue with the visual classification of products. He dived deep, initiated a customer survey and, based on the result, improved the image recognition system. The conversion rate increased from 5% to 12%.
Metrics are a powerful way to understand your customers and identify bottlenecks in a codebase.
If you don’t use metrics at work and can’t think of any example on any of your projects, you will come short to this question. Investigate if it is feasible in your current codebase at work, and if it is, call a meeting and work a POC. You will get a fantastic story for this question, and you will have a lasting impact on that company after you have left!
If what you work on doesn’t have a user interface, you can still think of metrics as latency (p90, etc.), fault rates, error rates, etc.