Question: Imagine you have a metric for determining positive customer experiences with Alexa. When you look at this week’s data, you see that the customer sales rates dropped by 10% from last week. How do you go about finding the issue that caused this drop?
Answer: I would look at the user data. What did the user ask for? Look at how their requests were answered.
Q: Which metric would you define for that? Which data would you use to determine this metric?
A: No data point.
Hint: So, it is basically about assessing whether the right intent was derived, so the issue in NLU?
Q: What other factors would you look at?
A: I would check whether the speed of service is OK. So whether the service provides a correct answer, but is too slow.
Comment: No data point on how the candidate would measure this and which data to use.
Q: What else could you look at?
A: Check whether it is a connectivity issue.
Q: What data would you need, for example in a log, to see that connectivity is an issue.
A: Login time, end time.
Q: What if “We know it’s Audible” – where to go from there?
A: I would check whether it is a user-specific problem, i.e. only one user would have the problem.
Q: OK, could be. But this is not specific about Audible, right?
A: Yes. Maybe there is a bug in Audible? Audible service is not working.
Q: What exactly do you mean by a bug? Could you provide more detail on what it means Audible is not working? Which data would you look at?
A: No data points. I am not so familiar with Audible (despite, later on, she mentioned that she subscribed to Audible herself).
Hint: So to use Audible, the user has to be logged in to the account.
A: There might be an authentication failure. Or the user is logged onto the wrong market place.
Q: Which data would you need to figure out that this is the issue?
A: No data points.
Competency asserted: Dealing with ambiguity, data analysis, data-driven evaluation
Job title: Language Engineer
Interviewer role: Software Development Manager
When asked how to investigate a 10% decrease in weekly CS, the candidate came up with high-level ideas but did not elaborate on specific data she would look at and how it should be made available via logs or any other means.
Concerning dealing with ambiguity in language data, she came up with correct answers but did not cover checking user subscriptions for user-specific information or recommender-like suggestions.
These replies represent good guesses, but I wouldn’t consider them raising the bar for data analysis skills.
If you are like me, you felt bad for the candidate when reading the transcript.
The candidate was lost. It wasn’t clear for the candidate what the questions were about and what direction the interviewer wanted to take.
If you find yourself in a similar situation, speak up. “I am not sure I understood the questions clearly. Are you expecting me to give you technical details about how to do x?”.
While it is a behavioral question, it is similar to this System and Design question: the site is slow.