Question: Tell me about a time you were troubleshooting an issue and went down several layers to understand the problem.
Answer: I was on-call for a system backed by Cassandra, while one of our schema migration failed.

Q: How did the issue manifest itself?
A: There was a latency spike during a schema migration (p95 latency). I reviewed traffic and logs, but the logs were not descriptive – just stated that there was a schema mismatch. The application logs were showing “SCHEMA NOT MATCHING.”

Q: What did you do to identify the issue?
A: I queried the nodes of our Cassandra server farm manually and noticed that one node had an old schema version.

Q: How did you fix it?
A: I manually applied the migration for that specific node, resolving the latency spike.

Q: What’s the correlation between the node with a schema out of date and the latency spike?
A: The node wasn’t usable, translating into higher latency.

Q: How long was the outage for?
A: 1 hour

Q: Customer impact?
A: Latency only. No faults.

Analysis

Competency asserted: Dive Deep
Job title: SDE II
Interviewer role: SDE II & SDE III

Vote: 👎

The example shows surface-level investigation but didn’t fully dive deep. There is no root cause of the underlying issue (why was one node not updated).

The candidate did not raise the bar for Dive Deep. However, during the coding, the candidate demonstrated a deep understanding of the different data structures. She naturally mentioned that “null” references would be garbage collected, and she wanted to improve the solution for handling thread concurrency.

Learnings

The example looks pretty good at first. Doesn’t it? The issue explained by the candidate didn’t require a deeper level of troubleshooting. The candidate didn’t mention anything about a follow-up root cause analysis. She did a short term fix without thinking long term to reduce operational pain. The candidate doesn’t raise the bar either when it comes to operational excellence.

As you get prepared for your next interview, volunteer to tackle more complex issues at work. Fresher experiences will increase your level of detail. In this occurrence, it may just be that this issue happened a few months or even years ago and that the candidate forgot the complexity of the problem.

Leave a Reply