Question: Tell me about a time when you really screwed up?
Answer: I was both developer and a student. My manager had a lot of trust in me. He’d let me do everything. I had to come up with most designs and to submit them to the developers. The service I was working on had to communicate directly with customers and users. It was business-to-business.
Q: What was the service about?
A: The idea behind the service was to store contract and payment information into an archive. Then store that archive in a service similar to S3, but proprietary to the company. We store those files for 10, 20 or even 30 years.
Q: What was an “archive” in this context?
A: It was like a zip file with payment metadata, contracts, etc. For example, who is your mobile phone provider?
Candidate: When you subscribed to a new mobile plan with Rogers, you signed a contract. That contract is then saved electronically for many years for legal reasons. Our service was hosting contracts for banks etc.
Q: I see, what happened then?
A: What I did was that I stored it in a database on my side. I did not realize the data was not stored at all. I realized after three days. We lost about 3000 archives from direct customers.
Q: What did you learn the most?
A: At that time, I was very arrogant and overconfident. Really rushing it because it was easy. Worse two weeks in life. I did not correctly read the specification. I need to take much more time in the future to understand the product correctly.
Q: Did you do anything once you found the bug?
A: My manager was nice and helped me recover; other people were harsher. Basically, try to ignore people influencing you the wrong way. I then backfilled the records.
Competency asserted: Earn Trust
Job title: Software Development Engineer (SDE II)
I asked the candidate for a time when he really screwed up. He mentioned a time when he did not properly integrate with a storage system, resulting in production data loss.
He quickly found out and was able to restore the data by pushing a local copy. However, I’m a bit concerned when he mentioned that he was not supposed to store the information locally.
We all make mistakes; what matters is how we respond to them. When you prepare your mini-stories for a behavioral interview, be ready to justify any decision or choice you talk about.
In this case, the candidate didn’t convince the interviewer when explaining why the data was stored locally in the first place. It’s hard to Earn Trust of the customers, but it takes one security incident to lose it forever.
The candidate involved the interviewer in the process, “who is your mobile phone provider?”. It changes the dynamic of an interview as it becomes more like a casual conversation with a coworker. It is a powerful technic to get someone’s attention and appreciation.
A year later, I had a chat with that same candidate as he was preparing for an interview with Google. He agreed for me to use his first name in this update, Chris. While reviewing this example together, Chris mentioned that the interviewer had misunderstood what he meant. That he never saved the data locally! Big deal!
Instead, the service he worked on saved the data on a pre-production server used for testing only. That server reset the data every day to save storage. A red flag raised by the interviewer was: “he was not supposed to store the information locally.” That is a significant misunderstanding as it was the one negative datapoint the interviewer brought up during the behavioral interview questions debrief.
Choose your stories wisely. Rehearse them often. Get used to explaining your stories in detail.
Another interesting point is how Chris and his team recovered the archives they presumed to be lost. They dove deeper and realized that the web server was configured to log all the HTTP requests for the pre-prod server with a retention period of a few days. By analyzing the logs, and replaying the multipart HTTP requests (because after all HTTP is plain-text), they managed to recover the archives they initially considered lost!
That final part is missing from the interviewer’s report. It would have shown Bias For Action, ability to Dive Deep, Ownership and more.