Good technical interview questions are hard to find. In the last month, I’ve seen some really great ones and equally bad ones. I want to go through some of them and write down my highly biased 5-star rating on their effectiveness.

First, for some context on how I evaluate interview questions. Your interview questions should be:

• Respectful of the candidates

• Objective and evaluated consistently

• Gathers relevant and insightful data

When it comes to data gathering, I’ve seen teams make two kinds of mistakes. The first mistake is coverage: The entire interview panel asks the same type of questions, with little coordination. The second mistake is that the interviewer interprets the results subjectively; They don’t have an objective and consistent way of evaluating answers and often rely on intuition to make a decision.

With that in mind, I’ll go over questions from my recent interviews, what I think they measure, and talk about how I scored them.

Implement Postorder Traversal of a Binary Tree

Setting: Phone interview

Summary: Given a binary tree, print each node using postorder traversal.

What does it measure?

1. Have you memorized the algorithm (recursive and iterative) for depth-first search?
2. If you memorized the algorithm, does your code look clean and compact?
3. Ability to write short snippets of code.

Review: 1 star

IMO it is fair to assume most people have memorized a recursive solution to DFS. Thus, it is only useful for measuring whether someone can write a short snippet of code. This is a pretty low bar. Fine for a phone interview though.

Wheel of Fortune Before-and-After Puzzle

Setting: Phone interview

Summary: Given a list of phrases, create all possible “before-and-after puzzle” using pairs of phrases.

It is a variation of this question from StackOverflow. In the original question, one is to join only on the last and first word. In this version, you consider more than the last and first words when joining two phrases, which increases the search space.

What does it measure?

1. Ability to analyze the problem and describe a solution
2. Ability to write code that are medium in length
3. Quickly evaluating different possible data structures and approaches that are possible solutions

Review: 2 stars

There are a number of ways one can solve this. It is implied that you are supposed to avoid a brute force solution and identify a data structure to make the algorithm more efficient. If someone makes progress, you can minimally say they know a few basic data structures, which, again, is a low bar to meet. It requires one to write quite a bit of code, which serves as a very strong signal on whether someone is comfortable coding.

One major problem is when someone solves this quickly. A quick solution doesn’t tell you much more about the candidate but is likely to create an impression that the candidate is really good. But we don’t know if we can trust that impression. Perhaps the candidate is simply good at algorithm puzzles.

Implement a Simple “Email Sender” API

Setting: Onsite coding interview

Summary: Implement code that emulates an email sender, capable of sending to email aliases, e.g., engineering@example.com

In this exercise, one should write working code that “sends” email using the API and verify that when sending an email to a group alias, all users in that group “received” the email.

What does it measure?

1. Data modeling of user-group relationships
2. Class design / separation of concerns
3. Ability to write code that are medium in length
4. Ability to communicate your class and API design
5. Ability to implement a working program within a reasonable time

Review: 3 stars

This is a very straight-forward problem. I gave it more than 2.5 stars because it pulls very strong signals on a few key qualities you want in an engineer: data modeling, communication, and fluency in their preferred programming language.

In this case, the problem was asked by an experienced interviewer, who knows how to guide the conversation as it went along. In this case, after seeing a working solution, the interviewer evaluated TDD and the performance characteristics of the underlying data structures.

It may be too simple though. There are not a lot of trade-offs to consider, and programming is all about making good trade-offs. Later on, I’ll contrast this with a problem with interesting trade-offs.

Select X winners out of Y Contestants

Setting: Onsite coding interview

Summary: Given Y contestants and at most Z prizes, randomly choose X winners

What does it measure?

1. Ability to write short snippets of code
2. (Maybe) memorization of a “swap-with-last-element” trick

Review: 1 star

Unless I am missing something. The only thing this question measures is your ability to write a very short snippet of code. Selecting X winners is basically an O(X) operation.

What bothered me about this question is that it had a non-obvious part to it: To avoid randomly selecting the same element more than once, you would 1) Swap the picked element with the tail of the array and 2) Updating the end range your random function. You either see this trick or you don’t. Take me for example, I finished this problem quickly. But my success told the interviewer nothing about my ability to write real code.

Variations of “Do You Have This Memorized”

Setting: Phone interview and onsite

Summary: Facts that can be easily looked up in API documentation or Google

A completely fictional conversation that I didn’t have recently:

Me: “Um… I guess you can use a thread pool to consume work from a queue and block when one tries to ‘checkout’ more than X threads."

Interviewer: “What if you don’t want to block?"

Me: “Ok, I guess you can use a callback based API. There’s a number of them. Its been a while since I used any specific one in Java. I’d review the Concurrency or Streams API."

Interviewer: “Sighs. OK, do you know what the Foo Bar Pattern is?"

Me: “Perhaps. I don’t have that name memorized. Do you want to talk about what it is?"

Interviewer: “Sighs. Well, if you don’t know. I’m just giving you the answer."

What does it measure?

1. Whether someone has memorized trivia
2. Whether someone has memorized the right trivia /sarcasm

Review: 0 star

Fortunately, I’ve only had two interviewers ask these type of questions. For what its worth, they are both from large, bureaucratic companies with slow moving engineering orgs.

Two-Part Schedule Management Problem

Setting: Phone interview

Summary: Given a schedule, which is a collection of time intervals, find all intersections of time intervals.

The problem is similar to doing a bitwise AND of two list of time intervals. The interesting thing about this problem is that the interviewer breaks the problem into two parts. The first part gives a hint on how to efficiently implement the second part.

What does it measure?

1. The ability to make connections from Part 1 to Part 2 quickly
2. Ability to write short snippets of code
3. Ability to listen and follow directions

Review: 3 stars

The interviewer put a lot of preparation into this question, allowing one to lead from solving a simple problem in Part 1 to a more complex problem in Part 2. You’ll have to write working code.

Making the connection between Part 1 and Part 2 requires a “gotcha” kind of leap. However, the interviewer tried hard to help me make that connection. For candidates who cannot make that leap, you can always help them with that, then evaluate them purely on implementation.

Simulating a Loaded Die

Setting: Onsite interview

Summary: Effectively solving this problem.

The interviewer was not looking for Vose’s Alias Method. I believe the interviewer would’ve been ok with a number of “naive” solutions. The important thing, I gathered, is one’s ability to discuss different suboptimal solutions and their trade-offs.

What does it measure?

1. The ability to identify more than one way to solve a problem
2. Communicate trade-offs of different solutions
3. Ability to write code that are medium in length

Review: 2.5 stars

The way the problem was given to me was biased heavily toward communication skills. Your ability to listen and understand the problem is key and you’re expected to come up with one or two ways to solve the problem and talk through them. Coding is minimal.

Design a Datastore with Transactions

Setting: Phone interview

Summary: Implement (code) a Key value store with transactions

However, this was asked in two parts. The first part was a 45-min session of coding. The coding portion is made easy as the interviewer provided working test cases and an interface to implement. The second part is a 15-min discussion on the edge cases of the implementation. We ended up going into how the code would work when the API is used concurrently, what are the trade-offs between different ways to handle concurrency modifications.

What does it measure?

1. Ability to write non-trivial code quickly
2. Communicate trade-offs of your design
3. Ability to think through a complex scenario in detail

Review: 4 stars

This problem got a high mark because the interviewer invested time to provide test cases. This mean you’re not wasting time on the minutiae of coding. You have a clear direction and you’re evaluated on your ability to write real-life code. The interviewer was experienced and knew how to guide me along (avoid getting bogged down).

Then we verbally discussed how the implemented code would work when used concurrently. We talked about how the interface should be changed to best support concurrent access. We talked a lot about different trade-offs. In contrast to the Email Sender problem, this one had a lot more potential for in-depth discussion.

The only downside is that I think a lot of people would do poorly on this one.

Implement Integer Multiplcation

Setting: Onsite interview

Summary: Given [6, 8] and [2, 3], return [1, 5, 6, 4]

What does it measure?

1. Ability to think through a straight-forward algorithm
2. Ability to write short snippets of code

Review: 1 stars

Because this is a very straight forward problem, the main task is to verbalize the algorithm you intent to code, followed by actually writing the code.

Design a Railroad Crossing Gate Service

Setting: Phone interview

Summary: Given sensors on a railroad, a set of control boxes, and a crossing gate, design a system to manage them.

What does it measure?

1. Communication
2. Ability to think through a complex scenario in detail
3. (Optional) your knowledge on distributed consensus algorithms

Review: 2.5 stars

There was no coding. The question heavily tests your ability to design and communicate. The interviewer was experienced and allowed flexibility in defining the problem space, then pushed hard on diving into details of the system design and why it is appropriate. Thus, this question is less useful on its own but relies heavily on the interviewer’s ability to pivot and ask hard questions.

I enjoyed this problem. From what I hear, the interviewer learned a lot about my strength and weaknesses. However, I took a star away from this problem. The interviewer intended for the discussion to go into distributed consensus protocols. However, the way the question is posed allowed it to be solved without distributed consensus. In our conversation, I accidentally steered the design into a non-distributed system. Fortunately, the interviewer was experienced enough to pivot quickly and still end up asking great questions about message authentication codes, TCP/IP, UDP, and keep-alives.

Full Day Pair Programming

Setting: Onsite interview

Summary: Read this Quora question. A candidate work with the team for a full day to see what pair programming is like.

What does it measure?

1. Communication
2. Your comfort with full-time pair programming and other agile-related processes
3. How well you get along with your future teammates

Review: 4 stars

This process is an intense investement from both sides. When you take preparation into accoount, committing an engineer to a full day of pair programming is a costly exercise. You need to find the right “get started” ticket that a new engineer can work on without domain knowledge. That is never easy.

On the candidate’s side, pair programming can be exhausting. For someone who is not used to it, doing it for a whole day can be challenging. You are asking a lot from the candidate. On the other hand, I don’t think there’s a better way to learn about me as a teammate.

Summary

What makes an interview question good depends on how much it teaches you about your candidate. Some questions are “complex” (e.g., Integer Multiplication), but doesn’t tell you much. Some questions are “simple” (Railing Crossing) but can tell you a lot. Some questions rely on a trick (Select X Winners). I think those are terrible. They teach you nothing at all.

I graded questions based on how informative they are. This is actually quite unfair. To be honest, it is ok to use a question that test for just a single thing, as long as the rest of your interview panel covers other areas. Like all things in life, the right thing to do is situational.