Things You Don’t Learn in Technical Interviews

I spend more time than I thought I would thinking about technical interviews lately, because something I’ve been doing is conducting technical interviews for a few startups including Glowforge (they’re hiring!). Since March I’ve done ~36 interviews.

Designing hiring systems is hard because you only have partial data – specifically you don’t know who your false negatives were. I read the articles about how we need to completely redesign hiring systems and how the technical interview is a terrible thing and think there’s merit to a lot of what is said. But ultimately I think there are two main problems with technical interviews (by which I mean – you write code together): 1) a lot of interviewers are bad, and 2) people think you learn more than you do.

My previous post on interviewing addressed (1). Let’s talk about (2).

We should start with the distinction between good questions and bad questions. Bad questions tell you essentially nothing. If you ask someone to code a solved computer science problem on a white board all you learn about is whether they have memorised the solution to that particular problem and can do it in a state of stress.

A good technical question is one with multiple answers, and tradeoffs. It is a problem large enough that it cannot be solved by a single call to a library function. It requires some thought and discussion.

Once you are asking a good question you can start to get a sense of:

• How does someone approach a problem? Do they dive in and start writing code or do they think about things first?
• How do they break it down?
• Can they discuss tradeoffs? Consider how different inputs would change?
• Do they understand what the benefits of different data-structures are? (Different from coding their own version – data structures are foundational and different ones are suitable for different problems).

But I think the most important question I aim to answer in each interview is: do they listen?

I don’t think you learn that much from someone who knows the answer, writes it down, and that is pretty much the extent of your communication. Maybe you’ve learned they are pretty smart. That they wrote code when they knew exactly what to do quickly.

But you don’t know how they will respond to a problem they don’t immediately know the answer to. And unless it’s a job they are going to be incredibly bored in, that’s a pretty important thing to learn about them.

When they can’t immediately solve something you can start to see:

• Can they look at their own work objectively and see problems?
• Do they listen to feedback?

The other question I aim to get a sense of is: How do they respond to new information? One approach I use for this is “unreasonable product manager”, which means I change the requirements fundamentally. I want to see how they incorporate this information into what they have.

Those are some things I think you can learn, and to be clear – I don’t always get a sense of all of these in each interview, and I’ll note that accordingly in my feedback. However, there are some glaring omissions that are clearly relevant in the day to day work of software engineering.

• How do they function in the last 20% of shipping a project (which can be pretty boring)?
• How do they work on a codebase of significant scope?
• Can they work with legacy code or will they insist on rewriting it all?
• Can they prioritise?
• Do they give thoughtful code reviews?
• Will they be a good mentor to other developers?
• Will they take direction? (This is partially covered above, but more in a “may show up if they really won’t” way).
• Will they adapt to the processes you have? (E.g. code coverage, testing standards).

It’s important to recognise the limitations of these interviews – as an individual, not just as an organisation – because part of being a good interviewer, of trying to be a good interviewer, is not to give feedback on aspects you really don’t know anything about, and not to delude yourself that you do.