Accidentally in Code

Tag: interviewing

  • Cate’s Career Coaching Process (AKA A Process for Finding Your Next Job)

    Cate’s Career Coaching Process (AKA A Process for Finding Your Next Job)

    A young woman sits accross from an older woman who has a notebook. Both are wearing old fashioned dress. It looks like a job interview.
    Credit: Flickr / Ethan

    I know a number of people who have been laid off recently – the market is on a downturn and this is more common than seems to be being talked about. I’ve spent time helping a few people try and find their next thing lately, and after some iterations I now have a process.

    1. What are you going towards?

    Whether someone is leaving their job voluntarily or not, there’s a good chance it hasn’t been great for a while. This makes it easy to focus on what you don’t want rather than what you’re going towards. But you make better decisions when you focus on what you want rather than what you’re trying to avoid.

    Be brutally honest – what do you really want to be doing? What do you want your life to look like? How do you want your job to fit into you life?

    Only once you’ve done this, consider what tradeoffs you are prepared to make.

    For me: I loved managing a distributed team, and I wanted to keep doing that if possible. But I knew there weren’t many of those kind of jobs, so I was realistic that I might have to compromise. There were two dimensions of compromise: on-site management, or remote IC (individual contributor) work. On-site IC wasn’t something I was going to consider until those options were ruled out.

    If you’re helping someone else: It’s easy to think you know what someone wants (if you think you know them well), but don’t assume. Ask questions, make observations about what they seemed to enjoy and what their strengths are, but let them figure out what they are going towards. It’s great if you are surprised! This sometimes takes people a while (and multiple conversations), but it’s really worth the time because it’s the foundation of everything else. Once they’ve figured it out, don’t help them be “realistic” even if you think that it’s going to be hard to find what they want. It’s unlikely to be helpful. Most people who you actually want to help have no trouble being realistic.

    2. Resume.

    Honestly, I hate resumes. Even as a hiring manager, I take very little interest in them (I think resumes are mainly a bias vector). I don’t have one, I got my last 4 job offers without one, and when I did have one it was one I paid someone else to create for me. If that’s an option for you, I really recommend it and I’m happy to connect you to the person who did mine (tweet me – @catehstn).

    Unfortunately most people still need resumes.

    The main points with resumes:

    • Focus on what you achieved, not what you did.
    • Keep it short.
    • Make it readable. Use clear and concise language.
      • Prefer short paragraphs to long bullet points.
      • Use a service to check the reading age.

    Once you’ve written up your experience, write your summary. The summary is two to three sentences that position you as a good person for the job you want, and should be backed up by the experience you have.

    When you think you’re done, send it to three friends whose opinion on this is worth having. Ideally they review or at least see resumes for the kind of job you want. They might point out stuff you’ve missed. Try and find at least one native speaker of the language your resume is in.

    If you’re helping someone else: This really depends on how much time you have. I don’t believe resume writing is a useful skill, so I’m happy to rewrite sections for someone. The process I followed was: I rewrite (or write) the section from their last job in discussion with them, they rewrite other jobs, and then I go through and edit those. Then, we write the summary together. If you’re just reviewing someone’s resume, focus on how they are coming across, or any information that appears to be missing, rather than how you personally would write a resume.

    3. “Soft” interview questions.

    Resources:

    The purpose of these interviews is normally to get a sense of your experience, personality, and ways of working. When you wrote the summary for your resume, you made a conscious effort to think about how you want to present yourself. You want this to come out in your answers. This means not just answering the question, but deciding what you want the takeaway from your answer to be – something that supports your summary – and then telling a story that showcases that. This means that the same story can be an appropriate answer to multiple questions, depending on what you highlight.

    The more open ended the question, the more scope you have for this. For example the question “tell me a bit about your background?” You can answer this question directly and chronologically (“I went to university at XXX and then I worked at YYY and did ZZZ and then…”) or you can frame the interview with your answer.

    When I get asked this question, I say something like: “I’ve spent my whole career working on mobile, including writing a J2ME app a long time ago! At Google, I built most of the first generations presentations experience on iOS and ran a team building a location-based B2B app, and also worked on that app on Android. Most recently, I ran the mobile team at Ride.”

    When answering these questions, one thing to keep in mind is that stories about dysfunctional environments are not helpful. For example a question like, “tell me about something you worked on that failed?” The answer that most springs to mind might be the one where the business didn’t know what they wanted, asked for unreasonable things, didn’t listen to engineering, and of course it didn’t ship or what eventually shipped was something that people didn’t want. If you don’t have a better lesson from this than “don’t work in that kind of environment” (a very fair lesson to learn from that experience), it’s not a good story to share. A less epic failure with more concrete takeaways is a better option.

    If you’re helping someone else: Don’t just consider if the answer seems good, consider if it relates to their summary. If you’ve worked with them or know them well, point out other stories or achievements they might have missed – people are often a little blinded to their strengths, because they come more easily.

    3b. Recommendation.

    If I’m writing (or likely to write) someone a recommendation, I write it after we’ve done this together – it’s when their achievements are top of my mind, and I can write something that a) supports their summary and b) highlights some stories they might want to talk about in an interview.

    If you’re asking for a recommendation: Make it easier on your recommender, and highlight some bigger achievements that you think they can speak to. These should relate to your summary. E.g. if your summary highlights your love of experimenting with new technology, include that experimental project you did that influenced something in a useful way.

    4. Technical interview preparation.

    Resources:

    When I give someone a mock technical interview, I give them my standard technical interview but I am less nice. Mainly, I do less time management for them – allowing them to spend too much time in places that aren’t helpful, if that’s their inclination.

    I think Time Management is the easiest and most overlooked skill of technical interviewing. When people don’t know the answer they are very tempted to spend a bunch of time on peripheral things like validation, or writing a bunch of different function definitions, or setting up an elaborate test suite. Sometimes they start writing a brute-force solution with the goal of better understanding, but this takes up a more time than it provides utility. All of these might be good and useful things to spend time on in Real Life Programming, but can work against you when you are trying to show your capabilities in 45 minutes to an hour.

    Other common mistakes:

    • Not taking the time to think of more than one option for solving the problem.
    • Not communicating clearly, meaning the interviewer has to guess what you are doing.
      • It’s easier to give someone feedback and guidance when you understand how they are thinking, what they are doing, and why.

    5. Introductions.

    Some people spread a broad net, my preference is to speak to a few places that I have strong introductions into. It’s more likely to be a good fit, and I’m more likely to be successful in the process. When I was looking for a job in August / September, I spoke to four places with strong connections and ended up with two offers that I felt great about. This is my preference with introductions – to introduce people to places where I think they have a good chance of success, and where I think they would be happy. I use a customised version of the recommendation I wrote to introduce people.

    Time

    I estimate that taking people through this process took ~8 hours per person (and these were people I knew well). As a result it’s not something that I would make time for other than for close friends or people I have a professional commitment to. They probably spent at least that, more, working on parts of it by themselves – technical interviews especially are a lot of work to prepare for.

    TL;DR

    Job hunting is a PITA and a ton of work. It helps to think about how to be effective at it.

  • Interviewing as a Manager

    Interviewing as a Manager

    Credit: Flickr / Marek Kubica
    Credit: Flickr / Marek Kubica

    The final part of interviewing for my last job involved coming to Colombia for 3 days and having 21 1:1s. FYI, this is the kind of thing that immigration finds very suspicious and resulted in me being detained and searched for drugs in Canada. But I digress.

    I firmly believe that people should interview their boss and have got to the point myself where I will schedule a call and up front say “I’m going to interview you to be my boss now.” – I want to understand things like how they will communicate with me, and what I can expect from them. So when I’m interviewing with people who might report to me I’ll encourage them to ask me anything they want to know – and I think I learn as much from the questions asked as they do from my answers.

    The thing is, most people will not ask hard questions in this context. So I wanted people to feel good about me joining, and I wanted to get to know them, so I prepared a set of questions.

    The Questions

    1) How long have you been at [company]?
    2) What were you doing before?
    Note: These Qs helped me get a sense of why people joined – e.g. they followed a friend they worked with before.

    3) What do you think your biggest contribution to the team is?
    Note: This Q (weirdly) helped me discover some things that were happening that I really needed to know about before joining.

    4) What do you think the team needs right now?
    Note: Hands down the best question I asked from which I got the most useful information. People telling me what solutions they wanted surfaced the problems they were having without asking them to complain at me (which given most of them were at least somewhat trying to sell me on working there I might have got less honest answers to). Most interesting was when people working on the same or nearby things had different views of the problem.

    5) What’s your next career goal (if you have one)?
    6) What do you most want to learn?
    Note: Helped me see where the team wants to grow, and the different approaches people want to take. Helps distinguish between people who go with the flow and people who are more goal-orientated.

    7) Did you have a manager you really liked? What kind of things did they do for you?
    Note: Reassuringly their current boss (who I reported to) was a pretty common answer to this. I think most people are not very intentional about the kind of manager they want. At least 2 people asked me this question from the opposite perspective – what kind of manager would I be and what could they expect from me?

    8) Is there anything you think would make a big difference to your happiness at work?
    9) … to your ability to do your job?
    Note: Helpful for people who didn’t have a good answer to 4, or who focused on what other people (or teams) needed not their own.

    The Process

    I had the questions (numbered) on a page, and then I could just number people’s answers in my notes.

    I didn’t ask everyone every question – the goal was to start off a conversation, and if they had questions for me then I wanted those to take precedence. I did ask everyone 1, 2, 4 and 5.

    We arranged it team by team, most recent joined to oldest. I picked this order for a couple of reasons: building up context is hard, and grouping by team makes understanding that context easier. Newer people will have more a more recent perspective of joining, longer-timers will be better able to explain why things are the way they are.

    Unsurprisingly, this many conversations was super exhausting. By day three, between each meeting I was hiding in the bathroom, fantasising about never speaking to another human ever again.

    The Results

    I learned a lot in these conversations and referred back to my notes as I started working there and even several months in. Question 4 is one that I continue to ask in basically every potential job conversation. It tells me so much about what people think the problems are without asking them to frame things negatively. The ordering worked as well as I hoped it would.

    The main thing I would change is to ask more questions about feelings – what do people most like about their job? What are they most proud of? What do they worry about?

  • Hiring Manager Interview Questions

    Hiring Manager Interview Questions

    Danbo vs Daruma
    Credit: Flickr / Takashi Hososhima

    Generally, I’ve really focused on doing technical interviews, but towards the end of last year and start of this I did a bunch of hiring manager interviews. This was a new and scary thing for me – whilst some things (like seeming warm) remained the same, the things I was evaluating were different. It was a much fuzzier process, mainly around whether the person I was speaking to was a potential match for what we were looking for, and whether we were likely to be the kind of environment they were looking for.

    I ended up coming up with the following list of questions, and I’ll talk a bit about each and why it was useful.

    Tell me about your background?

    Nice soft question to warm people up, but also an opportunity to get a sense of if they have the kind of experience we were looking for. E.g. we’re looking for an experienced Android developer, and they just got started with Android three months ago, it’s probably not a good fit. More subtly, a sense of whether they have worked on apps at some level of complexity. Did they work on a team, or solo?

    What do you want from your next job?

    I think people make better decisions if they are going towards something, so the main thing I’m interested in here is some kind of intentional thought into what would be a good next move. Based on what makes them happy, or where they want to go next in their career.

    Sometimes people will list all the things they don’t want, which I don’t think is a good sign.

    What is your favourite thing you’ve worked on?

    I think passion is bullshit, but I like to ask this question as a gauge of interest and engagement. Why $thing was their favourite is enlightening as to what motivates them – was it because they got to experience a whole project end to end? Was it because it was a product they liked and used themselves? Was it going deep on a specific technical problem?

    What do you like about Android / iOS?

    Very standard question for mobile developers. I find it interesting as to why people are drawn to that platform.

    Do you have any feelings on Android vs iOS?

    If someone develops for both, this question can lead to an interesting discussion of what is easier / possible on each platform. But mainly I like to see developers on one platform having some awareness of the other one, even if they don’t work with it day-to-day.

    Do you have experience with, or feelings about remote work?

    I totally get why remote work appeals, but to some people it appeals in theory more than in practise. Some people are anxious about it and it’s a good opportunity to gauge what kind of support they would need.

    Can you talk a bit about an interesting performance problem you solved?

    This is my favourite question! My theory of mobile development is that in any interesting app, there is at least one place where something is not standard, and not straightforward. This is where people have to go deepest into understanding the platform, and figuring out how to make it work. Good answers to this question are fascinating, and invariably teach me something.

  • We Hire The Best

    We Hire The Best

    The tech industry prides itself on its rationality, and yet is filled with trite slogans that are demonstrably untrue… and further, harmful.

    Originally published in Model View Culture, November 2014.

    Credit: Public Domain Pictures / Dawn Hudson
    Credit: Public Domain Pictures / Dawn Hudson

    “We hire the best.”

    It’s a slogan we can rally around in company meetings, a tagline to put on the jobs page… a shoring up of identity bound too much in employment.

    It’s also manifestly untrue. A more accurate way of putting it might be:

    Among people we know, we hire “the best” (as determined by our subjective process), who are willing and able to work in a specific place (or remotely), and who accept our offer.

    Sadly, that’s not as catchy.

    People We Know

    Who you hire is profoundly limited by your company’s network, so when looking to diversify your workforce, it’s worthwhile to broaden your network. It is unlikely that everyone has heard of you, and of those that have, many may not know that you are hiring. Among the people who do know: are they aware that you hire people with their skills and/or backgrounds? A historically elitist hiring process may well discourage people from non-traditional backgrounds from applying, even if you later change it. Given the prevalence of team pages filled with white men, it’s possible for someone to hear about your company, look at the team page, and determine it’s not worthwhile to apply. The later diversity in hiring is considered, the bigger problem it becomes.

    If you aim to hire mostly through referrals, those referrals are likely to reflect your current demographic — unless you work hard to change that. For example, women are often shut out of social gatherings (casual drink ups), or assumed to be service staff or partners of attendees at events where career connections are formed, and the typical social network of a white American is 1% black. Meanwhile, companies spend large amounts of time (or delegate to external recruiting firms at enormous expense) in order to find “passive” candidates – those who aren’t applying, but could be convinced to interview. This strategy is subject to serious bias, such as preferencing alums of pedigree universities and well-known companies. And recruiters often pitch minorities on too-junior roles — due to sexist and racist stereotypes, women are deemed less competent and black people are perceived to show less leadership. Such stereotypes and biases can play a huge role in if and how candidates are approached, and for what positions.

    …As Determined By Our Subjective Process

    All hiring processes are subjective which means: open to bias and flawed.

    In Shaft’s article Thoughts On Diversity Part 1 (tackling the Meritocracy Myth), he observed:

    “Yahoo > Google > LinkedIn > FaceBook > Twitter. After Yahoo each of these companies’ diversity numbers have been worse than the company that followed them. I believe this is because Google recruited from Yahoo, LinkedIn from Google, and so on. Each subsequent company becomes less diverse due to the sub-conscious amplification of educational, cultural and work history biases.”

    These biases are particularly evident in the educational pipeline leading to the tech industry. Just looking at the Ivy League, we see disproportionate representation of “legacy” enrollment: between 10 and 15% of students are children of alums. A 2011 study found that when a parent had attended that same university, students’ chance of admission went up 45.1%. In Dartmouth’s 2015 class, legacy admissions comprise 8.5% of the student body. Princeton legacies have a 33% admission rate, compared to 8.5% general admission rate, and Yale 20-25% compared to 6.7% (2013). And legacies make up 12-13% of Harvard undergraduates, with a 30% admission rate (2013).

    Meanwhile, racial diversity in the Ivy League is dismal. As of 2012, the highest enrollment rate of black students in an Ivy League school was 7.7%, compared to a US population that is 13.1% black. And the highest rate of enrollment of Hispanic students was 13.2%, compared to their 16.7% overall representation. Socio-economic status and parental income is also a significant factor in admissions to Ivy League universities. 69% of Yale undergrads come from families with incomes over $120,000, with just 15% of students coming from families with incomes below $65,000 (2014). Meanwhile, the US mean household income is $60,528, and is lower for black and Hispanic households.

    Further, admission rates are distinct from graduation rates, where we also see systemic inequality. For example, at University of California (Berkeley), white students have a graduation rate of 92%, while Hispanic and black students have a graduation rate of 81% and 71%, respectively. (For further reading, Whistling Vivaldi contains some good discussion). And while this data looks at a particular school system, it is a great illustration of how filtering resumes on pedigree companies and universities builds on the flawed, biased processes that have come before.

    Interviewing

    While much inequality enters the hiring process even before an interview, this is yet another place where we see significant bias enter the equation. Interviews range from unstructured conversations based around assessing “culture fit”, a vague practise that opens up almost unbounded opportunities for discrimination, to extremely structured interview questions which may or may not cover topics that are even relevant to the job.

    For example, the prevalent interview system in the Valley means assessing a candidate’s grasp of 2nd year Computer Science Algorithms and Data Structures courses through contrived problems. This is commonplace at companies including Microsoft, Google, Amazon, Facebook and Palantir. These interviews mainly test people’s abilities to remember, revise, or learn the algorithms and data structures, and to respond appropriately to artificial problems (pro-tip: the answer is usually a Hashtable). It is problematic in hiring people from non-academic backgrounds (who may not have taken 2nd year Algorithms and Data Structures), or who went to universities which do not cover that material to the necessary standard.

    This is also challenging for specialists; for example, much of this content has little bearing on front-end Javascript, and often the most challenging problems in mobile have less to do with algorithms than physical concerns like networking and battery life. In this subjective process, a tangentially related thing is used to assess someone’s ability to do a job that comprises 99.9% other things. In fact, Google’s own data from using this process has revealed interview performance has little bearing on job performance.

    There are numerous other issues with the design of interviews — many of which have been discussed elsewhere — including:

    1. Trick questions. May be based on knowing some esoteric factoid about the JVM, or that require an “aha” moment or are otherwise impossible to solve.
    2. Other kinds of bad questions: for example, questions that are too large for anyone to get through during the interview process, and so results are biased by which part of the question the interviewee focused on.
    3. Bad interviewers: interviewers who are not clear in their questions, who manage time poorly, etc.

    Finally, interview structure aside, feedback on applicant performance is riddled with “unconscious bias”. Having read much of the research around stereotypes and bias, when working at a major tech company I would spend around 30 minutes combing through my feedback for bias after interviewing minorities. Consistently, I found small comments and phrases that I would remove or rephrase to reduce bias. This doesn’t mean I invariably supported the hiring of minorities who I interviewed; I did not. Merely, I tried not to further stack the odds against them through my feedback and picking on things I would not have highlighted had the person I interviewed been a white male. This also does not mean that my feedback was without bias: bias runs too deep in all of us. It just means that I did my best to be conscious of my bias and work to combat it.

    It is unlikely that many, or even any of your interviewers are doing this: they lack the domain knowledge, and will not take the time. And the reward for employees who do do this kind of extra work is often being asked to do more of it: recruiters are incentivized to hire and learn which interviewers care and are quick and painstaking in their feedback. Relatedly, programs to improve the experience for women candidates often create additional work for the women who are already there. Well-intentioned companies interviewing women often aim to get a woman on each interview slate, which can be problematic and causes the work of interviewing to disproportionately fall on women. These extra tasks around hiring and outreach are something that is for the good of the collective, and are exactly the kind of task that women receive little or no recognition for but are penalized for not agreeing to (see Women Don’t Ask).

    Plus, it often doesn’t make for a good experience for women-interviewees, either – yes, we notice when we come in and don’t meet any women, but we also notice when we get a female interviewer who is clearly in the wrong domain, or inappropriately junior.

    Reviewing

    At this point in the hiring process, feedback is collected and another person or group of people review it and make a decision. Many of the filtering and interviewing biases are amplified here as more people get involved and look for adverse signals (like “shyness” or lack of technical credibility, or an unusual background), unless a serious and concerted effort is made to remove them.

    It’s also in this stage that we commonly see reference checks: another possible source of bias. As part of some volunteer work, I ran a committee assessing female CS students for scholarships. As part of the process, we reviewed recommendation letters from their (mostly male) professors: the prevalence of gendered feedback in these letters was frankly appalling.

    Are Willing to Work [In a specific place] / [remotely]

    The phrase “we hire the best” irks me everywhere, but most of all in the Valley. One would think that the competition and amount of movement between companies in the valley precludes this, but there are incredible people around the world who who do not wish to move there, and with good reason. There are the US issues, such as the difficulty of procuring visas, poor social policies around childcare and maternity leave, and the localized issues of high rents. In addition, professional women are more likely to face the “two body problem”, having a spouse with a job that is hard to move, or to be the primary carer for a relative that limits their geographic mobility.

    Remote work is one way of getting around this, however some people, particularly junior people, may not wish to work remotely, and want the benefit of in-person interaction and mentorship. Some may just find it too lonely, or not a fit for their working style. Remote work is not suited to everyone’s temperament.

    Those Who Accept

    In the end, hiring “the best” is dependent on “the best” accepting your offer in the first place.

    One of my friends has been interviewing recently and it’s fascinating how poorly some of the companies have treated her over the course of extending her an offer. One company lowballed her salary, was late on every deadline, and generally did not respect her time. Perhaps they think anyone who makes it through their arduous and time-consuming process must be committed to saying yes, or perhaps they’re just relying on their brand.

    During the process they used to determine she was a good fit for them, she figured out they were not going to be the best fit for her. She’ll go somewhere where they respect her time, and offer her fair compensation up front. Somewhere that communicates with her in a timely manner. As for startups she spoke to who treated her really well, there’s every chance that she’ll consider them again in 2 years or so, when she’s thinking about making a move again.

    There’s massive competition to hire engineers, and engineers talk amongst themselves. We know that Glassdoor and the like are full of bitter animadversions from people who didn’t get hired, but it is much more damning when someone gets an offer and chooses not to, based on their experiences in the process. Most people consider multiple options when making their next career move, and they will weigh things that you may have no control over: their commute time, their desire to start their own company. But they will also weigh up things you can control. Their compensation. Their impression of the company. And how recruiters and interviewers treated them.

    No Such Thing

    The tech industry prides itself on its rationality and yet is filled with trite slogans, like “hire the best,” that are demonstrably untrue… and further, harmful. White men declare loudly and proudly that the issue is “unconscious bias” and ignore the depressing data on sexual harassment. They claim to be trying to build more diverse companies and yet all effort focuses on the pipeline, children and university students, while the 56% of women who drop out by mid-career are ignored.

    “We hire the best” they cry. But they don’t. They never will. This mentality justifies the homogenous workplaces, where much is invested in maintaining this facade and propping up the status quo. The system works, and therefore I am here. I am here, and therefore the system works. It is used to justify lengthy processes of interviewing, of unpaid or poorly paid projects. The new trend of unpaid labour in less prestigious roles such as customer support reflects previous trends of demanding Open Source contributions, which is problematic and ineffective. Candidates must prove that they want it enough, prove that they are “the best”, where “the best” sometimes just means the most willing and able to work for free.

    The truth is, that there is no such thing as “the best,” except for the most arbitrary of metrics.

    Perhaps if we could admit that we all hire flawed humans, then we could do the work to create environments where those flawed humans treat each other well. We might not be able to claim “the best,” but we would all be better off.

    Thanks to Julia Evans for reviewing and giving feedback on this article, and Alex Wilson for helping with research.

  • Bad Interviews are a Company Problem, not a Candidate Problem

    Bad Interviews are a Company Problem, not a Candidate Problem

    two raccoons fighting
    Credit: Flickr / Tambako The Jaguar

    We know technical interviewing is a problem but rather than asking interviewers to do better, a lot of suggested solutions push that problem off onto people we interview rather than those who are doing the interviewing.

    This comes up a lot because the hiring process is the second most popular place to improve “diversity” after teaching children to code; the hiring process is the end of the pipeline.

    Bad hiring processes disproportionately affect people who don’t pattern match. Better hiring processes benefit everyone, although people who usually pattern match may disagree. I’ve seen a number of white dudes react angrily when it’s explicitly stated that diversity is valued – they take it a sign that they are unwelcome. The way the use of words like “hackers” and “rockstar”, while calling attention to pool table and the beer keg and amid endless reference to “he” and “guys” have made people who aren’t white dudes like them feel so unwelcome for so long. Except to them “unwelcome” seems to mean “not centred”.

    I have made these points general where I can, but some of them highlight the effects of seeking “diversity” without first understanding inclusion or the broader problems arising in a homogenous, often insular environment.

    Internships Rather than Full-Time Jobs

    Whether hiring new grads or for “returnships”, I understand why companies would like to have a three-month job interview rather than a single-day interview. But consider this: when you onboard a new full-time engineer into a team, they have a manager who (hopefully) has some experience and some training. Interns, on the other hand, tend to be assigned “mentors” who might have gone to some one-hour training sessions where nothing substantial had been covered. If companies are serious about internships like this, they need to set interns up to succeed. A big part of that would be to stop using the word “mentor”, start using the word “manager”, and to train, incentivise, and hold accountable accordingly.

    Take-Home Assignments

    I understand the appeal of take-home assignments from both sides: candidates get time to think and code without being watched, and companies can ask them to solve a bigger problem (and don’t have to allocate engineering time to watching someone solve it). The earlier take-home assignments are in the process, the less I like them. (Some well known companies have people complete an assignment before they’re given the chance to speak to an engineer at all).

    My reservation with take-home assignments is that you end up asking for the most work from the people who you’re least likely to hire. An assignment that may take an average person two hours may take a super star one hour. Someone who really doesn’t have the experience for the role might find themselves spending all weekend on it. (In comparison, at least technical interviews are time-boxed.)

    I worry that companies avoid training and incentivising their engineers to interview candidates well, and that they’re instead reducing engineers having to talk to potential colleagues. If your selection process is riddled with bias (and it is), the solution is to reduce your bias, not shove it onto other people with strategies like “have every $diversePerson who applies do the take-home assignment”.

    The other benefit of technical interviews is that they force some to decide whether a candidate is worth spending the most valuable resource your engineering team has – time. This is powerful from the perspective of the candidate—a take home assignment can feel impersonal. But also because it forces you to consider if it’s worth your spending time to continue the process. In my opinion, there’s a lot to be said for a fast, respectful no. Giving someone a chance that isn’t really a chance is a waste of everyone’s time.

    I want to be very clear about what this means: if you’re confident that someone doesn’t have the skills or experience that you need, it’s better to cut things off sooner rather than later. But where does that confidence come from? The more objective you can be, the better. Is it a vague feeling? That’s likely your bias talking. Or is there a concrete skill that you’ve identified you need, that you have given them the opportunity to demonstrate—or asked specifically about—and the answer was no?

    Whose Job Is It, Anyway?

    A little past the hiring process, but the other thing that I worry about is seeing junior engineers (particularly women) being given the burden of fixing the process that they just went through. Rather than hiring junior women and expecting them to do the thankless emotional labour of fixing your hiring process, first consider why there isn’t a woman who’s senior (and interested) enough that doing so might be part of her actual job.

    The answer might tell you a lot about your culture and what people are rewarded for.

    Competitive Advantage

    People throw around “diversity is a competitive advantage”, but it’s not clear that they know what that means. Here’s the reasoning I embrace: Inclusivity is a competitive advantage because it yields a diversity of ideas and insights at all levels within an organisation.

    A better hiring process is necessary but not sufficient in building an inclusive workplace. The hiring process is actually the least of it. I can accept that interviewing for jobs is inherently awful, and that as an interviewer or hiring manager I may only be able to make it less awful. But what I no longer accept is that being an engineer whilst female is inherently awful.

    So with that caveat, here are some ways that you can create a competitive advantage in hiring:

    • Check and reduce bias (for all interviewers).
    • Be transparent about your hiring process (and for startups, that’s especially true if you’re still defining what that process is).
    • Treat interviewees time with the respect it deserves. (Paying people to interview is one strategy, and it’s a good one.)
    • Be clear about what’s important to you—what kind of person do you need?
      • If you’re hiring to fill fewer, more specialist openings, be very clear about what kind of skills and experience you’re looking for. Importantly, ensure they’re realistic—does that kind of person even exist? And will they work within your constraints of geography and salary?
      • If you’re hiring “smart generalists” this is much fuzzier prospect, so define what you care about. Be mindful of what kind of criteria you use to assess what a “smart generalist” looks like, and what kind of problems you present to evaluate them.

    Structural and Individual

    I used to work at a place where I interviewed mostly women. There was no incentive for me to be a good interviewer, but I worked on that anyway. A bad move for my career at that company. But a good move as a human being. Not a rational decision, then. A moral one.

    Hiring processes are structural and individual. We design a process structurally—influenced by our bias to create a process we’d do well in—but interactions are individual, typically one-on-one. There’s no structure that eliminates the need for one-on-one interactions because we’re hiring humans—not building robots—and because the hiring process is a two-way evaluation where candidates learn about the company while the company learns about them. The people you hire aren’t going to work under the thumb of an automated computer evaluation system—they’re going to work with other human beings.

    If you design a hiring process that ignores the individual side of the equation, you’re just shifting the part of your process that’s broken. And if you consider the structure without the individual, then your theory will fail at the first interaction with someone who operates rationally rather than according to a moral code.

     

    Thanks to Ashley, Camille, Cristina and Jo for reading and giving feedback.

  • A Story

    A Story

    Credit: Flickr / Takashi Hososhima
    Credit: Flickr / Takashi Hososhima

    When I worked at The Conglomerate, I used to interview mostly women. Not slightly more. We’re talking a 2:1 or even 3:1 ratio.

    Why? Well The Conglomerate was (probably still is) a Pipeline Organization. They believed that the problem with diversity was that they just needed to Hire More Women. And so they would want to show these women that other women worked there, and voila: put a woman on every interview slate. This could be a challenge if, for example, there were very few women in an office, and might get even harder if, say, a number of them were harassed to the point where they took stress leave.

    So I did a lot of interviews. I could have said “no” sometimes, done a few less. But firstly saying no would have me characterised as “unhelpful”, or maybe “not a team player” (and I think the Team I was supposed to be on here was Team Female). Secondly, interviewing is stressful, and scary, and if me showing up and doing one sometimes made it a better experience for other women – because they saw someone else like them – it was something I would have a hard time living with myself not doing.

    This was the kind of thing that I characterise as “Thankless Emotional Labour” – if you don’t do it, you’re judged. But if you do… well the only “reward” you might get is more of it.

    Some standout experiences:

    • It’s Friday and I am frantically trying to get everything I need to do for the week, done. But I’m pressured and obligated into dropping everything and doing an interview in the middle of the afternoon, because otherwise the candidate will not encounter another woman that day.
    • I am the last interviewer on the slate. I arrive and the candidate has not been prepared for what to expect, at all. She’s had a terrible experience, is exhausted and has given up, I try and make her feel better and leave feeling wrung out myself, worrying about how traumatized she was by the experience.

    I generally believed that the odds were stacked against women in that system, but I had no power over anything other than the 45 minutes I spent with them and how I wrote my feedback after. So I worked at building a rapport with people, and tried to apply everything I knew about bias to writing up my feedback.

    This was a lot of work, but it was work I did because it was The Right Thing To Do. And something that was shocking to me once I left The Conglomerate – it was work that other people, other companies, appreciated.

    A lot has changed since then, and now I’m a hiring manager. I have control and influence over way more than the 45 minutes to an hour I spend with people. Two big things:

    Edited Jan 23, 2017 to fix GitHub link to point to the new repo.

  • Technical Interview Questions and Time Management

    Technical Interview Questions and Time Management

    danbo finds a lightbulb
    Credit: Flickr / Andrés Nieto Porras

    What Makes a Good Technical Interview Question?

    Three criteria of  a good interview question:

    1. Gives a sense of problem-solving and understanding.
    2. Explorable and extendable.
    3. Deeply understood by the interviewer.

    Problem Solving and Understanding

    What does this look like?

    • The problem presented needs to be decomposed into smaller problems in order to be solved.
    • There are a variety of different answers.

    This criteria eliminates:

    • Knowledge questions and factoids. If a question requires memory of a minor detail that rarely arises in day to day programming, it is a bad question.
    • Answers that are easily searchable, any question where there might be a specific answer on StackOverflow.

    Explorable and Extendable

    Assuming people’s performance on this question falls on a normal distribution:

    • A few people will really struggle to come up with an answer.
    • A few people will quickly produce a perfect answer.
    • Most people will fall somewhere in the middle.

    A good question needs to cater to this range of scenarios.

    One approach we could take here is to ask a huge question up front, and just expect hardly anyone to get to the end of it. I’m not a big fan about this approach because it means almost every candidate leaves berating themselves for not getting to the end of it… that’s not an experience I want to give people.

    Alternatives:

    • Build up.
      • The first question constitutes part of the problem.
      • Then situate it within a bigger problem.
        • Ask how does this change things?
      • … and repeat.
    • Change constraints.
      • The question is manageable in size, candidate comes up with a solution.
      • Interviewer changes…
        • … constraints.
        • … requirements.
      • … and repeat.

    Whichever approach is taken here, some things I like to see are:

    • Reasoning about what changes and what doesn’t as the situation changes.
    • Intentional consideration of when code can be reused… and when it shouldn’t be.

    Deeply Understood by the Interviewer

    It’s easy to think that if two people ask the same question, they give close to the same interview. I don’t believe this is true. Technical interviews are not standardised tests, they are a system involving the interviewer, the candidate, the environment, the programming language, and the question.

    Even when the interviewer, the question, and the environment are constant, with a question of any complexity, it’s going to be different each time, because different candidates will see the problem differently (possibly influenced by programming language, more on which later), and take a different route through it.

    This is why the interviewers understanding of the question is so important. You have to be able to follow fundamentally different implementations, often also in a variety of languages.

    The question is an island the candidate landed on for the first time. The interviewer needs to have a map, to help guide them through.

    This brings us to one of the most important skills of being a good interviewer.

    Interview Time Management

    Think about:

    • How much do you expect a “good” candidate to get through.
    • How long should each section take.

    Goals:

    • Time is spent in places proportional to what you learn from it.
    • The interview keeps moving along.

    What does this mean? When the candidate gets stuck, you need to move them on from whatever it is that they are stuck on. How long you spend on that should be proportional to how important that thing is. Is the mistake a fundamental misunderstanding of the problem? That deserves some time to explore. Is it a typo? Who cares – move on. (NB: the examples below are for remote pair programming).

    Fundamental misunderstanding of the problem:

    • Ask them what they understand the problem to be.
      • If they have misunderstood, explain again.
      • Ideally use a different way to express it than the first time.
      • Write it down.
    • Ask them if their code will work.
    • Suggest a test case that demonstrates the problem.
    • Highlight the section of code where the mistake is.
    • Tell them explicitly what is wrong.
    • Fix it yourself and move on.

    Off by one:

    • “I think you have an off by one”.
    • Suggest a test case that demonstrates the problem.
    • Highlight the section of code where the mistake is.
    • Fix it yourself and move on.

    Typo:

    • “You have a typo, here I’ve got it.”

    On Programming Languages

    It’s important to understand that the programming language used can fundamentally change the problem.

    It’s obvious that solving a problem in Haskell is fundamentally different from solving it in C. But solving a problem in Python is different from solving it in Java and both of these are very different from solving it in C. Languages don’t just differ in syntax but also in the library methods available, the ease of using them (see: sorting numbers in Javascript), and the choice of data-structures available in core libraries.

    This means that you can’t compare progress. If someone writing Python got through 80% of the question and someone writing C got through 70% you can’t conclude that the Python programmer was better than the C programmer. They were doing different things.

    One solution to this is that you constrain everyone to the same language. But if someone is writing code in the language they use every day, and the other person is not, then once again… they are doing different things.

    My solution:

    • People code in whatever language they want.
    • I’m familiar enough with the most common choices (Python / Java / C / Javascript) that I understand how they differ (what’s harder, how problems are generally approached).
    • I ask questions about things I’m unfamiliar with (bonus: I get to learn things).

    Takeaways

    • Finding a good technical interview question is hard. Expect to spend some time on it.
    • Deep understanding of the question is key to interview time management.
    • Two people asking the same question is not the same.
    • Two people answering the same question in different languages is not the same.

    My previous writing on interviews:

  • 12 Challenging Steps to Being a Better Interviewer

    12 Challenging Steps to Being a Better Interviewer

    Notes from my @TheLeadDev talk on interviewing.

    Do you follow the discussion about technical interviews? I find it interesting, and worthwhile. I think we kind of tiptoe around two fundamental problems though. 1. Most technical interviewers are bad. 2. We think that technical interviews teach us more than they do.

    Today we’re going to talk mainly about 1. This talk is about process, not about outcomes. It’s about us as individuals not about broken systems – although that is an important conversation to have. If we follow a process that is respectful to the people we interview, that is mindful of our own biases, we can feel better about the outcomes – even if those outcomes don’t change. Hopefully it’s a process that means those who participate in it feel less bad about. I’m not going to make grandiose claims here because if you don’t get a job you wanted, it sucks. But it sucks that much more if it never seemed like your interviewers were interested in talking to you.

    Essentially I’m going to tell you not to be a jerk.

    OK let’s frame that positively and get specific. We’re going to talk about being empathetic.

    Empathy is something we feel, but we can set ourselves up for empathy, and that comes from knowing ourselves – I think of this as it’s easier to be considerate of other people when my own needs are being met.

    And empathy in this context is kind of useless if we don’t apply it. How do we apply empathy here? Well we understand bias. We can use bias to make people feel better about the process but we don’t want bias to impact our conclusions.

    Know Yourself

    OK so we’re going to talk to someone for an hour. What are our own needs in that context?

    Maybe that we actually have an hour to talk to them. If someone is interviewing for a job, that’s probably one of the most important things they are going to do that day. We can reflect that importance in our own schedule, and allow time before and/or after. A friend goes to sit in a secret garden for 15 minutes before each interview. What a lovely way to get into the right headspace.

    What else might we need? Well we could refer to Maslow’s hierarchy of needs. Sleep. Food. Bathroom breaks etc.

    Maslow's hierarchy of needs: Self-actualisation, Self-esteem, Love, Safety, Physiology and ...WiFi!
    Credit: Flickr / Duncan Hull

    We’ve all heard about being hangry. But, Did you know that lack of sleep affects your morality?

    26 healthy adults, all active-duty military personnel, were presented with a variety of hypothetical dilemmas, first when well rested and later, after staying awake for 53 hours. Questions ranged from things like “is it OK to substitute ingredients in a chocolate brownie recipe?” to complex moral quandaries like whether to let one person die in order to save the lives of several others” [source].

    The headline is “affects morality” but participants did not become less “moral” when sleep deprived. They did require on average two seconds longer to answer complex moral questions. But those questions without a moral component did not take longer to answer after participants were kept awake.

    And do you know the biggest prediction of whether a prisoner will be released on parole? It’s time of day. How recently the judge ate. Although if we get specific, it’s not time, but the number of cases heard since the last break. Other interesting factoids: the average unfavourable decision took less time to arrive at (5.2 minutes) than the average favourable one (7.4 minutes). Also, it took more time to explain, averaging 90 words, compared with just 47 for unfavourable ones [source].

    I feel like techies often like to pretend we’re just one step away from being a computer ourselves, but let’s get real – we’re like other humans. We have the same limitations. It’s worth being aware of what those are.

    Now let’s talk about being in the right headspace. I think this is about believing that the person you are about to interview is a competent person who is capable of doing the job, and that you yourself are capable of evaluating them in a meaningful way.

    Proactively find and deal with things that challenge this. I stopped reading resumes before interviewing for two reasons. The first is that as someone doing an algorithms and data structures interview, the resume was a bias vector that didn’t change the interview in a meaningful way. The second is that a resume is a sales document for a human, so I would read the resume and feel like I was not qualified to interview that person, and it would put me in a bad headspace for talking to them.

    So one thing I do for a couple of startups is I help them with their interview process, including doing technical phone screens for them. One of my clients had a really high no-show rate, and I could feel it affecting me as an interviewer because I was getting stressed and frustrated with how many people weren’t showing up. I tried some things and they didn’t improve, so I ended up explaining to them that I was going to have to start charging a fee for no-shows above a certain percentage. Whilst this was also an effective strategy for motivating them to improve their no-show rate, the main benefit was that it helped me be in a better headspace to interview the people who did show up.

    Be Empathetic

    Let’s talk about being empathetic. Here we are going to talk about a functional kind of empathy, like… a fake it until you make it kind of empathy.

    Step one is about being in a state of warmth. There’s a book called The Charisma Myth (Amazon) which I recommend, but we’re going to do something quickly together to illustrate this idea.

    So in a moment I’m going to ask you to turn to the person next to you and introduce yourself. But first we’re going to make you seem nicer. So if you can, please stand up. We have more energy when we stand up.

    Someone in Florida snapped a picture of a raccoon riding a gator at the Ocala National Forest.
    Credit: Imgur

    Yes that racoon is riding a crocodile or an alligator something scary with lots of teeth anyway. don’t you want to give her a high five? My goal in life is to be as awesome as this raccoon.

    OK now I want you to close your eyes. Take a deep breath. Let it out. Now think about something or someone you love. Your child, partner, parent, pet, best friend, stuffed animal, whatever. If you can’t think of anything… I provide you this raccoon.

    OK so now really focus on this feeling of love, how great whatever you are thinking about is.

    Keep that feeling with you and extend it to the person you speak to as you turn and introduce yourself.

    Do you feel the difference from how you normally introduce yourself?

    Other fake it until you make it strategies include what I think of as basic manners.

    • “How are you today?” (care about the answer)
    • “Is now still a good time to talk?” (mean it)
    • “Oh is that not clear? Let me explain it again.” (be happy to explain it again)

    Let’s talk about this last one for a moment, because it’s important. In any relationship with a power dynamic, the person with the most power has the most impact on the quality of that relationship. Which means in this context the burden of building a good rapport is on you, the interviewer. Your interviewee has not come in determined to hate and misunderstand you. Assume they are doing their best, and be sympathetic to the fact that interviewing is stressful.

    Understand Bias

    Alright the final step, applying our empathy, and this is where understanding bias comes in. There is a lot of discussion about bias lately, but bias is essentially the human condition and sometimes it works for us and sometimes it works against us. If you want to understand how bias can be manipulated to make people feel better about things, look at the way Disney approaches queueing. If you want to understand how bias works against us, look at race relations and policing in America.

    So we want to use bias to make people feel better, but mitigate bias in our conclusions.

    One way we can use bias is that endings are disproportionately important to how people feel about something overall. So, think about that racoon again… “It was so great to speak to you! Thank you so much for making time.”

    Another way to use bias is that people feel better about situations where they have control, so look for places where they can make choices (not too many choices though! That’s overwhelming!). Whether it’s programming language, or approaches to solving it, look for places where you can highlight that they have a choice.

    Now removing bias. For me this is a three pronged approach:

    1. Factoring in time to write up my feedback immediately after the interview.
    2. Keeping detailed notes about the timing of everything so that I’m not going on feelings about how long things took, or what kind of hints I gave – I know.
    3. Taking an extra pass through to remove things like:
      1. Conclusions I cannot support (remember the 2nd problem? Thinking we learn more than we do? This is where that is addressed).
      2. Gendered and/or racially biased feedback. E.g. “lacking confidence” “aggressive”, suggestions that the person should be more junior. Educate yourself on the kind of words we are happier to use about women and minorities and conclusions we are much more ready to draw and make the effort to remove them from your feedback. This in and of itself is a huge topic. But that’s why this is called 12 challenging steps, right?

    Conclusions

    So I want to show you something that I found hilarious. It’s a snippet from an article about Google’s age discrimination law suit.

    ““The interviewer was 10 minutes late to the call, “barely fluent in English,” and “used a speaker phone that did not function well.” Heath politely asked him, repeatedly, to use the phone’s headset but the request was declined.

    Consequently, Heath and the interviewer had difficulty understanding each other. One part of the interview involved writing a short program to find the answer to a problem posed by the interviewer. Heath accomplished the task and offered to share it via Google Docs or email, but, instead, the interviewer required Heath “to read the program coding over the phone.” It did not go well. The interviewer “seemed not to understand” what was being read.””

    I found this hilarious because I expected age discrimination to look different. But actually it looks like a really bad interviewer. Rude, lacking empathy, and resentful of having to do it. What do you learn when your interviewee can’t hear anything you’re saying? I doubt anything meaningful about their ability as an engineer. Maybe how they react to having their time wasted and being set up to fail. I hate to think about the kind of environment where that is useful information.

    So this guy lawyered up, and y’know, good luck to him. I share this with you to emphasise the power we have as interviewers – we can give people terrible experiences and gain little information from it… or we can give people the best experience we can and learn as much as possible – but not more than we think we do. It is hard work, but I hope you have some more ideas about how and why to do that now. For me this is not a one-time learning, but a process that I continually work to live up to.

    I hope when you next interview, your interviewer is this considerate of you.

    Resources List

  • Things You Don’t Learn in Technical Interviews

    Things You Don’t Learn in Technical Interviews

    I can see now...
    Credit: Flickr / Adriane Dizon

    I spend more time than I thought I would thinking about technical interviews lately, because something I’ve been doing is conducting technical interviews for a few startups including Glowforge (they’re hiring!). Since March I’ve done ~36 interviews.

    Designing hiring systems is hard because you only have partial data – specifically you don’t know who your false negatives were. I read the articles about how we need to completely redesign hiring systems and how the technical interview is a terrible thing and think there’s merit to a lot of what is said. But ultimately I think there are two main problems with technical interviews (by which I mean – you write code together): 1) a lot of interviewers are bad, and 2) people think you learn more than you do.

    My previous post on interviewing addressed (1). Let’s talk about (2).

    We should start with the distinction between good questions and bad questions. Bad questions tell you essentially nothing. If you ask someone to code a solved computer science problem on a white board all you learn about is whether they have memorised the solution to that particular problem and can do it in a state of stress.

    A good technical question is one with multiple answers, and tradeoffs. It is a problem large enough that it cannot be solved by a single call to a library function. It requires some thought and discussion.

    Once you are asking a good question you can start to get a sense of:

    • How does someone approach a problem? Do they dive in and start writing code or do they think about things first?
    • How do they break it down?
    • Can they discuss tradeoffs? Consider how different inputs would change?
    • Do they understand what the benefits of different data-structures are? (Different from coding their own version – data structures are foundational and different ones are suitable for different problems).

    But I think the most important question I aim to answer in each interview is: do they listen?

    I don’t think you learn that much from someone who knows the answer, writes it down, and that is pretty much the extent of your communication. Maybe you’ve learned they are pretty smart. That they wrote code when they knew exactly what to do quickly.

    But you don’t know how they will respond to a problem they don’t immediately know the answer to. And unless it’s a job they are going to be incredibly bored in, that’s a pretty important thing to learn about them.

    When they can’t immediately solve something you can start to see:

    • Can they look at their own work objectively and see problems?
    • Do they listen to feedback?

    The other question I aim to get a sense of is: How do they respond to new information? One approach I use for this is “unreasonable product manager”, which means I change the requirements fundamentally. I want to see how they incorporate this information into what they have.

    Those are some things I think you can learn, and to be clear – I don’t always get a sense of all of these in each interview, and I’ll note that accordingly in my feedback. However, there are some glaring omissions that are clearly relevant in the day to day work of software engineering.

    • How do they function in the last 20% of shipping a project (which can be pretty boring)?
    • How do they work on a codebase of significant scope?
    • Can they work with legacy code or will they insist on rewriting it all?
    • Can they prioritise?
    • Do they give thoughtful code reviews?
    • Will they be a good mentor to other developers?
    • Will they take direction? (This is partially covered above, but more in a “may show up if they really won’t” way).
    • Will they adapt to the processes you have? (E.g. code coverage, testing standards).

    It’s important to recognise the limitations of these interviews – as an individual, not just as an organisation – because part of being a good interviewer, of trying to be a good interviewer, is not to give feedback on aspects you really don’t know anything about, and not to delude yourself that you do.