This is the second part in a two-part series, the first part is here.

Moishe Lettvin - What I learned doing 250 interviews at Google

I bring this piece up first because it speaks about Google’s process, which sets the stage for the status quo of technical interviews.

Moishe Lettvin used to interview for Google, and at the time of the video, interviewed for Etsy. His perspective is mostly created from the Google way of doing interviews, but he does mention at the end that “Is this the best way? Probably not.” One of his main points is that interviewing is about finding signal vs. noise; determining if a candidate will be productive at your company, fun to work with, and able to both learn and teach others on the team. He admits doing that is hard work, and likens interviewing to literary criticism rather than statistics: lots of subjectivity.

So, how can someone raise the level of the signal?

• Transparency - give candidates as much information about the interview process as possible
• Team effort - multiple people help round out the interview
• Be prepared - organizations should teach their interviewers how to perform the interviews

The quote from this video that I love is that there is “no reason to have an air of mystery in the interview process.” He says that there is no advantage in keeping secrets about the interview process. By being open, it allows you to let candidates know that you’re trying to allow them to show you their skills.

Dan Luu - hiring lemons

Dan Luu doesn’t shy away from talking about workplace problems. A recent post in that vein is called Hiring Lemons, where he discusses and debunks part of Joel Spolsky’s “Finding Great Developers.” It doesn’t so much discuss current hiring practices, but more so makes observations about the experiences of candidates. My notes are mostly interesting quotations.

• In his experience, Luu has found that companies rarely understand the value of their great employees. “When I worked at a small company, we regularly hired great engineers from big companies that were too clueless to know what kind of talent they had.”
• Teams that have openings are more likely to have dysfunction. Teams that are really good have low churn, and thus, rarely have openings.
  • My wife was able to relate to that: She had left a job despite knowing there were some very smart people around her, but the work was not interesting, and there was so much churn that she knew the project would not attract other people that she could learn from.

Matasano - hiring

In Dan Luu’s article, he mentions “Matasano famously solved their hiring problem by using a different set of filters and getting a different set of people.” Googling “Matasano Hiring” takes us to their blogpost about their hiring process. Sections 1 and 2 have some background on hiring and their hiring needs, but the good stuff starts in Section 3.

Section 3

• “Next, interviewers make up their own interviews. This is crazy.”
• “It gets worse. We’re unclear about selection criteria.”

Section 4

• “Engineering teams are not infantry squads. They aren’t selected for their ability to perform under unnatural stress. But that’s what most interview processes demand”

Section 5

• “Confidence bias selects for candidates who are good at interviewing.”
• “For every genuinely competent and effective developer who can ace a tough dev interview, there are many more genuinely competent and effective developers who can’t.”

At this point, you should really just read Sections 6 through 9 for their take on How can we interview better?, but here are my short notes.

Section 6 - Warm up your candidates

• “Not knowing what to expect makes candidates nervous. That’s a pointless handicap.” This echoes the comment from Lettvin above
• “We worked from the assumption that a candidate’s resume, background, and even their previous experience had no bearing on their ability to perform the difficult and specialized work we did.”
  • This one is pretty intriguing. They go on to say that they purchased $80 worth of books and sent them to candidates to study up before the interview.
  • This is complemented by the statement that most candidates did not know Matasano’s domain very well at all when starting the interview process, but this still resulted in great hires

Section 7 - Build work-sample tests

• “Your goal is to collect data you can use for apples-apples comparisons, which means every candidate has to be working on the same problems.”
• “This [work-sample test they wrote] is a couple hundred lines of code, written in a few hours. It out-predicts any interview we’ve ever done.”
• This is a really great section on how you can use work samples to gather data about candidates in a repeatable, objective way.

Section 8 - Standardize and discount interviews

• They talk about having interviewers read from a script, despite knowing they will dislike it.
• “Interviewers hate it. But we kept doing it, because we found that we were comfortable with “why” we were making hire/no-hire decisions when we had facts to look at.”
• “Phone screens carry almost every liability of interviews but further hamstring the candidate and the interviewer by occurring at a distance, being hastily scheduled, and, for the hiring team, having implicitly lower stakes than the in-person interview.”

Section 9 - Ask yourself questions about your interview process

• Nothing to add here, just ask yourself the questions in this section

Example - Slack’s engineering walkthrough

As Lettvin and Matasano explained, there is no advantage in keeping the interview process a secret. Slack has taken that principle to heart and wrote a great blog post in May 2016 about their hiring process. It works as a great template for implementing this at your own company.

What next?

I want to keep adding to the reading list each time I find another interesting article in this space. This is by no means an exhaustive list, just the articles I’ve had time to read and take notes on.

Have another hiring article you love that I missed? Have critical thoughts on these or my analysis? Let me know, I always want to try improving, especially on a subject I know so little about.

This is the first part in a two-part series, the second part is here.

One of my favorite things to read about is technical interviewing; both giving and receiving. I think the last five years have produced a lot of interesting reading material on the subject as smaller companies have been rejecting the Google-style interview marathon, just as Google rejected the 90s-era trivia-style interview.

Now, most of this was just entertainment to me, until two things happened:

1. At the start of 2016, I was looking for a new job; the first time since actually having work experience
2. After joining a growing company, I became part of the hiring and interview process

I’m going to focus on #2 here, since that’s the challenge I’m currently facing and trying to get better at.

So, looking back 10 years, Joel Spolsky makes an interesting observation about being an interviewer:

You are going to be making Major Decisions About [the candidate’s] Life soon

Wow, I better not mess this up; I remember the pressure of being on the other side of the table. Knowing this, when I started interviewing candidates, I knew to put in the effort of actually reading the resume and coming up with relevant questions before the interview. They really care about the result of this interview; I should respect them enough to do that minimal work.

What I’m finding, however, is that taking time to prep for an interview is simply not enough. Did I ask the right questions? Were they relevant to what information I want to get out? Do I even know what information I’m trying to get from the candidate? Were my questions, my expected answers, or even my interpretations of answers influenced by implicit bias?

My current hypothesis is that investing time to improve the entire interview process within an organization will pay off more than investing per-candidate time before interviews. In a good case, this will either reduce the per-candidate time required, or even reduce the number of candidates that need interviewing either through quicker hiring of open positions, or better filtering of candidates early.

So how can I help improve this process? There’s a lot of reading material in this space. My plan is to go through some of these articles, summarize their action items, and see what the overlap is.

The Problem

When running through your test suite on your own machine, everything passes and the code looks like it’s in a great place. As soon as its run by CI, you see the flaky integration tests fail day after day. What gives? I tend to see two situations (sometimes of my own making) that result in these failures.

• Thread.sleep() calls in the test
• A too-short timeout for some part of the test

The reason these problems crop up on CI is that the CI machine’s CPU tends to be fully utilized (or over-utilized). In that case, the timeouts or Thread.sleep() calls tend to not last long enough.

Let’s look at scenarios with Thread.sleep(). These tests need to be refactored, but how does one go about doing that? The first step is always reproducing the failure.

Refactor Part 1 - Reproducing the Failure

I very recently figured out a simple way to reproduce these failures with the help of a few Linux commands (which also work on OS X). To start, I would suggest running in a VM allocated with only 1 CPU. Vagrant is a good tool to help with that.

tool - dd

First, we need a simple way to start eating up CPU cycles. The dd command can be used in a clever way to completely use one CPU.

dd if=/dev/zero of=/dev/null

This tells dd to read from /dev/zero, which just constantly produces zero-bytes, and write them to /dev/null. This essentially does nothing, but can take up 100% of a CPU if nothing else is running.

tool - nice

Now with dd running, you can run your flaky test with a high nice value. nice gives a process (and any of its child processes) a scheduling priority. Zero is the highest priority of non-root processes, and is the default.

To run your process with a lower priority, use a higher number up to 19.

nice -n 1 script-to-run-just-my-test.sh

Put it together

With these tools in hand, we can go ahead and run the test, provoking a failure.

$> dd if=/dev/zero of=/dev/null &
$> nice -n 5 ./run-my-test.sh
...see the failure...
# stop the instance of dd
$> kill %1

Additionally, running multiple instances of dd can help here. There is some trial-and-error with finding the correct value for nice and/or how many instances of dd to run. Watching the test’s process in top or htop to see how much CPU it gets is helpful.

Now that you can reproduce the error, you can determine when a fix has most likely fixed the test. I say “most likely” to really mean that you’ve reduced the occurance of the test failure to some acceptable value, possibly even less than 0.1%.

Refactor Part 2 - Remove the flakiness

Refactoring flaky tests that rely on Thread.sleep() could be an entire series of blog posts. So, I’ll give some high-level advice on this that might get fleshed out in subsequent posts.

• Thread.sleep() is used to wait for something to occur. Figure out a way to recognize when that happens.
• With asynchronous operations, the best case is that some form of onSuccess() or onFailure() callback can be attached. This should remove the need for arbitrary waiting. This can usually be expressed with Java Futures or Guava ListenableFutures.
• Have an asynchronous operation output into some form of synchronized data structure. In the main test thread, do a blocking read from this data structure along with a timeout much longer than you’d expect the test to take. The BlockingQueue interface is useful here, and the Guava helper Queues.drain is great to do the blocking read.
• If nothing else, there should be some value to poll on to wait for the desired output. Here you can still use Thread.sleep(), but many iterations of it with a smaller timeout.
  • There should also still be a maximum timeout to fail the test
  • The maximum timeout should be longer than the timeout when you started fixing the test (since otherwise the flaky test would still be failing), but the average-case test completion time will improve because of the shorter poll duration

This post is just a collection of notes I took during a Meetup about Spark 2.0 performance. The main takeaway was if you don’t have to, don’t use the RDD API directly. The DataFrame and Dataset abstractions offer better performance because of optimizations they can perform. This is true in Spark 1.6 as well as 2.0.

• Project Tungsten is an effort since Spark 1.3 to improve performance in multiple different ways
  • Spark 1.5 introduced the Tungsten Binary Format that keeps data in binary blobs without deserializing it into JVM objects
  • Spark then uses the sun.misc.Unsafe module to directly access the memory and work with the data outside the scope of the GC
• Spark 1.x uses the Volcano Iterator Model (pdf) to compose operations
  • This results in many cache misses on CPU because of virtual function calls
  • This can be improved by removing virtual function calls
• Whole Stage Code Generation (in 2.0)
  • Spark can pack and optimize job logic at job submission time
  • By eliminating virtual function calls, this optimizes CPU in the Volcano model
• Not all operations can be optimized by WSCG. Can Vectorize those that cannot
  • Takes multiple row-oriented records, batches them together, and presents them in columnar format (Parquet)
  • Does not help with virtual function calls, but fewer memory accesses for the data

Here is the full presentation: https://tinyurl.com/markus-spark-2-0