Our Long National Nightmare Is Over

We just concluded a three-month-long marathon of recruiting developers here at TrackAbout. We hired six new developers. This post is a follow-up and summary of lessons learned from this most recent round of hiring. I began writing this in Evernote for my own reference so I could review my lessons learned the next time. I shared it with TrackAbout, and they said, “Blog it!” Who am I to argue?

Please see my post On Hiring Developers for background regarding how our hiring process has evolved and some of the tools we use.

The Statistics

Total Candidates Applied 240
I personally phone screened or interviewed 57
…before starting Codility 42
…after starting Codility 15
Total sent to Codility 34
Disappeared after sending to Codility 14
Passed Codility 14
Had 1st tech Interview 23
Had 2nd tech Interview 11
Extended offers 8
Hired 6

 

Codility

I’m really glad we introduced Codility. It added useful information to the process. Can someone follow directions? Solve a problem elegantly? Is their code readable? Did they name their variables and functions well? Did they optimize for performance? Did they anticipate every edge case?

Some candidates felt the test problems were a bit too “computer science-y”, like puzzles. They felt they weren’t like typical problems you’d have to solve in business application programming.

Other candidates said they enjoy challenges like Codility. They like solving puzzles. Give me a developer who enjoys challenges and solving puzzles for fun any day.

Some candidates realized they weren’t up to the task, and bowed out gracefully. One wrote, “I started through the coding tests, and realized that I really need to improve my chops if I want to work with this caliber of programming … humbling as it is, it’s honest. Thank you so very much for the opportunity and taking the time to talk to me the other day. I really appreciate it.”

Quite a few candidates emailed me after Codility to tell me it was fun and that they were impressed and happy that we ask candidates to actually code before hiring. Many employers hire developers without checking that they can code, crazy as that sounds.

Some candidates realized after the fact that they had screwed up the test. They continued to think about the problems after taking the test, or they did some research and realized their mistake. Several candidates sent me modified solutions via email to correct what they got wrong on the test. It carried weight with me that they cared enough to do so.

It’s unfortunate, but many of the tests in Codility are Google-able. It’s difficult to know whether a candidate cheated or not. I’m not sure if it matters. When faced with a challenging problem, a person should use all the (legal) resources available to prevail.

However, in life and business, the right answer is not always Google-able. The best coding test would be one that is unique, which we define ourselves, and which tests the kind of coding that we require on the job. There’s a company in beta called HackerHires which I’ve been told is working on this, but they haven’t launched yet. Nevertheless, Codility does the job weeding out those people who really can’t program and who aren’t resourceful enough to find an answer in time.

I’m not sure why 14 candidates fell off the face of the Earth after being sent to Codility. It may have been lack of human touch, that I didn’t attempt to speak to them beforehand. It may have been something in the tone of my email, which I edited iteratively until I came up with something I was happy with.

At some point after introducing Codility, I began taking the approach that if I had time in my schedule and the candidate looked especially promising, I’d reach out for a brief phone screen before throwing them to Codility. This wasn’t always possible.

The Resumator

I exercised much more of the functionality of The Resumator this hiring round.

I created several Custom Message Templates once I found myself writing the same messages again and again.

Image

I created some other canned emails which I keep in Evernote for communications outside The Resumator.

I also created a custom workflow in The Resumator so I could more accurately track the status of each candidate.

Image(1)

That last status, “Filled Job Before Contact” is for tagging those candidates who came into the pipeline late. I’ll review their submissions and if they pass muster, reach out to them first when we have open positions in the future.

I created browser bookmarks for filtered views in The Resumator based on the above statuses. This allowed me to more quickly check the candidates in each status.

image

I realized pretty late in the game that I should have created a custom URL for the job posting so The Resumator could automatically track referral sources. I couldn’t always tell when candidates came in from our Careers 2.0 job posting versus cross-posted sources if the candidate didn’t fill in the referral field. The Resumator has a link builder for just this purpose. Next time.

Careers 2.0

I’ve been using Careers 2.0 as my only paid job board when hiring developers. I’ve written about why in a previous blog post.

The Resumator can cross post for free to job boards like FlexJobs, Indeed and SimplyHired. This round of hiring further cemented my impression that candidates who come to us by way of those sources are nowhere near as good as those coming from Careers 2.0. There is the occasional gem, but they’re few and far between.

Paying extra for “Featured” treatment for our Careers 2.0 job posting seemed to boost our traffic immediately following. But it’s difficult to draw a direct correlation over such a short time period. There’s no argument it makes the job listing stand out a bit among the others.

I spoke to the Careers people about other ways they could help us get in front of more developers. They have an advertising model where your job posting appears in the side bar of Stack Overflow questions. However, those ads are targeted based on the user’s location. Our developers are telecommuters and so our job posting doesn’t have a single set location. Careers told me that using the ad model wouldn’t make sense. Kudos to them for not trying to sell it to me anyway. I wasn’t particularly attracted to an ad-based model anyway.

Scheduling Meetings

I miss tungle.me, the old scheduling service we used to use before RIM bought it and tanked it. It worked.

I tried the free edition of Doodle, but many of the appointments that got scheduled using Doodle didn’t contain any identifying information about the candidate. No email, name, phone number, nothing. Usually it’d just be a calendar entry with a generic title and no other info. Perhaps this was due to the free license level, but it didn’t entice me to pay for better.

ScheduleOnce seems to work well enough.

I’ve heard from the developers who have to coordinate three-person technical interviews that they’ve resigned themselves to using email because neither Doodle nor ScheduleOnce is sufficient for coordinating three peoples’ schedules.

Blogging + Hacker News

The On Hiring Developers blog post which I cross posted to Hacker News seemed to change the behavior of the campaign. This is anecdotal, but I feel like the caliber of the candidates who applied improved after I started blogging about our process. Many candidates mentioned the posts, citing them as positively influencing in their desire to work here.

Several candidates stated they’ve read some of our own developers’ independent blogs, or that they follow them on Twitter. Some candidates cited blog posts by developers who they know used to work here. The fact that our developers (past and present) have public personas outside of work seems to work for us. Good devs want to work with good devs, and getting a glimpse into the kind of talent we have helps.

I recall a pretty funny moment in one interview. A candidate was telling me about something he’d read on the blog of this awesome developer he’s been following for ages, and it sounded familiar. I said, “Oh, I know that post. That’s John Sonmez. He works here.” Mind: blown.

Our Corporate Site

I can’t wait for our new site design to go live sometime in the next couple of months. I feel compelled in interviews to work in an apologetic joke about how our corporate site is so very 2005, and the candidates laugh and say, “Yeah…” They know it, we know it, and I am thrilled we’re updating it.

Checking References

I always check a candidate’s references. In the past I’ve been overly deferential in scheduling times to check references. I’d send out emails with a link to a scheduling tool, do the whole, “When’s good for you?” thing. It takes forever, and for what? A 10 minute call?

This time around, I went old school. I just called the candidate’s references trying to catch them when they had 10 minutes to spare. It worked, and I got through the reference checks much more quickly. I guess this is what they did before email.

Trello

I knew I wanted to collect metrics about this round of hiring. I used a Trello board to manage the Codility candidate flow. This allowed me to easily track how many people I sent there, how many went dark, how many passed and how many failed.

We used another Trello board for managing the technical interview process, and it worked really well. It provided transparency and allowed for developers to sign up to interview candidates effectively. I could drag and drop candidate resumes and blog links right into the Trello card.

There were downsides of integrating Trello into the process. In the final analysis, I wanted to gather statistics by counting the various cards in their lists and come up with a spreadsheet that would tell the story of how many candidates got first interviews, second interviews, etc. Trello doesn’t do counting or filtering very well.  Also, in some cases, developers took their names off cards because they didn’t want the notifications, and that meant I had to go through card by card to get the right counts.

Keeping candidate statuses in sync between Trello and The Resumator was also a bit of a chore. I looked for integration possibilities, but it wasn’t feasible.

On Face-to-Face Meetings

Our developers work from home, 100% virtual. That means we don’t meet in person at any point in the interview process. I won’t usually even see the candidates’ faces until after they show up to work.

Part of me feels it’s kind of strange hiring people sight unseen.

On the flip side, these developers are all going to be working remotely, and rarely showing their faces while doing so. What’s important is that they can communicate well and demonstrate their ability to work remotely with members of our team. That is going to be their day-to-day existence for as long as they work here.

I didn’t feel strongly enough it to insist that we conduct a video conference at some point in the interview process. This may be a reflection of my own introverted nature, but for now I’m going to allow myself to focus on the results of the hiring process, that we have an awesome team, and not dwell too much on this one point.

Larry Silverman
Chief Technology Officer
TrackAbout, Inc.