What I Learned From Reading 8,000 Recruiting Emails
Aline Lerner used to code for a living. Now she works on hiring engineers. Last year, she wrote Lessons from a year’s worth of hiring data, where she showed that, in an engineering resume, pedigree isn’t a particularly valuable signal, whereas typos and grammatical errors are. Below is her next attempt to write a post with lots of numbers and graphs.
If you’ve ever been involved in hiring software engineers, you know how frustrating the process can be. One of the toughest parts is getting candidates’ attention. In an effort to understand what makes engineers respond to some recruiting messages and not others, I teamed up with the fine folks at Hired.
Hired is a marketplace where engineers create profiles and companies submit interview requests to those engineers. Each interview request includes salary information, title, and a personal message. This provides a rich data set comprised of thousands of candidates, hundreds of companies, and thousands of messages between them.
We analyzed a sample of about 8,000 recruiting messages, examining a number of factors including company prestige and size, engineers’ desired salaries, degree of message personalization, and whether messages came from recruiters or from engineers/founders themselves. Our goal was to determine which factors were predictive of an engineer engaging with a company by accepting an initial interview request (aka, introduction rate).
In the process, we discovered that:
- Given the large volume of opportunities available, few candidates find it worthwhile to pursue positions below their specified salary bar
- Message personalization is very important, but not all personalization is created equal
- Left to their own devices, engineers and founders sound just as spammy as recruiters in their attempts to woo new talent
Here’s a graph of all the factors we looked at and their relative importance in predicting introduction rates. These values come from running a logistic regression on the factors that were most statistically significant. The y-axis represents the standardized value of the coefficients in our regression. It’s on a scale of -1 to 1 and represents how heavily each of these factors weighed into predicting whether a candidate was interested in an opportunity. Bars that are below the x axis mean that that factor has a negative effect. Note that all the graphs in this post are interactive, so you can hover and do other fun things.
Mobile users, please click here to view this graph.
These findings require a bit of explanation, so below, I’ve drilled down into a few of the more interesting takeaways.
Pay People What They’re Worth
Of all the different factors that went into whether a candidate was interested in a job, the initial offer had the most measurable impact. The particular metric we used is initial salary offer from an employer divided by a given engineer’s self-reported salary bar. In other words, how well the offer meets the engineer’s expectations. A positive correlation indicates that the more an initial offer exceeds expectations, the more likely it is that s/he will respond affirmatively. No surprise here.
How much more likely? Assuming a preferred salary of $120,000, there’s a 45% introduction rate for interview requests right at that amount. Dropping the offer down to $110,000 results in a 34% introduction rate. Upping it to $130,000 shoots the introduction rate up to 54%. For an engineer with a $120,000 preferred salary, paying $10k more leads to a 20% higher chance of introduction, whereas paying $10k less leads to a 25% lower chance. To put it another way, very few people are willing to take pay cuts of even $10,000. Interestingly, a hypothetical offer of $160,000 would only have a 75% introduction rate, since candidates often decline offers for positions that are too senior. Candidates have a good idea of what they are worth and they are unwilling to sacrifice value. Expecting someone to take a pay cut isn’t reasonable in a competitive market. The following chart shows how this effect has panned out in practice – keep in mind 75% of offers are within 20% of the engineer’s salary bar.
Mobile users, please click here to view this graph.
Once a candidate’s salary bar has been met, they make a decision based on the company’s merits. As you can see below, lack of interest in the company’s value proposition is twice as common as any other reason for not engaging with a company. Salary is an objective metric that encapsulates not only the financial strength of the business behind the offer, but also how much they need your skills. I believe that if there were a way to quantify “how interesting the projects you’ll be working on are to you”, “how great the people you’re working with will be”, “the impact you can make”, and so on, those numbers would easily be as significant as cash.
Mobile users, please click here to view this graph.
It’s worth noting that discussing salary in the initial offer saves time for both the candidate and recruiter. There’s no need to spend time interviewing only to discover a $50,000 difference in expectation.
I’m a recruiter, and the nature of the beast is that I somewhat regularly send out cold emails. I used to be an engineer, however, so I viscerally despise recruiter spam. To prevent cognitive dissonance from melting my brain, I wanted to understand how much personalization is enough. To test how much personalization mattered when it came to introduction rates, I broke up the messages into 3 categories.
1. Not personal at all.This message could have gone to anyone on the platform who met the criteria for a given position and still would have made sense. Example:
This message mentions something about you that’s easily identifiable and maybe ties it back to the company. Example:
and
This message was clearly meant for you, you unique and beautiful snowflake. It might talk about your past work in depth or mention some projects that you would be interested in for very specific reasons or appeal to your specific sensibilities when talking about the company vision. We didn’t redact this particular sender’s info because this message is a shining beacon of all that is good and right in cold emails. Example:
Despite the title of this post, reading almost 8,000 messages and scoring them for how personal they were was clearly intractable. Instead I wrote a little script that hashed messages with an edit distance of less than 60 characters (effectively covering recipient and sender first & last names) to the same bucket and surfaced messages that varied by more than this threshold amount. Those messages I scored manually.
First, here’s the distribution of personalization across all messages (n=7818):
Mobile users, please click here to view this graph.
As you can see, the vast majority of messages were effectively form letters, and it was especially surprising how few genuinely personal messages there were (a whopping 60 out of 7818 or ~0.8%).
So how much did personalization matter? For context, the average introduction rate on the entire platform was about 49.6%. Here’s the breakdown of introduction rate by personalization level:
Mobile users, please click here to view this graph.
What really struck me here was that adding a little bit of personalization didn’t appear to matter at all; in fact, the introduction rates for both impersonal and somewhat personal messages were virtually identical. In other words, the time it takes to drop in the moral equivalent of “Ohai I see you went to Local Good College, well so did our founders, go Local College Sports Team!” or just casually mentioning a candidate’s past employer or projects should probably be spent on something else. On the other hand, truly personal messages had a 73% introduction rate on the platform.
Unfortunately, all told, there were only 60 truly personal messages, probably because taking the time to write something unique is really hard. If you’re not an engineer yourself, knowing what to say can also be really difficult. To that end, I broke up each personalization tier by whether it came from a recruiter or from an engineer/founder. Perhaps not entirely surprising is that more impersonal messages came from recruiters than from engineers and founders (per capita).
Mobile users, please click here to view this graph.
What was surprising is the sheer volume of form-letter like messages that DID come from engineers and founders. In fact, over 85% of the messages that engineers and founders sent out were form letters; this is especially surprising given that those groups are the ones who are often on the receiving end of this kind of barrage.
Based on these findings, it seems like if you know enough about the subject matter, putting in a bit of effort and making messages truly personal is probably worth it. This is quite hard to do in the real world, where info about a candidate can be scattered all over the internets, if it’s even there at all. On a platform like Hired, however, where the candidate goes out of their way to write up a self-summary, talk about their interests, provide a resume, and more often than not, a GitHub profile, and where most of the people doing the hiring are also the ones working on the product and are in a position to talk about projects in a meaningful way, going personal is very likely worth the mental effort.
Who the message comes from
Personalization notwithstanding, I was also curious about whether who the message came from (recruiter, engineer, or founder) mattered when it came to introduction rate. To control for personalization, I focused on just the impersonal messages (n=6827). The breakdown of introduction rates was surprising:
Recruiter: 53%
Founder: 49%
Engineer: 47%
In other words, recruiter introduction rates were the only ones that were statistically significant, and recruiter messages did better than messages from both engineers and founders. I was quite surprised by these results, so I did some sleuthing. First, let’s talk a bit about in what situations recruiters are likely to be the ones sending messages.
As a company grows, founders are less likely to be the ones doing recruiting. Below, you can see the what portion of messages came from recruiters as a function of funding stage:
Mobile users, please click here to view this graph.
Saying that recruiters get more or less introductions overall isn’t truly meaningful because perhaps major corporations might have really strong brands, happen to have more recruiters doing messaging, and can potentially pay higher salaries. When I controlled for these things, the effect largely went away, and introduction rates were pretty much the same across the board, except for some outlier recruiter messaging. Here’s a good example.
What made messages like this particularly compelling was beyond me, but then I looked at non-recruiter messages in the same batch.
So what’s the difference? I think the secret sauce is in the tone. The recruiter message sounds genuinely excited and what it lacks in technical depth, it makes up for in enthusiasm. In other words, this:
The non-recruiter message, on the other hand, is a bit more dry, canned, and tentative. It also doesn’t do a good job of getting across the scope of the opportunity. Perhaps the takeaway here is that, if you can’t go personal, you should at least go enthusiastic. The last thing I want to encourage here is a barrage of doge recruiting, but if push comes to shove, it may be in your best interest to hire good recruiters who know how to hit the right notes. While this kind of recruiting isn’t a substitute for genuinely engaging with your audience, having experience sending out tons of these is going to make you better at it than someone who is uncomfortable with reaching out to strangers and hasn’t done messages of it to overcome their gag reflex.
Conclusion
So, at the end of the day, what is it that makes recruiting messages more sticky? What drives engineers to be interested in certain opportunities and not others? What can you do now to get better at attracting engineers to your company?
- If you can, go deeply personal with your outbound recruiting messages. Talk about what you’re working on, how that ties into what the candidate has already done, and why what you’re doing matters to them. Name dropping and shallow, faux personalization attempts simply don’t cut it.
- Reachouts from founders aren’t intrinsically more valuable, unless they’re personal and targeted.
- Writing really good recruiting messages is hard, and when engineers and founders put on their recruiting hats, they don’t necessarily fare better than recruiters themselves. Good recruiters, on the other hand, are worth a lot. People who do it professionally are going to be able to craft more engaging messaging than those who don’t.
- Trying to underpay good people isn’t going to make you any friends. And, being transparent about salaries out of the gate, assuming those salaries are competitive, may be a good strategy for standing out.
Lastly, I’d like to throw out some acknowledgements. As always, Statwing made the statistical analysis required for this post a delight. If you liked the pretty, interactive graphs, check out Plotly. A huge thank you to Elliot Kulakow of Hired for all his help with the SCIENCE and the pretties. And finally, thanks to everyone who proofread this behemoth.
