The 2016 “Summer Swoon” of Data Science Jobs

I’m a data scientist from SF who relocated to NYC this spring. I prudently spent the prior 8 months scoping & planning, making sure there was a healthy appetite for data scientists in the region. But when I got here it didn’t seem like I was getting the responses to my outreach I had anticipated …

Why is No One Getting Back to Me?

I was a little skeptical the slow-start was attributed to just my own performance and the typical nature of a job search. From what I could tell, the pool of actively open jobs was quite shallow. Eagerly searching for an explanation, I decided to plot the number of data scientist job postings from this year and last year.

The data is from Gary’s Guide which does an excellent job of curating tech job postings in NYC (‘Data Scientist’ used for the search term). This isn’t indicative of all the jobs in NYC and is quite biased given the curation but I’d imagine there would be a similar trend for all data science jobs in NYC and insightful from seasonality perspective at the minimum.

What the Data Shows

Looking at hiring trends from last year, there’s two peaks: the lion’s share of hiring done in the spring, a lull in late summer/early fall, and another upswing just before the holidays – which is typical seasonality.

Now viewing this year, there’s a pick-up in spring just like last year but a stark drop in late March (end of Q1) – the spigot just shut-off for some reason – this lull continued to early August when demand just shot up.

I’m struggling to figure out an explanation for this sudden drop-off after March. It seems a bit too early for a “Summer Swoon”. I suspect the jump in August is from the pent-up demand from the prolonged lull.

Overlaying the charts above, the misalignment of lulls is readily apparent. On a more pleasant note, it seems like the overall supply of open jobs is better year-over-year.

Not all the job postings from the search results were for data science roles so I filtered to only retain titles with “Science” or “Scientist” - looks to be similar trend patterns year-over-year though notice the number of postings is just about halved. Perhaps a result of change in title name, but not actual job content.

I didn’t think a similar trend would follow for the Bay Area since Summer Swoon is more of a New York thing and it’s common for folks to travel out of the city but the data shows there’s a similar trend…Interesting! (only have SF data starting in March)

Hmm… maybe this phenomena is specific to data science and not engineering? Below are plots of LA ‘engineer’ postings versus ‘data scientists’. Conclusively, this seems to be an overall impact on tech… I mean who doesn’t want to hire engineers?

An interesting side-note, the demand for engineers absolutely dwarfs demand for a data scientist (just based off of LA’s graph), not sure if the same relation in magnitude holds in SF or NYC. (only have ‘engineer’ data for LA)

Some Potential Hypotheses

A few potential narratives for this phenomena: i) Forces of a natural hiring cycle ii) Troves of students from Masters, PhD, and Bootcamps iii) Funding environment

I suspect this may have something to do with funding… there’s an interesting story if we plot the funding volume by number of startups who raised Series A, B, or C+. Notice, A’s downward trend while B and C remain resilient throughout the year. (data from MatterMark)

There’s almost a 1-to-1 relationship between the directionality except for late spring through summer and the pivotal points (‘elbows’) nearly move in lock step with one another. It’s difficult to judge trajectory without full information as month-end approaches but Series A volume seems to be on an upward move.

Perhaps the overall downward trend of the Series A volume has led to negative sentiment in the broader VC/tech community consequently having an impact on hiring.

Why is Series A volume impacted and not B & C+? Series A is the riskiest category out of the trio. So if there’s tapering in the funding environment driven by risk-aversion, we’ll see a decrease in Series A volume first before B & C+. The logic being, B & C+ companies are more validated than As.

Another interesting tidbit: SF/Bay Area is hands-down a larger employer to data scientists which we can see by the higher peaks along the ends of chart #5. But consider lower troughs, SF seems to be more susceptible to this funding crunch.

This makes sense! NYC’s ecosystem is made up companies that aren’t purely reliant on VC funding. Hence they seem to be more resilient to the affects of a crunch. Here are some examples of NYC startups hiring during the lull: JOOR (Retail), Tapad (Ad-Tech), Ondeck (FinTech), Bonobos (Retail).

I’m also happy to report that while struggling to find traction when I initially set foot in spring things have changed dramatically since. I’ve been hit with a hailstorm of interested companies since the beginning of August!

Who can this info help? Anyone that’s a participant of the hiring market e.g. job seekers and hiring managers. For job seekers it answers the questions:

When’s the best time to look?
Should I take my first offer or continue pursuit of a better fit?
If you’re looking to switch cities and pound the pavement (like myself), positioning yourself in January seems ideal for either geo.

Analogously, for hiring managers: When’s the best time to expand headcount and make offers?

The period of a lull would be ideal for an employer to seek out and make offers given job seeker’s options are quite limited. Immediately after the market picks-up might also make sense, that way you’ll capture all those job seekers burned out and dissuaded from the lull period.

(i) Was this a result of the nature of hiring cycles? Typically, folks leave in the beginning of the year due to year-end bonuses, etc. This might be the case for folks in finance and/or other industries but in tech, people are more incentivized by their equity stake in a startup. What we see here is fairly abnormal.

(ii) Does it answer the question of supply and demand for data scientists? No. This analysis highlights trends in a broad terms. A far more rigorous analysis is needed to find the total universe of available data scientist jobs versus job seekers in the market in combination with the influx of newcomers. Moreover, Gary’s Guide focuses on a specific sample of employers.

(iii) Rumors have it, the phenomena was very much due to the funding environment. For more information check out this post. It mentions Series A volume has fallen 33% year-over-year though B’s have remained resilient and are continuing to increase. What’s causing this crunch?… I have my suspicions but I’ll save it for another post.

Though the question remains, is this an elimination or delay of the total open jobs available this year?… We shall see!

Noise, there is reposted jobs (about ~20 or so per year) and results are as good as GG’s search relevance algorithm; That said, there’s definitely some noise in the postings by capturing some non-relevant postings but the analysis does a fairly good job of showing the broader hiring trends for tech companies.

** Date of data: week ending 8/26/2016

Ajay Sharma is a Data Scientist with a background in quantitative trading and investments. He can be reached via Github or LinkedIn.