Don’t Become a Data Scientist

• Read original article here

OPINION
Don’t Become a Data Scientist
The advice I give when someone asks me how to get into data science. Become a software engineer instead.
May 4 · 6 min read
This is an opinion piece. I’d love to hear your counter arguments below.
Everyone and their grandmother wants to be a data scientist. But while data science may be the sexiest job of the 21 century , that discounts another rewarding and highly paid profession, the software engineer.
I often get messages from new grads and career changers asking me for advice on getting into data science. I tell them to become a software engineer instead.
Having experience in both, I’ll try to convince you to become the latter.
1. There Are More Software Engineering Jobs
There’s an order of magnitude more jobs in software engineering compared to data science.
Below are couple screenshots after googling “data scientist” and “software engineer” jobs.
google: data scientist usa indeed
google: software engineer usa indeed
Thats 7616 data science jobs compared with 53,8893 software engineering jobs. This is just jobs in the US but other countries showed similar results.
According to Glassdoor, data scientists make more money, but my untested hypothesis is that data science jobs are also on average more senior.
That said, if you’re offered a $1 million salary at Open AI , I recommend you take it.
2. There’s No Consensus What “Data Science” Means
Management often doesn’t have a consensus on what “data science” means. It’s also possible that given business constraints, they don’t have the luxury of rigidly following a framework of roles.
This means the responsibilities of a “data scientist” varies a lot from company-to-company.
A quick sketch
While an ideal spectrum of roles between software engineer to data scientist may exist, it‘s unlikely that it’s followed in reality. This particularly goes for scaling startups still building infrastructure.
Hired candidates end up working on the problems a company currently needs solved, rather than the “role” they may have been hired for.
Photo by Marc-Olivier Jodoin on Unsplash
Anecdotal evidence from colleagues in the field is that many data scientists find themselves writing the backend code like software engineers. I’ve known other “data scientists” who crunched financials in excel.
This is in stark contrast to what you’d expect if you grew up on Kaggle competitions.
3. Data Science Is Siloed
Most companies don’t need as many data scientists as software engineers. Other companies are hiring their first data scientist right now.
For this reason, many data scientists end up working alone, even if they sit at the same table as developers.
This can make it difficult to get feedback and second opinions. Software engineers either don’t understand predictive modelling, or are too busy working on completely different problems.
In contrast, one of the perks of being on a software engineering team is being able to say to colleagues, “I think we should implement ABC in XYZ way. What do you think?”.
Be prepared to have that conversation with yourself… or a rubber duck .
4. Data Science Is Exploratory
Be prepared for awkward conversations with management regarding why something you spent 2 weeks on cannot be used.
Working on solved vs un-solved problems is one of the fundamental differences between software development and AI.
Bugs and constraints aside, you know whether most software engineering projects are possible before beginning any work. The same can’t be said about ML where you don’t know if a model will be effective until after you build it.
5. Companies Aren’t Ready for AI
Even in an era when every company is an AI company , most don’t have the infrastructure to support it or even need it.
The Head of Data Science for a rapidly scaling startup recently shared some advice over coffee.
First you figure out the problem, then you build the infrastructure, then you bring in the data scientists. This is not a quick process. (I paraphrase)
Another 1st-data-science-hire at a well-known company recently vented to me. She was forced to train AI models on big data on her laptop rather than in the cloud.
If you’re brought in without specific problems to solve, or the company isn’t prepared for data science, you may find yourself struggling to add value.
6. Software Engineering Teaches Generic Skills
Becoming a junior software engineer is like getting an MBA in technology. You learn a little bit of everything.
You’ll learn databases, cloud technology, deployment, security, and writing clean code.
You’ll learn to manage building software by watching your scrum leader, senior developer or PM.
You’ll get mentorship via code reviews.
If you land in a company with an established engineering team, it’s almost guaranteed that you’ll level up your skills quickly and build a generalist background.
7. Software Engineering Is More Transferrable
By providing a more holistic experience with technology, software engineering provides better exit opportunities when you’ve decided it’s time for a change.
DevOps, Security, Front End, Back End, Distributed Systems, Business Intelligence, Data Engineering, Data Science…
I’ve know a number of developers who moved from software to data science. If you skim data science job descriptions you’ll immediately notice they’re littered with core software development skills.
Source: Indeed (I’m cherry picking here)
If you can build end-to-end projects you can also do more than build a model for Kaggle. You can take that model, productionize it, setup authorization and Stripe, then start charging users for access. That’s your own startup.
I’d never argue that data science isn’t transferrable. Making decisions based on data is a killer skill. But it’s also something that will become more of every job as we become more data driven.
8. Machine Learning Will Become A Tool For Software Engineers
As AI becomes commoditized and easier to use, software engineers will begin using it to solve their problems.
I can teach a developer to build Sklearn classifiers in an afternoon. This doesn’t mean they can build the next AlphaGo, but it does give them an alternative to hard coded conditional logic based on user input.
Data scientists have specialized knowledge like statistics and an intuition for how models work. But DevOps and Security engineers have their own specialized knowledge as well.
I’d argue that these are more common than different. A seasoned software professional could move between specialties an order of magnitude faster than a new entrant could pick one up.
While I don’t think we’ll see a complete merger of data science into software engineering, it does feel like data science could become another software engineering specialty.
9. AI Is Not Replacing Software Engineers
As silly as it sounds, I got into software engineering in 2014 because I worried AI would render every other job obsolete.
Photo by Dominik Scythe on Unsplash
Yet since then, the dial barely moved outside specific environments. Technology adoption is slow and AI is narrower than the media would have you believe.
Compared to other professions, machine learning is even further from automating software engineering. While we have startups building cool products like AI enabled code completion , writing code isn’t the real job. The job is solving problems using technology.
Pre-singularity, that will remain a valuable and highly paid skill.
Conclusion
Firstly, this is anecdotal. Secondly, I realize I conflated data scientists, ML engineers and AI researchers. But I think these arguments are still worth considering, given this is your career.
Don’t take it too seriously. I’d prefer you research and make your own decision. That’s part of being a data scientist after all :).
At the end of the day, we’re paid to solve problems.