Data engineering is one of the highest-paying and most sought-after careers in technology. Depending on the company, data engineering can involve anything from strictly ETL pipeline work to architecting data platforms or building machine learning models. When you see job postings for data engineers, you will notice quickly that the job responsibilities for this role vary a lot between companies.
Why is it so hard to break into data engineering?
Skill-sets
Data engineers need a broad range of skill sets. Some of the common skills are
- Building and maintaining data pipelines
- Writing SQL queries
- Data Warehousing
- Programming (Python, R, etc.)
- Cloud technologies
- Data modeling
Lack of college courses/online curriculum
Given the huge demand for data engineers, it’s surprising that colleges don’t offer more data engineering courses. Colleges have courses for data science and programming, But these courses don’t cover all the skills you need to be a data engineer. Even online platforms don’t have comprehensive courses for data engineers. As a result, most data engineers are self-taught.
Lack of junior data engineer roles
When companies advertise for a data engineer role, they are looking for at least half a dozen skill sets and at least five years of experience. These roles are usually senior roles. There are fewer opportunities for junior data engineers on the market.
Data engineer job descriptions are full of buzzwords, the latest tools, and technology. Most employers expect hands-on experience with these tools and technologies even though most of these can be easily learned on the job.
Where do data engineers come from?
The most common path to a data engineer job is data analyst to data engineer. That’s because data analysts manipulate data, build pipelines, build reports and dashboards, write SQL queries, and are generally knowledgeable in databases.
But because of the variety of skills needed to become a data engineer, software engineers, database administrators, and even data scientists can become data engineers.
How to break into data engineering?
Build projects and share your portfolio on GitHub. The project doesn’t have to involve anything you did at your job. Pick a problem and try to solve it. Get your hands on a public data set, build some data pipeline, do some data transformations, and show your results in reports or dashboards.
Get some data engineering certifications if you can. Certifications are not as valuable as real projects but can distinguish you from other candidates for the job.
Build a solid LinkedIn profile showcasing your skills. Link to your GitHub repos. If you have certifications, remember to include them in your profile.
Read about the company’s hiring practices and prepare accordingly. Does the company give job candidates take-home projects? Do they have a SQL or Python test? If you put in the effort and prepare well, you will land your dream data engineer job.