From smart speakers to fitness trackers to self-driving cars, the proliferation of digital technologies is creating a wild world of data.
To be valuable, though, that data has to be tamed. That’s where the data engineer steps in.
“There’s literally millions and millions of devices sending thousands and thousands of bytes a minute, or a day, or a second,” said Jason Kolter, an instructor for the Certificate in Big Data Technologies, offered through UW Professional & Continuing Education. “Making sense of all that and getting it to everybody who needs it — that’s the job of the data engineer.”
Engineers Make Big Data Move
Data engineers are software specialists who serve as “high-tech plumbers” for troves of digital data, said Kolter. Data engineers design, build and maintain the behind-the-scenes infrastructure that delivers data reliably and quickly to those who depend on it, whether it be data analysts, upper management or consumers.
A decade ago, Kolter said, data engineers brought together various sources of data in a repository known as a data warehouse to get it ready for analysis. But today, with the volume, velocity and variety of big data, data engineers need more than the traditional data warehouse; they need to be able to support increasingly advanced queries and real-time data analysis. So data engineers have to be well-versed in a wide range of data storage technologies, as well as pros at the code that keeps data constantly moving and available on demand.
“Software is being distributed across many, many machines,” he said. “You need to be very knowledgeable, if not an expert, on how they all talk to each other to be able to get data from A to B successfully.”
Kolter said a typical day for a data engineer includes a lot of working on one’s own, writing code and configuring software that keeps data moving. But data engineers can also expect to collaborate on teams with hardware experts, as well as data scientists, who mine data and translate it into information that’s useful to an organization.
Data Engineers Are In Demand
Throughout Seattle’s ecosystem of data-driven industries — from Amazon to Zillow and all kinds of enterprises in between — demand for data engineers grew by 118 percent between 2015 and 2017, according to data from the labor market analytics company EMSI. It’s a trend that’s expected to stick around: In 2017, LinkedIn ranked big data engineer in the top five of emerging jobs nationally.
Common Job Titles
Data Engineer, Big Data Engineer, Big Data Developer
Median ANNual salary
“Every company, in any industry, can benefit from using data to improve their business,” said Kolter, whose career includes 15 years in software engineering for companies like Cray, Verizon and Northrop Grumman, with over half of that focused on data engineering.
“Data engineer is kind of an underappreciated role,” he said. “It’s not a flashy job. But it’s important. There’s definitely this huge drive to be able to do more intelligent things with data.”
Getting Started in Data Engineering
As more companies move to take advantage of big data, the Certificate in Big Data Technologies prepares students for jobs that require scalable skills to connect multiple systems and meet the need for speed.
The certificate focuses on core data engineering concepts and emphasizes the most popular technologies, most of which are open source and continually improving, Kolter said. Certificate students will learn about a range of leading data processing frameworks, including Hadoop, Hive and Spark.
“There’s always new things coming out,” Kolter said. “Understanding which ones are actually useful, understanding how to use these to solve your problems and improve things, that’s the part of this job that makes it so interesting.”