Big data engineer
Work at the cutting edge of technology and maths and use your analytical skills to solve complex problems.
About the job
What it's like
More than 2.5 quintillion bytes of data are created every day. As a big data engineer, you’ll develop software that collects and analyses large quantities of information. Sometimes, you’ll use Artificial Intelligence (AI) to help you do this.
Businesses can then use this to inform decisions or predict behaviour. It’s how websites like YouTube or Netflix show you recommendations based on things you’ve watched before. They use AI software to tap into banks of data, created by a big data engineer, to learn about your preferences and habits.
You’ll work with organisations to help them develop, maintain, test and evaluate big data solutions. Your work will involve collaborating with data architects, data scientists and analysts. You’ll make sure any solution meets the business’ needs.
Your tasks might include:
creating and maintaining complex computer systems that use AI to collect data
programming software to process high volumes of important information
using AI-powered tools to design mathematical algorithms that simplify data analysis
working with data architects, data scientists and analysts
making sure the data solutions you build meet the business needs or requirements it was intended to
You can do this job across a variety of industries. For example, many big data engineers work in finance, financial technology, IT and healthcare. But there are also opportunities in retail, manufacturing, education and sales and marketing.
Explore more information about this job
Here are some useful links to learn more about this career:
FinTech Scotland – hear from financial technology professionals working in Glasgow
BCS, The Chartered Institute for IT – join the global and diverse home for digital, technical and IT professionals
GOV.UK – read about the UK Government's data science community
Like the sound of this career?
Browse courses in Scotland related to 'Big data engineer'
Select qualification level(s)
Other careers that you might like
-
Systems analyst -
Solution architect -
IT project analyst -
Infrastructure engineer -
Systems engineer -
Data visualisation specialist -
Internet-of-things architect -
Cloud services engineer -
Cyber security architect -
Data architect
Related industries
Many jobs can be done in lots of different industries. We've highlighted the ones we think are most important for this job.
-
Financial and professional services -
Energy -
Digital technologies
Top skills
Skills are things you're good at. Whether you know what yours are or not, everyone has them!
It's useful to learn which ones are important in a job so you know the areas you need to brush up on. It can also help you work out if you're suited to a career.
Meta skills
Here are some of the meta skills you'll need to do this job.
- analysing
- attention to detail
- researching
- working with numbers
- problem solving
- working with technology
Your skills are important
Our unique skillsets are what make us stand out from the crowd. Learn about each skill in depth and discover what employers look for in your applications and interviews.
Getting in
Explore each section to find more information about getting into this career.
Colleges and universities will list subjects you'll need for entry to a course. Some useful subjects include:
-
Business Management
-
Economics
-
Maths (Mathematics)
-
Statistics
-
Foundation Apprenticeship: Software Development
You can get a head start in this career by doing a Foundation Apprenticeship in S5 and S6.
You'll get an SCQF level 6 qualification which is the same level as a Higher. You'll also learn new skills and gain valuable experience in a work environment.
Discover what's on offer at your school on Apprenticeships.scot.
There are many ways to become a big data engineer. You could go to college, university or get work-based qualifications, like an apprenticeship.
University
A degree may be helpful for getting into this job. You could study subjects, such as:
Computing Science
Data Science
Data Engineering
Software Engineering
Artificial Intelligence
Mathematics
Postgraduate data engineering courses are available. A background in a related field such as business, quantitative science or computing is helpful when applying.
Use the My World of Work course search to find degrees.
College
Getting qualifications at college, such as an HNC or HND, could help you secure a job or progress to more advanced study. Useful subjects include:
Computing: Software Development
Computing: Data Science
Computing: Technical Support
Computing: Cyber Security
Apprenticeships
Graduate Apprenticeships (GAs) are designed for industry. You'll spend most of your time learning on the job but you'll also go to uni or college. You'll have a job, get paid and work towards a qualification at the same time.
You might want to consider a GA in Data Science.
Not all employers list specific qualifications they need. They they might ask for relevant experience, usually work-based, that demonstrates a range of transferable skills.
You'll likely need to show up-to-date knowledge and experience of software and programming developments. It's also helpful to have this for many scripting languages.
Find the right course for you
Browse courses in Scotland related to 'Big data engineer'
