Role Overview
Reporting to the Data and Analytics Manager, the Senior Data Engineer is responsible for designing, building, and maintaining scalable data infrastructure and pipelines to support analytics, reporting, and machine learning initiatives. Their main objective is to ensure high-quality data is efficiently extracted, transformed, and loaded, making it readily accessible for analytics, machine learning, and business applications. It plays a critical role in enabling data-driven decision-making by making high-quality data accessible and usable across the organization.
Key Responsibilities
- Design, build, and maintain robust data pipelines and ETL processes.
- Leverage technologies such as Apache Airflow, dbt, Kafka, Spark, and cloud-native ETL tools (e.g., AWS Glue, Azure Data Factory, GCP Dataflow) to ensure reliability and fault tolerance.
- Develop and manage data models and data warehouse architecture.
- Build and maintain modern cloud data warehouse platforms (e.g., Snowflake, Redshift, BigQuery, Azure Synapse) following dimensional modeling and data vault methodologies.
- Ensure data quality, integrity, and governance across all data systems.
- Establish data quality checks, validation rules, and anomaly detection frameworks to maintain high standards of data trustworthiness.
- Contribute to and enforce data governance policies, metadata management, and data lineage documentation to support regulatory compliance and transparency.
- Collaborate with data scientists, analysts, and business units to understand data needs.
- Co-design data solutions that align with stakeholder goals, streamline experimentation, and enable self-service analytics while minimizing data silos.
- Continuously monitor system performance and pipeline health using observability tools (e.g., Datadog, Prometheus, OpenTelemetry).
- Rapidly diagnose and resolve issues in production environments to ensure uninterrupted data flow and availability.
- Implement data security, access controls, and compliance standards.
- Promote best practices in coding standards, CI/CD pipelines, version control (e.g., Git), and DevOps for data infrastructure using tools like Terraform or Kubernetes.
- Stay ahead of the curve by investigating emerging data engineering technologies, open-source frameworks, and cloud-native capabilities.
Qualifications and Experience
- Bachelor’s degree in computer science, engineering, or a related field.
- 4+ years of experience in data engineering.
- Advanced proficiency in SQL and data modeling.
- Strong knowledge of ETL/ELT development and orchestration tools.
- Experience with big data technologies.
- Proficiency in programming languages such as Python, Scala, or Java.
- Hands-on experience with cloud platforms (e.g., AWS, Azure, GCP) for data engineering workloads.
- Understanding of data warehouse and lakehouse architectures.
- Familiarity with DevOps and CI/CD practices for data pipelines.
- Strong grasp of data governance, security, and privacy standards.
- Ability to design scalable and efficient data pipelines for large, complex datasets.
- Ability to interpret business requirements and translate them into data engineering solutions.
- Excellent communication skills for cross-functional collaboration with data scientists, analysts, and business stakeholders.
