Senior Data Engineer – AWS Data Lake & Pipeline Architecture

We are seeking a Senior Data Engineer to support the development of a Data Intelligence Platform. This role focuses on data modeling, data services, data pipelines, and cloud-based data infrastructure for reporting, analytics, and data science.

Responsibilities

• Participate in the architecture design and implementation of high-performance, scalable, and optimized data solutions.
• Create data models from scratch using strong SQL fundamentals.
• Write and optimize in-application SQL statements.
• Ensure the performance, security, and availability of databases.
• Prepare documentation and technical specifications.
• Handle database procedures such as upgrades, backups, recovery, and migration.
• Profile server resource usage and optimize configurations as necessary.
• Design, build, and automate the deployment of data pipelines and applications.
• Integrate data from on-premise databases and external data sources using REST APIs and harvesting tools.
• Collaborate with business units and data science teams on data access, transformation, processing, and reporting needs.
• Support implementation, technical issues, and training related to the data lake ecosystem.
• Work with the team to manage AWS resources, including EMR and ECS clusters.
• Support provisioning, monitoring, configuration, and maintenance of AWS tools.
• Evaluate and promote new cloud technologies that improve capabilities and lower operating costs.
• Support automation efforts using Infrastructure as Code with Terraform and CI/CD tools such as Jenkins.
• Work with the team to implement data governance, access control, and security risk reduction.

Requirements

• 7-9 years of experience designing and developing cloud-based data models, ETL pipelines, and infrastructure.
• Experience working with both structured and unstructured data.
• Strong proficiency with SQL across popular databases.
• Experience optimizing large, complex SQL statements.
• Knowledge of best practices for relational databases.
• Experience configuring database engines and orchestrating clusters.
• Ability to plan resource requirements from high-level specifications.
• Ability to troubleshoot common database issues.
• Experience with Spark, Glue, EMR, and Apache Kafka or AWS Kinesis.
• Experience with version control tools such as Git or Subversion.
• Experience using automated build systems and CI/CD workflows.
• Experience programming in Java, Python, and Scala.
• Knowledge of data structures and algorithms.
• Knowledge of relational, NoSQL, graph, document, key-value, and time-series databases.
• Knowledge of scalable data model design and management.
• Knowledge of ML model deployment.
• Knowledge of AWS cloud platforms.
• Knowledge of TDD and BDD.
• Strong interest in improving software development skills, frameworks, and technologies.

How to Apply https://weworkremotely.com/remote-jobs/toptal-senior-data-engineer-aws-data-lake-pipeline-architecture