Data Engineer

We are seeking a highly skilled and experienced Data Engineer to join our growing data team. This role will be pivotal in building and maintaining our data infrastructure, focusing on leveraging Azure cloud technologies and Snowflake. The ideal candidate will have at least 5 years of experience in data engineering, with a strong background in Azure Data Factory, Azure Synapse, Snowflake, Python, Java, and CI/CD pipelines.

Responsibilities:

Data Pipeline Development and Management:

• Design, develop, and maintain robust and scalable data pipelines using Azure Data Factory and Azure Synapse.
• Implement data ingestion, transformation, and loading (ETL/ELT) processes from various data sources into Snowflake.
• Optimize data pipelines for performance, reliability, and cost-effectiveness.
• Monitor and troubleshoot data pipeline issues.

Data Warehousing and Modeling:

• Design and implement data models in Snowflake to support business intelligence and analytics needs.
• Optimize Snowflake performance through query tuning and data partitioning.
• Ensure data quality and consistency within the data warehouse.

Software Development and Automation:

• Develop and maintain data processing applications using Python and Java.
• Automate data engineering tasks using scripting and orchestration tools.
• Implement and maintain CI/CD pipelines for data pipeline deployments.
• Develop and maintain infrastructure as code.

Cloud Platform Management:

• Manage and optimize Azure data services, including Data Factory, Synapse, and storage.
• Implement and maintain data security and access controls in Azure and Snowflake.
• Monitor cloud resource utilization and optimize costs.

Collaboration and Communication:

• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements.
• Communicate effectively with technical and non-technical audiences.
• Participate in code reviews and knowledge sharing sessions.

Qualifications:

• Bachelor's degree in Computer Science, Data Engineering, or a related field (or equivalent experience).
• Minimum of 5 years of experience in data engineering.
• Extensive experience with Azure Data Factory and Azure Synapse.
• Strong proficiency in Snowflake data warehousing.
• Proficiency in Python and Java programming.
• Solid understanding of CI/CD pipelines and DevOps practices.
• Experience with data modeling and data warehousing concepts.
• Strong understanding of cloud computing concepts and technologies.
• Excellent problem-solving and analytical skills.
• Strong communication and interpersonal skills.

Preferred Skills:

• Experience with data streaming technologies (e.g., Kafka, Azure Event Hubs).
• Experience with data governance and data quality tools.
• Experience with infrastructure as code tools (e.g. Terraform).
• Azure certifications (e.g., Azure Data Engineer Associate).
• Experience with databricks.