Data Engineer

This role involves working within the industry to integrate data engineering solutions into our broader product architecture.

• Collaborate closely with data scientists to prepare datasets for model training, validation, and deployment
• Develop, design, and sustain scalable data pipelines to support dynamic pricing models
• Oversee and optimize ETL (Extract, Transform, Load) processes to assure data reliability and accuracy
• Contribute to best practices and document data engineering processes
• Engage with relevant stakeholders to comprehend data requirements and convert these into technical specifications
• Ensure adherence to data governance and compliance with appropriate data privacy and security regulations
• Integrate data engineering solutions into the broader product architecture in collaboration with the software development team
• Continuously monitor and troubleshoot data workflows to ensure reliable data integration
• Experience with Databricks for big data analytics and artificial intelligence solutions required (Azure preferred, but open to AWS or GCP)
• Proficiency in Apache Spark is required for handling large amounts of data processing tasks
• Familiarity with Cloud Technologies for managing and storing big data
• Comprehensive understanding of Database structures and theories
• Proficiency in Python for writing scripts and automation tasks
• Skill in Algorithm Implementation to solve complex data problems
• Strong analytical abilities for interpreting significant amounts of information
• Proficiency in Apache Hadoop for the processing and storage of big data
• Experience in API Development for integrating different software platforms
• Familiarity with Machine Learning algorithms for predictive data modeling
• Experience with Power BI for creating interactive visualizations and business intelligence capabilities
• Knowledge of Apache Kafka for building real-time data pipelines and streaming apps