Data Engineer (from Media and Entertainment Industry Background)
Location: 100% Remote
Duration: 10+ Months
No. of Positions: 2
Job Summary:We are seeking a highly skilled Senior Data Engineer to join our dynamic technology team. The ideal candidate will have a strong background in data engineering, data architecture, and cloud-based data solutions, particularly with AWS services.
This role is critical for building and maintaining robust data pipelines, ensuring data quality, and enabling data-driven decision-making across the organization.
Key Responsibilities:Data Pipeline Development: Design, implement, and maintain scalable data pipelines to process large volumes of data using AWS services, ensuring high availability and performance.
Data Processing: Utilize Apache Spark, Databricks, or Snowflake to transform and analyze data from various sources, supporting business analytics and reporting needs.
Data Modeling: Use DBT for data transformation, ensuring that data models are accurate, efficient, and maintainable.
Collaboration: Work closely with data analysts, data scientists, and business stakeholders to understand data needs and deliver data solutions that meet those requirements.Data Quality Assurance: Monitor data quality metrics, identify discrepancies, and implement corrective measures to maintain data integrity and accuracy.
Documentation: Create and maintain detailed documentation of data architectures, pipelines, and workflows for future reference and knowledge transfer.Optimization: Continuously evaluate and optimize data storage, processing, and retrieval strategies for efficiency and scalability.
Troubleshooting: Investigate and resolve data issues in a timely manner, providing support for data-related queries and challenges.
Best Practices: Promote best practices in data engineering, including coding standards, testing methodologies, and data governance.
Must-Have Skills:AWS Services: Strong experience with AWS data services (e.g., S3, Redshift, Glue, Lambda) for data storage, processing, and orchestration.
SQL: Proficient in writing complex SQL queries for data extraction, manipulation, and analysis.
Python: Solid programming skills in Python for data processing, automation, and API integration.
Apache Spark: Hands-on experience with Apache Spark for distributed data processing and analytics.
Data Lakes: Familiarity with Databricks or Snowflake as data warehousing solutions to support analytical workloads.
DBT: Proficiency in using DBT for data transformation and building modular, testable data models.
Soft Skills:Excellent Communication: Ability to convey complex technical concepts to non-technical stakeholders and work collaboratively within cross-functional teams.
Adaptability: Comfortable working in a fast-paced, evolving environment and adapting to changing priorities and technologies.Collaboration: Strong teamwork and interpersonal skills to foster a collaborative work environment.
Analytical Skills: Exceptional analytical and problem-solving skills with a focus on detail, consistency, and quality.
Critical Thinking: Ability to think critically and strategically about data architecture and engineering challenges.
Nice-to-Have Skills:Kafka: Familiarity with Apache Kafka for real-time data streaming and processing.
CI/CD: Experience with Continuous Integration and Continuous Deployment tools and practices to automate the deployment of data solutions.
Airflow: Knowledge of Apache Airflow for orchestrating and scheduling data workflows and pipelines.
Fivetran: Understanding of Fivetran for seamless data integration from various sources to data warehouses.
Qualifications: Bachelors degree in Computer Science, Engineering, Information Technology, or a related field.
[Specify years of experience, e.g., 10+ Years in data engineering, data architecture, or a related field.Experience working with cloud-based data solutions and big data technologies.