Platform Automation Engineer 24-12578
APN Consulting Inc Raritan
APN Consulting has an immediate need for a direct client requirement: Title - Platform Automation Engineer Location: Raritan/Bridgwater/Trenton NJ Contract Remote/Hybrid(1 day onsite) Note - Candidate needs to be 1 day onsite to any of the location (Raritan/Bridgwater/Trenton) About the Role We are seeking a skilled Platform Automation Engineer to design, implement, and maintain automated workflows and scalable solutions for our Databricks and Data Lake environments.
You will play a pivotal role in optimizing data platform performance, ensuring data integrity, and enabling efficient data pipelines through automation. Key Responsibilities Platform Automation & Management: Develop and maintain Infrastructure-as-Code (IaC) solutions for Databricks and cloud-based Data Lakes using tools like Terraform or CloudFormation.
Automate deployment, scaling, and management of Databricks clusters and services. Implement monitoring, logging, and alerting systems for platform performance and uptime. Data Engineering Support: Collaborate with data engineering teams to design and deploy automated ETL pipelines for data ingestion, transformation, and processing.
Optimize Data Lake storage patterns and structures for high performance and cost efficiency. Cloud Integration: Integrate Databricks and Data Lake solutions with cloud platforms (AWS, Azure, GCP Configure secure connections for external data sources and downstream applications.
Security & Compliance: Implement automated security controls, ensuring compliance with organizational and regulatory standards. Conduct regular audits to verify access controls, encryption, and data integrity. Continuous Improvement: Evaluate new tools and technologies to enhance platform efficiency and automation.
Design and implement CI/CD pipelines for Databricks notebooks and data workflows. Required Qualifications Education: Bachelor's degree in Computer Science, Engineering, or related field. Experience: 3+ years of experience in platform automation, infrastructure engineering, or DevOps roles.
Hands-on experience with Databricks, Spark, and large-scale data platforms. Expertise in Data Lake architecture and management (e.g., Delta Lake, Apache Iceberg Proficiency in scripting/programming languages (Python, Scala, Shell Strong knowledge of cloud platforms (AWS S3, Azure Data Lake, GCP BigQuery Preferred Skills Familiarity with orchestration tools like Apache Airflow or Azure Data Factory.
You will play a pivotal role in optimizing data platform performance, ensuring data integrity, and enabling efficient data pipelines through automation. Key Responsibilities Platform Automation & Management: Develop and maintain Infrastructure-as-Code (IaC) solutions for Databricks and cloud-based Data Lakes using tools like Terraform or CloudFormation.
Automate deployment, scaling, and management of Databricks clusters and services. Implement monitoring, logging, and alerting systems for platform performance and uptime. Data Engineering Support: Collaborate with data engineering teams to design and deploy automated ETL pipelines for data ingestion, transformation, and processing.
Optimize Data Lake storage patterns and structures for high performance and cost efficiency. Cloud Integration: Integrate Databricks and Data Lake solutions with cloud platforms (AWS, Azure, GCP Configure secure connections for external data sources and downstream applications.
Security & Compliance: Implement automated security controls, ensuring compliance with organizational and regulatory standards. Conduct regular audits to verify access controls, encryption, and data integrity. Continuous Improvement: Evaluate new tools and technologies to enhance platform efficiency and automation.
Design and implement CI/CD pipelines for Databricks notebooks and data workflows. Required Qualifications Education: Bachelor's degree in Computer Science, Engineering, or related field. Experience: 3+ years of experience in platform automation, infrastructure engineering, or DevOps roles.
Hands-on experience with Databricks, Spark, and large-scale data platforms. Expertise in Data Lake architecture and management (e.g., Delta Lake, Apache Iceberg Proficiency in scripting/programming languages (Python, Scala, Shell Strong knowledge of cloud platforms (AWS S3, Azure Data Lake, GCP BigQuery Preferred Skills Familiarity with orchestration tools like Apache Airflow or Azure Data Factory.
Experience with containerization and Kubernetes for scalable application deployments. Certification in Databricks, AWS, Azure, or GCP.
Berkeley Heights, 13 mi from Raritan (NJ)
Will Need:
Required
• 5+ years of experience as Test Automation Engineer utilizing Harness and Ansible, PowerShell for scripting on Microsoft stack
• Extensive experience working with Product Management, Business Analysts, Developers and Solution...
Intone NetworksNew York, 34 mi from Raritan (NJ)
Sr. SDET Automation Engineer Onsite (Hybrid 1 north castle drive ,Armonk, NY. 10504 - Candidate need to go onsite 3 day in week. Duration : 6 months (possible extension) Need local candidate only Need strong experience. 10+ years' experience...
Synerfac Technical StaffingEatontown, 36 mi from Raritan (NJ)
Manufacturer of Dispensing solutions within the cosmetics industry in Eatontown, NJ is seeking a Direct Hire Manufacturing Automation Engineer.
We have manufacturing facilities in North America, Europe, Asia and South America and over 14,000...