Principal Engineer – Data and AI Platforms

Principal Engineer – Data and AI Platforms

Requirements

• 8+ years of hands-on experience in Data Engineering, DataOps, MLOps, cloud architecture, DevOps, or related fields, with a consistent record of delivering large-scale enterprise data and ML/AI solutions.
• 5+ years of experience with CI/CD pipelines, IaC tools, configuration management tools, containerization technologies, and infrastructure automation, with a focus on data and ML/AI workloads.
• 5+ years of experience with at least one major cloud platform (e.g., AWS, Azure, GCP) and its suite of products, including data and ML/AI services.
• 4+ years of experience with data platforms like Snowflake, Databricks, or similar technologies.
• Bachelor’s degree in Computer Science, Engineering, Business, or a related field.

Responsibilities

• Define and champion the vision for a modern, scalable, and secure data platform, incorporating cloud-native technologies and drive adoption of best practices in data modeling, data warehousing, and data lake architecture.
• Champion effective Data Platform principles and practices, ensuring automated data pipelines, CI/CD, and robust monitoring for efficient operations. Optimize data ingestion, transformation, and serving layers.
• Design and implement scalable Data and ML/AI platforms to accelerate the development, deployment, and management of ML/AI solutions in production. Ensure the data infrastructure supports the full ML/AI lifecycle.
• Optimize data infrastructure, applying concepts such as FinOps, Infrastructure as a Code (IaC) for data and ML/AI infrastructure, promoting automation and CI/CD. Lead the governance of data architecture.
• Lead cross-functional collaboration between data scientists, platform engineers, data engineers, operations, security and business teams to ensure alignment and effective data delivery.
• Keep abreast of the latest data and AI technologies, proactively identifying and evaluating new tools and methodologies to improve efficiency and performance of the data platform.
• Partner with engineering and product development teams to understand their needs and translate requirements into the relevant data and ML/AI workflows.
• Lead the selection, integration, and optimization of DevOps tools and technologies specific to data and ML/AI, including CI/CD platforms (e.g., Jenkins, GitLab CI/CD) with integrations for data and ML/AI workflows, Infrastructure as Code (IaC) tools (e.g., Terraform) for provisioning and handling data and ML/AI infrastructure, and Containerization technologies (e.g., Docker, Kubernetes) for deploying and scaling data and ML/AI applications.
• Establish and enforce architectural standards, guidelines, best practices, and frameworks specific to data workloads, ensuring consistency and quality across all projects and systems.
• Promote a culture of automation, continuous integration, continuous delivery, and robust path to production across our data platforms. Drive the implementation of DataOps and MLOps methodologies.
• Foster a culture of collaboration and shared responsibility between data scientists, platform engineers, data engineers, operations, and security teams, promoting effective communication and knowledge sharing.
• Effectively communicate technical concepts and solutions to both technical and non-technical audiences, including business stakeholders and senior management.
• Stay up-to-date with the latest DevOps trends and technologies relevant to data and ML/AI, proactively proposing, evaluating and designing proof-of-concepts on new tools and methodologies to improve efficiency and performance.
• Drive continuous improvement initiatives to optimize DataOps and MLOps processes, enhance automation, and reduce technical debt within the data platforms.
• Mentor and coach other engineers on architectural and engineering best practices specific to data and AI workloads.

Preferred Qualifications

• Proven ability to establish standard processes and lead multi-functional teams, particularly within data and AI domains.
• Strong understanding of data management principles, data governance, and data security standard methodologies.
• Strong understanding of software development lifecycles and project delivery methodologies (Agile, Waterfall, etc.).
• Experience working collaboratively with third-party vendors, as well as data scientists, ML engineers, software engineers, product development, and cloud operations teams.
• Excellent communication, interpersonal, and leadership skills with the ability to influence and mentor others.
• Strong problem-solving and troubleshooting skills in technical environments.
• Ability to work in a fast-paced, multi-functional team environment.