Skip to content
Data Platform Engineer
| Company | Disqo |
|---|
| Location | Los Angeles, CA, USA |
|---|
| Salary | $160000 – $180000 |
|---|
| Type | Full-Time |
|---|
| Degrees | |
|---|
| Experience Level | Senior |
|---|
Requirements
- At least 5+ years of experience in Data Platform owning and enabling capabilities within an enterprise data ecosystem including best Data Management practices.
- At least 3+ years of experience as SRE, DevOps or equivalent engineering roles
- At least 3+ years of production experience with AWS
- Strong hands on experience managing infrastructure and configuration of Data Platforms in the Cloud – Data Lake, Data Warehouse, Data Mart, Graph Database, Time Series Database, Object Store, etc.
- Strong experience and deep knowledge in the Hadoop ecosystem including performance tuning and operational efficiency.
- Strong experience in Spark with Python or Scala
- Strong hands-on experience in Infrastructure-as-Code and Configuration management tools and technologies: Terraform, Helm, Strimzi, AWS CloudFormation, Packer, Chef, Puppet, Ansible, etc.
- Strong experience in CI/CD tools like Jenkins
- Hands-on experience with observability systems like New Relic, Prometheus/Grafana
- Deep knowledge in various ETL/ELT tools and concepts, data modeling, SQL, query performance optimization
- Experience with building real-time processing applications using Spark Stream and Kinesis/Kafka
- Experience with workflow management tools (Airflow, Oozie, Azkaban, Luigi, etc.)
- Comfortable working in Linux environment
- Ability to thrive in an agile, entrepreneurial start-up environment
Responsibilities
- Leverage your data engineering and infrastructure skills to impact our business by taking ownership of key projects requiring environment build and data pipelines
- Collaborate with product managers, software engineers and data engineers to design, implement, and deliver successful data solutions.
- Identify, design and implement internal process improvements including re-designing Data/Observability infrastructure for greater scalability, optimizing data delivery, and automating manual processes.
- Design, build and optimize performant databases, data models, integrations and ETL pipelines in RDBMS and NoSQL environments
- Play a critical role in designing and evolving our data infrastructure on AWS, orchestrating Spark-based pipelines in Kubernetes (EKS), and enabling our data engineering and analytics teams to deliver insights with confidence and speed.
- Proven expertise in Spark and managing Spark workloads on Kubernetes (EKS).
- Use your Big Data skills to help optimize poor performing SQLs for the business
- Maintain detailed documentation of your work and changes to support data quality and governance.
- Participate in an on-call rotation to provide operational support for critical data platform services and data pipeline jobs.
- Ensure high operational efficiency and quality of your solutions to meet SLAs and support commitment to the customers
- Be an engaged participant and advocate of agile/scrum practices to ensure health and process improvements for your team
- Define best practices around making our systems and services measurable and work with our various teams to get those best practices applied
- Collaborate with other Engineering teams to ensure our Jobs are emitting the right metrics
- Collect, aggregate and visualize the collected metrics to provide actionable insight
Preferred Qualifications
No preferred qualifications provided.
