Software Engineer – Data Platform
| Company | Middesk |
|---|---|
| Location | San Francisco, CA, USA, New York, NY, USA |
| Salary | $Not Provided – $Not Provided |
| Type | Full-Time |
| Degrees | |
| Experience Level | Senior |
Requirements
- 5+ years of experience working in a data engineering or backend engineering role
- Strong programming skills in at least one backend language and web application framework
- Knowledge of SQL and experience with RDMS like postgresql, mySQL, etc.
Responsibilities
- Build the batch and streaming data pipelines critical to Middesk’s data infrastructure using Airflow, BigQuery, Dataflow, Dataproc and VertexAI.
- Build reliable, scalable, maintainable, and cost-efficient systems across the stack.
- Build, scale and extend existing web scraping platforms and capabilities used for real-time data acquisition.
- Build, update and maintain our interface layer between our application and data platform stack to improve performance and as we continue to scale our platform and datasets.
Preferred Qualifications
- Fluency with Ruby, Python or Go
- Experience building and designing collections stored on ElasticSearch
- Experience working with Airflow or other orchestration platforms
- Experience working on data pipelines and data streaming with tools like Spark or Kafka
- Experience building machine learning systems
- Experience with Terraform, Datadog, or Kubernetes
- Experience with scraper technologies, including agentic AI