Software Engineer – Data Acquisition
| Company | OpenAI |
|---|---|
| Location | San Francisco, CA, USA |
| Salary | $325000 – $405000 |
| Type | Full-Time |
| Degrees | Bachelor’s, Master’s, PhD |
| Experience Level | Mid Level, Senior |
Requirements
- BS/MS/PhD in Computer Science or a related field
- 4+ years of industry experience in software development
- Strong expertise in large stateful distributed systems and data processing
- Proficiency in Kubernetes, and Infrastructure-as-Code concepts
- Ability to handle multiple tasks and adapt to changing priorities
- Strong communication skills, both written and verbal
Responsibilities
- Own and lead engineering projects in the area of data acquisition including web crawling, data ingestion, and search
- Collaborate with other sub-teams, such as Data Processing, Architecture, and Scaling, to ensure smooth data flow and system operability
- Work closely with the legal team to handle any compliance or data privacy-related matters
- Develop and deploy highly scalable distributed systems capable of handling petabytes of data
- Architect and implement algorithms for data indexing and search capabilities
- Build and maintain backend services for data storage, including work with key-value databases and synchronization
- Deploy solutions in a Kubernetes Infrastructure-as-Code environment and perform routine system checks
- Conduct and analyze experiments on data to provide insights into system performance
Preferred Qualifications
- Experience with large web crawlers a plus
- Willingness and enthusiasm for trying new approaches and technologies