What You’ll Do
- Design, develop, and maintain scalable data pipelines and workflows to ingest, transform, and store large datasets.
- Collaborate with data scientists, analysts, and software engineers to understand data needs and deliver effective solutions.
- Optimize and enhance existing data processes for performance, scalability, and cost-efficiency.
- Implement data quality checks, validation, and monitoring to ensure data accuracy and reliability.
- Develop and manage data warehouses, databases, and other storage solutions.
- Ensure compliance with data governance and security policies.
- Stay up-to-date with emerging technologies and best practices in data engineering and apply them as appropriate.
An Ideal Candidate Should Have
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
- Proven experience as a Data Engineer or in a similar role and experience with ETL.
- Proficiency in programming languages such as Python and experience in SQL
- Big data tools: Data- and Delta-lakes
- Cloud: Bare-Metal, Hybrid infrastructure
Good to Have
- Experience working with media files (transformations)
- Torch dataset experience
