We are looking for an experienced Data Engineer (Lead) with strong expertise in Databricks to design, develop, and maintain scalable data pipelines and data processing solutions.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks.
- Develop data processing solutions using PySpark and Python.
- Work with Apache Spark for large-scale data processing.
- Build and optimize ETL/ELT pipelines.
- Work with Delta Lake for data storage and processing.
- Develop and manage workflows using Databricks Workflows.
- Perform data transformation, cleansing, and validation.
- Optimize Spark jobs and improve pipeline performance.
- Collaborate with data engineers, analysts, and other technical teams.
- Troubleshoot data pipeline issues and ensure data quality.
Required Skills
- 3–5 years of experience in Data Engineering.
- Strong hands-on experience with Databricks.
- Good knowledge of PySpark and Python.
- Strong understanding of Apache Spark.
- Experience with Delta Lake and data lake architecture.
- Good knowledge of SQL.
- Experience in building ETL/ELT data pipelines.
- Understanding of cloud platforms such as Azure, AWS, or GCP.
- Experience with Git and CI/CD is an added advantage.
Good to Have
- Experience with Azure Databricks.
- Knowledge of Azure Data Factory, ADLS, or similar cloud data services.
- Experience with DBT, Snowflake, or other modern data platforms.
- Databricks certification is a plus.
What We Offer
- Opportunity to work on real-world Data Engineering and AI projects.
- Exposure to modern data technologies and cloud platforms.
- Collaborative and learning-focused work environment.
- Opportunity to work with a growing technology team.