Experience : 7 - 11 Years

Location : Hyderabad / Chennai / Bangalore

Role & Responsibilities :

- Design, develop, and maintain scalable data pipelines using Azure Databricks, PySpark, and Spark SQL.

- Build and orchestrate complex, parameterized data pipelines using Azure Data Factory (ADF).

- Develop and implement end-to-end data engineering solutions using Azure Data Lakehouse and Medallion Architecture.

- Process structured and unstructured data across batch and streaming environments.

- Optimize Spark jobs, SQL queries, and data pipelines for performance, scalability, and reliability.

- Implement metadata-driven frameworks for data ingestion, transformation, and data quality.

- Develop exception handling, logging, monitoring, and troubleshooting mechanisms for data pipelines.

- Work with Databricks Notebooks, Workflows, Unity Catalog, and Delta Lake.

- Ensure data security, governance, and lifecycle management across the Azure environment.

- Collaborate with data architects, data scientists, BI teams, and other stakeholders to deliver data solutions.

- Take ownership of critical modules, conduct code reviews, and guide team members on best practices.

Required Skills :

- Strong experience in Azure Databricks, PySpark, and Spark SQL.

- Hands-on experience in Azure Data Factory (ADF) and pipeline orchestration.

- Strong understanding of Apache Spark architecture, including RDDs and DataFrames.

- Experience in designing and developing scalable data pipelines and ETL/ELT workflows.

- Strong SQL skills, including complex query development, optimization, and troubleshooting.

- Experience with Azure Data Lakehouse and Medallion Architecture.

- Knowledge of Delta Lake, Parquet, Avro, JSON, and CSV file formats.

- Experience in batch and streaming data processing.

- Strong understanding of data modeling, data warehousing, and dimensional modeling.

- Experience with Databricks Workflows, Unity Catalog, and cloud data platforms.

- Proficiency in Python, PySpark, SQL, debugging, and unit testing.

- Familiarity with Git, CI/CD, and automated deployment practices.

- Strong problem-solving, collaboration, and ownership skills.