Key Responsibilities :

- Design, develop, and maintain scalable data processing pipelines using PySpark and Apache Spark.

- Build and optimize ETL/ELT workflows to process large volumes of structured and unstructured data.

- Develop complex SQL queries and perform data transformation, cleansing, validation, and aggregation.

- Optimize Spark jobs for performance, scalability, and efficient resource utilization.

- Integrate data from multiple sources, including databases, APIs, data lakes, and enterprise systems.

- Implement data quality checks, error handling, logging, and monitoring for reliable data pipelines.

- Collaborate with data engineers, data architects, analysts, and business teams to deliver data solutions.

- Support production deployments, troubleshoot pipeline failures, and resolve data processing issues.

- Follow coding standards, version control, testing, documentation, and CI/CD best practices.

Required Skillsets :

- Strong hands-on experience in PySpark, Python, SQL, and Apache Spark.

- Experience developing ETL/ELT pipelines and processing large datasets.

- Strong understanding of Spark architecture, DataFrames, RDDs, Spark SQL, joins, partitioning, and caching.

- Experience in performance tuning, debugging, and optimizing distributed data processing jobs.

- Knowledge of data warehousing, data modeling, and data lake concepts.

- Experience with databases such as Oracle, PostgreSQL, or other relational/NoSQL databases.

- Familiarity with cloud platforms such as AWS, Azure, or GCP and their data engineering services.

- Knowledge of workflow orchestration tools such as Airflow and version control tools such as Git.

- Understanding of data quality, security, and governance practices.