Python Developer - Big Data Ecosystem

About the Role :

We are seeking highly skilled Python Developers with extensive hands-on experience in PySpark, Python scripting, and Big Data technologies to join our innovative and fast-paced team. The ideal candidate will be passionate about developing scalable data solutions and automating data processes in complex data environments.

Key Responsibilities :

- Collaborate with data engineers, analysts, and other stakeholders to build robust data pipelines and ETL processes.

- Develop automation scripts to streamline data processing, transformation, and validation.

- Work extensively with Big Data ecosystems including Hadoop, Hive, and Spark to manage and manipulate large datasets.

- Manage and optimize database operations across various platforms such as AWS Redshift, AWS RDS, Oracle, and Azure SQL.

- Monitor and troubleshoot performance issues in data pipelines and workflows.

- Ensure data integrity and security by implementing best practices across all data handling processes.

- Stay up-to-date with the latest trends and technologies in Big Data and cloud platforms to continuously improve data engineering processes.

Required Skills and Qualifications :

- Strong hands-on experience with Big Data ecosystems including Hadoop, Hive, and Spark.

- Proficiency in scripting for automation, data processing, and ETL workflows.

- Experience with cloud-based database platforms such as AWS Redshift, AWS RDS, Oracle, and Azure SQL.

- Solid understanding of data pipeline architecture, ETL processes, and cloud platform services.

- Familiarity with version control tools (e.g., Git) and development best practices.

- Excellent problem-solving skills and ability to work in a collaborative team environment.

- Strong communication skills, both verbal and written.

Preferred Qualifications :

- Knowledge of containerization and orchestration tools such as Docker and Kubernetes.

- Familiarity with data visualization and reporting tools.

Education :