HiLabs - Lead Data Engineer - Python/SQL/ETL

Description :

- Design, develop, and maintain scalable ETL/ELT pipelines for structured and unstructured data.

- Build and optimize data architectures, data lakes, and data warehouses.

- Lead the development of batch and real-time data processing systems.

- Collaborate with cross-functional teams including Data Science, Analytics, and Product teams to understand data requirements.

- Ensure data quality, integrity, and governance across platforms.

- Implement and manage data orchestration workflows (e.g., Airflow).

- Optimize performance of data pipelines and queries for efficiency and scalability.

- Work with cloud platforms like AWS, Azure, or GCP for data infrastructure.

- Mentor and guide junior data engineers and contribute to best practices.

- Drive data security and compliance standards.

Required Skills & Qualifications :

- Strong programming experience in Python, Scala, or Java.

- Hands-on experience with SQL and NoSQL databases.

- Expertise in Big Data technologies such as Hadoop, Spark, Kafka.

- Experience with data warehousing solutions like Snowflake, Redshift, or BigQuery.

- Proficiency in ETL tools and frameworks.

- Strong experience with cloud platforms (AWS/Azure/GCP).

- Familiarity with data modeling techniques and schema design.

- Experience with CI/CD pipelines and version control (Git).

- Solid understanding of data governance and security practices.

Preferred Qualifications :

- Experience in building real-time streaming pipelines.

- Exposure to containerization tools like Docker and Kubernetes.

- Knowledge of DataOps and MLOps practices.

- Prior experience in leading small teams or projects.

Education :

- Bachelors or Masters degree in Computer Science, Information Technology, or related field.

Key Competencies :

- Strong problem-solving and analytical skills

- Leadership and mentoring abilities

- Excellent communication and stakeholder management

- Ability to work in a fast-paced, agile environment

Nice to Have :

- Experience with BI tools (Tableau, Power BI)

- Exposure to AI/ML data pipelines

- Certification in cloud platforms (AWS/Azure/GCP)