Husk Power Systems - Data Engineer - Python/Scala

Description :

Role Summary :

We are seeking a skilled Data Engineer to design, build, and maintain scalable data pipelines and data platforms that power analytics, reporting, and machine learning initiatives.

The ideal candidate will have strong experience working with large datasets, cloud data ecosystems, and modern data engineering tools.

This role is critical for enabling high-quality, reliable, and accessible data for business decision-making.

Key Responsibilities :

- Design and build robust ETL/ELT pipelines to ingest, process, and transform data from multiple sources.

- Develop batch and streaming data workflows using modern tools and frameworks.

- Automate data workflows to ensure efficiency, reliability, and scalability.

- Work with architects to design scalable data architectures, including data lakes, data warehouses, and lakehouse solutions.

- Develop and maintain logical and physical data models, schemas, and metadata.

- Implement best practices for data partitioning, indexing, and performance optimization.

- Ensure high levels of data quality, integrity, accuracy, and completeness.

- Implement data validation, lineage, and governance best practices.

- Collaborate with data governance teams to enforce compliance and security standards.

- Build and operate data solutions on cloud platforms such as AWS, Azure, or GCP.

- Work with cloud-native services (e.g., AWS Glue, Redshift, EMR, Azure Data Factory, Databricks).

- Manage data storage and compute environments for scalability and cost-efficiency.

- Work closely with data scientists, analysts, BI teams, and application developers to support their data needs.

- Participate in design discussions, sprint planning, and code reviews.

- Troubleshoot data pipeline issues and provide ongoing support.

Required Skills & Experience :

- Bachelors or Masters in Computer Science, Engineering, Information Systems, or related field.

- Strong programming experience in Python, Scala, or Java.

- Expertise in SQL and working with structured/unstructured datasets.

- Hands-on experience with ETL/ELT tools and frameworks (e.g., Apache Airflow, dbt, Kafka, Spark, NiFi).

- Proficiency with cloud data platforms :

1. AWS (Glue, Redshift, EMR, Athena, Lambda, S3)

2. Azure (ADF, Databricks, Synapse)

3. GCP (BigQuery, Dataflow, Dataproc)

- Experience with data warehousing technologies (Snowflake, Redshift, BigQuery, Synapse).

- Familiarity with containerization (Docker) and orchestration (Kubernetes).

- Experience working with version control (Git) and CI/CD pipelines