Altysys - Senior Data Engineer - Python/SQL

Description :

Senior Data Engineer Python

Location : Bangalore Electronic City

Work Model : Work from Office

Experience : 5 to 8 Years

Position Type : C2H (Contract to Hire)

Notice Period : Immediate Joiners Only

Role Summary :

The Senior Data Engineer will be a cornerstone of our data architecture team, taking a lead role in the design, development, and implementation of enterprise-grade data solutions. This position requires a "code-first" approach to data engineering, utilizing Python to build high-performance, scalable, and reliable data platforms. You will be responsible for transforming complex raw data into refined assets that power critical business intelligence and advanced analytics, ensuring that the underlying architecture is both efficient and future-proof.

Responsibilities :

- Lead the end-to-end design and implementation of distributed data processing systems and complex data pipelines using Python as the primary development language.

- Architect scalable ETL/ELT workflows that can handle massive datasets, ensuring high availability, fault tolerance, and data consistency.

- Develop custom Python-based frameworks for data ingestion, validation, and transformation, moving beyond standard off-the-shelf tools to meet specific platform requirements.

- Design and maintain robust data models (Star, Snowflake, or Data Vault) that support high-performance querying for Business Intelligence tools.

- Optimize data processing performance by refactoring legacy code, tuning database queries, and implementing efficient memory management in Python scripts.

- Implement comprehensive data quality and observability frameworks to proactively monitor pipeline health and ensure the integrity of the data lake/warehouse.

- Collaborate with Data Scientists and BI Analysts to understand complex data requirements and translate them into technical pipeline specifications.

- Conduct rigorous code reviews and provide technical mentorship to junior engineers, enforcing clean code standards and architectural best practices.

- Manage the deployment of data services within containerized environments (Docker/Kubernetes) and integrate them into CI/CD pipelines for automated delivery.

- Engineer secure data access patterns, ensuring compliance with organizational data privacy policies and industry-standard security protocols.

Technical Requirements :

- 5 to 8 years of professional experience in Data Engineering, with a deep mastery of Python for data manipulation (Pandas, NumPy, or PySpark).

- Strong expertise in SQL and deep experience with at least one enterprise-grade RDBMS (Postgres, Oracle) or Cloud Data Warehouse (Snowflake, BigQuery).

- Proven track record of building and managing large-scale distributed systems and high-throughput data pipelines.

- Hands-on experience with workflow orchestration tools, preferably Apache Airflow or Prefect, for managing complex job dependencies.

- Solid understanding of data serialization formats such as Parquet, Avro, and JSON, and their impact on storage and processing efficiency.

- Proficiency in version control systems (Git) and a strong understanding of DevOps principles within a data engineering context.

- Experience with Linux/Unix environments, including shell scripting and system-level performance troubleshooting.

Preferred Skills :

- Experience in Big Data technologies such as Apache Spark, Hadoop, or Hive.

- Familiarity with Cloud infrastructure services (AWS, Azure, or GCP) and their native data components (e.g., S3, Lambda, Redshift).

- Knowledge of NoSQL databases (MongoDB, Cassandra, or DynamoDB) for specific high-velocity data use cases.

- Experience with Infrastructure as Code (Terraform) to automate the provisioning of data platforms.

- Understanding of stream processing concepts using technologies like Apache Kafka or Amazon Kinesis.

- Relevant certifications in Python Development or Cloud Data Engineering are a significant plus.