Azure Databricks Engineer/Technical Lead - Apache Spark/Informatica

Job Role : Senior Data Bricks Engineer

Type : Contract to hire (6 months)

Location : Work from Office (BLR/HYD)

Experience : Total 8+ Years (6+ years (Databricks, AWS) 4+ years (Apache spark, Informatica))

Job Title : Senior Databricks Engineer / Technical Lead

Job Overview :

Key Responsibilities :

- Architecture & Design: Develop scalable data pipelines and Lakehouse architectures.

- Data Engineering: Lead ingestion and transformation processes for both batch and streaming data.

- Performance Optimization: Troubleshoot and optimize resource usage for high-performance

workloads.

- Security & Compliance: Apply best practices for access control, data governance, and regulatory compliance.

- Collaboration: Work with cross-functional teams, including data engineers, analysts, and business stakeholders.

- Cloud Integration: Manage and operate Databricks environments on AWS (preferred), Azure, or GCP.

- Monitoring & Automation: Set up automated workflows and monitoring tools to enhance operational efficiency.

Qualifications :

- 4+ years of hands-on experience with Apache Spark and Informatica.

- Strong understanding of distributed systems, data warehousing, and data lake architecture.

- Proficiency in SQL, Python, and Spark-based data processing.

- Experience with job orchestration tools like Airflow or similar is a plus.

- Excellent problem-solving and leadership skills.

- Strong verbal and written communication abilities.