Broadridge - Lead Data Engineer - Python/Metadata

Role Overview :

We are seeking an experienced and dynamic Lead Data Engineer to join our team. This role is pivotal in driving our data strategy, ensuring optimal data management and governance, and leading the development of scalable data infrastructure. You will be responsible for designing, implementing, and maintaining robust data solutions that empower our organization with actionable insights.

Key Responsibilities :

- Develop and maintain metadata management solutions to provide a unified view of all data assets.

- Implement tools and processes to capture, standardize, and manage metadata effectively.

2. Data Pipelines :

- Design, build, and maintain scalable and robust data pipelines to support data acquisition, transformation, and distribution.

- Optimize data processing performance, reliability, and scalability.

3. Data Provenance and Lineage :

- Ensure data provenance and lineage are accurately tracked, documented, and maintained to enable transparency and traceability.

- Integrate lineage tools and practices within existing systems.

4. Data Quality Controls :

- Develop and enforce data quality frameworks and automated testing protocols to ensure high data integrity and reliability.

- Monitor data quality and address anomalies promptly.

5. Data Management :

- Oversee the management of master data, reference data, and metadata to support business objectives and analytics.

- Implement best practices for data governance and stewardship.

6. Codifying Data Policies :

- Collaborate with stakeholders to define and codify data policies and guidelines aligned with organizational goals and compliance requirements.

- Ensure adherence to data policies and facilitate user awareness and education.

7. Data Lakes and Management :

- Architect and manage data lake solutions that enable flexible data storage and processing capabilities.

- Optimize data lake performance for efficient data retrieval and analytics.

8. Data Catalogs and Products :

- Build and manage data catalogs to improve data discoverability and accessibility across the organization.

- Develop data products that deliver actionable insights and drive business value.

9. Logic to Physical Data Mapping :

- Lead the translation of logical data models into physical database designs that meet performance and scalability requirements.

- Collaborate with application developers and architects to optimize database designs.

Qualifications :

- Bachelor's or masters degree in computer science, Engineering, Information Systems, or related field.

- Extensive experience in data engineering, data management, or a similar role.

- Proficiency in cloud platforms (e.g., AWS, Azure, Google Cloud) and big data technologies (e.g., Hadoop, Spark).

- Strong programming skills in languages like Python, Java, or Scala.

- Expertise in database technologies (e.g., SQL, NoSQL, data warehousing).

- Familiarity with data governance, data cataloging, and data lineage tools.

- Demonstrated experience in leading data engineering teams and projects.

- Excellent problem-solving skills and attention to detail.

- Strong communication and collaboration skills to work effectively with cross-functional teams.

Preferred Skills :

- Knowledge of machine learning and data analytics techniques.

- Understanding of regulatory requirements and data privacy issues.