Senior Software Engineer - Data Engineering/Migration

Job Profile :

- Data Engineer/ Migration Specialist

Contract Duration :

- 6-7 months (ext)

Mode of work :

- Remote

Working Hours :

- UK Shift

Job Overview :

We are seeking an experienced Data Engineer / Migration Specialist to lead the assessment, planning, execution, and validation of a complex data migration project involving Databricks Unity Catalog and Hive Metastore. The ideal candidate will have deep technical expertise in cloud platforms (Azure), data governance, and hands-on experience with data platform engineering and migrations.

Qualifications & Skills :

- 9-10 years of hands-on experience in data platform engineering or migration projects

- Strong proficiency with Databricks, Unity Catalog, Hive Metastore (HMS), and UCX assessment tools

- Expertise in Azure cloud platform, including Identity and Access Management (IAM) and networking configurations

- Familiarity with BI/reporting tools integration and enforcing security policy governance

- Excellent documentation skills and experience collaborating across cross-functional teams

Good to Have :

- Experience designing cluster policies and managing Unity Catalog-enabled environments

- Knowledge of ACL mapping strategies and data governance best practices

- Ability to create migration dashboards or reports for leadership visibility

Roles & Responsibilities :

HMS Assessment & Migration Planning :

- Execute UCX Assessment Tool runs to evaluate Hive Metastore states, migration complexity, and readiness

- Identify external and managed tables, BI dependencies, and environmental separation requirements

- Prepare and deliver a comprehensive migration plan and finalize a detailed migration runbook

Platform Setup :

- Configure necessary cloud resources : storage accounts, IAM roles, and networking

- Set up Unity Catalog Metastore(s) and assign workspace mappings

- Establish and validate admin roles such as Account Admin, Metastore Admin, and Workspace Admin

Unity Catalog Design & Cluster Enablement :

- Inventory and categorize data assets including catalogs, schemas, tables, views, and files

- Design catalog/schema hierarchy aligned with business units or data domains

- Develop and test ACL/permission migration strategies and validate cluster policies to enable Unity Catalog

Migration Execution :

- Use UCX tools and upgrade notebooks to migrate tables and views from Hive Metastore to Unity Catalog

- Migrate and validate ACLs and permissions using audit scripts

- Update BI tool configurations and modify application connection strings as needed

- Transition and test job cluster policies for post-migration stability and functionality

Post-Migration Validation & Handover :

- Verify data integrity, job executions, BI report accuracy, and access control mechanisms

- Document rollback procedures and highlight any unresolved issues

- Conduct handover workshops and knowledge transfer sessions with operational and support teams