HamburgerMenu
hirist

CloudSufi - Python Architect/Lead - PySpark/Fivetran

CLOUDSUFI
7 - 13 Years
Multiple Locations

Posted on: 26/05/2026

Job Description

About the Role :


Job Title : Python Architect/Lead


Location : Noida(Hybrid)


Experience : 7-12 years


Must have :


- 7+ years of data engineering; at least 3 years working on connector or integration framework development


- Deep Python expertise including PySpark, pyarrow, and an understanding of Spark's execution model (driver vs executor, serialization constraints, partition fan-out)


- Hands-on experience with at least one SaaS ingestion platform Fivetran, Airbyte, Google DTS, AWS Glue connectors, or equivalent at the connector-build level, not just configuration


- Strong understanding of OAuth 2.0 flows (auth code, PKCE, client credentials, JWT), rate limiting strategies (token bucket, leaky bucket, per-endpoint quotas), and incremental sync patterns (cursor, watermark, CDC)


- Experience designing shared connector frameworks reusable auth managers, rate governors, state stores not just per-connector scripts


- Ability to author and own TDDs and PRDs that can be handed to a junior engineer with minimal back-and-forth


Nice to have :


- Prior exposure to Databricks Asset Bundles / Declarative Automation Bundles or Lakeflow pipelines


- Experience with the Databricks Python Data Source API (DBR 15.4 LTS+) extremely rare, so treat practical Spark DSv2 Java/Scala background as equivalent


- GCP DTS or Cloud Data Fusion connector experience


- Knowledge of the specific source systems, particularly Social Ads APIs (Meta, LinkedIn, X) or enterprise SaaS (Salesforce, Oracle)

info-icon

Did you find something suspicious?

Similar jobs that you might be interested in

Loading chat...