Apache NiFi Engineer

Description :

Role - Apache Nifi Engineer

Location - Dubai

Exp - 5 to 13 years

Notice - 0-30 days

Job Description :

Required Experience :

- 5+ years of hands-on experience with Apache NiFi, including developing, managing, and optimizing complex data flows in production environments.

- Proven experience with Cloudera NiFi (CDP Data Flow) in enterprise environments, including integration with Cloudera Manager.

- Experience migrating NiFi flows across major version upgrades with strong understanding of backward compatibility

- Strong proficiency in Groovy scripting, used for ExecuteScript and InvokeScriptedProcessor processors.

- Solid understanding of SSH and SFTP protocols, including authentication schemes (key-based, password), session negotiation, and file permissions handling in NiFi processors (e.g., ListSFTP, FetchSFTP, PutSFTP)

- Good grasp of data encryption mechanisms, key management, and secure flowfile handling using processors like EncryptContent.

- Experience integrating NiFi with MongoDB, including reading/writing documents via processors like GetMongo, PutMongo, and QueryMongo.

- Experience working with Apache Kafka, including producing and consuming from Kafka topics using NiFi (PublishKafka, ConsumeKafka), and handling schema evolution with Confluent Schema Registry.

- Strong knowledge of Red Hat Enterprise Linux (RHEL) environments, including systemd services, filesystem permissions, log rotation, and resource tuning for JVMbased applications like NiFi.

NiFi-Specific Technical Requirements :

- In-depth knowledge of NiFi flow design principles, including proper use of queues, back pressure, prioritizers, and connection tuning.

- Mastery of controller services, including SSLContextService, DBCPConnectionPool, and RecordReader/RecordWriter services.

- Experience with Record-based processing using Avro, JSON, CSV schemas and Record processors like ConvertRecord, QueryRecord, and LookupRecord.

- Ability to debug and optimize NiFi flows using Data Provenance, bulletins, and log analysis.

- Familiarity with custom processor development in Java/Groovy (optional but preferred).

- Experience setting up secure NiFi clusters, configuring user authentication (LDAP, OIDC), TLS certificates, and access policies.

- Proficiency in parameter contexts, variable registry, and flow versioning using NiFi Registry.

- Understanding of Zero-Master clustering model, node coordination, and site-to-site protocol.

- Experience deploying and monitoring NiFi in high-availability, production-grade environments, including using Prometheus/Grafana or Cloudera Manager for metrics and alerting.

Preferred Qualifications :

- Experience working in regulated or secure environments, with strict data handling and audit requirements.

- Familiarity with DevOps workflows, including version-controlled flow templates (JSON/XML), CI/CD integration for NiFi Registry, and automated deployment strategies.

- Strong written and verbal communication skills, with ability to document flows and onboard other engineers.