Data Architect - ETL/InfluxDB

Description :

Key Responsibilities :

Architecture & Design :

- Design end-to-end data architecture on AWS (batch + real-time)

- Define data ingestion patterns (Kafka/MSK, Kinesis, APIs)

- Build scalable data lake architecture (S3 + Glue + Lake Formation)

- Design time-series and monitoring systems (InfluxDB / Prometheus / Timestream)

Create multi-layered architecture :

- Raw (ingestion)

- Processed (ETL)

- Curated (analytics-ready)

Data Engineering & Pipelines :

Architect and guide implementation of :

- Streaming pipelines (MSK, Kafka, Telegraf, Flink/Spark Streaming)

- Batch pipelines (Glue, EMR, Athena)

- Define schema evolution and data contracts

- Optimize data partitioning, compression, and storage formats (Parquet/ORC)

Real-Time Monitoring & Analytics :

- Design real-time device/event monitoring systems

Integrate :

- Kafka/Time-series DB (InfluxDB/Timestream)

- Prometheus/AMP for alerting

- Grafana/AMG for visualization

- Define alerting strategies (SNS, Alertmanager)

AWS Services Expertise :

Architect solutions using :

- MSK (Kafka)

- S3 (Data Lake)

- Glue (ETL & Catalog)

- Athena/Redshift

- Lambda/Step Functions

- Timestream/DynamoDB (as needed)

- AMP (Managed Prometheus), AMG (Grafana)

- SNS/SQS/EventBridge

Data Governance & Security :

Implement :

- Data encryption (at rest & in transit)

- IAM-based access control

- Lake Formation governance

- Define data quality, lineage, and cataloging strategies

Performance & Cost Optimization :

Optimize :

- Query performance (Athena/Redshift)

- Storage cost (S3 lifecycle, compression)

- Streaming throughput (Kafka partitions, retention)

- Design for high availability and fault tolerance