Data Automation Engineer

Key Responsibilities:

·      Design and implement scalable data automation workflows using AWS services with integration to Azure platforms.

·      Develop and maintain ETL/ELT processes to ingest, transform, and move data across DynamoDB, SQL Server, Azure SQL, and other enterprise data sources.

·      Build and support batch and near real-time data pipelines using Apache Spark and streaming technologies such as Kafka or Flume.

·      Develop scalable data processing solutions utilizing Amazon EMR, AWS Fargate, Kubernetes, and related cloud technologies.

·      Integrate REST APIs, external services, and Amazon Connect customer interaction data into analytics platforms.

·      Evaluate and prototype Generative AI solutions using Amazon Bedrock, Azure OpenAI, Hugging Face, LangChain, and related technologies.

·      Implement source control, CI/CD, and automation practices using GitHub, Azure DevOps, Jenkins, Docker, and Agile methodologies.

·      Ensure data solutions meet security and compliance requirements, including IAM, encryption, RBAC, VPC isolation, and firewall policies.

Requirements

Core Qualifications:

·      Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related technical field.

·      5+ years of experience in data engineering, data automation, software engineering, or a related discipline.

·      Strong hands-on experience developing Python- and SQL-based data pipelines in AWS environments.

·      Experience with Apache Spark and at least one streaming/ingestion technology such as Apache Kafka or Apache Flume.

·      Hands-on experience with AWS services including several of the following: S3, AWS Glue, Lambda, EMR, Step Functions, and AWS database services.

·      Experience integrating REST APIs and external services into automated data workflows.

·      Experience working with CI/CD and source control platforms such as GitHub, Azure DevOps, or Jenkins.

·      Strong troubleshooting, performance tuning, and production support experience across data platforms and pipelines.

·      Excellent communication and collaboration skills.

·      U.S. Citizenship and ability to obtain a Federal Public Trust clearance.

Preferred Qualifications:

·      Experience integrating Large Language Models (LLMs) such as Amazon Bedrock, Azure OpenAI, Hugging Face, or similar AI services into production workflows.

·      Experience with Retrieval-Augmented Generation (RAG), embeddings, vector databases, Apache Solr, OpenSearch, pgvector, or similar technologies.

·      Multi-cloud experience supporting both AWS and Azure environments.

·      Experience with Docker, Kubernetes, and containerized application deployments.

·      Experience operationalizing Generative AI solutions, including monitoring, evaluation, observability, and lifecycle management.

·      Familiarity with data governance and cataloging solutions such as Microsoft Purview, Unity Catalog, or AWS Glue Catalog.

·      Experience with Infrastructure as Code tools including Terraform, AWS CloudFormation, or Bicep.

·      Knowledge of FedRAMP, HIPAA, PCI-DSS, or other regulated compliance environments.

·      Relevant cloud, AI, or data engineering certifications.