_{22nd January, 2025}

This is an independent consultant position for contract work with a Kyndryl client and is not a direct staff position with Kyndryl
 
 
Position: AWS Data Consultant
Client: Public Sector
Location: Remote, National Capital Region Preferred
Duration: 9-month initial contract
Security Clearance: Enhanced reliability
Language: English
 
 
Background: 
 
Kyndryl Canada is supporting our public sector client who is working on the design, development, and implementation of a Data Layer, a secure and scalable data integration and management solution.
 
They require an AWS Data Consultant to create a robust architecture to automate data ingestion, transformation, storage, and access using AWS services such as Glue, S3, Athena, Lambda, and API Gateway.
 
 
 
Qualifications: 
 

• Relevant certification in AWS (Preferably relevant data related AWS tools)
• Extensive experience working with AWS Glue, S3, Athena, Lambda
• Degree or Diploma in technology, engineering or project management related field.
• Active Government of Canada security clearance (Mandatory)

• Configure and maintain ingestion pipelines in Amazon Web Services (AWS) Glue
  • Developing and configuring AWS Glue jobs using scripts to automate data extraction from systems of record and loading into S3 buckets (Landing, Raw, Transformed).
  • Improve existing AWS Glue jobs, focusing on performance and cost-efficiency.
  • Validate the functionality of ingestion pipelines to ensure data is pulled correctly.
• Manage Secure Data Connections
  • Setting up connections via API/File Transfer Protocol (FTP)/etc. for data transfer into the S3 buckets and for delivery to users.
  • Ensuring secure data transfer methods are in place (e.g., encryption, secure tokens).
• Monitor and Optimize Data Connections using the Power BI Gateway
  • Regularly reviewing data retrieval queries for efficiency.
  • Optimizing Athena query settings and caching strategies to prevent timeouts.
  • Troubleshooting data connectivity issues with the Power BI Gateway.
• Process Raw Data into Refined Formats
  • Ensuring the data in the Raw bucket is processed into formats like Parquet for optimized storage.
  • Conducting checks to confirm that transformed data meets quality standards.
• Set up and optimize Amazon Athena Views
  • Developing views using Structure Query Language (SQL)-like queries to give users access to the transformed data.
  • Maintaining documentation for all Athena views for future reference.
• Develop and manage AWS Lambda functions and the API Gateway
  • Creating AWS Lambda functions to handle querying the Athena data using SQL and providing it to users in a JavaScript Object Notation (JSON) format.
  • Setting up the API Gateway to expose Lambda functions securely.
  • Validating that APIs function correctly and securely expose data.
• Configure S3 Bucket Policies
  • Establishing and maintaining access controls for Landing, Raw, and Transformed data buckets.
  • Ensuring data is encrypted both at rest and in transit.
  • Setting up data partitioning within S3 buckets.
• Document Processes and Practices
  • Maintaining detailed documentation of ETL processes, data ingestion configurations, and API configurations using Azure DevOps Wiki pages.
  • Creating guides for team members on the API management lifecycle and data catalogue management practices.
• Optimize VPC and Security Configurations
  • Assessing and configuring VPC settings for optimal performance and to ensure low latency.
  • Implementing configurations to maintain persistent connectivity with the Power BI Gateway, preventing connection drops and timeouts.
• Conduct Knowledge Transfer Sessions
  • Organizing knowledge transfer sessions with team members, covering pipeline maintenance, troubleshooting, and usage of AWS data management tools.
  • Creating resources and documentation to support training efforts.