For a client, we are seeking a Data Engineer with a strong background in big data and experience in developing data pipelines. This is a project-based opportunity with a focus on delivering high-quality data solutions in a dynamic and agile environment.

Responsibilities:

• Develop data pipelines using Databricks and/or PySpark libraries.
• Design and implement unit/component/integration tests.
• Collaborate with other engineers on creating a common codebase using Github SCM.
• Address non-functional requirements such as security, compliance, performance, and maintenance.
• Learn new technologies/tools and provide support within a short time frame.
• Experience in Data Analytic work is considered a plus.

Required Qualifications and Experience:

• Bachelor's degree in statistics, business, economics, mathematics, computer science, engineering, research, or a related field.
• 5+ years of industry experience, with at least 3 years of relevant big data experience.
• Strong programming experience in Python and a good knowledge of PySpark.
• Experience with Hadoop and Big Data processing frameworks (Spark, Hive, Nifi, Spark-Streaming, Flink, etc.).
• Knowledge of SCM Tools – Github, Bitbucket.
• Experience with NoSQL data stores such as HBase, DynamoDB, etc.
• Proficiency in SQL and SQL Analytical functions.
• Experience building domain-driven Microservices.
• Experience working with structured and unstructured datasets, including CSV, TXT, JSON, and XML.
• Experience provisioning RESTful APIs to enable real-time data consumption.

Desired Skills:

• Ability to collaborate with key partners to understand, identify, and capture requirements.
• Ability to work in cross-functional teams following Agile Methodologies (SAFe, SCRUM).
• A flexible, dynamic personality who works well in a matrixed team environment.
• Ability to work independently with minimal direction, execute large projects, and take the initiative to solve problems.
• Strong desire to learn and share knowledge with others.

Language Proficiency:

• Fluency in English is required.

This project offers the opportunity to work with cutting-edge technologies and to make a significant impact in a fast-paced, agile setting. If you have the required skills and experience, we look forward to your application.