Senior Data Engineer at Paystack

About the Senior Data Engineer role

• Data engineering at Paystack focuses on building and extending platforms for managing data at scale.
• This involves data ingestion, processing, storage and egress. Data engineers are also responsible for creating and maintaining the infrastructure our data platforms run on.
• Data engineers operate across a diverse tech stack.
• They are expected to be adaptable and drawn to learning new skills and technologies.
• The role requires a proactive individual who can work independently and collaboratively within a remote-first environment, has a strong software engineering background with good experience building and maintaining data pipelines, expertise in Python and experience in streaming technologies.

We’ll trust you to

• Data Pipeline Development: Design, develop, and maintain robust data pipelines using ETL and ELT methodologies to process and integrate data from various sources into a data lake, a central data warehouse, operational data stores, analytical data marts and various application interfaces.
• Streaming Data Processing: Implement and manage real-time data streaming solutions utilising Kafka, Debezium, Kafka Connect.
• Workflow Orchestration: Build, schedule and maintain custom workflows using Apache Airflow to ensure timely and accurate data processing and delivery.
• Database Management: Work with a variety of database technologies, including relational databases (MySQL, PostgreSQL), NoSQL databases (MongoDB) and analytical/big data systems (Redshift, BigQuery, SingleStore).
• Infrastructure as Code: Employ tools like Terraform, Kubernetes, and Helm to manage and provision infrastructure efficiently.
• CI/CD Implementation: Develop and maintain continuous integration and deployment pipelines to streamline development processes.
• Testing and Quality Assurance: Conduct unit and integration testing to ensure high code quality, data integrity and system reliability.
• Collaboration: Engage with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data needs and deliver solutions.
• Documentation: Maintain clear and comprehensive documentation of data processes, workflows and systems.
• Monitoring and Support: Monitoring system performance and addressing faults and failures in production systems as part of an on-call rotation.

You’ll thrive in this role if you have

• Bachelor's degree in Computer Science, Engineering or a related field.
• Proficiency in Python is essential. JavaScript and Scala development experience is advantageous.
• Minimum of 3 years of experience in data engineering roles, with a focus on building and managing data pipelines.
• Minimum of 2 years experience in software and/or application development roles (can be concurrent with data engineering experience)
• Hands-on experience with Kafka, Debezium, and Kafka Connect.
• Proficiency in a data pipeline orchestration tool or suitable workflow orchestration tool like Apache Airflow (preferred), Databricks, Dagster or Airbyte.
• Strong understanding and hands-on experience working with various database technologies, including MySQL, PostgreSQL, MongoDB and Redshift (BigQuery and SingleStore advantageous)
• Experience with Terraform, Kubernetes, and Helm for infrastructure management.
• Solid knowledge of cloud computing concepts, with experience in AWS services being advantageous.
• Ability to write complex SQL queries across different dialects.
• Familiarity with unit and integration testing methodologies.
• Experience in setting up and maintaining CI/CD pipelines.
• Exposure to analytical systems and basic data science tooling. Familiarity with basic machine learning and analytical modelling concepts advantageous.
• Exposure to self-service reporting tools like Tableau, Looker and DOMO.

Soft Skills:

• Good verbal and written communication skills, with the ability to convey complex technical concepts to non-technical stakeholders.
• Demonstrated ability to work collaboratively within a team and across departments.
• Comfortable working in a fast-paced environment with changing priorities, technologies and tooling. Life-long learners will do well here.
• Strong analytical and problem-solving skills.

Benefits

• Competitive compensation package and benefits
• 13th month bonus
• TSG Equity compensation
• Full medical coverage
• Wellbeing stipend
• Generous leave and sabbatical policies
• Hybrid working environment
• Smart, kind colleagues who are invested in your growth.