Senior Data Engineer

Senior Data Engineer

Requirements

• Deep expertise in the Databricks platform, including Jobs and Workflows, Cluster Management, Catalog Design and Maintenance, Apps, Hive Metastore Management, Network Management, Delta Sharing, Dashboards, and Alerts.
• Proven experience working with big data technologies, i.e., Databricks and Apache Spark.
• Proven experience working with Azure data platform services, including Storage, ADLS Gen2, Azure Functions, Kubernetes.
• Background in cloud platforms and data architectures, such as Corporate DataLake, Medallion Architecture, Metadata Driven Platform, Event-driven architecture.
• Proven experience of ETL/ELT, including Lakehouse, Pipeline Design, Batch/Stream processing.
• Strong working knowledge of programming languages, including Python, SQL, PowerShell, PySpark, Spark SQL.
• Good working knowledge of data warehouse and data mart architectures.
• Good experience in Data Governance, including Unity Catalog, Metadata Management, Data Lineage, Quality Checks, Master Data Management.
• Experience using Azure DevOps to manage tasks and CI/CD deployments within an Agile framework, including utilising Azure Pipelines (YAML), Terraform, and implementing effective release and branching strategies.
• Knowledge of security practices, covering RBAC, Azure Key Vault, Private Endpoints, Identity Management.
• Experience working with relational and non-relational databases and unstructured data.
• Ability to compile accurate and concise technical documentation.
• Strong analytical and problem-solving skills.
• Good interpersonal and communication skills.

Responsibilities

• Design, develop, and maintain a high-performing, secure, and scalable data platform, leveraging Databricks Corporate Lakehouse and Medallion Architectures.
• Utilise our metadata-driven data platform framework combined with advanced cluster management techniques to create and optimise scalable, robust, and efficient data solutions.
• Implement comprehensive logging, monitoring and alerting tools to manage the platform, ensuring resilience and optimal performance are maintained.
• Integrate and transform data from multiple organisational SQL databases and SaaS applications using end-to-end dependency-based data pipelines, to establish an enterprise source of truth.
• Create ETL and ELT processes using Azure Databricks, ensuring audit-ready financial data pipelines and secure data exchange with Databricks Delta Sharing and SQL Warehouse endpoints.
• Ensure compliance with information security standards in our highly regulated financial landscape by implementing Databricks Unity Catalog for governance, data quality monitoring, and ADLS Gen2 encryption for audit compliance.
• Evaluate requirements, create technical design documentation, and work within Agile methodologies to deploy and optimise data workflows, adhering to data platform policies and standards.
• Collaborate with stakeholders to develop data solutions, maintain professional knowledge through continual development, and advocate best practices within a Centre of Excellence.

Preferred Qualifications

• Exposure to Azure Purview, Power BI, and Profisee is an advantage.