Job Description

• Design and implement data ingestion, transformation, and movement using Azure Data Factory (ADF), Azure Synapse, and Data Lake.
• Collaborate with business stakeholders, data scientists, and engineers to build robust data solutions.
• Develop and manage high-volume, real-time, and batch data ingestion pipelines using Azure Data Factory (ADF).
• Implement event-driven architectures for real-time data movement and processing.
• Develop large-scale data processing solutions using Azure Databricks and Apache Spark with PySpark, Scala, or Python.
• Optimize data partitioning, caching, and indexing for efficient performance.
• Manage complex transformations and aggregations for structured and unstructured datasets.
• Design and implement high-performance data models in Azure Synapse Analytics using dedicated SQL Pools and Spark Pools.
• Optimize query performance, workload management, and cost efficiency in Synapse Analytics.
• Implement column store indexes, partitioning strategies, and data caching to enhance performance.
• Design and manage secure, scalable data lakes using Azure Data Lake Storage Gen2 (ADLS Gen2).
• Implement RBAC (Role-Based Access Control), encryption, and data masking to ensure security.
• Implement Azure Monitor, Log Analytics, and Application Insights for data pipeline monitoring and troubleshooting.
• Optimize cost management, auto-scaling, and performance tuning across Azure services.

Requirements

• 8+ years of experience in data engineering, ETL development, and cloud-based data integration,
• Strong experience with Azure Data Factory (ADF): ETL/ELT pipeline orchestration and data movement.
• Azure Databricks: Large-scale data transformation and big data processing with Apache Spark, PySpark, Scala, or Python.
• Azure Synapse Analytics: Data warehousing, SQL Pools, Spark Pools, and performance tuning.
• Azure Data Lake Storage Gen2 (ADLS Gen2): Secure, scalable data lake architecture.
• SQL Server: Advanced T-SQL, stored procedures, indexing, and query optimization.
• Power BI, Tableau, or other BI tools for data visualization.
• Financial/Banking experience.

Job Tags

Similar Jobs