Azure Databricks Developer

About the role

We use cookies to offer you the best possible website experience. Your cookie preferences will be stored in your browser’s local storage. This includes cookies necessary for the website's operation. Additionally, you can freely decide and change any time whether you accept cookies or choose to opt out of cookies to improve website's performance, as well as cookies used to display content tailored to your interests. Your experience of the site and the services we are able to offer may be impacted if you do not accept all cookies. Modify Cookie Preferences Reject All Cookies Accept All Cookies Job Opportunities Jobs by Country Join Talent Community View Profile Search by Keyword Show More Options Clear Select how often (in days) to receive an alert: Create Alert Apply now » Country/Region: IN Requisition ID: 37467 Work Model: Position Type: Salary Range: Location: INDIA - NOIDA- BIRLASOFT OFFICE Title: Azure Databricks Developer Description: Area(s) of responsibility Role Summary We are seeking a seasoned Senior Azure Data Engineer / Sr. Technical Lead with extensive expertise in Azure Databricks, PySpark, ADF, SQL, and modern data engineering practices. This role requires strong technical leadership, hands on engineering capabilities, and the ability to design, architect, and deliver enterprise-scale data solutions. The ideal candidate will lead complex data initiatives, mentor engineering teams, and collaborate with cross-functional stakeholders to build a robust and scalable data ecosystem on Azure. ________________________________________ Key Responsibilities 1. Solution Architecture & Technical Leadership • Lead the design, development, and deployment of large-scale data engineering solutions using Azure Databricks, PySpark, SQL, ADF, and Azure Data Lake. • Architect end-to-end Modern Data Warehouse (MDW) and Lakehouse solutions, ensuring scalability, performance, security, and cost optimization. • Define technical standards, coding best practices, reusable frameworks, and architectural guidelines for engineering teams. • Provide technical leadership across the project lifecycle—requirements analysis, solution blueprinting, estimation, development, and deployment. 2. Data Pipeline Engineering • Build, optimize, and maintain scalable, high performance ELT/ETL pipelines to process large volumes of structured and unstructured data. • Set up complex data ingestion frameworks, enabling seamless integration with on-premise systems, cloud services, APIs, and third-party sources. • Ensure high availability, data reliability, and error-resilient orchestration workflows in Azure Data Factory. 3. Azure Databricks & PySpark Expertise • Design and implement advanced transformation logic using PySpark on Databricks, ensuring efficient data processing and code modularity. • Utilize Delta Lake capabilities—ACID transactions, schema evolution, versioning, time travel—to manage enterprise-grade datasets. • Perform cluster-level tuning, optimization of shuffle operations, caching, partitioning, and job parallelization. • Manage Databricks job pipelines, notebooks, clusters, job scheduling, and integration with CI/CD pipelines ________________________________________ Mandatory Skills & Experience • 10–13 years of overall experience in data engineering and enterprise data platforms. • Minimum 3 years of hands-on project experience in Azure Databricks (beyond POCs). • Minimum 5 years of experience building and orchestrating pipelines in Azure Data Factory (ADF). • Minimum 2+ years of strong PySpark experience with complex data transformation logic. • 6+ years of ETL & Data Warehouse experience, including dimensional modelling, data partitioning, and performance optimization. • Strong SQL expertise—complex queries, optimization, stored procedures, analytical functions. • Proven experience working with Azure Data Lake Storage (ADLS), Delta Lake, and modern data processing patterns. ________________________________________ Good-to-Have Skills • Experience working in Agile/Scrum environments, including sprint planning and backlog management. • Knowledge of Azure Synapse, Event Hub, Databricks workflows, and Azure DevOps CI/CD. • Databricks certifications (Associate/Professional) or Azure certifications (DP 203/DP 900). Apply now » Find similar jobs: India, Data & Analytics Home Cookie Policy Privacy Policy Cookie Consent Manager Recruitment Fraud Alert Opens in a new tab. Opens in a new tab. Opens in a new tab. Opens in a new tab. Share Post Email Share Share

Key responsibilities

• check_circleCollaborate with the team on day-to-day project tasks
• check_circleLearn tools and processes used by the organization
• check_circleDocument work and participate in team meetings
• check_circleSupport quality checks and continuous improvement

Requirements

• check_circleLead the design, development, and deployment of large-scale data engineering solutions using Azure Databricks, PySpark, SQL, ADF, and Azure Data Lake.
• check_circleArchitect end-to-end Modern Data Warehouse (MDW) and Lakehouse solutions, ensuring scalability, performance, security, and cost optimization.
• check_circleDefine technical standards, coding best practices, reusable frameworks, and architectural guidelines for engineering teams.
• check_circleProvide technical leadership across the project lifecycle—requirements analysis, solution blueprinting, estimation, development, and deployment.
• check_circleBuild, optimize, and maintain scalable, high performance ELT/ETL pipelines to process large volumes of structured and unstructured data.
• check_circleSet up complex data ingestion frameworks, enabling seamless integration with on-premise systems, cloud services, APIs, and third-party sources.
• check_circleEnsure high availability, data reliability, and error-resilient orchestration workflows in Azure Data Factory.
• check_circleDesign and implement advanced transformation logic using PySpark on Databricks, ensuring efficient data processing and code modularity.
• check_circleUtilize Delta Lake capabilities—ACID transactions, schema evolution, versioning, time travel—to manage enterprise-grade datasets.
• check_circlePerform cluster-level tuning, optimization of shuffle operations, caching, partitioning, and job parallelization.
• check_circleManage Databricks job pipelines, notebooks, clusters, job scheduling, and integration with CI/CD pipelines
• check_circle10–13 years of overall experience in data engineering and enterprise data platforms.

Skills & keywords

Benefits & perks

• check_circleMentorship
• check_circleCertificate of completion
• check_circleFlexible work arrangement where applicable