Databricks Engineer

Databricks Engineer

Job Description

Overview:
We are seeking a dynamic and highly skilled Databricks Engineer to join our data team. In this role, you will be at the forefront of designing, developing, and maintaining scalable big data solutions using Databricks and related technologies. Your expertise will enable us to harness the power of large datasets, optimize data workflows, and deliver actionable insights that drive strategic decision-making. This position offers an exciting opportunity to work with cutting-edge cloud platforms and innovative data tools in a fast-paced, collaborative environment.

Responsibilities

Design, develop, and optimize data pipelines leveraging Databricks Unified Analytics Platform to process large-scale datasets efficiently.
Implement robust ETL workflows using Spark, Python, SQL, and shell scripting to ensure high-quality data ingestion and transformation.
Collaborate with cross-functional teams to integrate diverse data sources such as AWS, Azure Data Lake, Hadoop, Oracle, Microsoft SQL Server, and Informatica into unified data warehouses.
Develop and maintain scalable data models and database schemas aligned with best practices in database design for analytical applications.
Utilize analytics tools like Looker for creating dashboards and visualizations that facilitate insightful business analysis.
Conduct model training and analysis to support machine learning initiatives, ensuring data readiness and quality.
Write RESTful APIs for seamless data access across platforms and applications while adhering to security standards.
Support agile development processes by participating in sprint planning, code reviews, and continuous integration efforts.
Maintain comprehensive documentation of data workflows, system configurations, and technical specifications to ensure operational transparency.

Experience:

8+ Years of experience
Proven experience working with big data technologies and cloud platforms like AWS or Azure.
Hands-on experience with Databricks platform for developing scalable analytics solutions in a cloud environment.
Experience working with Spark, Unstructured format, Unity Catalog, Genie, Immuta, lambda, dynamodb, aws data sync, aws sftp
Strong proficiency in programming languages including Java, Python, Bash (Unix shell), Shell Scripting, and VBA for automation and scripting tasks.
Extensive knowledge of ETL processes, SQL query optimization, database design principles for Data Warehouses, and familiarity with Oracle and Microsoft SQL Server databases.

Notes:

Hybrid 3 Days/Week on-site

VIVA is an equal opportunity employer. All qualified applicants have an equal opportunity for placement, and all employees have an equal opportunity to develop on the job. This means that VIVA will not discriminate against any employee or qualified applicant on the basis of race, color, religion, sex, sexual orientation, gender identity, national origin, disability or protected veteran status.