Install, configure, and maintain Databricks clusters and workspaces.
Monitor cluster performance, resource utilization, and troubleshoot issues to ensure optimal performance.
Implement and manage access controls and security policies to protect sensitive data.
Perform regular software updates and patches.
Optimize cluster configurations for performance and cost-effectiveness.
Implement and enforce security policies, access controls, and encryption mechanisms.
Stay up-to-date with security best practices and compliance requirements.
Develop and maintain backup and disaster recovery strategies to ensure data integrity and availability.
Collaborate with data engineers to integrate Databricks with other data sources, data warehouses, and data lakes.
Maintain detailed documentation of configurations, procedures, and best practices.
Collaborate with cross-functional teams, including data scientists, data engineers, and business analysts to understand their requirements and provide technical solutions
Requirements
Bachelor's degree in a relevant field (computer science, data engineering, etc.)
2 to 3 years experience with Databricks Platform
Strong knowledge of Apache Spark and Databricks architecture.
Familiar with managing clusters, workspaces, notebooks, user group management.
Familiar with configuring permissions.
Can create notebooks related to cleanup/housekeeping and schedule/manage jobs through Databricks