Databricks

​

Catalog options

• Unity Catalog: the modern, recommended governance layer for Databricks. Supports both serverless and classic SQL warehouses.
• Legacy Hive Metastore: the original hive_metastore catalog. Requires a classic (cluster-based) SQL warehouse, as serverless SQL warehouses do not support the legacy Hive Metastore.

​

Setup

​

Step 1: Get Databricks connection details

1. Server hostname
2. HTTP path
3. Service principal Application ID
4. Service principal OAuth secret

​

Step 2: Find your Databricks connection details

​

Server Hostname

1. Log in to your Databricks workspace
2. In the sidebar, click SQL Warehouses (or Compute → SQL Warehouses)
3. Click on your SQL warehouse name
4. Go to the Connection Details tab
5. Copy the Server hostname (e.g., your-workspace.cloud.databricks.com)

​

HTTP Path

1. In the same Connection Details tab of your SQL warehouse
2. Copy the HTTP path (e.g., /sql/1.0/warehouses/abc123def456)

​

Step 3: Create service principal credentials

​

Create service principal for ClarityQ

1. In your Databricks workspace, go to Settings → Identity and access
2. Click the Service principals tab
3. Click Add service principal
4. Enter name: clarityq-service-principal
5. Click Add
6. Click on the newly created service principal
7. Copy the Application ID - you’ll need this for the ClarityQ connection, where it will be entered in the Client ID input field (UUID format like 065e73cc-a713-4ae0-8212-ba7f98e32bee)
8. Go to the Secrets tab
9. Click Generate secret
10. Set expiration date (recommended for security)
11. Click Generate and copy the secret immediately
12. In the same service principal details page, go to Permissions tab
13. Click Grant access or Add permissions
14. Search for and select Service Principal: Use
15. Save the changes
16. This allows the service principal to execute queries (required for ClarityQ)

​

Grant SQL Warehouse access

1. In the sidebar, click SQL Warehouses (or Compute → SQL Warehouses)
2. Click on your SQL warehouse name
3. Go to the Permissions tab
4. Search for and select clarityq-service-principal
5. In the permission dropdown, select Can use
6. Click Save

​

Grant data access permissions to ClarityQ

GRANT USAGE ON CATALOG hive_metastore TO `<application-id>`;
GRANT USAGE ON SCHEMA hive_metastore.default TO `<application-id>`;
GRANT SELECT ON SCHEMA hive_metastore.default TO `<application-id>`;
GRANT READ_METADATA ON SCHEMA hive_metastore.default TO `<application-id>`;

​

Step 4: Configure connection in ClarityQ

​

Required Fields

• Connection Name: Choose a name for this connection (e.g., “Production Databricks”)
• Server Hostname: Your Databricks server hostname (e.g., your-workspace.cloud.databricks.com)
• HTTP Path: Your SQL warehouse HTTP path (e.g., /sql/1.0/warehouses/abc123def456)
• Client ID: Your service principal Application ID (UUID format, e.g., 065e73cc-a713-4ae0-8212-ba7f98e32bee)
• Client Secret: The service principal OAuth secret generated in Step 3
• Catalog type: Select Unity Catalog (default) or Legacy Hive Metastore
  • Choose Legacy Hive Metastore only if your data resides in the original hive_metastore catalog and you are using a classic (cluster-based) SQL warehouse

​

Step 5: Test the connection

• Databricks workspace accessibility
• SQL warehouse availability
• Token authentication
• Catalog and schema access
• Query execution capabilities

​

Connection parameters reference

​

Required parameters

​

Troubleshooting

​

Common connection issues

• Verify your service principal OAuth secret is correct and not expired
• Check if the service principal has necessary permissions through Admin Settings
• Ensure the user/service principal exists and is active

• Most common cause: Service principal not added to workspace yet
  • Ask your Databricks administrator to add the service principal to the workspace
• Missing permissions: Ask your admin to use the role-based approach described above
• Insufficient permissions: Ensure ClarityQ has Service Principal Use role + read access to desired data
• SQL Warehouse access denied: Ensure the service principal has Can use permission on the SQL Warehouse (see Step 3)

• Tables not visible: Ensure READ_METADATA has been granted on the schema. Without it, ClarityQ can connect but cannot discover tables.
• Serverless warehouse errors: The legacy Hive Metastore is not supported on serverless SQL warehouses. Switch to a classic (cluster-based) SQL warehouse.
• Permission denied on GRANT statements: Hive Metastore grants require workspace admin or the GRANT privilege. Ask your administrator to run the SQL grant statements.