> ## Documentation Index
> Fetch the complete documentation index at: https://docs.clarityq.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Databricks

> Connect Databricks to ClarityQ

## Catalog options

ClarityQ supports two Databricks catalog types:

* **Unity Catalog**: the modern, recommended governance layer for Databricks. Supports both serverless and classic SQL warehouses.
* **Legacy Hive Metastore**: the original `hive_metastore` catalog. Requires a **classic (cluster-based) SQL warehouse**, as serverless SQL warehouses do not support the legacy Hive Metastore.

You'll select your catalog type in the ClarityQ connection form (Step 4).

## Setup

### Step 1: Get Databricks connection details

To connect ClarityQ to your Databricks workspace using a service principal, you'll need the following information:

1. **Server hostname**
2. **HTTP path**
3. **Service principal Application ID**
4. **Service principal OAuth secret**

### Step 2: Find your Databricks connection details

#### Server Hostname

1. Log in to your Databricks workspace
2. In the sidebar, click **SQL Warehouses** (or **Compute** → **SQL Warehouses**)
3. Click on your SQL warehouse name
4. Go to the **Connection Details** tab
5. Copy the **Server hostname** (e.g., `your-workspace.cloud.databricks.com`)

#### HTTP Path

1. In the same **Connection Details** tab of your SQL warehouse
2. Copy the **HTTP path** (e.g., `/sql/1.0/warehouses/abc123def456`)

<Note>
  If you plan to use the **Legacy Hive Metastore**, make sure your HTTP path points to a **classic (cluster-based) SQL warehouse**, not a serverless one. Serverless SQL warehouses do not support the legacy Hive Metastore.
</Note>

### Step 3: Create service principal credentials

Create a dedicated service principal for ClarityQ (recommended for production environments):

#### Create service principal for ClarityQ

1. In your Databricks workspace, go to **Settings** → **Identity and access**
2. Click the **Service principals** tab
3. Click **Add service principal**
4. Enter name: `clarityq-service-principal`
5. Click **Add**
6. Click on the newly created service principal
7. **Copy the Application ID** - you'll need this for the ClarityQ connection, where it will be entered in the **Client ID** input field (UUID format like `065e73cc-a713-4ae0-8212-ba7f98e32bee`)
8. Go to the **Secrets** tab
9. Click **Generate secret**
10. Set expiration date (recommended for security)
11. Click **Generate** and copy the secret immediately
12. In the same service principal details page, go to **Permissions** tab
13. Click **Grant access** or **Add permissions**
14. Search for and select **Service Principal: Use**
15. Save the changes
16. This allows the service principal to execute queries (required for ClarityQ)

#### Grant SQL Warehouse access

You need to grant the service principal permission to use the SQL Warehouse:

1. In the sidebar, click **SQL Warehouses** (or **Compute** → **SQL Warehouses**)
2. Click on your SQL warehouse name
3. Go to the **Permissions** tab
4. Search for and select `clarityq-service-principal`
5. In the permission dropdown, select **Can use**
6. Click **Save**

#### Grant data access permissions to ClarityQ

**Note**: These steps require **workspace admin** or **account admin** permissions. If you don't have these permissions, ask your Databricks administrator to perform these steps.

Choose the tab that matches your catalog type:

<Tabs>
  <Tab title="Unity Catalog">
    Grant SELECT permissions to your ClarityQ service principal for the data you want to analyze:

    **Step 1: Navigate to your catalog**

    1. Go to **Catalog** in your Databricks workspace
    2. Choose the catalog containing data you want ClarityQ to access

    **Step 2: Grant SELECT permissions**

    1. Select the **Permissions** tab
    2. Click the **Grant** button
    3. Search for and select your `clarityq-service-principal`
    4. Check the **SELECT** privilege
    5. Click **Grant** to save

    **Granular permission options:**

    * **Whole catalog**: Grants ClarityQ access to all schemas and tables in the catalog
    * **Specific schema**: Navigate to the schema level and repeat the above steps to grant access to specific schemas only
    * **Specific tables**: Navigate to individual tables and grant access to only the tables you want ClarityQ to analyze.

    **For metadata discovery (recommended):**

    Also grant SELECT permissions on `information_schema` so ClarityQ can discover available tables and columns.
  </Tab>

  <Tab title="Legacy Hive Metastore">
    The legacy Hive Metastore does not use the Unity Catalog permissions UI. Instead, you must grant permissions via SQL.

    Run the following SQL statements in a Databricks SQL editor or notebook, replacing `<application-id>` with your service principal's Application ID:

    ```sql theme={null}
    GRANT USAGE ON CATALOG hive_metastore TO `<application-id>`;
    GRANT USAGE ON SCHEMA hive_metastore.default TO `<application-id>`;
    GRANT SELECT ON SCHEMA hive_metastore.default TO `<application-id>`;
    GRANT READ_METADATA ON SCHEMA hive_metastore.default TO `<application-id>`;
    ```

    Replace `default` with the name of each schema you want ClarityQ to access, and repeat for additional schemas as needed.

    <Note>
      The `READ_METADATA` grant is required for the Hive Metastore. It allows ClarityQ to discover table names, column names, and column types, equivalent to the `information_schema` SELECT access used in Unity Catalog. Without it, ClarityQ can authenticate successfully but won't be able to see which tables exist.
    </Note>
  </Tab>
</Tabs>

### Step 4: Configure connection in ClarityQ

In the ClarityQ interface, fill out the connection form with the following fields:

#### Required Fields

* **Connection Name**: Choose a name for this connection (e.g., "Production Databricks")
* **Server Hostname**: Your Databricks server hostname (e.g., `your-workspace.cloud.databricks.com`)
* **HTTP Path**: Your SQL warehouse HTTP path (e.g., `/sql/1.0/warehouses/abc123def456`)
* **Client ID**: Your service principal Application ID (UUID format, e.g., `065e73cc-a713-4ae0-8212-ba7f98e32bee`)
* **Client Secret**: The service principal OAuth secret generated in Step 3
* **Catalog type**: Select **Unity Catalog** (default) or **Legacy Hive Metastore**
  * Choose **Legacy Hive Metastore** only if your data resides in the original `hive_metastore` catalog and you are using a classic (cluster-based) SQL warehouse

### Step 5: Test the connection

Verify the connection in ClarityQ to ensure:

* Databricks workspace accessibility
* SQL warehouse availability
* Token authentication
* Catalog and schema access
* Query execution capabilities

## Connection parameters reference

### Required parameters

| Parameter         | Type   | Description                                                              | Example                                |
| ----------------- | ------ | ------------------------------------------------------------------------ | -------------------------------------- |
| `connection_name` | string | Unique identifier for this connection                                    | `Production Databricks`                |
| `server_hostname` | string | Databricks workspace hostname                                            | `your-workspace.cloud.databricks.com`  |
| `http_path`       | string | SQL warehouse HTTP path                                                  | `/sql/1.0/warehouses/abc123def456`     |
| `client_id`       | string | Service principal Application ID, entered as **Client ID** in ClarityQ   | `065e73cc-a713-4ae0-8212-ba7f98e32bee` |
| `client_secret`   | string | Service principal OAuth secret, entered as **Client Secret** in ClarityQ | `dapis...`                             |

## Troubleshooting

### Common connection issues

**Authentication failed**

* Verify your service principal OAuth secret is correct and not expired
* Check if the service principal has necessary permissions through Admin Settings
* Ensure the user/service principal exists and is active

**Service principal access issues**

* **Most common cause**: Service principal not added to workspace yet
  * Ask your Databricks administrator to add the service principal to the workspace
* **Missing permissions**: Ask your admin to use the role-based approach described above
* **Insufficient permissions**: Ensure ClarityQ has **Service Principal Use** role + **read access** to desired data
* **SQL Warehouse access denied**: Ensure the service principal has **Can use** permission on the SQL Warehouse (see Step 3)

**Legacy Hive Metastore issues**

* **Tables not visible**: Ensure `READ_METADATA` has been granted on the schema. Without it, ClarityQ can connect but cannot discover tables.
* **Serverless warehouse errors**: The legacy Hive Metastore is not supported on serverless SQL warehouses. Switch to a classic (cluster-based) SQL warehouse.
* **Permission denied on GRANT statements**: Hive Metastore grants require workspace admin or the `GRANT` privilege. Ask your administrator to run the SQL grant statements.