Release/0.3.3 (#188)

* serverless compatability check * update logger methods for serverless * changes to support serverless - 1 * changes to sat driver to make it serverless compataible * import time from driver * fix logging utils to create file , fix to dbsql client to parse the correct response * fixes related to logger * fix logger regression issues * bug fixes on the left over file -part 1 * bug fixes and code refactor * drop staging database before each run * move drop intermediate schema to initialize * Update TERRAFORM_Azure.md updated Azure terraform instruction with SP requirements * Added dashboard and corrected the name in the setup notebook * Update TERRAFORM_Azure.md more readable format. * updated sdk ,removed local references and minot refactoring * Bug Fixes using sdk * logfile changes * removed stale secrets and moved to variables * utility to dynamically infer json schema from rest response * Terraform changes to support sat run on serverless * New dashboard and code to create * fixed edge case for PAT expiry when only one token meets the condition. * GCP tests completed * Removed internal workspace link in dashboard * Fixed removed changes * Removed logger * Add serverless configuration options for DABs * Refactor job cluster configuration for serverless compatibility * Added detailed instructions to add SP to workspaces * Fixed serverless compute with TF * added run on serverless to the provider * update serverless instructions --------- Co-authored-by: sudharshanraja-db <sudharshanraja-db> Co-authored-by: Chris Moon <[email protected]> Co-authored-by: andres-zuniga <[email protected]> Co-authored-by: ramdas.murali <[email protected]> Co-authored-by: arunpamulapati <arunpamulapati> Co-authored-by: David Leiva <[email protected]>
databricks-industry-solutions · Jan 13, 2025 · d7ce6b1 · d7ce6b1
1 parent 61b9122
commit d7ce6b1
Show file tree

Hide file tree

Showing 53 changed files with 611 additions and 5,403 deletions.
diff --git a/dabs/dabs_template/databricks_template_schema.json b/dabs/dabs_template/databricks_template_schema.json
@@ -20,6 +20,10 @@
         "node_type": {
             "type": "string",
             "description": "Node Type"
+        },
+        "serverless": {
+            "type": "boolean",
+            "description": "Serverless"
         }
     },
     "success_message": ""

diff --git a/dabs/dabs_template/template/tmp/resources/sat_driver_job.yml.tmpl b/dabs/dabs_template/template/tmp/resources/sat_driver_job.yml.tmpl
@@ -7,13 +7,16 @@ resources:
         timezone_id: "America/New_York"
       tasks:
         - task_key: "sat_initializer"
+          {{- if eq .serverless false }}
           job_cluster_key: job_cluster
           libraries:
             - pypi:
                 package: dbl-sat-sdk
+          {{- end }}
           notebook_task:
             notebook_path: "../notebooks/security_analysis_driver.py"
 
+      {{- if eq .serverless false }}
       job_clusters:
         - job_cluster_key: job_cluster
           new_cluster:
@@ -26,3 +29,4 @@ resources:
             gcp_attributes:
               google_service_account: {{.google_service_account}}
             {{- end }}
+      {{- end }}
diff --git a/dabs/dabs_template/template/tmp/resources/sat_initiliazer_job.yml.tmpl b/dabs/dabs_template/template/tmp/resources/sat_initiliazer_job.yml.tmpl
@@ -5,13 +5,18 @@ resources:
 
       tasks:
         - task_key: "sat_initializer"
+
+          {{- if eq .serverless false }}
           job_cluster_key: job_cluster
+
           libraries:
             - pypi:
                 package: dbl-sat-sdk
+          {{- end }}
           notebook_task:
             notebook_path: "../notebooks/security_analysis_initializer.py"
-
+
+      {{- if eq .serverless false }}
       job_clusters:
         - job_cluster_key: job_cluster
           new_cluster:
@@ -23,4 +28,5 @@ resources:
             {{- if eq .cloud "gcp" }}
             gcp_attributes:
               google_service_account: {{.google_service_account}}
-            {{- end }}
+            {{- end }}
+      {{- end }}
diff --git a/dabs/main.py b/dabs/main.py
@@ -25,6 +25,7 @@ def install(client: WorkspaceClient, answers: dict, profile: str):
             photon_driver_capable=True,
             photon_worker_capable=True,
         ),
+        "serverless": answers.get("enable_serverless", False),
     }
 
     config_file = "tmp_config.json"

diff --git a/dabs/sat/config.py b/dabs/sat/config.py
@@ -48,6 +48,11 @@ def form():
             message="Schema name for SAT",
             default="security_analysis",
         ),
+        Confirm(
+            name="enable_serverless",
+            message="Run on serverless?",
+            default=True,
+        ),
         List(
             name="warehouse",
             message="Select warehouse",

diff --git a/dashboards/SAT_Dashboard_definition.json b/dashboards/SAT_Dashboard_definition.json
diff --git a/docs/deprecated_old_setup.md b/docs/deprecated_old_setup.md
@@ -1,3 +1,4 @@
+Note: The manual setup is out dates and is deprecated. Please use [Terraform or Standard](https://github.com/databricks-industry-solutions/security-analysis-tool/tree/main?tab=readme-ov-file#security-analysis-tool-sat) setup. 
 ## Checklist to prepare for SAT setup
 
 **Note**: SAT creates a new **security_analysis** databses and Delta tables. 

diff --git a/docs/setup.md b/docs/setup.md
@@ -2,7 +2,7 @@
 
 > **SAT v0.2.0 or higher** brings full support for Unity Catalog. Now you can pick your catalog instead of hive_metastore. Plus, you get to choose your own schema name.
 
-> **Note**: SAT requires at least one SAT set up in a workspace per **account** in AWS or GCP and at least one SAT set up in a workspace per Azure **subscription**.  
+> **Note**: SAT requires at least one SAT set up in a workspace per **account** in AWS or GCP and at least one SAT set up in a workspace per Azure **subscription**. SAT uses the Service Principal configured with SAT to access and analyze configurations by calling account and workspace APIs. Please make sure to add the Service Principal to the workspaces you wish to analyze as instructed in the setup guides below. SAT running on serverless or classic compute can't analyze account and destination workspaces with IP ACLs configured unless the ACLs are updated to allow access from the SAT workspace. SAT running on Serverless compute can't access other workspaces if the SAT workspace has serverless egress controls configured. In these scenarios, a separate SAT can be configured to analyze the respective target workspace by setting up SAT in that workspace.
 
 >  Please make sure to review the SAT report with your business stakeholders, administrators, security team and auditors about SAT report and assess your organizational security requirements before making any security improvements bases on the report, not all deviations required to be mitigated. Some of the recommendations may have cost implications, some of the security features recommended may have dependency feature limitations, please thoroughly review individual feature documentation before making changes to your security configurations.
 

diff --git a/docs/setup/aws.md b/docs/setup/aws.md
@@ -32,7 +32,7 @@ The first step is to create a Service Principal in Databricks. This will allow S
 
 ![AWS_SP_Workspace](../images/gcp_ws.png)
 
-> The Service Principle requires an [Accounts Admin role](https://docs.gcp.databricks.com/en/admin/users-groups/service-principals.html#assign-account-admin-roles-to-a-service-principal), [Admin role](https://docs.gcp.databricks.com/en/admin/users-groups/service-principals.html#assign-a-service-principal-to-a-workspace-using-the-account-console) for **each workspace** and needs to be a member of the [metastore admin group](https://docs.gcp.databricks.com/en/data-governance/unity-catalog/manage-privileges/admin-privileges.html#who-has-metastore-admin-privileges) is required to analyze many of the APIs
+> The Service Principle requires an [Accounts Admin role](https://docs.gcp.databricks.com/en/admin/users-groups/service-principals.html#assign-account-admin-roles-to-a-service-principal), [Admin role](https://docs.gcp.databricks.com/en/admin/users-groups/service-principals.html#assign-a-service-principal-to-a-workspace-using-the-account-console) for **each workspace** and needs to be a member of the [metastore admin group](https://docs.gcp.databricks.com/en/data-governance/unity-catalog/manage-privileges/admin-privileges.html#who-has-metastore-admin-privileges) is required to analyze many of the APIs. Please [add this Service Princple](https://docs.databricks.com/en/admin/users-groups/service-principals.html#assign-a-service-principal-to-a-workspace-using-the-account-console) to **each workspace** so that SAT can access the APIs for analysis.  
 
 ## Installation
 

diff --git a/docs/setup/azure.md b/docs/setup/azure.md
@@ -63,7 +63,7 @@ After creating the App Registration and client secret, you will need to add the
 
 See the [Databricks documentation](https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/service-principals#--databricks-and-microsoft-entra-id-formerly-azure-active-directory-service-principals) for more information on adding service principals.
 
-> The Service Principle requires an [Accounts Admin role](https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/service-principals#--assign-account-admin-roles-to-a-service-principal), [Admin role](https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/service-principals#assign-a-service-principal-to-a-workspace-using-the-account-console) for **each workspace** and needs to be a member of the [metastore admin group](https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/manage-privileges/admin-privileges#who-has-metastore-admin-privileges) is required to analyze many of the APIs.
+> The Service Principle requires an [Accounts Admin role](https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/service-principals#--assign-account-admin-roles-to-a-service-principal), [Admin role](https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/service-principals#assign-a-service-principal-to-a-workspace-using-the-account-console) for **each workspace** and needs to be a member of the [metastore admin group](https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/manage-privileges/admin-privileges#who-has-metastore-admin-privileges) is required to analyze many of the APIs. Please [add this Service Princple](https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/service-principals) to **each workspace** so that SAT can access the APIs for analysis. 
 
 ## Installation
 

diff --git a/docs/setup/gcp.md b/docs/setup/gcp.md
@@ -26,7 +26,7 @@ The first step is to create a Service Principal in GCP. This will allow SAT to a
   - You will need to create a new user and add the service account email as the user email.
 - The Service Principal must be granted the `Account Admin` role. This role provides the ability to manage account-level settings and permissions.
 - Assign the Workspace Admin Role: The Service Principal must be assigned the `Workspace Admin` role for each workspace it will manage. This role provides the ability to manage workspace-level settings and permissions.
-- Add to the Metastore Admin Group: The Service Principal must be added to the `Metastore Admin` group or role. This role provides the ability to manage metastore-level settings and permissions.
+- Add to the Metastore Admin Group: The Service Principal must be added to the `Metastore Admin` group or role. This role provides the ability to manage metastore-level settings and permissions. 
 
 ### Databricks Service Principal
 
@@ -41,7 +41,7 @@ The first step is to create a Service Principal in Databricks. This will allow S
 - Add to the Metastore Admin Group: The Service Principal must be added to the `Metastore Admin` group or role. This role provides the ability to manage metastore-level settings and permissions.
 - Create a new OAuth Secret.
 - Save the `Secret` and `Client ID`
-- To deploy SAT in a workspace, you must add the Service Principal to the workspace.
+- To analyze a workspace with SAT, you must add the Service Principal to the workspace. Please [add this Service Princple](https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/service-principals) to **each workspace** so that SAT can access the APIs for analysis.
 
 ## Installation
 

diff --git a/notebooks/Includes/install_sat_sdk.py b/notebooks/Includes/install_sat_sdk.py
@@ -9,20 +9,25 @@
 import os
 #Get databricks runtime configured to run SAT
 dbr_version = os.environ.get('DATABRICKS_RUNTIME_VERSION','0.0')
+is_sat_compatible = False
 #sanity check in case there is major and minor version
 #strip minor version since we need to compare as number
-dbrarray = dbr_version.split('.')
-dbr_version =  f'{dbrarray[0]}.{dbrarray[1]}'
-dbr_version = float(dbr_version)
+if(dbr_version.startswith("client")):
+    is_sat_compatible = True
+else:
+    dbrarray = dbr_version.split('.')
+    dbr_version =  f'{dbrarray[0]}.{dbrarray[1]}'
+    dbr_version = float(dbr_version)
+    is_sat_compatible = True if dbr_version >= RECOMMENDED_DBR_FOR_SAT else False
 
 #test version
 
-if dbr_version < RECOMMENDED_DBR_FOR_SAT:
+if is_sat_compatible== False:
     dbutils.notebook.exit(f"Detected DBR version {dbr_version} . Please use the DBR {RECOMMENDED_DBR_FOR_SAT} for SAT and try again , please refer to docs/setup.md")
 
 # COMMAND ----------
 
-SDK_VERSION='0.1.37' 
+SDK_VERSION='0.0.102' 
 
 # COMMAND ----------