Release v0.37.0 (#2680)

* Added ability to run create-missing-principals command as collection
([#2675](#2675)). This
release introduces the capability to run the `create-missing-principals`
command as a collection in the UCX (Unified Cloud Experience) tool with
the new optional flag `run-as-collection`. This allows for more control
and flexibility when managing cloud resources, particularly in handling
multiple workspaces. The existing `create-missing-principals` command
has been modified to accept a new `run_as_collection` parameter,
enabling the command to run on multiple workspaces when set to True. The
function has been updated to handle a list of `WorkspaceContext`
objects, allowing it to iterate over each object and execute necessary
actions for each workspace. Additionally, a new `AccountClient`
parameter has been added to facilitate the retrieval of all workspaces
associated with a specific account. New test functions have been added
to `test_cli.py` to test this new functionality on AWS and Azure cloud
providers. The `acc_client` argument has been added to the test
functions to enable running the tests with an authenticated AWS or Azure
client, and the `MockPrompts` object is used to simulate user responses
to the prompts displayed during the execution of the command.
* Added storage for direct filesystem references in code
([#2526](#2526)). The
open-source library has been updated with a new table
`directfs_in_paths` to store Direct File System Access (DFSA) records,
extending support for managing and collecting DFSAs as part of
addressing issue
[#2350](#2350) and
[#2526](#2526). The changes
include a new class `DirectFsAccessCrawlers` and methods for handling
DFSAs, as well as linting, testing, and a manually verified schema
upgrade. Additionally, a new SQL query deprecates the use of direct
filesystem references. The commit is co-authored by Eric Vergnaud, Serge
Smertin, and Andrew Snare.
* Added task for linting queries
([#2630](#2630)). This
commit introduces a new `QueryLinter` class for linting SQL queries in
the workspace, similar to the existing `WorkflowLinter` for jobs. The
`QueryLinter` checks for any issues in dashboard queries and reports
them in a new `query_problems` table. The commit also includes the
addition of unit tests, integration tests, and manual testing of the
schema upgrade. The `QueryLinter` method has been updated to include a
`TableMigrationIndex` object, which is currently set to an empty list
and will be updated in a future commit. This change improves the quality
of the codebase by ensuring that all SQL queries are properly linted and
any issues are reported, allowing for better maintenance and development
of the system. The commit is co-authored by multiple developers,
including Eric Vergnaud, Serge Smertin, Andrew Snare, and Cor.
Additionally, a new linting rule, "direct-filesystem-access", has been
introduced to deprecate the use of direct filesystem references in favor
of more abstracted file access methods in the project's codebase.
* Adopt `databricks-labs-pytester` PyPI package
([#2663](#2663)). In this
release, we have made updates to the `pyproject.toml` file, removing the
`pytest` package version 8.1.0 and updating it to 8.3.3. We have also
added the `databricks-labs-pytester` package with a minimum version of
0.2.1. This update also includes the adoption of the
`databricks-labs-pytester` PyPI package, which moves fixture usage from
`mixins.fixtures` into its own top-level library. This affects various
test files, including `test_jobs.py`, by replacing the
`get_purge_suffix` fixture with `watchdog_purge_suffix` to standardize
the approach to creating and managing temporary directories and files
used in tests. Additionally, new fixtures have been introduced in a
separate PR for testing the `databricks.labs.ucx` package, including
`debug_env_name`, `product_info`, `inventory_schema`,
`make_lakeview_dashboard`, `make_dashboard`, `make_dbfs_data_copy`,
`make_mounted_location`, `make_storage_dir`, `sql_exec`, and
`migrated_group`. These fixtures simplify the testing process by
providing preconfigured resources that can be used in the tests. The
`redash.py` file has been removed from the `databricks/labs/ucx/mixins`
directory as the Redash API is being deprecated and replaced with a new
library.
* Assessment: crawl UDFs as a task in parallel to tables instead of
implicitly during grants
([#2642](#2642)). This
release introduces changes to the assessment workflow, specifically in
how User Defined Functions (UDFs) are crawled/scanned. Previously, UDFs
were crawled/scanned implicitly by the GrantsCrawler, which requested a
snapshot from the UDFSCrawler that hadn't executed yet. With this
update, UDFs are now crawled/scanned as their own task, running in
parallel with tables before grants crawling begins. This modification
addresses issue
[#2574](#2574), which
requires grants and UDFs to be refreshable but only once within a given
workflow run. A new method, crawl_udfs, has been introduced to iterate
over all UDFs in the Hive Metastore of the current workspace and persist
their metadata in a table named $inventory_database.udfs. This inventory
is utilized when scanning securable objects for issues with grants that
cannot be migrated to Unit Catalog. The crawl_grants task now depends on
crawl_udfs, crawl_tables, and setup_tacl, ensuring that UDFs are
crawled/scanned before grants are.
* Collect direct filesystem access from queries
([#2599](#2599)). This
commit introduces support for extracting Direct File System Access
(DirectFsAccess) records from workspace queries, adding a new table
`directfs_in_queries` and a new view `directfs` that unions
`directfs_in_paths` with the new table. The DirectFsAccessCrawlers class
has been refactored into two separate classes:
`DirectFsAccessCrawler.for_paths` and
`DirectFsAccessCrawler.for_queries`, and a new `QueryLinter` class has
been introduced to check queries for DirectFsAccess records. Unit tests
and manual tests have been conducted to ensure the correct functioning
of the schema upgrade. The commit is co-authored by Eric Vergnaud, Serge
Smertin, and Andrew Snare.
* Fixed failing integration test:
`test_reflect_account_groups_on_workspace_skips_groups_that_already_exists_in_the_workspace`
([#2624](#2624)). In this
release, we have made updates to the group migration workflow,
addressing an issue
([#2623](#2623)) where the
integration test
`test_reflect_account_groups_on_workspace_skips_groups_that_already_exists_in_the_workspace`
failed due to unhandled scenarios where a workspace group already
existed with the same name as an account group to be reflected. The
changes include the addition of a new method,
`_workspace_groups_in_workspace()`, which checks for the existence of
workspace groups. We have also modified the `group-migration` workflow
and integrated test
`test_reflect_account_groups_on_workspace_skips_account_groups_when_a_workspace_group_has_same_name`.
To enhance consistency and robustness, the `GroupManager` class has been
updated with two new methods:
`test_reflect_account_groups_on_workspace_warns_skipping_when_a_workspace_group_has_same_name`
and
`test_reflect_account_groups_on_workspace_logs_skipping_groups_when_already_reflected_on_workspace`.
These new methods check if a group is skipped when a workspace group
with the same name exists and log a warning message, as well as log
skipping groups that are already reflected on the workspace. These
improvements ensure that the system behaves as expected during the group
migration process, handling cases where workspace groups and account
groups share the same name.
* Fixed failing solution accelerator verification tests
([#2648](#2648)). This
release includes a fix for an issue in the LocalCodeLinter class that
was unable to normalize Python code at the notebook cell level. The
solution involved modifying the LocalCodeLinter constructor to include a
notebook loader, as well as adding a conditional block to the lint_path
method to determine the correct loader to use based on whether the path
is a notebook or not. These changes allow the linter to handle Python
code more effectively within Jupyter notebook cells. The tests for this
change were manually verified using `make solacc` on the files that
failed in CI. This commit has been co-authored by Eric Vergnaud. The
functionality of the linter remains unchanged, and there is no impact on
the overall software functionality. The target audience for this
description includes software engineers who adopt this open-source
library.
* Fixed handling of potentially corrupt `state.json` of UCX workflows
([#2673](#2673)). This
commit introduces a fix for potential corruption of `state.json` files
in UCX workflows, addressing issue
[#2673](#2673) and resolving
[#2667](#2667). It updates
the import statement in `install.py`, introduces a new `with_extra`
function, and centralizes the deletion of jobs, improving code
maintainability. Two new methods are added to check if a job is managed
by UCX. Additionally, the commit removes deprecation warnings for direct
filesystem references in pytester fixtures and adjusts the known.json
file to accurately reflect the project's state. A new `Task` method is
added for defining UCX workflow tasks, and several test cases are
updated to ensure the correct handling of jobs during the uninstallation
process. Overall, these changes enhance the reliability and
user-friendliness of the UCX workflow installation process.
* Let `create-catalog-schemas` command run as collection
([#2653](#2653)). The
`create-catalog-schemas` and `validate-external-locations` commands in
the `databricks labs ucx` package have been updated to operate as
collections, allowing for simultaneous execution on multiple workspaces.
These changes, which resolve issue
[#2609](#2609), include the
addition of new parameters and flags to the command signatures and
method signatures, as well as updates to the existing functionality for
creating catalogs and schemas. The changes have been manually tested and
accompanied by unit tests, with integration tests to be added in a
future update. The `create-catalog-schemas` command now accepts a list
of workspace clients and a `run_as_collection` parameter, and skips
existing catalogs and schemas while logging a message. The
`validate-external-locations` command also operates as a collection,
though specific details about this change are not provided.
* Let `create-uber-principal` command run on collection of workspaces
([#2640](#2640)). The
`create-uber-principal` command has been updated to support running on a
collection of workspaces, allowing for more efficient management of
service principals across multiple workspaces. This change includes the
addition of a new flag, `run-as-collection`, which, when set to true,
allows the command to run on a collection of workspaces with UCX
installed. The command continues to grant STORAGE_BLOB_READER access to
Azure storage accounts and identify S3 buckets used in AWS workspaces.
The changes also include updates to the testing strategy, with manual
testing and unit tests added. Integration tests will be added in a
future PR. These modifications enhance the functionality and reliability
of the command, improving the user experience for managing workspaces.
In terms of implementation, the `create_uber_principal` method in the
`access.py` and `cli.py` files has been updated to support running on a
collection of workspaces. The modification includes the addition of a
new parameter, `run_as_collection`, which, when set to True, allows the
method to retrieve a collection of workspace contexts and execute the
necessary operations for each context. The changes also include updates
to the underlying methods, such as the `aws_profile` method, to ensure
the correct cloud provider is being utilized. The behavior of the
command has been isolated from the underlying `ucx` functionality by
introducing mock values for the uber service principal ID and policy ID.
The changes also include updates to the tests to reflect these
modifications, with new tests added to ensure that the command behaves
correctly when run on a collection of workspaces and to test the error
handling for unsupported cloud providers and missing subscription IDs.
* Let `migrate-acls` command run as collection
([#2664](#2664)). The
`migrate-acls` command in the `labs.yml` file has been updated to
facilitate the migration of access control lists (ACLs) from a legacy
metastore to a UC metastore for a collection of workspaces with Unity
Catalog (UC) installed. This command now supports running as a
collection, enabled by a new optional flag `run-as-collection`. When set
to true, the command will run for all workspaces with UC installed,
enhancing efficiency and ease of use. The new functionality has been
manually tested and verified with added unit tests. However, integration
tests are yet to be added. The command is part of the
`databricks/labs/ucx` module and is implemented in the `cli.py` file.
This update addresses issue
[#2611](#2611) and includes
both manual and unit tests.
* Let `migrate-dbsql-dashboards` command to run as collection
([#2656](#2656)). The
`migrate-dbsql-dashboards` command in the `databricks labs ucx` command
group has been updated to support running as a collection, allowing it
to migrate queries for all dashboards in one or more workspaces. This
new feature is achieved by adding an optional flag `run-as-collection`
to the command. If set to True, the command will be executed for all
workspaces with ucx installed, resolving issue
[#2612](#2612). The
`migrate-dbsql-dashboards` function has been updated to take additional
parameters `ctx`, `run_as_collection`, and `a`. The `ctx` parameter is
an optional `WorkspaceContext` object, which can be used to specify the
context for a single workspace. If not provided, the function will
retrieve a list of `WorkspaceContext` objects for all workspaces. The
`run_as_collection` parameter is a boolean flag indicating whether the
command should run as a collection. If set to True, the function will
iterate over all workspaces and migrate queries for all dashboards in
each workspace. The `a` parameter is an optional `AccountClient` object
for authentication. Unit tests have been added to ensure that the new
functionality works as expected. This feature will be useful for users
who need to migrate many dashboards at once. Integration tests will be
added in a future update after issue
[#2507](#2507) is addressed.
* Let `migrate-locations` command run as collection
([#2652](#2652)). The
`migrate-locations` command in the `databricks labs ucx` library for AWS
and Azure has been enhanced to support running as a collection of
workspaces, allowing for more efficient management of external
locations. This has been achieved by modifying the existing `databricks
labs ucx migrate-locations` command and adding a `run_as_collection`
flag to specify that the command should run for a collection of
workspaces. The changes include updates to the `run` method in
`locations.py` to return a list of strings containing the URLs of
missing external locations, and the addition of the
`_filter_unsupported_location` method to filter out unsupported
locations. A new `_get_workspace_contexts` function has been added to
return a list of `WorkspaceContext` objects based on the provided
`WorkspaceClient`, `AccountClient`, and named parameters. The commit
also includes new test cases for handling unsupported cloud providers
and testing the `run as collection` functionality with multiple
workspaces, as well as manual and unit tests. Note that due to current
limitations in unit testing, the `run as collection` tests for both
Azure and AWS raise exceptions.
* Let `migrate-tables` command run as collection
([#2654](#2654)). The
`migrate-tables` command in the `labs.yml` configuration file has been
updated to support running as a collection of workspaces with UCX
installed. This change includes adding a new flag `run_as_collection`
that, when set to `True`, allows the command to run on all workspaces in
the collection, and modifying the existing command to accept an
`AccountClient` object and `WorkspaceContext` objects. The function
`_get_workspace_contexts` is used to retrieve the `WorkspaceContext`
objects for each workspace in the collection. Additionally, the
`migrate_tables` command now checks for the presence of hiveserde and
external tables and prompts the user to run the
`migrate-external-hiveserde-tables-in-place-experimental` and
`migrate-external-tables-ctas` workflows, respectively. The command's
documentation and tests have also been updated to reflect this new
functionality. Integration tests will be added in a future update. These
changes improve the scalability and efficiency of the `migrate-tables`
command, allowing for easier and more streamlined execution across
multiple workspaces.
* Let `validate-external-locations` command run as collection
([#2649](#2649)). In this
release, the `validate-external-locations` command has been updated to
support running as a collection, allowing it to operate on multiple
workspaces simultaneously. This change includes the addition of new
parameters `ctx`, `run_as_collection`, and `a` to the
`validate-external-locations` command in the `cli.py` file. The `ctx`
parameter determines the current workspace context, obtained through the
`_get_workspace_contexts` function when `run_as_collection` is set to
True. The function queries for all available workspaces associated with
the given account client `a`. The
`save_as_terraform_definitions_on_workspace` method is then called to
save the external locations as Terraform definitions on the workspace.
This enhancement improves the validation process for external locations
across multiple workspaces. Additionally, the command's implementation
has been updated to include the `run_as_collection` parameter, which
controls whether the command is executed as a collection, ensuring
sequential execution of each statement within the command. The unit
tests have been updated to include a test case that verifies this
functionality. The `validate_external_locations` function has also been
updated to include a `ctx` parameter, which is used to specify the
workspace context. These changes improve the functionality of the
`validate-external-locations` command, ensuring sequential execution of
statements across workspaces.
* Let `validate-groups-membership` command to run as collection
([#2657](#2657)). The latest
commit introduces an optional `run-as-collection` flag to the
`validate-groups-membership` command in the `labs.yml` configuration
file. This flag, when set to true, enables the command to run for a
collection of workspaces equipped with UCX. The updated
`validate-groups-membership` command in `databricks/labs/ucx/cli.py` now
accepts new arguments: `ctx`, `run_as_collection`, and `a`. This change
resolves issue
[#2613](#2613) and includes
updated unit and manual tests, ensuring thorough functionality
verification. The new feature allows software engineers to validate
group memberships across multiple workspaces simultaneously, enhancing
efficiency and ease of use. When run as a collection, the command
validates groups at both the account and workspace levels, comparing
memberships for each specified workspace context.
* Removed installing on workspace log message in `_get_installer`
([#2641](#2641)). In this
enhancement, the `_get_installer` function in the `install.py` file has
undergone modification to improve the clarity of the installation
process for users. Specifically, a confusing log message that
incorrectly indicated that UCX was being installed when it was not, has
been removed. The log message has been relocated to a more accurate
position in the codebase. It is important to note that the
`_get_installer` function itself has not been modified, only the log
message has been removed. This change eliminates confusion about the
installation of UCX, thus enhancing the overall user experience.
* Support multiple subscription ids for command line commands
([#2647](#2647)). The
`databricks labs ucx` tool now supports multiple subscription IDs for
the `create-uber-principal`, `guess-external-locations`,
`migrate-credentials`, and `migrate-locations` commands. This change
allows users to specify multiple subscriptions for scanning storage
accounts, improving management for users who handle multiple
subscriptions simultaneously. Relevant flags in the `labs.yml`
configuration file have been updated, and unit tests, as well as manual
testing, have been conducted to ensure proper functionality. In the
`cli.py` file, the `create_uber_principal` and `principal_prefix_access`
functions have been updated to accept a list of subscription IDs,
affecting the `create_uber_principal` and `principal_prefix_access`
commands. The `azure_subscription_id` property has been renamed to
`azure_subscription_ids`, modifying the `azureResources` constructor and
ensuring correct handling of the subscription IDs.
* Updated databricks-labs-lsql requirement from <0.11,>=0.5 to
>=0.5,<0.12
([#2666](#2666)). In this
release, we have updated the version requirement for the
`databricks-labs-lsql` library in the 'pyproject.toml' file from a
version greater than or equal to 0.5 and less than 0.11 to a version
greater than or equal to 0.5 and less than 0.12. This change allows us
to use the latest version of the `databricks-labs-lsql` library while
still maintaining a version range constraint. This library provides
functionality for managing and querying data in Databricks, and this
update ensures compatibility with the project's existing dependencies.
No other changes are included in this commit.
* Updated sqlglot requirement from <25.21,>=25.5.0 to >=25.5.0,<25.22
([#2633](#2633)). In this
pull request, we have updated the `sqlglot` dependency requirement in
the `pyproject.toml` file. The previous requirement was for a minimum
version of 25.5.0 and less than 25.21, which has now been changed to a
minimum version of 25.5.0 and less than 25.22. This update allows us to
utilize the latest version of `sqlglot`, up to but not including version
25.22. While the changelog and commits for the latest version of
`sqlglot` have been provided for reference, the specific changes made to
the project as a result of this update are not detailed in the pull
request description. Therefore, as a reviewer, it is essential to verify
the compatibility of the updated `sqlglot` version with our project and
ensure that any necessary modifications have been made to accommodate
the new version.
* fix test_running_real_remove_backup_groups_job timeout
([#2651](#2651)). In this
release, we have made an adjustment to the
`test_running_real_remove_backup_groups_job` test case by increasing the
timeout of an inner task from 90 seconds to 3 minutes. This change is
implemented to address the timeout issue reported in issue
[#2639](#2639). Furthermore,
to ensure the correct functioning of the code, we have incorporated
integration tests. It is important to note that the functionality of the
code remains unaffected. This enhancement aims to provide a more
reliable and efficient testing process, thereby improving the overall
quality of the open-source library.

Dependency updates:

* Updated sqlglot requirement from <25.21,>=25.5.0 to >=25.5.0,<25.22
([#2633](#2633)).
* Updated databricks-labs-lsql requirement from <0.11,>=0.5 to
>=0.5,<0.12 ([#2666](#2666)).