turbot · binaek · Sep 8, 2023 · Sep 13, 2023 · Sep 22, 2023 · Sep 25, 2023
diff --git a/docs/develop/writing-plugins.md b/docs/develop/writing-plugins.md
@@ -486,6 +486,89 @@ Currently supported data types are:
 
 ---
 
+
+## Client-Side Rate Limiting
+
+The Steampipe Plugin SDK supports a [client-side rate limiting implementation](/docs/guides/limiter) to allow users to define [plugin `limiter` blocks](/docs/reference/config-files/plugin#limiter) to control concurrency and rate limiting.  Support for limiters is build in to the SDK and basic functionality requires no changes to the plugin code;  Just indluding the SDK will enable users to create limiters for your plugin using the built in `connection`, `table`, and `function_name` scopes.  You can add additional flexibility by adding [function tags](#function-tags) and by [accounting for pageing in List calls](#accounting-for-paged-list-calls).
+## Function Tags
+
+Hydrate function tags provide useful diagnostic metadata, and they can also be used as scopes in rate limiters.  Rate limiting requirements vary by plugin because the underlying APIs that they access implement rate limiting differently.  Tags provide a way for a plugin author to scope rate limiters in a way that aligns with the API implementation.
+
+Tags can be added to a ListConfig, GetConfig, or HydrateConfig.
+
+```go
+//// TABLE DEFINITION
+func tableAwsSnsTopic(_ context.Context) *plugin.Table {
+	return &plugin.Table{
+		Name:        "aws_sns_topic",
+		Description: "AWS SNS Topic",
+		Get: &plugin.GetConfig{
+			KeyColumns: plugin.SingleColumn("topic_arn"),
+			IgnoreConfig: &plugin.IgnoreConfig{
+				ShouldIgnoreErrorFunc: shouldIgnoreErrors([]string{"NotFound", "InvalidParameter"}),
+			},
+			Hydrate: getTopicAttributes,
+			Tags:    map[string]string{"service": "sns", "action": "GetTopicAttributes"},
+		},
+		List: &plugin.ListConfig{
+			Hydrate: listAwsSnsTopics,
+			Tags:    map[string]string{"service": "sns", "action": "ListTopics"},
+		},
+
+		HydrateConfig: []plugin.HydrateConfig{
+			{
+				Func: listTagsForSnsTopic,
+				Tags: map[string]string{"service": "sns", "action": "ListTagsForResource"},
+			},
+			{
+				Func: getTopicAttributes,
+				Tags: map[string]string{"service": "sns", "action": "GetTopicAttributes"},
+			},
+		},
+    ...
+```
+
+Once the tags are added to the plugin, you can use them in the `scope` and `where` arguments for your rate limiter.
+
+```hcl
+plugin "aws" {
+  limiter "sns_get_topic_attributes_us_east_1" {
+    bucket_size  = 3000
+    fill_rate    = 3000
+
+    scope  = ["connection", "region", "service", "action"]
+    where  = "action = 'GetTopicAttributes' and service = 'sns' and region = 'us-east-1' "
+  }
+}
+```
+
+## Accounting for Paged List calls
+The Steampipe plugin SDK transparently handles the most of the details around waiting for limiters.  List calls, however, usually iterate through pages of results, and each call to fetch a page must wait for any limiters that are defined.  The SDK provides a hook, `WaitForListRateLimit`, which should be called before paging to apply rate limiting to the list call:
+
+```go
+// List call
+for paginator.HasMorePages() {
+
+  // apply rate limiting
+  d.WaitForListRateLimit(ctx)
+
+  output, err := paginator.NextPage(ctx)
+  if err != nil {
+    plugin.Logger(ctx).Error("List error", "api_error", err)
+    return nil, err
+  }
+  for _, items := range output.Items {
+    d.StreamListItem(ctx, items)
+
+    // Context can be cancelled due to manual cancellation or the limit has been hit
+    if d.RowsRemaining(ctx) == 0 {
+      return nil, nil
+    }
+  }
+}
+```
+
+--
 ## Logging
 
 A logger is passed to the plugin via the context.  You can use the logger to write messages to the log at standard log levels:
@@ -521,3 +604,4 @@ For example, consider a `myplugin` plugin that you have developed.  To install i
     ```
 - Your connection will be loaded the next time Steampipe runs.  If Steampipe is running service mode, you must restart it to load the connection.
 
+---
diff --git a/docs/guides/limiter.md b/docs/guides/limiter.md
diff --git a/docs/guides/overview.md b/docs/guides/overview.md
@@ -9,4 +9,5 @@ Guides provide expanded explanations for common cases.
 
 - **[Use the search_path to target specific connections (or aggregators) →](guides/search-path)**
 - **[Use the Steampipe CLI with AWS Organizations →](guides/aws-orgs)**
-- **[Understand and Configure the Query Cache →](guides/caching)**
+- **[Understand and Configure the Query Cache →](guides/caching)**
+- **[Implement client-side rate limiting with `limiter` →](guides/limiter)**
diff --git a/docs/managing/plugins.md b/docs/managing/plugins.md
@@ -60,7 +60,7 @@ Steampipe plugins are packaged in OCI format and can be hosted and installed fro
 $ steampipe plugin install us-docker.pkg.dev/myproject/myrepo/myplugin@mytag
 ```
 
-### Installing from a file
+### Installing from a File
 A plugin binary can be installed manually, and this is often convenient when developing the plugin. Steampipe will attempt to load any plugin that is referred to in a `connection` configuration:
 - The plugin binary file must have a `.plugin` extension
 - The plugin binary must reside in a subdirectory of the `~/.steampipe/plugins/` directory and must be the ONLY `.plugin` file in that subdirectory
@@ -76,6 +76,21 @@ For example, consider a `myplugin` plugin that you have developed.  To install i
     }
     ```
 
+### Installing Missing Plugins
+
+You can install all missing plugins that are referenced in your configuration files:
+
+```bash
+$ steampipe plugin install
+```
+
+Running `steampipe plugin install` with no arguments will cause Steampipe to read all `connection` and `plugin` blocks in all `.spc` files in the `~/.steampipe/config` directory and install any that are referenced but are not installed.  Note that when doing so, any default `.spc` file that does not exist in the configuration will also be copied.  You may pass the `--skip-config` flag if you don't want to copy these files:
+
+```bash
+$ steampipe plugin install --skip-config
+```
+
+
 ## Viewing Installed Plugins
 You can list the installed plugins with the `steampipe plugin list` command:
 

diff --git a/docs/reference/cli/plugin.md b/docs/reference/cli/plugin.md
@@ -18,7 +18,7 @@ steampipe plugin [command]
 
 | Command | Description
 |-|-
-| `install`     | Install or update a plugin
+| `install`     | Install one or more plugins
 | `list`        | List currently installed plugins
 | `uninstall`   | Uninstall a plugin
 | `update `     | Update one or more plugins
@@ -39,6 +39,10 @@ steampipe plugin [command]
       <td nowrap="true"><inlineCode>--progress</inlineCode></td>
       <td>Enable or disable progress information. By default, progress information is shown - set <inlineCode>--progress=false</inlineCode> to hide the progress bar. Applies only to <inlineCode>plugin install</inlineCode> and <inlineCode>plugin update</inlineCode>.</td>
     </tr>
+      <tr>
+      <td nowrap="true"><inlineCode>--skip-config </inlineCode></td>
+      <td>Applies only to <inlineCode>plugin install</inlineCode>,  skip creating the default config file for plugin.</td>
+    </tr>
   </tbody>
 </table>
 
@@ -54,6 +58,12 @@ Install a specific version of a plugin:
 steampipe plugin install [email protected]
 ```
 
+Install all missing plugins that specified in configuration files. Do not download their default configuration files:
+
+```bash
+steampipe plugin install --skip-config
+```
+
 List installed plugins:
 ```bash
 steampipe plugin list

diff --git a/docs/reference/cli/service.md b/docs/reference/cli/service.md
@@ -31,12 +31,12 @@ steampipe service [command]
 | `--dashboard` |  `start` | Start the `dashboard` web server with the database service
 | `--dashboard-listen string` | `start` | Accept dashboard connections from: `local` (localhost only) or `network` (open)
 | `--dashboard-port int` |  `start` | Dashboard web server port (default `9194`)
-| `--database-listen string` |  `start` | Accept database connections from: `local` (localhost only) or `network` (open)
+| `--database-listen string` |  `start` | Accept connections from: local (alias for `localhost` only), `network` (alias for `*`), or a comma separated list of hosts and/or IP addresses (default `network`)
 | `--database-password string`  |  `start` |  Set the steampipe database password for this session.  See [STEAMPIPE_DATABASE_PASSWORD](reference/env-vars/steampipe_database_password) for additional information
-| `--database-port int` | `start` |  Database service port (default 9193)
+| `--database-port int` | `start` |  Database service port (default `9193`)
 | `--force` |  `stop`, `restart` | Forces the service to shutdown, releasing all open connections and ports
 | `--foreground` |  `start` | Run the service in the foreground
-| `--show-password` |  `start`, `status` | View database password for connecting from another machine (default false)
+| `--show-password` |  `start`, `status` | View database password for connecting from another machine (default `false`)
 | `--var stringArray` |  `start` | Specify the value of a variable (only applies if '--dashboard' flag is also set)
 | `--var-file strings` |  `start` | Specify an .spvar file containing variable values (only applies if '--dashboard' flag is also set)
 | `--all` |  `status` | Bypass the `--install-dir` and print status of all running services

diff --git a/docs/reference/config-files/connection.md b/docs/reference/config-files/connection.md
@@ -13,14 +13,13 @@ Most `connection` arguments are plugin-specific, and they are used to specify cr
 | Argument | Default | Values | Description 
 |-|-|-|-
 | `import_schema` | `enabled` | `enabled`, `disabled` |  Enable or disable the creation of a Postgres schema for this connection. When `import_schema` is disabled, Steampipe will not create a schema for the connection (and will delete it if it exists), but the connection will still be queryable from any aggregator that includes it.  For installations with a large number of connections, setting `import_schema` to `disabled` can decrease startup time and increase performance.
-| `plugin` | none     | [plugin version string](#plugin-version-strings) |  The plugin version that this connection uses.  This must refer to an [installed plugin version](/docs/managing/plugins#installing-plugins).
+| `plugin` | none     | [plugin version string](#plugin-version-strings) or [plugin reference](/docs/reference/config-files/plugin) |  The plugin version / instance that this connection uses.  This must refer to an [installed plugin version](/docs/managing/plugins#installing-plugins).
 | `type`   | `plugin` | `plugin`, `aggregator` | The type of connection  - [plugin connection](/docs/managing/plugins#installing-plugins) or [aggregator](/docs/managing/connections#using-aggregators).
 | `{plugin argument}`| varies |  varies|  Additional options are defined in each plugin - refer to the documentation for your plugin on the [Steampipe Hub](https://hub.steampipe.io/plugins).
 
 
 ### Plugin Version Strings 
 
-
 Steampipe plugin versions are in the format:
 ```
 [{organization}/]{plugin name}[@release stream]
@@ -65,6 +64,9 @@ connection "myplugin" {
 ```
 
 ## Examples
+
+Connections using [plugin version strings](#plugin-version-strings):
+
 ```hcl
 connection "aws_all" {
   type        = "aggregator"
@@ -92,4 +94,30 @@ connection "aws_03" {
   regions     = ["us-east-1", "us-west-2"]
 }
 
+```
+
+Connections using [plugin reference](/docs/reference/config-files/plugin):
+```hcl
+plugin "aws" {
+  memory_max_mb = 2048
+}
+
+connection "aws_all" {
+  type        = "aggregator"
+  plugin      = plugin.aws
+  connections = ["aws_*"]
+}
+
+connection "aws_01" {
+  plugin        = plugin.aws
+  profile       = "aws_01"
+  regions       = ["*"]
+}
+
+connection "aws_02" {
+  plugin        = plugin.aws
+  import_schema = "disabled"
+  profile       = "aws_02"
+  regions       = ["us-*", "eu-*"]
+}
 ```
diff --git a/docs/reference/config-files/options.md b/docs/reference/config-files/options.md
@@ -21,6 +21,7 @@ The following `options` are currently supported:
 | [database](#database-options)     | Database options.
 | [dashboard](#dashboard-options)   | Dashboard options.
 | [general](#general-options)       | General CLI options, such as auto-update options.
+| [plugin](#plugin-options)         | Plugin options.
 
 
 
@@ -97,6 +98,7 @@ options "dashboard" {
 | Argument | Default | Values | Description
 |-|-|-|-
 | `log_level` | `warn` | `trace`, `debug`, `info`, `warn`, `error` | Sets the output logging level. Standard log levels are supported. This can also be set via the  [STEAMPIPE_LOG_LEVEL](reference/env-vars/steampipe_log) environment variable.
+| `memory_max_mb` | `1024` | Set a memory soft limit for the `steampipe` process.  Set to `0` to disable the memory limit.  This can also be set via the [STEAMPIPE_MEMORY_MAX_MB](/docs/reference/env-vars/steampipe_memory_max_mb) environment variable.
 | `telemetry` | `none` | `none`, `info` | Set the telemetry level in Steampipe. This can also be set via the  [STEAMPIPE_TELEMETRY](reference/env-vars/steampipe_telemetry) environment variable. See also: [Telemetry](https://steampipe.io/blog/release-0-15-0#telemetry).
 | `update_check` | `true` | `true`, `false` | Enable or disable automatic update checking. This can also be set via the  [STEAMPIPE_UPDATE_CHECK](reference/env-vars/steampipe_update_check) environment variable.
 
@@ -112,12 +114,31 @@ options "dashboard" {
 ```hcl
 options "general" {
   log_level     = "warn"  # trace, debug, info, warn, error
+  memory_max_mb = 512     # megabytes
   telemetry     = "info"  # info, none
   update_check  = true    # true, false
 }
 ```
 
 ---
+## Plugin Options
+
+**Plugin** options are used to set plugin default options, such as memory soft limits.
+
+### Supported options
+| Argument | Default | Values | Description
+|-|-|-|-
+| `memory_max_mb` | `1024` | Set a default memory soft limit for each plugin process. Note that each plugin can have its own `memory_max_mb` set in [a `plugin` definition](/docs/reference/config-files/plugin), and that value would override this default setting. Set to `0` to disable the memory limit.  This can also be set via the [STEAMPIPE_PLUGIN_MEMORY_MAX_MB](/docs/reference/env-vars/steampipe_plugin_memory_max_mb) environment variable.
+
+
+### Example: Plugin Options
+
+```hcl
+options "plugin" {
+  memory_max_mb = 2048  # megabytes
+}
+```
+
 
 
 

diff --git a/docs/reference/config-files/overview.md b/docs/reference/config-files/overview.md
@@ -10,5 +10,5 @@ Configuration file resource are defined using HCL in one or more Steampipe confi
 
 Typically, config files are laid out as follows:
 - Steampipe creates a `~/.steampipe/config/default.spc` file for setting [options](reference/config-files/options).
-- Each plugin creates a `~/.steampipe/config/{plugin name}.spc` (e.g. `aws.spc`, `github.spc`, `net.spc`, etc). Define your [connections](reference/config-files/connection) in these files.
+- Each plugin creates a `~/.steampipe/config/{plugin name}.spc` (e.g. `aws.spc`, `github.spc`, `net.spc`, etc). Define your [connections](reference/config-files/connection) and [plugins](reference/config-files/plugin) in these files.
 - Define your [workspaces](reference/config-files/workspace) in `~/.steampipe/config/workspaces.spc`.