Resource Management

Many of the ASIC's internal resources are limited and are shared between several hardware procedures. In many cases the user can provide a partitioning scheme for such a resource in order to perform fine tuning for his application. In such cases performing driver reload is needed for the changes to take place.

Such an abstraction can be coupled with devlink's dpipe interface, which models the ASIC's pipeline as a graph of match/action tables. By modeling the hardware resource object, and by coupling it with several dpipe tables, further visibility can be achieved in order to debug ASIC-wide issues.

Table of Contents

1. Spectrum's Resources
2. Resource
   1. Resource Parameters
3. Dpipe Resource Interaction
4. Suggested Spectrum Resource Profiles

Spectrum's Resources

The main database in the Spectrum ASIC is a centralized KVD database which is shared between different entries such as FDB, routes, neighbours and more. The KVD is divided into several regions which are the linear, hash single and hash double parts.

                         +---------------+
                         |  KVD  memory  |
                         |               |
                         |  Size: 240K   |
                         +---------------+
                        /        |        \
                       /         |         \
                      /          |          \
                     /           |           \
                    /            |            \
         +--------------+ +--------------+  +--------------+
         |    Linear    | | Hash single  |  | Hash double  |
         |              | |              |  |              |
         |   Size: 96K  | | Size: 85K    |  |  Size:59K    |
         +--------------+ +--------------+  +--------------+
        /        |       \
       /         |        \
  +--------+ +--------+ +--------+
  | Single | | Chunks | | Large  |
  |        | |        | | Chunks |
  +--------+ +--------+ +--------+

The next table presents the mapping between resources and hardware entry types:

Resource Name	Entry Type
Linear-Single	Nexthops (Adjacency entries)
Linear-Chunks	Multiple nexthops for multipath routes requiring up to 32 entries each
Linear-Large-Chunks	Multiple nexthops for multipath routes requiring up to 512 entries each
Hash Single	FDB, IPv4 routes, IPv4 neighbours, IPv6 routes with prefix length <= 64
Hash Double	IPv6 routes with prefix length > 64, IPv6 neighbours

Resource

The ASIC's resources can be observed per device:

$ devlink -v resource show pci/0000:03:00.0
pci/0000:03:00.0:
  name kvd resource_path /kvd size 245760 unit entry dpipe_tables none
  resources:
    name linear resource_path /kvd/linear size 98304 occ 0 unit entry size_min 0 size_max 147456 size_gran 128
    dpipe_tables:
      table_name mlxsw_adj
    resources:
      name singles resource_path /kvd/linear/singles size 16384 occ 0 unit entry size_min 0 size_max 147456 size_gran 1 dpipe_tables none
      name chunks resource_path /kvd/linear/chunks size 49152 occ 0 unit entry size_min 0 size_max 147456 size_gran 32 dpipe_tables none
      name large_chunks resource_path /kvd/linear/large_chunks size 32768 occ 0 unit entry size_min 0 size_max 147456 size_gran 512 dpipe_tables none

    name hash_double resource_path /kvd/hash_double size 60416 unit entry size_min 32768 size_max 180224 size_gran 128
    dpipe_tables:
      table_name mlxsw_host6

    name hash_single resource_path /kvd/hash_single size 87040 unit entry size_min 65536 size_max 212992 size_gran 128
    dpipe_tables:
      table_name mlxsw_host4

Resource Parameters

• name - Resource's name
• resource_path - The full path of the resource
• size - The size of the resource
• size_new - The pending size. Reload is required for the change to happen
• size_valid - For nested resources only. In case the children's sizes exceed the size of the parent
• unit - The units in which the size is represented
• occ - Real time occupancy, represented in units. May not be available for all resources
• size_min - Minimum size. Not available for constant size resources
• size_max - Maximum size. Not available for constant size resources
• size_gran - Required granularity of the size. Not available for constant size resources
• dpipe_tables - The list of tables which use this resource
• resources - The list of child resources

The resources are presented in a tree based structure in order to represent the nested property, with a file system like path.

Resources whose size can change will present size properties which include minimum, maximum and granularity.

Example for changing a resource's size:

$ devlink resource set pci/0000:03:00.0 path /kvd/hash_double size 65536

In order for the changes to take affect reload is required:

$ devlink dev reload pci/0000:03:00.0

The reload process performs hot reload of the driver, thus it is recommended to perform the desired changes before any other configuration. The reload will fail in case one of the nested resources in the hierarchy is marked as having an invalid size:

$ devlink resource show pci/0000:03:00.0
pci/0000:03:00.0:
  name kvd size 245760 unit entry dpipe_tables none size_valid false
  resources:
    name linear size 98304 size_new 147456 occ 0 unit entry size_min 0 size_max 147456 size_gran 128
    dpipe_tables:
      table_name mlxsw_adj

    name hash_double size 60416 unit entry size_min 32768 size_max 180224 size_gran 128
    dpipe_tables:
      table_name mlxsw_host6

    name hash_single size 87040 unit entry size_min 65536 size_max 212992 size_gran 128
    dpipe_tables:
      table_name mlxsw_host4

In this case the linear's size was changed, but the overall configuration is invalid (kvd is marked as having an invalid size).

Dpipe Resource Interaction

Each dpipe table can be mapped to a single resource and specify the number of units it consumes for a single table entry. For example, the mlxsw_host6 table uses the /kvd/hash_double resource and consumes two units per table entry:

$ devlink dpipe table show pci/0000:03:00.0
pci/0000:03:00.0:
  name mlxsw_erif size 1000 counters_enabled false
  match:
    type field_exact header mlxsw_meta field erif_port mapping ifindex
  action:
    type field_modify header mlxsw_meta field l3_forward
    type field_modify header mlxsw_meta field l3_drop

  name mlxsw_host4 size 0 counters_enabled false resource_path /kvd/hash_single resource_units 1
  match:
    type field_exact header mlxsw_meta field erif_port mapping ifindex
    type field_exact header ipv4 field destination ip
  action:
    type field_modify header ethernet field destination mac

  name mlxsw_host6 size 0 counters_enabled false resource_path /kvd/hash_double resource_units 2
  match:
    type field_exact header mlxsw_meta field erif_port mapping ifindex
    type field_exact header ipv6 field destination ip
  action:
    type field_modify header ethernet field destination mac

  name mlxsw_adj size 0 counters_enabled false resource_path /kvd/linear resource_units 1
  match:
    type field_exact header mlxsw_meta field adj_index
    type field_exact header mlxsw_meta field adj_size
    type field_exact header mlxsw_meta field adj_hash_index
  action:
    type field_modify header ethernet field destination mac
    type field_modify header mlxsw_meta field erif_port mapping ifindex

Suggested Spectrum Resource Profiles

As the KVD can be re-partitioned using the devlink tool, here are several recommended sets of partition sizes that can be configured along-side with the number of networking features supported by each:

Profile	Linear	Linear Singles	Linear Chunks	Linear Large Chunks	Hash Single	Hash Double
Default	98304	16384	49152	32768	87040	60416
Scale	64000	15616	32000	16384	128000	48000
IPv4 Max	64000	15616	32000	16384	144000	32768

	Default	Scale	IPv4 Max
FDB	75K	105K	120K
Neighbor IPv4	78336	115200	129600
Neighbor IPv6	27187	21600	14745
Routes IPv4	73984	108800	133200

The following features are independent of the KVD partitioning, but the numbers supported by the driver is set here for simplicity of reach:

Feature	Amount
ACL rules	5631
MDB	6992