An overview of using SAS and the csi-lib-sas example program for attaching to a SAS storage device. This guide uses Linux commands to display server file system and device information. This guide attempts to use SCSI terminology.
The sas.go Attach()
function of this library uses the following information and device paths to discover SAS devices. The function also performs a SCSI bus rescan if a device cannot be discovered on the first pass. All of the information in this guidebook helps to explain the source code logic used in sas.go.
This guide provides system device details after creating a storage volume and mapping it to a server using a single transport path. That same volume is then mapped to 2nd SAS target port, resulting in the creation of a multipath device. The example program is used to display information using this SAS library.
Table of Contents
- (1) Storage Volume Creation aka Logical Unit (Single Path)
- (2) Storage Volume Mapping (Dual Path)
- (3) Running the Example Program
- (4) Additional Information
The first step is to create and map a storage volume from an external source so that is is visible on an attached computer system. The commands used to create and map a storage volume are vendor specific, so this section only shares the information needed from the server perspective. That information is needed so that the storage volume can be discovered in the host operating system.
For this example, here is the information pertaining to the storage volume created on an external storage array and mapped to a server.
Item |
Value |
Notes |
---|---|---|
Volume | ph_00000000000000000000000000000 | The name given to the storage volume created. |
World Wide Name (WWN) | 600c0ff000546067a079e36201000000 | The WWN of the storage volume created. |
World Wide Port Name (WWPN) 1 | 500c0ff0afe43000 | 1st SAS port used to connect this volume to the server. |
World Wide Port Name (WWPN) 2 | 500c0ff0afe43400 | 2nd SAS port used to connect this volume to the server. |
Logical Unit Number (LUN) | 2 | The LUN used to connect this volume to the server. |
After the storage volume is created and mapped on the external storage device, a SCSI rescan is performed on the server to discover and map that storage to an operating system device. The following is a table of the new operating system device discovered and created after the SCSI bus rescan.
Item | Value |
---|---|
LUN 2 WWPN 2 | sdb |
When running a rescan of the SCSI bus, the output indicates old and new devices discovered. In this example, only the scanning of host 8 is included to reduce the lines of output. Host 8 is the SAS host that has cables connected to the storage. Two lines of output are highlighted below followed by the entire host 8 scan results.
- NEW: Host: scsi8 Channel: 00 Id: 00 Lun: 02
$ sudo /usr/bin/rescan-scsi-bus.sh
Scanning SCSI subsystem for new devices
Scanning host 8 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 8 0 0 0 ...
OLD: Host: scsi8 Channel: 00 Id: 00 Lun: 00
Vendor: SEAGATE Model: 4006 Rev: I210
Type: Enclosure ANSI SCSI revision: 06
Scanning for device 8 0 0 2 ...
NEW: Host: scsi8 Channel: 00 Id: 00 Lun: 02
Vendor: SEAGATE Model: 4006 Rev: I210
Type: Direct-Access ANSI SCSI revision: 06
Scanning for device 8 0 2 0 ...
OLD: Host: scsi8 Channel: 00 Id: 02 Lun: 00
Vendor: SEAGATE Model: 4006 Rev: I210
Type: Enclosure ANSI SCSI revision: 06
1 new or changed device(s) found.
[8:0:0:2]
0 remapped or resized device(s) found.
0 device(s) removed.
The next step is to review a list of devices created using the ls -lR /dev/disk
command. This produces a recursive list of devices for each subfolder. In this example, many lines were removed if they did not relate to the new device created, so this is example text that was reduced in line count.
One explanation of the output listed below. There are several different ways sdb
is listed as a device name.
Listing |
Notes |
---|---|
wwn-0x600c0ff000546067a079e36201000000 -> ../../sdb | The WWN of the storage volume mapped to sdb |
$ ls -lR /dev/disk
/dev/disk/by-id:
lrwxrwxrwx. 1 root root 9 Jul 29 10:02 scsi-0SEAGATE_4006_113061666534330000c0ff54606500000aebc87600c0ff54606700000aebc875 -> ../../sdb
lrwxrwxrwx. 1 root root 9 Jul 29 10:02 scsi-3600c0ff000546067a079e36201000000 -> ../../sdb
lrwxrwxrwx. 1 root root 9 Jul 29 10:02 scsi-SSEAGATE_4006_00c0ff5460670000a079e36201000000 -> ../../sdb
lrwxrwxrwx. 1 root root 9 Jul 29 10:02 wwn-0x600c0ff000546067a079e36201000000 -> ../../sdb
/dev/disk/by-path:
lrwxrwxrwx. 1 root root 9 Jul 29 10:02 pci-0000:09:00.0-sas-0x500c0ff0afe43400-lun-2 -> ../../sdb
The same device is also listed under the /dev/
folder and the /sys/block
folder. We can also see that sdb
has no associated devices, indicating it is not a multipath device.
$ ls -l /dev/sd*
brw-rw----. 1 root disk 8, 16 Jul 29 10:02 /dev/sdb
$ ls -l /sys/block/sd*
lrwxrwxrwx. 1 root root 0 Jul 29 10:02 /sys/block/sdb -> ../devices/pci0000:00/0000:00:03.0/0000:05:00.0/0000:06:09.0/0000:09:00.0/host8/port-8:0/end_device-8:0/target8:0:0/8:0:0:2/block/sdb
$ ls -l /sys/block/sdb/slaves
total 0
The tool lsscsi
produces useful information mapping operating system devices to the storage volume. For this guidebook, only host 8 lines are listed. A user can see how the device sdb is linked to the sas disk port 0x500c0ff0afe43400 and lun 2. The tool also produces additional details if lsscsi -t --list
is used.
$ lsscsi -t
[8:0:0:0] enclosu sas:0x500c0ff0afe43400 -
[8:0:0:2] disk sas:0x500c0ff0afe43400 /dev/sdb
[8:0:2:0] enclosu sas:0x500c0ff0afe43000 -
At this point, we have created a storage volume and mapped it to a single SAS port. The next step is to map the same storage volume to a second SAS port, thus creating a dual path from the server to the same storage volume. With the multipath daemon installed and running, this step displays the system information after the second map and scsi bus rescan.
After the storage volume is mapped again on the external storage device, a SCSI rescan is performed on the server to discover and map that storage to operating system devices. This example is running the mulitpath daemon which creates an additional multipath device. Here is the list of devices created. The following is a table of new operating system devices discovered and created after the second SCSI bus rescan. The device sdb
is listed a second time for reference although it already exists.
Item | Value |
---|---|
LUN 2 WWPN 1 | sdc |
LUN 2 WWPN 2 | sdb |
Multipath Device | dm-5 |
When running a rescan of the SCSI bus, the output indicates old and new devices discovered. In this example, only the scanning of host 8 is included to reduce the lines of output. Host 8 is the SAS host that has cables connected to the storage. Two lines of output are highlighted below followed by the entire host 8 scan results.
- NEW: Host: scsi8 Channel: 00 Id: 02 Lun: 02
$ sudo /usr/bin/rescan-scsi-bus.sh
Scanning SCSI subsystem for new devices
Scanning host 8 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 8 0 0 0 ...
OLD: Host: scsi8 Channel: 00 Id: 00 Lun: 00
Vendor: SEAGATE Model: 4006 Rev: I210
Type: Enclosure ANSI SCSI revision: 06
Scanning for device 8 0 0 2 ...
OLD: Host: scsi8 Channel: 00 Id: 00 Lun: 02
Vendor: SEAGATE Model: 4006 Rev: I210
Type: Direct-Access ANSI SCSI revision: 06
Scanning for device 8 0 2 0 ...
OLD: Host: scsi8 Channel: 00 Id: 02 Lun: 00
Vendor: SEAGATE Model: 4006 Rev: I210
Type: Enclosure ANSI SCSI revision: 06
Scanning for device 8 0 2 2 ...
NEW: Host: scsi8 Channel: 00 Id: 02 Lun: 02
Vendor: SEAGATE Model: 4006 Rev: I210
Type: Direct-Access ANSI SCSI revision: 06
1 new or changed device(s) found.
[8:0:2:2]
0 remapped or resized device(s) found.
0 device(s) removed.
The next step is to review a list of devices created using the ls -lR /dev/disk
command. This produces a recursive list of devices for each subfolder. In this example, many lines were removed if they were not related to the new devices created, so this is example text that was reduced in line count.
Some explanations of the output listed below. There are several different ways dm-5
is listed as a /dev/disk/by-id
device name.
Listing |
Notes |
---|---|
wwn-0x600c0ff000546067a079e36201000000 -> ../../dm-5 | The WWN of the storage volume mapped to dm-5 |
$ ls -lR /dev/disk
/dev/disk/by-id:
lrwxrwxrwx. 1 root root 10 Jul 29 10:26 dm-name-3600c0ff000546067a079e36201000000 -> ../../dm-5
lrwxrwxrwx. 1 root root 10 Jul 29 10:26 dm-uuid-mpath-3600c0ff000546067a079e36201000000 -> ../../dm-5
lrwxrwxrwx. 1 root root 10 Jul 29 10:26 scsi-3600c0ff000546067a079e36201000000 -> ../../dm-5
lrwxrwxrwx. 1 root root 10 Jul 29 10:26 wwn-0x600c0ff000546067a079e36201000000 -> ../../dm-5
/dev/disk/by-path:
lrwxrwxrwx. 1 root root 9 Jul 29 10:26 pci-0000:09:00.0-sas-0x500c0ff0afe43000-lun-2 -> ../../sdc
lrwxrwxrwx. 1 root root 9 Jul 29 10:26 pci-0000:09:00.0-sas-0x500c0ff0afe43400-lun-2 -> ../../sdb
The ls -lR /dev/disk
command can be used to discover the SCSI devices created after the storage volume is mapped to the server. These devices are referenced by their World Wide Port Name (WWPN) and Logical Unit Number (LUN).
Listing |
Notes |
---|---|
pci-0000:09:00.0-sas-0x500c0ff0afe43000-lun-2 -> ../../sdc | WWPN 1, and lun, mapped to sdc |
pci-0000:09:00.0-sas-0x500c0ff0afe43400-lun-2 -> ../../sdb | WWPN 2, and lun, mapped to sdb |
Once we know the multipath device dm-5
we can use the /sys/block
information to discover the linkage between dm-5
and sdc
and sdb
. For each multipath device, the following path can be explored to discover the linked devices using /sys/block/<dm-#>/slaves
.
$ ls -l /sys/block
lrwxrwxrwx. 1 root root 0 Jul 29 10:26 dm-5 -> ../devices/virtual/block/dm-5
lrwxrwxrwx. 1 root root 0 Jul 29 10:02 sdb -> ../devices/pci0000:00/0000:00:03.0/0000:05:00.0/0000:06:09.0/0000:09:00.0/host8/port-8:0/end_device-8:0/target8:0:0/8:0:0:2/block/sdb
lrwxrwxrwx. 1 root root 0 Jul 29 10:26 sdc -> ../devices/pci0000:00/0000:00:03.0/0000:05:00.0/0000:06:09.0/0000:09:00.0/host8/port-8:2/end_device-8:2/target8:0:2/8:0:2:2/block/sdc
$ ls -l /sys/block/dm-5/slaves
lrwxrwxrwx. 1 root root 0 Jul 29 10:26 sdb -> ../../../../pci0000:00/0000:00:03.0/0000:05:00.0/0000:06:09.0/0000:09:00.0/host8/port-8:0/end_device-8:0/target8:0:0/8:0:0:2/block/sdb
lrwxrwxrwx. 1 root root 0 Jul 29 10:26 sdc -> ../../../../pci0000:00/0000:00:03.0/0000:05:00.0/0000:06:09.0/0000:09:00.0/host8/port-8:2/end_device-8:2/target8:0:2/8:0:2:2/block/sdc
Once we know the WWN of storage volume, we can prepend 3
to it and find the multipath device using the /dev/mapper
information. In this case, we see /dev/mapper/3<wwn>/
is linked to dm-5
, our multipath device for this storage volume.
$ ls -l /dev/mapper
lrwxrwxrwx. 1 root root 7 Jul 29 10:26 3600c0ff000546067a079e36201000000 -> ../dm-5
The tool lsscsi
produces useful information mapping operating system devices to the storage volume. For this guidebook, only host 8 lines are listed. A user can see how the device sdb is linked to the sas disk port 0x500c0ff0afe43400 and lun 2. And see how the device sdc is linked to the sas disk port 0x500c0ff0afe43000 and lun 2. Use lsscsi -t --list
to list additional details.
$ lsscsi -t
[8:0:0:0] enclosu sas:0x500c0ff0afe43400 -
[8:0:0:2] disk sas:0x500c0ff0afe43400 /dev/sdb
[8:0:2:0] enclosu sas:0x500c0ff0afe43000 -
[8:0:2:2] disk sas:0x500c0ff0afe43000 /dev/sdc
For multipath devices, additional information is obtained using the multipath
tool. This output produces information relating the WWN of the storage volume (600c0ff000546067a079e36201000000) to the multipath device (dm-5) and also lists each linked device (sdc and sdb) used to produce dm-5.
$ sudo multipath -ll
3600c0ff000546067a079e36201000000 dm-5 SEAGATE,4006
size=1.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 8:0:2:2 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
`- 8:0:0:2 sdb 8:16 active ready running
The example program allows you to attach and detach to a storage volume using the sas.go library functions Attach()
and Detach()
.
$ ./example -h
====================================================================================================
example - Run the SAS Library Example Program
Examples:
./example -wwn 600c0ff000546067369fe36201000000
./example -wwn 600c0ff000546067369fe36201000000 -v=4
./example -wwn 600c0ff000546067369fe36201000000 -v=4 -detach
Low level library commands require sudo or root privilege. You must provide a wwn to be succeed.
Options:
-add_dir_header
If true, adds the file directory to the header of the log messages
-alsologtostderr
log to standard error as well as files (no effect when -logtostderr=true)
-detach
automatically detach after a successful attach
-log_backtrace_at value
when logging hits line file:N, emit a stack trace
-log_dir string
If non-empty, write log files in this directory (no effect when -logtostderr=true)
-log_file string
If non-empty, use this log file (no effect when -logtostderr=true)
-log_file_max_size uint
Defines the maximum size a log file can grow to (no effect when -logtostderr=true). Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
-logtostderr
log to standard error instead of files (default true)
-one_output
If true, only write logs to their native severity level (vs also writing to each lower severity level; no effect when -logtostderr=true)
-skip_headers
If true, avoid header prefixes in the log messages
-skip_log_headers
If true, avoid headers when opening log files (no effect when -logtostderr=true)
-stderrthreshold value
logs at or above this threshold go to stderr when writing to files and stderr (no effect when -logtostderr=true or -alsologtostderr=false) (default 2)
-v value
number for the log level verbosity
-vmodule value
comma-separated list of pattern=N settings for file-filtered logging
-wwn string
Specify a WWN
$ ./example -wwn 600c0ff000546067a079e36201000000
I0729 15:14:42.580104 226903 main.go:60] "[] sas test example" wwn="600c0ff000546067a079e36201000000" detach=false
I0729 15:14:42.580908 226903 main.go:72] "SAS Attach success" device="/dev/sdb" connector={VolumeName: TargetWWN:600c0ff000546067a079e36201000000 Multipath:false TargetDevice:/dev/sdb SCSIDevices:[] IoHandler:<nil>}
$ ./example -wwn 600c0ff000546067a079e36201000000 -v=3
I0729 15:15:13.931817 226915 main.go:60] "[] sas test example" wwn="600c0ff000546067a079e36201000000" detach=false
I0729 15:15:13.932071 226915 sas.go:63] "Attaching SAS storage volume"
I0729 15:15:13.932105 226915 sas.go:265] "find disk by id" wwnPath="wwn-0x600c0ff000546067a079e36201000000" devPath="/dev/disk/by-id/"
I0729 15:15:13.932522 226915 sas.go:272] "found device, evaluate symbolic links" devPath="/dev/disk/by-id/" name="wwn-0x600c0ff000546067a079e36201000000"
I0729 15:15:13.932730 226915 sas.go:291] "find linked devices on multipath" dm="/dev/sdb"
I0729 15:15:13.932769 226915 sas.go:301] "linked scsi devices path" linkPath="/sys/block/sdb/slaves"
I0729 15:15:13.932886 226915 sas.go:307] "linked scsi devices found" devices=[]
I0729 15:15:13.932916 226915 sas.go:213] "find disk by id returned" dm="/dev/sdb" devices=[]
I0729 15:15:13.932977 226915 main.go:72] "SAS Attach success" device="/dev/sdb" connector={VolumeName: TargetWWN:600c0ff000546067a079e36201000000 Multipath:false TargetDevice:/dev/sdb SCSIDevices:[] IoHandler:<nil>}
$ ./example -wwn 600c0ff000546067a079e36201000000
I0729 15:17:35.043454 227729 main.go:60] "[] sas test example" wwn="600c0ff000546067a079e36201000000" detach=false
I0729 15:17:35.044334 227729 main.go:72] "SAS Attach success" device="/dev/dm-5" connector={VolumeName: TargetWWN:600c0ff000546067a079e36201000000 Multipath:true TargetDevice:/dev/dm-5 SCSIDevices:[/dev/sdb /dev/sdc] IoHandler:<nil>}
$ ./example -wwn 600c0ff000546067a079e36201000000 -v=3
I0729 15:17:42.491035 227743 main.go:60] "[] sas test example" wwn="600c0ff000546067a079e36201000000" detach=false
I0729 15:17:42.491287 227743 sas.go:63] "Attaching SAS storage volume"
I0729 15:17:42.491321 227743 sas.go:265] "find disk by id" wwnPath="wwn-0x600c0ff000546067a079e36201000000" devPath="/dev/disk/by-id/"
I0729 15:17:42.491736 227743 sas.go:272] "found device, evaluate symbolic links" devPath="/dev/disk/by-id/" name="wwn-0x600c0ff000546067a079e36201000000"
I0729 15:17:42.491815 227743 sas.go:291] "find linked devices on multipath" dm="/dev/dm-5"
I0729 15:17:42.491849 227743 sas.go:301] "linked scsi devices path" linkPath="/sys/block/dm-5/slaves"
I0729 15:17:42.491993 227743 sas.go:307] "linked scsi devices found" devices=[/dev/sdb /dev/sdc]
I0729 15:17:42.492029 227743 sas.go:213] "find disk by id returned" dm="/dev/dm-5" devices=[/dev/sdb /dev/sdc]
I0729 15:17:42.492054 227743 sas.go:216] "add scsi device" device="/dev/sdb"
I0729 15:17:42.492077 227743 sas.go:216] "add scsi device" device="/dev/sdc"
I0729 15:17:42.492103 227743 sas.go:226] "multipath device was discovered" dm="/dev/dm-5" SCSIDevices=[/dev/sdb /dev/sdc]
I0729 15:17:42.492147 227743 main.go:72] "SAS Attach success" device="/dev/dm-5" connector={VolumeName: TargetWWN:600c0ff000546067a079e36201000000 Multipath:true TargetDevice:/dev/dm-5 SCSIDevices:[/dev/sdb /dev/sdc] IoHandler:<nil>}
$ sudo ./example -wwn 600c0ff000546067a079e36201000000 -v=3 -detach
I0729 15:17:54.470808 227761 main.go:60] "[] sas test example" wwn="600c0ff000546067a079e36201000000" detach=true
I0729 15:17:54.471010 227761 sas.go:63] "Attaching SAS storage volume"
I0729 15:17:54.471039 227761 sas.go:265] "find disk by id" wwnPath="wwn-0x600c0ff000546067a079e36201000000" devPath="/dev/disk/by-id/"
I0729 15:17:54.471427 227761 sas.go:272] "found device, evaluate symbolic links" devPath="/dev/disk/by-id/" name="wwn-0x600c0ff000546067a079e36201000000"
I0729 15:17:54.471504 227761 sas.go:291] "find linked devices on multipath" dm="/dev/dm-5"
I0729 15:17:54.471537 227761 sas.go:301] "linked scsi devices path" linkPath="/sys/block/dm-5/slaves"
I0729 15:17:54.471702 227761 sas.go:307] "linked scsi devices found" devices=[/dev/sdb /dev/sdc]
I0729 15:17:54.471739 227761 sas.go:213] "find disk by id returned" dm="/dev/dm-5" devices=[/dev/sdb /dev/sdc]
I0729 15:17:54.471766 227761 sas.go:216] "add scsi device" device="/dev/sdb"
I0729 15:17:54.471787 227761 sas.go:216] "add scsi device" device="/dev/sdc"
I0729 15:17:54.471814 227761 sas.go:226] "multipath device was discovered" dm="/dev/dm-5" SCSIDevices=[/dev/sdb /dev/sdc]
I0729 15:17:54.471856 227761 main.go:72] "SAS Attach success" device="/dev/dm-5" connector={VolumeName: TargetWWN:600c0ff000546067a079e36201000000 Multipath:true TargetDevice:/dev/dm-5 SCSIDevices:[/dev/sdb /dev/sdc] IoHandler:<nil>}
I0729 15:17:54.471886 227761 sas.go:82] "Detaching SAS volume" devicePath="/dev/dm-5"
I0729 15:17:54.471923 227761 sas.go:291] "find linked devices on multipath" dm="/dev/dm-5"
I0729 15:17:54.471951 227761 sas.go:301] "linked scsi devices path" linkPath="/sys/block/dm-5/slaves"
I0729 15:17:54.472050 227761 sas.go:307] "linked scsi devices found" devices=[/dev/sdb /dev/sdc]
I0729 15:17:54.472080 227761 sas.go:97] "sas: DetachDisk" devicePath="/dev/dm-5" dstPath="/dev/dm-5" devices=[/dev/sdb /dev/sdc]
I0729 15:17:54.472106 227761 sas.go:326] "remove device from scsi-subsystem" path="/sys/block/sdb/device/delete"
I0729 15:17:54.491171 227761 sas.go:326] "remove device from scsi-subsystem" path="/sys/block/sdc/device/delete"
E0729 15:17:54.512141 227761 main.go:76] "SAS Detach success"
Here are several development troubleshooting tips that may prove useful, but each should be use with extreme caution if used on servers that have live user data is use. Some or all of these commands may require sudo
or root privileges in order to run.
You can use the echo
command to rescan a single SCSI host. In fact, this library performs a SCSI bus rescan if a device cannot be discovered on a first pass. The placeholder <hostX>
must be replaced with the name of the host to scan. Use ls -l /sys/class/scsi_host
to determine a list of all active SCSI hosts.
echo "- - -" > /sys/class/scsi_host/<hostX>/scan
After removing a storage volume, unmapping it, and performing a SCSI bus rescan; you may find that the operating system still lists the device under one or more /dev/disk subfolders. You can use the following command to remove that stale device. The placeholder <sdX>
must be replaced with the name of your device. For example, the devices created in this guide where sdb
and sdc
.
Note: Be careful, don’t remove devices you are using!
With root privilege, call echo directly, otherwise use a sudo shell.
echo 1 > /sys/block/<sdX>/device/delete
sudo sh -c 'echo 1 > /sys/block/<sdX>/device/delete'
To see host specific SAS details, use cat
of the host SAS address file.
$ ls -l /sys/class/sas_host
lrwxrwxrwx. 1 root root 0 Jul 25 15:36 host8 -> ../../devices/pci0000:00/0000:00:03.0/0000:05:00.0/0000:06:09.0/0000:09:00.0/host8/sas_host/host8
$ cat /sys/class/scsi_host/host8/host_sas_address
0x500605b0091fbb20