Tuesday, August 21, 2012

Device Mapper Multupath configuration and Administration

Device Mapper Multipathing (DM-Multipath) allows you to configure multiple I/O paths between server nodes and storage arrays into a single device. These I/O paths are physical SAN connections that can include separate cables, switches, and controllers. Multipathing aggregates the I/O paths, creating a new device that consists of the aggregated paths.


Overview of DM-Multipath

DM-Multipath can be used to provide:


Redundancy

DM-Multipath can provide failover in an active/passive configuration. In an active/passive configuration, only half the paths are used at any time for I/O. If any element of an I/O path (the cable, switch, or controller) fails, DM-Multipath switches to an alternate path.


Improved Performance

DM-Multipath can be configured in active/active mode, where I/O is spread over the paths in a round-robin fashion. In some configurations, DM-Multipath can detect loading on the I/O paths and dynamically re-balance the load.


DM-Multipath Setup Overview

 DM-Multipath includes compiled-in default settings that are suitable for common multipath configurations. Setting up DM-multipath is often a simple procedure.

 The basic procedure for configuring your system with DM-Multipath is as follows:

 1.Install device-mapper-multipath rpm. 

 2.Edit the multipath.conf configuration file:

  • comment out the default blacklist
  • change any of the existing defaults as needed
  • save the configuration file

 3.Start the multipath daemons.

 4.Create the multipath device with the multipath command. 


Multipath Device Identifiers


 Each multipath device has a World Wide Identifier (WWID), which is guaranteed to be globally unique and unchanging. By default, the name of a multipath device is set to its WWID. Alternately, you can set the user_friendly_names option in the multipath configuration file, which sets the alias to a node-unique name of the form mpath n.


 For example, a node with two HBAs attached to a storage controller with two ports via a single unzoned FC switch sees four devices: /dev/sda, /dev/sdb, dev/sdc, and /dev/sdd. DM-Multipath creates a single device with a unique WWID that reroutes I/O to those four underlying devices according to the multipath configuration. When the user_friendly_names configuration option is set to yes, the name of the multipath device is set to mpath n.


 When new devices are brought under the control of DM-Multipath, the new devices may be seen in three different places under the /dev directory: /dev/mapper/mpath n, /dev/mpath/mpath n, and /dev/dm- n.

  •  The devices in /dev/mapper are created early in the boot process. Use these devices to access the multipathed devices, for example when creating logical volumes.
  • The devices in /dev/mpath are provided as a convenience so that all multipathed devices can be seen in one directory. These devices are created by the udev device manager and may not be available on startup when the system needs to access them. Do not use these devices for creating logical volumes or filesystems.
  •  Any devices of the form /dev/dm- n are for internal use only and should never be used.

You can also set the name of a multipath device to a name of your choosing by using the alias option in the multipaths section of the multipath configuration file


Consistent Multipath Device Names in a Cluster


 When the user_friendly_names configuration option is set to yes, the name of the multipath device is unique to a node, but it is not guaranteed to be the same on all nodes using the multipath device. Similarly, if you set the alias option for a device in the multipaths section of the multipath.conf configuration file, the name is not automatically consistent across all nodes in the cluster. This should not cause any difficulties if you use LVM to create logical devices from the multipath device, but if you require that your multipath device names be consistent in every node it is recommended that you not set the user_friendly_names option to yes and that you not configure aliases for the devices. By default, if you do not set user_friendly_names to yes or configure an alias for a device, a device name will be the WWID for the device, which is always the same.


If you want the system-defined user-friendly names to be consistent across all nodes in the cluster, however, you can follow this procedure: 


1.Set up all of the multipath devices on one machine.


2.Disable all of your multipath devices on your other machines by running the following commands:

# service multipathd stop

# multipath -F


3.Copy the bindings file from the first machine to all the other machines in the cluster. By default, the location of this file is /var/lib/multipath/bindings. If /var is a separate partition on your system, however, you should change this value with the bindings_file option in the defaults section of the multipath.conf configuration file, This file needs to be located on your root file system partition, for example: 


bindings_file "/etc/multipath_bindings"


4.Re-enable the multipathd daemon on all the other machines in the cluster by running the following command:

# service mutipathd start


 If you add a new device, you will need to repeat this process.


 Similarly, if you configure an alias for a device that you would like to be consistent across the nodes in the cluster, you should ensure that the /etc/multipath.conf file is the same for each node in the cluster by following the same procedure:


1.Configure the aliases for the multipath devices in the in the multipath.conf file on one machine.

2.Disable all of your multipath devices on your other machines by running the following commands:

# service multipathd stop

# multipath -F


3.Copy the /etc/multipath.conf file from the first machine to all the other machines in the cluster.


4.Re-enable the multipathd daemon on all the other machines in the cluster by running the following command:

# service mutipathd start


 When you add a new device you will need to repeat this process.


Multipath Devices in Logical Volumes


 After creating multipath devices, you can use the multipath device names just as you would use a physical device name when creating an LVM physical volume. For example, if /dev/mapper/mpath0 is the name of a multipath device, the following command will mark /dev/mapper/mpath0 as a physical volume.

pvcreate /dev/mapper/mpath0


 You can use the resulting LVM physical device when you create an LVM volume group just as you would use any other LVM physical device.


Note


 If you attempt to create an LVM physical volume on a whole device on which you have configured partitions, the pvcreate command will fail. Note that the Anaconda and Kickstart installation programs create empty partition tables if you do not specify otherwise for every block device. If you wish to use the whole device rather than a partition, you must remove the existing partitions from the device. You can remove existing partitions with the kpartx -d and the fdisk commands. If your system has block devices that are greater that 2Tb, you can use the parted command to remove partitions.


 When you create an LVM logical volume that uses active/passive multipath arrays as the underlying physical devices, you should include filters in the lvm.conf to exclude the disks that underlie the multipath devices. This is because if the array automatically changes the active path to the passive path when it receives I/O, multipath will failover and failback whenever LVM scans the passive path if these devices are not filtered. For active/passive arrays that require a command to make the passive path active, LVM prints a warning message when this occurs.


 To filter all SCSI devices in the LVM configuration file ( lvm.conf), include the following filter in the devices section of the file.


filter = [ "r/disk/", "r/sd.*/", "a/.*/" ]


Setting Up DM-Multipath


 Before setting up DM-Multipath on your system, ensure that your system has been updated and includes the device-mapper-multipath package.


 Use the following procedure to set up DM-Multipath for a basic failover configuration.



1.Edit the /etc/multipath.conf file by commenting out the following lines at the top of the file. This section of the configuration file, in its initial state, blacklists all devices. You must comment it out to enable multipathing.


blacklist {

        devnode "*"

}


 After commenting out those lines, this section appears as follows.


# blacklist {

#        devnode "*"

# }



2.The default settings for DM-Multipath are compiled in to the system and do not need to be explicitly set in the /etc/multipath.conf file.


 The default value of path_grouping_policy is set to failover, so in this example you do not need to change the default value. For information on changing the values in the configuration file to something other than the defaults. 


The initial defaults section of the configuration file configures your system that the names of the multipath devices are of the form mpath n; without this setting, the names of the multipath devices would be aliased to the WWID of the device.


3.Save the configuration file and exit the editor.


4.Execute the following commands:


modprobe dm-multipath

service multipathd start

multipath -v2


 The multipath -v2 command prints out multipathed paths that show which devices are multipathed. If the command does not print anything out, ensure that all SAN connections are set up properly and the system is multipathed.


5.Execute the following command to ensure sure that the multipath daemon starts on bootup:


chkconfig multipathd on


 Since the value of user_friendly_name is set to yes in the configuration file the multipath devices will be created as /dev/mapper/mpath n. 

Ignoring Local Disks when Generating Multipath Devices

Some machines have local SCSI cards for their internal disks. DM-Multipath is not recommended for these devices. The following procedure shows how to modify the multipath configuration file to ignore the local disks when configuring multipath.


1.Determine which disks are the internal disks and mark them as the ones to blacklist.


 In this example, /dev/sda is the internal disk. Note that as originally configured in the default multipath configuration file, executing the multipath -v2 shows the local disk, /dev/sda, in the multipath map.


[root@rh4cluster1 ~]# multipath -v2

create: SIBM-ESXSST336732LC____F3ET0EP0Q000072428BX1

[size=33 GB][features="0"][hwhandler="0"]

\_ round-robin 0

  \_ 0:0:0:0 sda  8:0    [---------


device-mapper ioctl cmd 9 failed: Invalid argument

device-mapper ioctl cmd 14 failed: No such device or address

create: 3600a0b80001327d80000006d43621677

[size=12 GB][features="0"][hwhandler="0"]

\_ round-robin 0

  \_ 2:0:0:0 sdb  8:16   

  \_ 3:0:0:0 sdf  8:80   


create: 3600a0b80001327510000009a436215ec

[size=12 GB][features="0"][hwhandler="0"]

\_ round-robin 0

  \_ 2:0:0:1 sdc  8:32   

  \_ 3:0:0:1 sdg  8:96   


create: 3600a0b80001327d800000070436216b3

[size=12 GB][features="0"][hwhandler="0"]

\_ round-robin 0

  \_ 2:0:0:2 sdd  8:48   

  \_ 3:0:0:2 sdh  8:112  


create: 3600a0b80001327510000009b4362163e

[size=12 GB][features="0"][hwhandler="0"]

\_ round-robin 0

  \_ 2:0:0:3 sde  8:64   

  \_ 3:0:0:3 sdi  8:128


2.In order to prevent the device mapper from mapping /dev/sda in its multipath maps, edit the blacklist section of the /etc/multipath.conf file to include this device. Although you could blacklist the sda device using a devnode type, that would not be safe procedure since /dev/sda is not guaranteed to be the same on reboot. To blacklist individual devices, you can blacklist using the WWID of that device. 


 Note that in the output to the multipath -v2 command, the WWID of the /dev/sda device is SIBM-ESXSST336732LC____F3ET0EP0Q000072428BX1. To blacklist this device, include the following in the /etc/multipath.conf file.


blacklist {

      wwid SIBM-ESXSST336732LC____F3ET0EP0Q000072428BX1

}



3.After you have updated the /etc/multipath.conf file, you must manually tell the multipathd daemon to reload the file. The following command reloads the updated /etc/multipath.conf file.


service multipathd reload


4.Run the following commands:

multipath -F

multipath -v2


 The local disk or disks should no longer be listed in the new multipath maps, as shown in the following example.


[root@rh4cluster1 ~]# multipath -F

[root@rh4cluster1 ~]# multipath -v2

create: 3600a0b80001327d80000006d43621677

[size=12 GB][features="0"][hwhandler="0"]

\_ round-robin 0

  \_ 2:0:0:0 sdb  8:16   

  \_ 3:0:0:0 sdf  8:80   


create: 3600a0b80001327510000009a436215ec

[size=12 GB][features="0"][hwhandler="0"]

\_ round-robin 0

  \_ 2:0:0:1 sdc  8:32   

  \_ 3:0:0:1 sdg  8:96   


create: 3600a0b80001327d800000070436216b3

[size=12 GB][features="0"][hwhandler="0"]

\_ round-robin 0

  \_ 2:0:0:2 sdd  8:48   

  \_ 3:0:0:2 sdh  8:112  


create: 3600a0b80001327510000009b4362163e

[size=12 GB][features="0"][hwhandler="0"]

\_ round-robin 0

  \_ 2:0:0:3 sde  8:64   

  \_ 3:0:0:3 sdi  8:128


Adding Devices to the Multipathing Database 


 By default, DM-Multipath includes support for the most common storage arrays that support DM-Multipath. The default configuration values, including supported devices, can be found in the multipath.conf.defaults file.


 If you need to add a storage device that is not supported by default as a known multipath device, edit the /etc/multipath.conf file and insert the appropriate device information.


 For example, to add information about the HP Open-V series the entry looks like this:

devices {

       device {

               vendor "HP"

               product "OPEN-V."

               getuid_callout "/sbin/scsi_id -g -u -p0x80 -s /block/%n"

       }

}




Configuration File Overview


 The multipath configuration file is divided into the following sections:


blacklist


 Listing of specific devices that will not be considered for multipath. By default all devices are blacklisted. Usually the default blacklist section is commented out.


blacklist_exceptions


 Listing of multipath candidates that would otherwise be blacklisted according to the parameters of the blacklist section.


defaults


 General default settings for DM-Multipath.


multipaths


 Settings for the characteristics of individual multipath devices. These values overwrite what is specified in the defaults and devices sections of the configuration file.


devices


 Settings for the individual storage controllers. These values overwrite what is specified in the defaults section of the configuration file. If you are using a storage array that is not supported by default, you may need to create a devices subsection for your array.


 When the system determines the attributes of a multipath device, first it checks the multipath settings, then the per devices settings, then the multipath system defaults.



 Troubleshooting with the multipathd Interactive Console


Resizing an Online Multipath Device


If you need to resize an online multipath device, use the following procedure. 


1.Resize your physical device.


2. Use the following command to find the paths to the LUN:

# multipath -l


3. Resize your paths. For SCSI devices, writing a 1 to the rescan file for the device causes the SCSI driver to rescan, as in the following command:

# echo 1 > /sys/block/device_name/device/rescan


4.Resize your multipath device by running the multipathd resize command:

# multipathd -k'resize map mpath0'


5.Resize the filesystem (assuming no LVM or DOS partitions are used):

# resize2fs /dev/mapper/mpath0


Multipath Command Output


 When you create, modify, or list a multipath device, you get a printout of the current device setup. The format is as follows.


 For each multipath device:

 action_if_any: alias (wwid_if_different_from_alias) [size][features][hardware_handler]


 For each path group:

\_ scheduling_policy [path_group_priority_if_known] [path_group_status_if_known]


 For each path:

\_ host:channel:id:lun devnode major:minor [path_status] [dm_status_if_known]


 For example, the output of a multipath command might appear as follows:

mpath1 (3600d0230003228bc000339414edb8101) [size=10 GB][features="0"][hwhandler="0"]

\_ round-robin 0 [prio=1][active]

 \_ 2:0:0:6 sdb 8:16 [active][ready]

\_ round-robin 0 [prio=1][enabled]

 \_ 3:0:0:6 sdc 8:64 [active][ready]


 If the path is up and ready for I/O, the status of the path is ready or active. If the path is down, the status is faulty or failed. The path status is updated periodically by the multipathd daemon based on the polling interval defined in the /etc/multipath.conf file.


 The dm status is similar to the path status, but from the kernel's point of view. The dm status has two states: failed, which is analogous to faulty, and active which covers all other path states. Occasionally, the path state and the dm state of a device will temporarily not agree.


Note


 When a multipath device is being created or modified, the path group status and the dm status are not known. Also, the features are not always correct. When a multipath device is being listed, the path group priority is not known.


Multipath Queries with multipath Command


 You can use the -l and -ll options of the multipath command to display the current multipath configuration. The -l option displays multipath topology gathered from information in sysfs and the device mapper. The -ll option displays the information the -l displays in addition to all other available components of the system.


 When displaying the multipath configuration, there are three verbosity levels you can specify with the -v option of the multipath command. Specifying -v0 yields no output. Specifying -v1 outputs the created or updated multipath names only, which you can then feed to other tools such as kpartx. Specifying -v2 prints all detected paths, multipaths, and device maps.


 The following example shows the output of a multipath -l command.


# multipath -l

mpath1 (3600d0230003228bc000339414edb8101)

[size=10 GB][features="0"][hwhandler="0"]

\_ round-robin 0 [prio=1][active]

 \_ 2:0:0:6 sdb 8:16 [active][ready]

\_ round-robin 0 [prio=1][enabled]

 \_ 3:0:0:6 sdc 8:64 [active][ready]


Multipath Command Options


Table 5.1. Useful multipath Command Options
Option           Description 

-l              Display the current multipath configuration gathered                     from sysfs and the device mapper. 


-ll             Display the current multipath configuration gathered                   from sysfs, the device mapper, and all other available                   components on the system. 


-f device      Remove the named multipath device.


-F             Remove all unused multipath devices.



Determining Device Mapper Entries with the dmsetup Command


 You can use the dmsetup command to find out which device mapper entries match the multipathed devices.


 The following command displays all the device mapper devices and their major and minor numbers. The minor numbers determine the name of the dm device. For example, a minor number of 3 corresponds to the multipathed device /dev/dm-3.


# dmsetup ls

mpath2  (253, 4)

mpath4p1        (253, 12)

mpath5p1        (253, 11)

mpath1  (253, 3)

mpath6p1        (253, 14)

mpath7p1        (253, 13)

mpath0  (253, 2)

mpath7  (253, 9)

mpath6  (253, 8)

VolGroup00-LogVol01     (253, 1)

mpath5  (253, 7)

VolGroup00-LogVol00     (253, 0)

mpath4  (253, 6)

mpath1p1        (253, 10)

mpath3  (253, 5)


No comments:

Post a Comment