[Update September 24th, 2009: With OpenSolaris the format of the config file has changed: Here's an update.]
With two RAID devices in different locations onsite you may want to use a ZFS mirror to access them - as well as Sun MPxIO to have redundant links to them.
The relevant configuration is /kernel/drv/scsi_vhci.conf as scsi_vhci is the MPxIO driver for redundant scsi/fc links.
Yes Solaris 10 is quite good but this configuration is like telling your grandma to disguise herself as a man to be able to use trousers:
Using Sun Arrays all works without any problem. The latter do arise when you want to use other devices (non-Sun), which is not really an uncommon practice...
You have to tell scsi_vhci to ACCEPT your device based on the vendor- und product-id which it tells you at SCSI inquiry stage.
In my case, I had to add this to scsi_vhci.conf:
You see the grandma's trousers?
These lines tell the driver that whenever it encounters a "ADVUNI OXYGENRAID" device it should use the value "0x1000000" as an option - which stands for "behaves like a sun fc array and is able to handle redundant links" (I think Sun believes that only Sun devices can do that...).
You notice the TWO SPACES between Vendor and Product-ID? They are part of the game. The scsi_vhci driver expects the vendor and product id as 8 character-words each. You have to fill up with spaces. ADVUNI has only 6 characters, so you have to use two spaces. If your SCSI vendor id has 8 chars, just don't use any space! Otherwise the driver just ignores your devices. Without a warning or error, of course.
To find out SCSI Vendor and Product-ID,just use format(1), select one of the instances of your device (each link to your device shows up as a different entry):
Only use the first part of "Product:", until the first space!
After doing a reboot -- -r
it should work:
My two different(!) hardware devices are using a new virtual controller (c6) to access the multiple links (each c6 device has 4 redundant links to it, without scsi_vhci you would see all these links as different disk instances).
[Update Apr 19th, 2008:] You may check the multiple paths handled by scsi_vhci by using vhci_stat.
Now I found performance was abysmal. I smiled too early. It just would have been too easy until now.
The solution is a technique called "logical block" round-robin.
First round-robin should speed up things as all fiberchannel links are used and after a failing link recovery is faster as you don't have to transfer all outstanding operations to the next functioning link. But it did not work well.
I found that each of the controllers of these raid devices do have their own RAM cache which won't be used efficiently when every subsequent 512-byte-block request is done by the next controller... And yes, scsi_vhci has a solution for this, if you have much luck. This option came in with a patch or by installing at least Solaris 10 8/07. Otherwise it just won't recognize the option:
Complicated? This line tells scsi_vhci to override some default options for device type "ADVUNI OXYGENRAID" (also with 2 spaces, see above!).
What's logical-block round-robin? You define a block size to summarize requests in this block size range to be served by ONE link. region-size=20 means 2^20 bytes = 1 MB. With this "trick" the onboard controller caches do their thing of prefetching and caching right.
With two RAID devices in different locations onsite you may want to use a ZFS mirror to access them - as well as Sun MPxIO to have redundant links to them.
The relevant configuration is /kernel/drv/scsi_vhci.conf as scsi_vhci is the MPxIO driver for redundant scsi/fc links.
Yes Solaris 10 is quite good but this configuration is like telling your grandma to disguise herself as a man to be able to use trousers:
Using Sun Arrays all works without any problem. The latter do arise when you want to use other devices (non-Sun), which is not really an uncommon practice...
You have to tell scsi_vhci to ACCEPT your device based on the vendor- und product-id which it tells you at SCSI inquiry stage.
In my case, I had to add this to scsi_vhci.conf:
device-type-scsi-options-list =
"ADVUNI OXYGENRAID", "symmetric-option";
symmetric-option = 0x1000000;
You see the grandma's trousers?
These lines tell the driver that whenever it encounters a "ADVUNI OXYGENRAID" device it should use the value "0x1000000" as an option - which stands for "behaves like a sun fc array and is able to handle redundant links" (I think Sun believes that only Sun devices can do that...).
You notice the TWO SPACES between Vendor and Product-ID? They are part of the game. The scsi_vhci driver expects the vendor and product id as 8 character-words each. You have to fill up with spaces. ADVUNI has only 6 characters, so you have to use two spaces. If your SCSI vendor id has 8 chars, just don't use any space! Otherwise the driver just ignores your devices. Without a warning or error, of course.
To find out SCSI Vendor and Product-ID,just use format(1), select one of the instances of your device (each link to your device shows up as a different entry):
format> inquiry
Vendor: ADVUNI
Product: OXYGENRAID 416F4
Revision: 347B
Only use the first part of "Product:", until the first space!
After doing a reboot -- -r
it should work:
# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c3t0d0 <DEFAULT cyl 8872 alt 2 hd 255 sec 63>
/pci@7b,0/pci1022,7458@11/pci1000,3060@2/sd@0,0
1. c6t600D0230006B66680C50AB7821F0E900d0 <ADVUNI-OXYGENRAID 416F4-347B-3.41TB>
/scsi_vhci/disk@g600d0230006b66680c50ab7821f0e900
2. c6t600D0230006C1C4C0C50BE5BC9D49100d0 <ADVUNI-OXYGENRAID 416F4-347B-3.41TB>
/scsi_vhci/disk@g600d0230006c1c4c0c50be5bc9d49100
My two different(!) hardware devices are using a new virtual controller (c6) to access the multiple links (each c6 device has 4 redundant links to it, without scsi_vhci you would see all these links as different disk instances).
[Update Apr 19th, 2008:] You may check the multiple paths handled by scsi_vhci by using vhci_stat.
Now I found performance was abysmal. I smiled too early. It just would have been too easy until now.
The solution is a technique called "logical block" round-robin.
First round-robin should speed up things as all fiberchannel links are used and after a failing link recovery is faster as you don't have to transfer all outstanding operations to the next functioning link. But it did not work well.
I found that each of the controllers of these raid devices do have their own RAM cache which won't be used efficiently when every subsequent 512-byte-block request is done by the next controller... And yes, scsi_vhci has a solution for this, if you have much luck. This option came in with a patch or by installing at least Solaris 10 8/07. Otherwise it just won't recognize the option:
device-type-mpxio-options-list =
"device-type=ADVUNI OXYGENRAID", "load-balance-options=logical-block-options";
logical-block-options="load-balance=logical-block", "region-size=20";
Complicated? This line tells scsi_vhci to override some default options for device type "ADVUNI OXYGENRAID" (also with 2 spaces, see above!).
What's logical-block round-robin? You define a block size to summarize requests in this block size range to be served by ONE link. region-size=20 means 2^20 bytes = 1 MB. With this "trick" the onboard controller caches do their thing of prefetching and caching right.
This was very informative comment, because even I use a 3rd party target on which ALUA is disabled, LUN Affinity is disabled and its developed for SCSI 3 command set. Did configure the host which is a SPARC T2000 as per your suggestion but still nothing seems to be working. Also checked few other SUN webpages which said the same stuff. Still I am not seeing devices listed under multipath(i.e 'mpathadm list lu'), it just lists the local devices connected to the SPARC, but I do see the initiator-ports getting listed. I did enter the product-id, vendor-id in the 'scsi_vhci.conf' as per the guidelines, no go. I also went thru the 'fp.conf' and made sure that the MPxIO is enabled. Devices are getting listed in Solaris and not under mpxio. Please share your experience.
Without your scsi_vhci.conf and the output of a SCSI inquiry (format) it is difficult to give a hint...
Did you ever have to run "stmsboot -e"?
Another thing we had to do was add
mpxio-disable="no";
to /kernel/drv/fp.conf
Yes, you're right, mpxio-disable="no" is needed in fp.conf.
And, no, I did not need stmsboot -e, adding the right device name for my 3rd party device in scsi_vhci.conf was sufficient.
This week, I got two Sun StorageTek 6340 arrays and here no change to scsi_vhci.conf was needed.
We are currently struggeling getting a SE6130 properly connected to a T2000 with Solaris10 and for the volume to be properly recognized after reboot. Since this is a asymmetric storage device, MPxIO has to be instructed to have a primary and a secondary path. Unfortunately this seems to be nowhere explained. Not even in the Sun StorEdge documentation.
Any idea if there are other options for this file, other than the symmetric example(s) at Sun of keyword "symmetric-option = 0x1000000;"?
I have not found a list anywhere and am very intrigued to see you using the parameter "device-type-mpxio-options-list =", as I am only aware of the parameter "device-type-scsi-options-list =" in my configurations. Let me know if there is a list out there?
I have a StorageTek 6140 here and scsi_vhci.conf "knows" the ONLINE/STANDBY configuration of this array automatically:
/scsi_vhci/disk@g600a0b800048a692000006404905840a
(Vendor = SUN Model = CSM200_R Rev. = 0710 VolID = )
ZFS: pool name = fc, ZFS Version: 4
ZFS: size = 1.36T, member type = disk
/pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0 201700a0b848a692,1 STANDBY
/pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0 201600a0b848a692,1 ONLINE
"device-type-mpxio-options-list =" is used to give special scsi_vhci-Options per Device Identifier. It is used to set the logical-block size for symmetric multipathing in my example.
Do not use "symmetric-option" with your Sun StorageTek Array. It will not work, the vhci-Driver will not use MPxIO for your array in that case.
Feel free to contact me by mail, including your scsi_vhci.conf and the result of an SCSI inquiry of your SE6130 device. Mine (6140) has "SUN" as Vendor and "CRM200_R" as Model Identifier.
I have several systems running Solaris 10 08/07. They each have two StorageTek 6140's that are fiber attached using Brocade switches for redundancy (Once in each Control room). When one of the switches is failed, we lose connectivity for a period of time that which causes the applications to fail. You mention that your scsi_vhci.conf file shows a STANDBY and ONLINE. Mine does not. I am trying to understand how to set the primary path for each of the devices to be their local switch.
SunOS xxxxxxxx 5.10 Generic_137111-05 sun4u sparc SUNW,Sun-Fire-V245
#
# Copyright 2004 Sun Microsystems, Inc. All rights reserved.
# Use is subject to license terms.
#
#pragma ident "@(#)scsi_vhci.conf 1.9 04/08/26 SMI"
#
name="scsi_vhci" class="root";
#
# Load balancing global configuration: setting load-balance="none" will cause
# all I/O to a given device (which supports multipath I/O) to occur via one
# path. Setting load-balance="round-robin" will cause each path to the device
# to be used in turn.
#
load-balance="round-robin";
#
# Automatic failback configuration
# possible values are auto-failback="enable" or auto-failback="disable"
auto-failback="enable";
#
# For enabling MPxIO support for 3rd party symmetric device need an
# entry similar to following in this file. Just replace the "SUN SENA"
# part with the Vendor ID/Product ID for the device, exactly as reported by
# Inquiry cmd.
#
# device-type-scsi-options-list =
# "SUN SENA", "symmetric-option";
#
# symmetric-option = 0x1000000;
According to this, it's not 2^20 BYTES, but 512 byte blocks. So 2^20 would result in 512MB.
http://forums.hds.com/index.php?showtopic=264
Thank you for your correction, you seem to be right!
Are the mpxio and vhci drivers affected when a Solaris 10 host boots into cluster mode? I'm running SC 3.2u2 on a pair of sunfire x2200s, each configured with dladm aggregated links over pairs of nge/e1000g interfaces. The targets reside on a Dell/EqualLogic ISCSI array. When the cluster is active, the initiator issues resets to the targets (more specifically, sends an Abort Task command over the iscsi session, then closes the session's TCP connection) every 2-4 minutes. This stops when I reboot the systems with clustering disabled, and the iscsi sessions work as expected.
SunOS nfs12 5.10 Generic_139556-08 i86pc i386 i86pc Solaris
Dmesg output shows:
Sep 25 12:33:18 nfs12 scsi_vhci: [ID 734749 kern.warning] WARNING: vhci_scsi_reset 0x1
Sep 25 12:37:25 nfs12 last message repeated 5 times
Is there a workaround for this? I've also verified that all interfaces use the same MTU end-to-end (currently 1500, tried 9000) and tried using one scsi session per target.
Thank you so much for explaining WHY we are doing this...I finally got a T2000 working on a VMAX thanks to your post here. Good work!!!
-Jim
Thanks for this useful information. I have just inherited a pair of uspv's for second line storage. I finally have them plumbed up and am ready to use them. I already have 3Par storage attached to these machines and after configuring scsi-vhci with scsi-device and mpxio-device info its been working perfect for two years.
When I add the USPV's I see 48 paths to 4 Luns. I understand I need to configure multipathing. The Hitachi rep said nothing had to be done on the Solaris side if mpxio-disable was set to no. I told him it was and he said your good. I am not good I still see 48 paths to 4 luns. I talked to our Oracle rep. They said nothing had to be done because the Luns show up with device names and type in the format command.
I showed our Oracle on site rep. I can mount a Lun using one name, read and write some files. Mount it using another name read and read and write files. He said HMM this isn't right. But maybe it because we have a split/dual fabric.
I said no duh. So if any body has configured multipathing and MPXIO on an USPV. please comment.
Thank you very much for this article.
i was spending all day trying to figure out why i can't see the path for my NETAPP machine.