When using some relatively inexpensive parts to build a test three server Ceph solution, it seems that drive controllers and SSD drive combinations are finicky. Specifically, it seems that a Dell H310 controller may have issues with some or all Samsung SSD drives.
At Slow writes PERC H700 and Samsung 850 PRO SSD, a Dell response indicates:
The Samsung 850 Pro SSDs are not validated or certified to work with Dell controllers and as such there is a communication mismatch between the drives and the controller at the firmware level. As a result, you are bound to realize unexpected poor Read and Write performance regardless of controller cache settings.
It said that the Samsung 840 drive will work with the controllers. With Broadcom taking over LSI, they've removed old links, so documents are hard to find now. In trying to follow a document trail:
To use the H310 for non-cache passthrough, it is recommended to flash the card in IT mode. I'll try this once I obtain the card. In the meantime, some possibly relevant links to the drivers and process:
- Disk Controller features and Queue Depth? - from 2014, discusses queue depth and diagnostic commands.
- Flashing the H310 Mono Mini to IT mode has a link to probably the
best link to a flashing tutorial. But is complicated for flashing the mini H310. From the notes: the R610 is an 11th generation server, and when the mini H310 is reflashed to IT mode, the server probably won't reboot.
- Crossflashing the Dell PERC H200 and H310 to the LSI 9211-8i (from year 2018) implies that the H310 is the equivalent of an LSI 9211-8i. Also, a Dell H200 and an H310 must be similar from a firmware perspective.
- Flashing IT Firmware to the LSI SAS9211-8i HBA - 2012 - flash to a pci card
- How to crossflash PERC H310 to IT mode LSI 9211-8i firmware (HBA for FreeNAS, UnRAID) - an article from 2017 with better, more native instructions
- Crossflash Dell PERC H310 to LSI 9211-8i IT Mode Using Legacy (DOS) and UEFI Method (HBA Firmware + BIOS)
- PERC H310 - LSI 9211-8i - $50 - notes on purchase, install, and upgrade of card.
- Problems Flashing Dell Perc H310 - discusses H310 in PCI slot instead
- How-to Flash Dell Perc H310 with IT Firmware To Change Queue Depth from 25 to 600
- Updated: SAS HBA crossflashing or flashing to IT mode, Dell Perc H200 and H310 from 2016 where is is suggested that "Integrated, Mini or Mini Mono Perc H310 do NOT try to crossflash with these steps".
- DELL PERC H310/H710/H710P/H810 Controllers driver version 5.1.112.64,A00 - a starting point for the search of appropriate drivers
- LINUX PERCCLI Utility For All Dell HBA/PERC Controllers - and the starting point for command line utilities
- LSI MegaRAID SAS - some notes on the CLI utilities
- Doing battle with a Dell R620 and Ubuntu - has some remarks on H310 testing.
- H310 and Fan Noise
- Low random read/write on samsung 840 250gb ssd - spectre bios upgrades can destroy random read/write times
Drive Information:
Cluster test:
rados bench -p vm_storage 10 write -b 4M -t 16"
2022/04/13 - challenges
In a Dell R620, with an LSI SAS card, it took a while to find a utility to communicate with the card. Over the fold are some results:
# modinfo mpt3sas
filename: /lib/modules/5.13.19-2-pve/kernel/drivers/scsi/mpt3sas/mpt3sas.ko
alias: mpt2sas
version: 38.100.00.00
license: GPL
description: LSI MPT Fusion SAS 3.0 Device Driver
author: Avago Technologies
srcversion: FFE8DCA2163C60394647C0E
alias: pci:v00001000d000000E7sv*sd*bc*sc*i*
alias: pci:v00001000d000000E4sv*sd*bc*sc*i*
...
alias: pci:v00001000d00000072sv*sd*bc*sc*i*
alias: pci:v00001000d00000070sv*sd*bc*sc*i*
depends: scsi_transport_sas,raid_class
retpoline: Y
intree: Y
name: mpt3sas
vermagic: 5.13.19-2-pve SMP mod_unload modversions
# uname -a
Linux host03 5.13.19-2-pve #1 SMP PVE 5.13.19-4 (Mon, 29 Nov 2021 12:10:09 +0100) x86_64 GNU/Linux
The module has a parameter for queue depth. Not sure how to show current value though. I believe it should be 600. Some ramblings at LSI 2008 Dell H310 VSAN rebuild performance concerns.
The following shows a depth of 3422 with 32 for each drive:
dmesg |egrep -i 'mpt|scsi|sas'
[ 0.014463] Device empty
[ 0.559450] SCSI subsystem initialized
[ 0.595952] MPTCP token hash table entries: 32768 (order: 7, 786432 bytes, vmalloc)
[ 0.633072] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
[ 2.296355] mpt3sas version 38.100.00.00 loaded
[ 2.296620] mpt3sas 0000:03:00.0: can't disable ASPM; OS doesn't have ASPM control
[ 2.297188] mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (32869156 kB)
[ 2.349843] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 2.349858] mpt2sas_cm0: MSI-X vectors supported: 1
[ 2.349863] mpt2sas_cm0: 0 1
[ 2.350001] mpt2sas_cm0: High IOPs queues : disabled
[ 2.350004] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 45
[ 2.350007] mpt2sas_cm0: iomem(0x00000000ddffc000), mapped(0x000000007f46ae63), size(16384)
[ 2.350014] mpt2sas_cm0: ioport(0x000000000000fc00), size(256)
[ 2.381782] scsi host1: ahci
[ 2.382193] scsi host2: ahci
[ 2.382506] scsi host3: ahci
[ 2.382812] scsi host4: ahci
[ 2.383178] scsi host5: ahci
[ 2.383497] scsi host6: ahci
[ 2.399778] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 2.399787] mpt2sas_cm0: sending message unit reset !!
[ 2.401320] mpt2sas_cm0: message unit reset: SUCCESS
[ 2.428741] mpt2sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(9), sge_per_io(128), chains_per_io(15)
[ 2.429024] mpt2sas_cm0: request pool(0x00000000c1074382) - dma(0x113c80000): depth(3492), frame_size(128), pool_size(436 kB)
[ 2.458138] mpt2sas_cm0: sense pool(0x000000002aa5df49) - dma(0x114400000): depth(3367), element_size(96), pool_size (315 kB)
[ 2.458145] mpt2sas_cm0: sense pool(0x000000002aa5df49)- dma(0x114400000): depth(3367),element_size(96), pool_size(0 kB)
[ 2.458420] mpt2sas_cm0: reply pool(0x00000000cbf50222) - dma(0x114480000): depth(3556), frame_size(128), pool_size(444 kB)
[ 2.458436] mpt2sas_cm0: config page(0x00000000bfb9533c) - dma(0x1143c7000): size(512)
[ 2.458439] mpt2sas_cm0: Allocated physical memory: size(7579 kB)
[ 2.458441] mpt2sas_cm0: Current Controller Queue Depth(3364),Max Controller Queue Depth(3432)
[ 2.458444] mpt2sas_cm0: Scatter Gather Elements per IO(128)
[ 2.503148] mpt2sas_cm0: overriding NVDATA EEDPTagMode setting
[ 2.503758] mpt2sas_cm0: LSISAS2008: FWVersion(20.00.07.00), ChipRevision(0x03), BiosVersion(07.27.01.01)
[ 2.503769] mpt2sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[ 2.503882] scsi host0: Fusion MPT SAS Host
[ 2.506321] mpt2sas_cm0: sending port enable !!
[ 2.507180] mpt2sas_cm0: hba_port entry: 00000000895d8150, port: 255 is added to hba_port list
[ 2.509392] mpt2sas_cm0: host_add: handle(0x0001), sas_addr(0x590b11c032019600), phys(8)
[ 2.524727] mpt2sas_cm0: port enable: SUCCESS
[ 2.526473] scsi 0:0:0:0: Direct-Access ATA SAMSUNG MZ7LM240 204Q PQ: 0 ANSI: 6
[ 2.526484] scsi 0:0:0:0: SATA: handle(0x0009), sas_addr(0x4433221101000000), phy(1), device_name(0x5002538c40794acd)
[ 2.526488] scsi 0:0:0:0: enclosure logical id (0x590b11c032019600), slot(2)
[ 2.526584] scsi 0:0:0:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 2.526590] scsi 0:0:0:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
[ 2.527861] end_device-0:0: add: handle(0x0009), sas_addr(0x4433221101000000)
[ 2.529618] scsi 0:0:1:0: Direct-Access ATA SAMSUNG MZ7KM480 003Q PQ: 0 ANSI: 6
[ 2.529627] scsi 0:0:1:0: SATA: handle(0x000a), sas_addr(0x4433221104000000), phy(4), device_name(0x5002538c4007ca8c)
[ 2.529631] scsi 0:0:1:0: enclosure logical id (0x590b11c032019600), slot(7)
[ 2.529724] scsi 0:0:1:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 2.529729] scsi 0:0:1:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
[ 2.530898] end_device-0:1: add: handle(0x000a), sas_addr(0x4433221104000000)
[ 2.532479] scsi 0:0:2:0: Direct-Access ATA SAMSUNG MZ7KM480 003Q PQ: 0 ANSI: 6
[ 2.532487] scsi 0:0:2:0: SATA: handle(0x000b), sas_addr(0x4433221105000000), phy(5), device_name(0x5002538c40232b19)
[ 2.532491] scsi 0:0:2:0: enclosure logical id (0x590b11c032019600), slot(6)
[ 2.532582] scsi 0:0:2:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 2.532588] scsi 0:0:2:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
[ 2.533756] end_device-0:2: add: handle(0x000b), sas_addr(0x4433221105000000)
[ 2.535282] scsi 0:0:3:0: Direct-Access ATA SAMSUNG MZ7KM480 003Q PQ: 0 ANSI: 6
[ 2.535290] scsi 0:0:3:0: SATA: handle(0x000c), sas_addr(0x4433221106000000), phy(6), device_name(0x5002538c4007cc60)
[ 2.535293] scsi 0:0:3:0: enclosure logical id (0x590b11c032019600), slot(5)
[ 2.535384] scsi 0:0:3:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 2.535389] scsi 0:0:3:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
[ 2.536558] end_device-0:3: add: handle(0x000c), sas_addr(0x4433221106000000)
[ 2.538080] scsi 0:0:4:0: Direct-Access ATA SAMSUNG MZ7LM240 1F3Q PQ: 0 ANSI: 6
[ 2.538088] scsi 0:0:4:0: SATA: handle(0x000d), sas_addr(0x4433221107000000), phy(7), device_name(0x5002538c40a4325e)
[ 2.538091] scsi 0:0:4:0: enclosure logical id (0x590b11c032019600), slot(4)
[ 2.538183] scsi 0:0:4:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 2.538188] scsi 0:0:4:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
[ 2.539445] end_device-0:4: add: handle(0x000d), sas_addr(0x4433221107000000)
[ 2.539870] sd 0:0:0:0: Attached scsi generic sg0 type 0
[ 2.540119] scsi 0:0:1:0: Attached scsi generic sg1 type 0
[ 2.540380] scsi 0:0:2:0: Attached scsi generic sg2 type 0
[ 2.540622] sd 0:0:3:0: Attached scsi generic sg3 type 0
[ 2.540936] sd 0:0:4:0: Attached scsi generic sg4 type 0
[ 2.946995] sd 0:0:2:0: [sdc] Attached SCSI disk
[ 2.951004] sd 0:0:3:0: [sdd] Attached SCSI disk
[ 2.983013] sd 0:0:1:0: [sdb] Attached SCSI disk
[ 2.983481] sd 0:0:4:0: [sde] Attached SCSI disk
[ 2.983652] sd 0:0:0:0: [sda] Attached SCSI disk
[ 3.257973] scsi host7: usb-storage 1-1.2:1.0
[ 4.271422] scsi 7:0:0:0: Direct-Access Generic- USB3.0 CRW -SD 1.00 PQ: 0 ANSI: 4
[ 4.279439] scsi 7:0:0:1: Direct-Access Generic- USB3.0 CRW -SD 1.00 PQ: 0 ANSI: 4
[ 4.280097] sd 7:0:0:0: Attached scsi generic sg5 type 0
[ 4.280334] sd 7:0:0:1: Attached scsi generic sg6 type 0
[ 5.810740] sd 7:0:0:1: [sdg] Attached SCSI removable disk
[ 5.887114] sd 7:0:0:0: [sdf] Attached SCSI removable disk
[ 10.313703] Loading iSCSI transport class v2.0-870.
[ 10.463713] iscsi: registered transport (tcp)
[ 10.707980] iscsi: registered transport (iser)
[ 21.214909] Timeout policy base is empty
[ 3886.419636] Fusion MPT base driver 3.04.20
[ 3886.427650] Fusion MPT misc device (ioctl) driver 3.04.20
[ 3886.427780] mptctl: Registered with Fusion MPT base driver
[ 3886.427783] mptctl: /dev/mptctl @ (major,minor=10,220)
[ 4878.254979] mpt2sas_cm0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101)
[ 4983.762983] mpt2sas_cm0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101)
[ 5007.794703] mpt2sas_cm0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101)
Will need to try something like the following on the kernel command line:
mpt3sas.max_queue_depth=8000
via in the grub file, something like:
GRUB_CMDLINE_LINUX_DEFAULT="mpt3sas.max_queue_depth=10000"
lsblk shows six drives:
# lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINT
sda
├─sda1
└─sda2 ntfs New Volume BA0EB3740EB32871
sdb
sdc
sdd
sde
└─sde1 ntfs New Volume D25CDEFD5CDEDB75
sdf
├─sdf1
├─sdf2 vfat FAT32 014E-E0AD
└─sdf3 LVM2_member LVM2 001 8RwG2R-kVB8-NJ5U-RcrW-wZd7-veWn-rJuWBo
├─pve-swap swap 1 d0013833-8e3a-4ceb-be8a-35ff2d18bb4e [SWAP]
├─pve-root ext4 1.0 d0e54e34-601f-404c-b7fa-a3bcba0a05d9 10.5G 21% /
├─pve-data_tmeta
│ └─pve-data-tpool
│ ├─pve-data
│ └─pve-vm--100--disk--0 ext4 1.0 d2022da3-cc7d-444a-bec2-3dbd2210e57e
└─pve-data_tdata
└─pve-data-tpool
├─pve-data
└─pve-vm--100--disk--0 ext4 1.0 d2022da3-cc7d-444a-bec2-3dbd2210e57e
Another helpful command:
lshw -c disk
LinuxRaidForAdmins offers some tools. For the H310 card, the only one seemed to be:
# sas2ircu LIST
LSI Corporation SAS2 IR Configuration Utility.
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2009-2013 LSI Corporation. All rights reserved.
Adapter Vendor Device SubSys SubSys
Index Type ID ID Pci Address Ven ID Dev ID
----- ------------ ------ ------ ----------------- ------ ------
0 SAS2008 1000h 72h 00h:03h:00h:00h 1028h 1f51h
SAS2IRCU: Utility Completed Successfully.
root@host03:~# sas2ircu 0 DISPLAY
LSI Corporation SAS2 IR Configuration Utility.
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2009-2013 LSI Corporation. All rights reserved.
Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
Controller type : SAS2008
BIOS version : 7.39.02.00
Firmware version : 20.00.07.00
Channel description : 1 Serial Attached SCSI
Initiator ID : 0
Maximum physical devices : 255
Concurrent commands supported : 3432
Slot : Unknown
Segment : 0
Bus : 3
Device : 0
Function : 0
RAID Support : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0
Device is a Hard disk
Enclosure # : 1
Slot # : 2
SAS Address : 4433221-1-0100-0000
State : Ready (RDY)
Size (in MB)/(in sectors) : 228936/468862127
Manufacturer : ATA
Model Number : SAMSUNG MZ7LM240
Firmware Revision : 204Q
Serial No : S2TWNX0J815597
GUID : 5002538c40794acd
Protocol : SATA
Drive Type : SATA_SSD
HWRAID helped with package selection. I viewed
Index of /debian/ directly, downloaded the desired packages, and installed with dpkg -i.
LSI MPT SAS Linux management offered some examples.