So we got our hardware and got busy. First up was testing if the S130 RAID1 setup could perform enough because there were some rumours the software RAID on the S130 was terrible. I did a quick test with CrystalDiskMark. I compared to a few systems, both R630 (one with SAS and SSD) and the figures did seem acceptable. This was just for an indication, but it was enough for me to be put me at ease for now.

So we went forward, updated all the firmware using the Dell Lifecycle controller, created the network config, configured RDMA, disabled IPv6 like on our other servers, fixed wsus (it needed KB4022715 to get it to detect wsus btw), configure the VMQ settings, test/configure the cluster, etc. After our basic config we could finally issue the following command;

Enable-ClusterStorageSpacesDirect

But after a reboot we got a BSOD with: “Stop code: MEMORY MANAGEMENT”

 

We couldn’t get the system back to work, F8 save mode did not work. The only option was to use the “last known good configuration” option but as that overwrote the logs we couldn’t found out what happened. The other nodes did not have any errors but all had BSOD’s after rebooting. Funnily enough we could prevent the BSOD by issuing “Disable-ClusterS2D” on the nodes that hadn’t rebooted yet.

Re-enabling S2D by running the command in -verbose mode yielded no results either;

PS C:\Windows\system32> Enable-ClusterStorageSpacesDirect -verbose
VERBOSE: 2017/06/23-16:30:22.057 Ensuring that all nodes support S2D
VERBOSE: 2017/06/23-16:30:22.073 Querying storage information
VERBOSE: 2017/06/23-16:30:33.972 Sorted disk types present (fast to slow): NVMe,SSD,HDD. Number of types present: 3
VERBOSE: 2017/06/23-16:30:33.972 Checking that nodes support the desired cache state
VERBOSE: 2017/06/23-16:30:33.988 Checking that all disks support the desired cache state

Confirm
Are you sure you want to perform this action?
Performing operation 'Enable Cluster Storage Spaces Direct' on Target 'S2D01'.
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"):
VERBOSE: 2017/06/23-16:31:01.478 Creating health resource
VERBOSE: 2017/06/23-16:31:01.571 Setting cluster property
VERBOSE: 2017/06/23-16:31:01.571 Setting default fault domain awareness on clustered storage subsystem
VERBOSE: 2017/06/23-16:31:02.712 Waiting until physical disks are claimed
VERBOSE: 2017/06/23-16:31:05.729 Number of claimed disks on node 'Server01': 18/18
VERBOSE: 2017/06/23-16:31:05.729 Number of claimed disks on node 'Server03': 18/18
VERBOSE: 2017/06/23-16:31:05.744 Number of claimed disks on node 'Server04': 18/18
VERBOSE: 2017/06/23-16:31:05.744 Number of claimed disks on node 'Server02': 18/18
VERBOSE: 2017/06/23-16:31:05.760 Node 'Server01': Waiting until cache reaches desired state (HDD:'ReadWrite' SSD:'WriteOnly')
VERBOSE: 2017/06/23-16:31:05.776 SBL disks initialized in cache on node 'Server01': 18 (18 on all nodes)
VERBOSE: 2017/06/23-16:31:05.791 SBL disks initialized in cache on node 'Server03': 18 (36 on all nodes)
VERBOSE: 2017/06/23-16:31:05.791 SBL disks initialized in cache on node 'Server04': 18 (54 on all nodes)
VERBOSE: 2017/06/23-16:31:05.807 SBL disks initialized in cache on node 'Server02': 18 (72 on all nodes)
VERBOSE: 2017/06/23-16:31:05.807 Cache reached desired state on Server01
VERBOSE: 2017/06/23-16:31:05.807 Node 'Server03': Waiting until cache reaches desired state (HDD:'ReadWrite' SSD:'WriteOnly')
VERBOSE: 2017/06/23-16:31:05.854 Cache reached desired state on Server03
VERBOSE: 2017/06/23-16:31:05.885 Node 'Server04': Waiting until cache reaches desired state (HDD:'ReadWrite' SSD:'WriteOnly')
VERBOSE: 2017/06/23-16:31:05.932 Cache reached desired state on Server04
VERBOSE: 2017/06/23-16:31:05.932 Node 'Server02': Waiting until cache reaches desired state (HDD:'ReadWrite' SSD:'WriteOnly')
VERBOSE: 2017/06/23-16:31:05.979 Cache reached desired state on Server02
VERBOSE: 2017/06/23-16:31:05.979 Waiting until SBL disks are surfaced
VERBOSE: 2017/06/23-16:31:09.026 Disks surfaced on node 'Server01': 72/72
VERBOSE: 2017/06/23-16:31:09.073 Disks surfaced on node 'Server03': 72/72
VERBOSE: 2017/06/23-16:31:09.120 Disks surfaced on node 'Server04': 72/72
VERBOSE: 2017/06/23-16:31:09.167 Disks surfaced on node 'Server02': 72/72
VERBOSE: 2017/06/23-16:31:12.599 Waiting until all physical disks are reported by clustered storage subsystem
VERBOSE: 2017/06/23-16:31:16.636 Physical disks in clustered storage subsystem: 72
VERBOSE: 2017/06/23-16:31:16.636 Querying pool information
VERBOSE: 2017/06/23-16:31:17.011 Starting health providers
VERBOSE: 2017/06/23-16:33:51.265 Creating S2D pool
VERBOSE: 2017/06/23-16:33:57.748 Checking that all disks support the desired cache state

We tried a lot, firstly to check all available firmware updates and drivers, but all seemed in order. Checked the hardware, but as all nodes would BSOD’d after a reboot we considered a memory failure to be unlikely. Disabled the RDMA cards, installed the Mellanox drivers instead of the Dell drivers, etc.

We had a gut feeling that it might had to do with the RAID1 setup of the S130 so we made some backups of the system and reinstalled the system with AHCI enabled on the S130 instead of RAID1. We repeated the same config and ran the command Enable-ClusterStorageSpacesDirect and prayed.. And guess what, everything rebooted with no problem!

I’m gonna try to speak to the engineer who assured me this was a supported configuration in the Dell R730xd and S2D, hopefully he can tell me some useful things. On the Dell Tech center is a document who shows this to be a supported configuration too;

But maybe I’ve understood them wrong as the firmware document mentions a HBA330 controller instead of the S130.. And the deployment guide too;

Which would explain why I was surprised to have to resort to Software RAID. But we haven’t found a RAID HBA330 controller, but maybe we did things wrong. I’m gonna find out!

 

Btw, if you need to re-create the S2D config (as we had to do multiple times) you need to clear the configuration. Which means;

  • Remove the storage from the Failover Cluster manager
  • Remove the storage pool;
    Get-StoragePool | ? IsPrimordial -eq $false | Set-StoragePool -IsReadOnly:$false
    Get-StoragePool | ? IsPrimordial -eq $false | Get-VirtualDisk | Remove-VirtualDisk
    Get-StoragePool | ? IsPrimordial -eq $false | Remove-StoragePool -Confirm:$false
  • Wipe disks
    Get-PhysicalDisk | Reset-PhysicalDisk
    Get-Disk | ? Number -ne $null | ? IsBoot -ne $true | ? IsSystem -ne $true | ? PartitionStyle -ne RAW | % {
    $_ | Set-Disk -isoffline:$false
    $_ | Set-Disk -isreadonly:$false
    $_ | Clear-Disk -RemoveData -RemoveOEM -Confirm:$false
    $_ | Set-Disk -isreadonly:$true
    $_ | Set-Disk -isoffline:$true
    }
  • Reboot the nodes (I think the failover cluster manager needs to free the claim on the storage, a service restart may be enough).

 

Categories: S2D, Windows Server

9 Comments

  1. Stephan

    Hope you will find a solution. I have nearly the same hardware and will try setup also an S2D Cluster but with 3 nodes

    1. Dennis Pennings

      I haven’t got a solution yet, but the new reference designs mention that there have to be 2 HBA330 (one in JBOD mode and one in RAID mode) controllers in the system. My contact at Dell has escalated the issue to see what has to be done. If we get an solution or more info, I will update this blogpost.

      It took us way too much time to wait for an answer, so at the moment the OS disks are not in raid, but in SATA-AHCI mode.

      Btw, for another S2D implementation we tried to add an HBA330 (in RAID mode) to the system but the Dell solutions configurator (the Dell partner thing where you configure your system) wouldn’t allow it. So yes, there is finally a reference design but an easy way to order the hardware? Don’t think so.. 😡

      1. Brian

        I’m building almost an identical system. R730xd with 4 x 4TB SSD in the front bays, 2 x 2TB internal NVMe and 2 x 200GB SSDs in the flex bays. The factory had the HBA in RAID mode, setting the S2Dbustypes to allow RAID let me set it up but I don’t want to go to production like that. If you hear from Dell on exactly how those OS disks can be in RAID1 with the other disks on the same H330 in HBA mode I would love to hear it. If the OS disks could pass through to the on-board S130 controller to do the RAID 1 that would be cool, but I don’t see how I could cable it that way…

        1. Dennis Pennings

          I will share all info I get from Dell, but so far it’s been quiet..

          1. Dennis Pennings

            In the 14G servers, there seems to be a “BOSS” option for the OS-RAID1 setup
            https://www.360ict.nl/blog/dell-14g-servers-are-here/

  2. Stephan

    Only to be sure, it has nothing to do with this https://community.mellanox.com/thread/3593 behavior?

    1. Dennis Pennings

      I don’t think so, we actually disabled the mellanox cards before running the command to enable s2d. Also he mentiones stress testing as his bsod. We cloud prevent the bsod after disabling s2d or by reverting to a previous restore point.

      But it is interesting indeed. We also wanted to use the existing n4032 switches, but we were told to use the s4048 instead of the larger buffers. We also dont have the pro cards (a mistake because of the lack of rocev2) and it looks like he is doing a windows os mirror (so software mirror from within the dis manager), but in my opnion this results in bad performance..

  3. Andre

    Interesting system behavior you are sharing with us. I have ordered a very similar config to yours using the 24x 2.5″ + 2x 2.5″ flexbay R730xd chassis configured with 2x OS SSD in flexbay and other SSD in the front. The first issue I got was with the S130. The system was configured using solution configurator as you did with the 24+2 chassis with the 2 flexbay drives in SW RAID1 with the S130. Delivered was all drives connected to the HBA330 as yours. ProSupport gave me the same hint as to you: connecting the flexbay to the S130 direct and ignoring the POST error message. Later I did some testing because I had the same concerns as you about poor performance, but the test went fine.

    After setting up the S2D cluster I got exact the same error as you. So next ProSupport call got opend and still is. Tested different configs, analyzed memory dumps and at the moment the case in in 3rd level. We also ruled out the Mellanox X3 pro cards as the issue is the same with using onboard NICs. The behavior you found out with beeing the S130 the root cause, I will test tomorrow and consult with ProSupport. Many thanks for sharing this with us.

    1. Dennis Pennings

      I can’t believe it takes them this long, also the knowledge behind the s2d reference designs were completely ignored when i talked to dell for the last 6 months.. and let is know what your test results are, i will bug my technical contact again. It will take a bit longer because i’m on a holiday..

Leave a Reply

Your email address will not be published. Required fields are marked *

*