Intel Optane + vExpert = Awesome!
Thank you to @Intel and @vExpert for the opportunity to test some unique technology in the Intel Optane drives.
I have been part of the VMware vExpert program for four years now, and in 2023, there was a webinar done by Intel on their Optane drives. After the webinar, there was the chance to be chosen to receive a gift from Intel of some Optane drives for homelab testing.
I decided to submit for the chance to receive the Optane drives and was lucky enough to be selected. At the time of selection, you had to choose your drive choice, which was PCIe or U2 format. My homelab consists of four Intel Skull Canyon NUCs, so using PCIe was not going to work, but dumb of me was the selection that I accidentally chose. After a discussion with Corey Romero, the vExpert Community Manager, I got in touch with Intel, which sent the U2 format drives to me.
Once receiving the Intel Optane U2 form factor drives, it was time to determine a USB adapter to use them with my Intel NUCs. One of the first things I needed to do to use a USB drive within VMware ESXi8 was to turn off the USBArbitrator service on all my hosts. After that, it will recognize the USB adapter and drive, at which point I can change the drive to ” FLASH ” to be used with vSAN.
So I tried the first adapter from Sabarent, which can be found here –
While this adapter did work for my U2 drives, it was very problematic within VMware ESXi8, wherein it would randomly disappear from the storage list. I would need to unplug the drive from the adapter and then plug it back in to get it working again. This was not going to suffice for me to be able to properly test vSAN without it constantly going into error states.
This would just not do, so I started to explore other adapters. I found another one which is also USB but is like a dock for the U2 drives, keeping it vertical, so I decided to just order one to try before committing to four total.
This adapter is from StorageMall and can be found here –
StorageMall Adapter – https://www.amazon.ca/NVMe-Adapter-SFF-8639-Docking-Station/dp/B0B31LRRGZ/ref=sr_1_1?crid=36RVYLRAIITRR&keywords=storagemall+u2+to+usb&qid=1680287840&sprefix=storagemall+u2+t%2Caps%2C406&sr=8-1
So I ordered it, and it was time to connect it to one of my Intel NUCs for testing.
This is what it looks like connected to my Intel NUC –
So after testing just one adapter, the U2 drive never disconnected from ESXi8 while the other drives, which were still connected via the Sabrent adapter, did! So that sealed the deal, and I ordered three more adapters – one for each Intel NUC. I already had a 1TB NVME drive in each Intel NUC for the capacity tier, and the Intel Optane drives would be used for the cache tier.
I received my other StorageMall adapters and got all my drives connected to each Intel NUC. I then set up vSAN correctly, and it has been rock solid since getting these little docking units. Highly recommend them if you want something reasonably priced and not overly expensive for USB connectivity.
So I could configure VMware vSAN with just minimal settings as I did not need anything fancy, and the status eventually appeared as OK.
One of my first tests was with a Windows VM using DiskSpd. The configuration is as follows –
- Windows 10 (64-bit)
- VMware Paravirtual SCSI controllers (x2)
- OS drive (C), Data drive (E)
So I used the following command to run the tests on both the C drive and E drive, just changing the command for the drive letter each time, and the command was piped to a text file for future reference –
diskspd.exe -d60 -W15 -C15 -c128M -t4 -o4 -b8k -L -r -Sh -w50 c:\disk-speed-test.dat
The results were not bad, to say the least, with the VM being on vSAN and having USB Intel Optane U2 drives. An example of the tests is below for the C drive –
Input parameters:
timespan: 1
-------------
duration: 60s
warm up time: 15s
cool down time: 15s
measuring latency
random seed: 0
path: 'c:\disk-speed-test.dat'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
performing mix test (read/write ratio: 50/50)
block size: 8KiB
using random I/O (alignment: 8KiB)
number of outstanding I/O operations per thread: 4
threads per file: 4
using I/O Completion Ports
IO priority: normal
System information:
computer name: TON-7UZ
start time: 2023/03/21 18:37:39 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 60.02s
thread count: 4
proc count: 4
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 17.29%| 3.07%| 14.22%| 82.71%
1| 18.15%| 3.20%| 14.95%| 81.85%
2| 17.42%| 2.79%| 14.63%| 82.58%
3| 16.95%| 3.05%| 13.90%| 83.05%
-------------------------------------------
avg.| 17.45%| 3.03%| 14.42%| 82.55%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 1355177984 | 165427 | 21.53 | 2756.41 | 1.450 | 3.899 | c:\disk-speed-test.dat (128MiB)
1 | 1347731456 | 164518 | 21.42 | 2741.26 | 1.458 | 3.943 | c:\disk-speed-test.dat (128MiB)
2 | 1348632576 | 164628 | 21.43 | 2743.09 | 1.457 | 3.926 | c:\disk-speed-test.dat (128MiB)
3 | 1348214784 | 164577 | 21.42 | 2742.24 | 1.457 | 3.976 | c:\disk-speed-test.dat (128MiB)
-----------------------------------------------------------------------------------------------------
total: 5399756800 | 659150 | 85.80 | 10983.00 | 1.455 | 3.936
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 678486016 | 82823 | 10.78 | 1380.03 | 0.209 | 0.552 | c:\disk-speed-test.dat (128MiB)
1 | 673710080 | 82240 | 10.71 | 1370.31 | 0.210 | 0.914 | c:\disk-speed-test.dat (128MiB)
2 | 672129024 | 82047 | 10.68 | 1367.10 | 0.211 | 1.162 | c:\disk-speed-test.dat (128MiB)
3 | 674004992 | 82276 | 10.71 | 1370.91 | 0.211 | 0.894 | c:\disk-speed-test.dat (128MiB)
-----------------------------------------------------------------------------------------------------
total: 2698330112 | 329386 | 42.88 | 5488.35 | 0.210 | 0.906
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 676691968 | 82604 | 10.75 | 1376.38 | 2.693 | 5.201 | c:\disk-speed-test.dat (128MiB)
1 | 674021376 | 82278 | 10.71 | 1370.95 | 2.705 | 5.210 | c:\disk-speed-test.dat (128MiB)
2 | 676503552 | 82581 | 10.75 | 1376.00 | 2.694 | 5.130 | c:\disk-speed-test.dat (128MiB)
3 | 674209792 | 82301 | 10.71 | 1371.33 | 2.703 | 5.263 | c:\disk-speed-test.dat (128MiB)
-----------------------------------------------------------------------------------------------------
total: 2701426688 | 329764 | 42.93 | 5494.65 | 2.699 | 5.201
total:
%-ile | Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
min | 0.025 | 0.810 | 0.025
25th | 0.089 | 1.980 | 0.125
50th | 0.125 | 2.337 | 1.283
75th | 0.197 | 2.952 | 2.343
90th | 0.383 | 3.534 | 3.111
95th | 0.652 | 3.959 | 3.541
99th | 1.380 | 6.024 | 4.755
3-nines | 3.349 | 28.245 | 18.948
4-nines | 14.731 | 242.986 | 240.320
5-nines | 213.511 | 244.998 | 244.357
6-nines | 239.836 | 340.090 | 340.090
7-nines | 239.836 | 340.090 | 340.090
8-nines | 239.836 | 340.090 | 340.090
9-nines | 239.836 | 340.090 | 340.090
max | 239.836 | 340.090 | 340.090The results from the E drive –
Input parameters:
timespan: 1
-------------
duration: 60s
warm up time: 15s
cool down time: 15s
measuring latency
random seed: 0
path: 'e:\disk-speed-test.dat'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
performing mix test (read/write ratio: 50/50)
block size: 8KiB
using random I/O (alignment: 8KiB)
number of outstanding I/O operations per thread: 4
threads per file: 4
using I/O Completion Ports
IO priority: normal
System information:
computer name: TON-7UZ
start time: 2023/03/31 19:33:49 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 60.00s
thread count: 4
proc count: 4
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 76.69%| 30.42%| 46.28%| 23.31%
1| 76.46%| 32.14%| 44.32%| 23.54%
2| 75.63%| 33.41%| 42.21%| 24.38%
3| 76.72%| 36.41%| 40.31%| 23.28%
-------------------------------------------
avg.| 76.37%| 33.09%| 43.28%| 23.63%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 1076240384 | 131377 | 17.11 | 2189.62 | 1.824 | 2.812 | e:\disk-speed-test.dat (128MiB)
1 | 1060225024 | 129422 | 16.85 | 2157.03 | 1.852 | 2.830 | e:\disk-speed-test.dat (128MiB)
2 | 1046495232 | 127746 | 16.63 | 2129.10 | 1.877 | 2.965 | e:\disk-speed-test.dat (128MiB)
3 | 1010950144 | 123407 | 16.07 | 2056.78 | 1.943 | 3.253 | e:\disk-speed-test.dat (128MiB)
-----------------------------------------------------------------------------------------------------
total: 4193910784 | 511952 | 66.66 | 8532.53 | 1.873 | 2.967
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 539795456 | 65893 | 8.58 | 1098.22 | 0.499 | 1.017 | e:\disk-speed-test.dat (128MiB)
1 | 530014208 | 64699 | 8.42 | 1078.32 | 0.514 | 1.017 | e:\disk-speed-test.dat (128MiB)
2 | 521920512 | 63711 | 8.30 | 1061.85 | 0.514 | 1.139 | e:\disk-speed-test.dat (128MiB)
3 | 505266176 | 61678 | 8.03 | 1027.97 | 0.529 | 1.360 | e:\disk-speed-test.dat (128MiB)
-----------------------------------------------------------------------------------------------------
total: 2096996352 | 255981 | 33.33 | 4266.35 | 0.513 | 1.139
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 536444928 | 65484 | 8.53 | 1091.40 | 3.158 | 3.357 | e:\disk-speed-test.dat (128MiB)
1 | 530210816 | 64723 | 8.43 | 1078.72 | 3.190 | 3.377 | e:\disk-speed-test.dat (128MiB)
2 | 524574720 | 64035 | 8.34 | 1067.25 | 3.233 | 3.544 | e:\disk-speed-test.dat (128MiB)
3 | 505683968 | 61729 | 8.04 | 1028.82 | 3.356 | 3.914 | e:\disk-speed-test.dat (128MiB)
-----------------------------------------------------------------------------------------------------
total: 2096914432 | 255971 | 33.33 | 4266.18 | 3.233 | 3.551
total:
%-ile | Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
min | 0.034 | 0.730 | 0.034
25th | 0.137 | 2.079 | 0.240
50th | 0.240 | 2.618 | 1.564
75th | 0.517 | 3.483 | 2.676
90th | 1.023 | 4.860 | 3.865
95th | 1.572 | 6.412 | 5.025
99th | 4.330 | 13.682 | 10.370
3-nines | 15.588 | 30.139 | 25.808
4-nines | 29.935 | 223.382 | 68.381
5-nines | 58.715 | 238.147 | 236.003
6-nines | 77.962 | 238.896 | 238.896
7-nines | 77.962 | 238.896 | 238.896
8-nines | 77.962 | 238.896 | 238.896
9-nines | 77.962 | 238.896 | 238.896
max | 77.962 | 238.896 | 238.896So the overall performance, while not great, is pretty decent using U2 over USB. I will try to run some further benchmarking tests, and since I did receive the PCIe Intel Optane drives, I will be testing a couple of those in a Synology DS923+ NAS that I am getting from them for testing and more blogs. It will be interesting to see how they boost the performance in the NAS using them as possible cache drives or even volumes may be depending on the number I can squeeze in there.
I hope this article was informative, and I will do more on the Intel Optane drives in future posts. Until then, happy testing!