Hi,

I have a ESXi 6.7 server hosting several 2016 servers. I have the OS drive on the sata controller but all other data share volumes on paravirtual controllers. In windows under device manager they show up in the "Disk drives" section as vmware virtual disk but the disk on the paravirtual drives also show up as "Portable devices". One of which shows a yellow exclamation warning saying the device has extra configuration required. They seem to work OK but is definitely wrong.

Anyone seen this?

I found this from another thread on here   VMware Knowledge Base  as a possible workaround.

We have 2 other environments with 2012r2 servers with the same setup not showing the same issues.

Hello, we had an issue with a guest CPU ID changing on a guest which I have listed below.

• We have an application running on Windows Server 2016 that used CPUID as part of the application licensing.
• HA kicked in due to a network issue and this caused the machine to start up on host 3 of a 3 host cluster.
• The application license on the guest VM had become invalidated to to CPUID change.
• Checked the vsphere logs and saw that in the months past the VM had VMotioned successfully between hosts 1 and 2 without issue many times.
• Moved VM back to host 2 hoping to reactivate the license but the CPUID stayed the same.
• CPUD ID was: 0FABFBFF000306F0  is now: 0F8BFBFF000306F0
• EVC is enabled on the cluster and is set to intel "Haswell" Generation.

My questions are:

How are CPUID's generated for the guest machines? I have read a post from 2014 that said the CPUID was in the .vmx file. I had a look but it was not there so I assume that has changed over the years.

I checked other 2016 servers running on the same host and found each VM has a different CPUID, are unique values assigned to each guest during provisioning? Shouldn't the CPUID's be the same for machines running on the same host?

And my most important question: How can the CPUID change and stay changed after the server restarts on another host?

Thanks for the help.

Hi everyone,

Sorry if this is a trivial question, but I am new to VMWare and still trying to get to know it a bit better.

I have a server with VMWare ESXi 6.5U2 (free Hypervisor) installed, which is connected to my internet router. Since this is a free license, I do not have vCenter available. Now, I would like to host a linux server as a virtual machine (web server etc.) that is protected by a transparent firewall setup (OPNsense or similar) placed as a virtual machine "before" the virtual host. Incomfing traffice would be routed to the ESXi host (port forwarding), through the transparent firewall and to the linux server.

However, I am at loss on how I can possibly configure the vSwitches in the ESXi Web Interface to allow for such a scenario. Can anyone point me in the right direction? Any help is appreciated!

Best,

Andreas

Hi,

I detach a disk from one VM and move it to other VM. while doing that the disk was changing from thin to thick. I want to mention that the disk is full capacity.

Why does VMware changing it to thick without mention it anywhere?

What will happen when i delete files from the disk? is it still going to stay thick?

thanks.

I'm relatively new to ESXi. I ordered a new workstation PC and tried to install ESXi 6.7 in UEFI mode on it. The installation won't start in UEFI with the following message.

using simple offset uefi rts mapping policy 

It gets stuck after this message.

Right afterwards I tried to install in legacy. The installation finishes successfully however it won't boot afterwards stating that it can't find a disc to boot from.

The PC is on the latest BIOS version.

The following hardware is used:

• Intel Xeon E-2176G 6C CPU
• (4x16GB) DDR4 2666 DIMM ECC
• NVIDIA Quadro P400 2GB (3)mDP GFX
• 2x Intel I210-T1 Ethernet Server Adapter

Many thanks in advance!

We have a standalone ESXi 6.5.0 server with 3TB datastore (VMFS 5.81).  I attempted to deploy an OVF package but it failed twice with a file not found error.  Before I attempted the deployments, I had over 900GB free.  Now I only have 750GB free in the datastore.  Is there a way to reclaim that space?

I've tried to execute "esxcli storage vmfs reclaim config get -l=mydatastore" but it fails stating that version VMFS5 doesn't support the unmap property.

Any help in the matter is much appreciated.

ESXi 6.7.0 U2 and ESXi 6.5.0 U2 both give the same error.  I am using an MSI B450-A-Pro motherboard, AMD Ryzen 7 1700 and 32G RAM.  I get the following error, " no network adapters were detected.  either no network adapters are physically connected or a suitable driver could be located."

Any help would be appreciated.

Brian

Is it possible to use the Update Manager to update the host it is running on?  Or is my only option to use CLI or a USB/ISO ?

Thanks!

Hi,

I've been running ESXi Hypervisior for sometime now in a home lab. I've recently reinstalled onto larger storage (same physical machine), imported my OVF's from the previous storage. But when I try and start them I get "Failed to power on virtual machine xxx. License key has expired".

When I log into myvmare I can't seem to locate how to get the free license back - any ideas?

Greetings,

     This is a lab environment. ESXi 6.7 version is running on Dell R710 SFF Server. We created a storage on another Dell R710 LFF with a capacity of 8 TB one volume and 5 TB one volume as virtual disks. Using 5 TB virtual disk and FreeNAS we created an iSCSI Target on top of it. Now we are trying to setup the iSCSI storage and datastore as one Dell R710 SFF is acting as initiator and other Dell R710 LFF acting as a target.

Initiator (10.39.130.35) Dell R710 SFF specifications:

vCPUs 12 with 96 GB RAM

VMKernal2 with MTU 9000 and vDS with MTU of 9000

Target (10.39.130.222 iqn.2005-10.org.freenas.ctl:all-esxi) Dell R710 LFF specifications:

vCPUs 8 and 128 GB RAM

FreeNAS with em1 and MTU of 9000

Issue: The iSCSI Target is getting timeout and Disk is getting ONLINE and OFFLINE intermittently

Below are the logs of esxi vmkernal.log

 

 

[root@esxi1:~] cat /var/log/vmkernel.log

2019-05-07T17:48:32.912Z cpu4:2247842)Vol3: 1255: Could not open device '5cc6f8b0-8e513860-6903-0026b93f7b73' for volume open: Failure

2019-05-07T17:48:32.912Z cpu4:2247842)LVM: 5919: Could not open device naa.6589cfc000000f7547481ec53d72549e:1, vol [5cc6f8b0-8e513860-6903-0026b93f7b73, 5cc6f8b0-8e513860-6903-0026b93f7b73, 1]: LVM device unreachable

2019-05-07T17:48:32.912Z cpu4:2247842)Vol3: 1255: Could not open device '5cc6f8b0-8e513860-6903-0026b93f7b73' for volume open: Failure

2019-05-07T17:48:32.912Z cpu4:2247842)LVM: 5919: Could not open device naa.6589cfc000000f7547481ec53d72549e:1, vol [5cc6f8b0-8e513860-6903-0026b93f7b73, 5cc6f8b0-8e513860-6903-0026b93f7b73, 1]: LVM device unreachable

2019-05-07T17:48:32.912Z cpu4:2247842)FSS: 6092: No FS driver claimed device '5cc6f8b0-8e513860-6903-0026b93f7b73': No filesystem on the device

2019-05-07T17:48:32.918Z cpu0:2098497)PLOG: PLOGOpenDevice:4346: Disk handle open failure for device naa.6782bcb05bcabf0021f7e31c06bb079d:2, status:Busy

2019-05-07T17:48:32.919Z cpu0:2098497)PLOG: PLOGInitAndAnnounceMD:7958: Successfully announced VSAN MD (naa.6782bcb05bcabf0021f7e31c06bb079d:2) with UUID: 5259f2c5-80bd-17b5-20c5-77b4fd68b317. kt 1, en 0, enC 0.

2019-05-07T17:48:32.922Z cpu4:2097199)NMP: nmp_ThrottleLogForDevice:3727: last error status from device mpx.vmhba1:C0:T0:L0 repeated 2 times

2019-05-07T17:48:32.922Z cpu4:2097199)NMP: nmp_ThrottleLogForDevice:3781: Cmd 0x9e (0x459b98d365c0, 0) to dev "mpx.vmhba1:C0:T0:L0" on path "vmhba1:C0:T0:L0" Failed: H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

2019-05-07T17:48:32.922Z cpu4:2097199)ScsiDeviceIO: 3082: Cmd(0x459b98d365c0) 0x9e, CmdSN 0x16b2 from world 0 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x7 D:0x0 P:0x0 Invalid sense data: 0x73 0x65 0x20.

2019-05-07T17:48:32.933Z cpu4:2097199)NMP: nmp_ThrottleLogForDevice:3727: last error status from device mpx.vmhba1:C0:T0:L0 repeated 2 times

2019-05-07T17:48:32.933Z cpu4:2097199)NMP: nmp_ThrottleLogForDevice:3781: Cmd 0x9e (0x459b98d365c0, 0) to dev "mpx.vmhba1:C0:T0:L0" on path "vmhba1:C0:T0:L0" Failed: H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

2019-05-07T17:48:32.933Z cpu4:2097199)ScsiDeviceIO: 3082: Cmd(0x459b98d365c0) 0x9e, CmdSN 0x16b4 from world 0 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x7 D:0x0 P:0x0 Invalid sense data: 0x73 0x65 0x20.

2019-05-07T17:48:32.942Z cpu4:2097199)NMP: nmp_ThrottleLogForDevice:3727: last error status from device mpx.vmhba1:C0:T0:L0 repeated 2 times

2019-05-07T17:48:32.942Z cpu4:2097199)NMP: nmp_ThrottleLogForDevice:3781: Cmd 0x9e (0x459b98d365c0, 0) to dev "mpx.vmhba1:C0:T0:L0" on path "vmhba1:C0:T0:L0" Failed: H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

2019-05-07T17:48:32.942Z cpu4:2097199)ScsiDeviceIO: 3082: Cmd(0x459b98d365c0) 0x9e, CmdSN 0x16b6 from world 0 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x7 D:0x0 P:0x0 Invalid sense data: 0x73 0x65 0x20.

2019-05-07T17:48:32.954Z cpu4:2097199)NMP: nmp_ThrottleLogForDevice:3727: last error status from device mpx.vmhba1:C0:T0:L0 repeated 2 times

2019-05-07T17:48:32.954Z cpu4:2097199)NMP: nmp_ThrottleLogForDevice:3781: Cmd 0x9e (0x459b98d365c0, 0) to dev "mpx.vmhba1:C0:T0:L0" on path "vmhba1:C0:T0:L0" Failed: H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

2019-05-07T17:48:32.954Z cpu4:2097199)ScsiDeviceIO: 3082: Cmd(0x459b98d365c0) 0x9e, CmdSN 0x16b8 from world 0 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x7 D:0x0 P:0x0 Invalid sense data: 0x73 0x65 0x20.

2019-05-07T17:48:32.965Z cpu4:2097199)NMP: nmp_ThrottleLogForDevice:3727: last error status from device mpx.vmhba1:C0:T0:L0 repeated 2 times

2019-05-07T17:48:32.965Z cpu9:2247840)WARNING: NFS: 1227: Invalid volume UUID mpx.vmhba1:C0:T0:L0

2019-05-07T17:48:32.965Z cpu4:2097199)NMP: nmp_ThrottleLogForDevice:3781: Cmd 0x9e (0x459b98d365c0, 0) to dev "mpx.vmhba1:C0:T0:L0" on path "vmhba1:C0:T0:L0" Failed: H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

2019-05-07T17:48:32.965Z cpu4:2097199)ScsiDeviceIO: 3082: Cmd(0x459b98d365c0) 0x9e, CmdSN 0x16ba from world 0 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x7 D:0x0 P:0x0 Invalid sense data: 0x73 0x65 0x20.

2019-05-07T17:48:32.976Z cpu4:2097199)NMP: nmp_ThrottleLogForDevice:3727: last error status from device mpx.vmhba1:C0:T0:L0 repeated 2 times

2019-05-07T17:48:32.977Z cpu4:2097199)NMP: nmp_ThrottleLogForDevice:3781: Cmd 0x9e (0x459b98d365c0, 0) to dev "mpx.vmhba1:C0:T0:L0" on path "vmhba1:C0:T0:L0" Failed: H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

2019-05-07T17:48:32.977Z cpu4:2097199)ScsiDeviceIO: 3082: Cmd(0x459b98d365c0) 0x9e, CmdSN 0x16bc from world 0 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x7 D:0x0 P:0x0 Invalid sense data: 0x73 0x65 0x20.

2019-05-07T17:48:32.995Z cpu4:2097199)NMP: nmp_ThrottleLogForDevice:3727: last error status from device mpx.vmhba1:C0:T0:L0 repeated 2 times

2019-05-07T17:48:32.995Z cpu4:2097199)NMP: nmp_ThrottleLogForDevice:3781: Cmd 0x9e (0x459b98d365c0, 0) to dev "mpx.vmhba1:C0:T0:L0" on path "vmhba1:C0:T0:L0" Failed: H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

2019-05-07T17:48:32.995Z cpu4:2097199)ScsiDeviceIO: 3082: Cmd(0x459b98d365c0) 0x9e, CmdSN 0x16be from world 0 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x7 D:0x0 P:0x0 Invalid sense data: 0x73 0x65 0x20.

2019-05-07T17:48:33.005Z cpu4:2097199)NMP: nmp_ThrottleLogForDevice:3727: last error status from device mpx.vmhba1:C0:T0:L0 repeated 2 times

2019-05-07T17:48:33.005Z cpu4:2097199)NMP: nmp_ThrottleLogForDevice:3781: Cmd 0x9e (0x459b98d365c0, 0) to dev "mpx.vmhba1:C0:T0:L0" on path "vmhba1:C0:T0:L0" Failed: H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

2019-05-07T17:48:33.005Z cpu4:2097199)ScsiDeviceIO: 3082: Cmd(0x459b98d365c0) 0x9e, CmdSN 0x16c5 from world 0 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x7 D:0x0 P:0x0 Invalid sense data: 0x73 0x65 0x20.

2019-05-07T17:48:33.017Z cpu4:2097199)NMP: nmp_ThrottleLogForDevice:3727: last error status from device mpx.vmhba1:C0:T0:L0 repeated 2 times

2019-05-07T17:48:33.017Z cpu9:2097204)NMP: nmp_ThrottleLogForDevice:3781: Cmd 0x9e (0x459b98d365c0, 0) to dev "mpx.vmhba1:C0:T0:L0" on path "vmhba1:C0:T0:L0" Failed: H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

2019-05-07T17:48:33.017Z cpu9:2097204)ScsiDeviceIO: 3082: Cmd(0x459b98d365c0) 0x9e, CmdSN 0x16c7 from world 0 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x7 D:0x0 P:0x0 Invalid sense data: 0x73 0x65 0x20.

2019-05-07T17:48:33.026Z cpu9:2097204)NMP: nmp_ThrottleLogForDevice:3727: last error status from device mpx.vmhba1:C0:T0:L0 repeated 2 times

2019-05-07T17:48:33.026Z cpu9:2097204)NMP: nmp_ThrottleLogForDevice:3781: Cmd 0x9e (0x459b98d365c0, 0) to dev "mpx.vmhba1:C0:T0:L0" on path "vmhba1:C0:T0:L0" Failed: H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

2019-05-07T17:48:33.026Z cpu9:2097204)ScsiDeviceIO: 3082: Cmd(0x459b98d365c0) 0x9e, CmdSN 0x16c9 from world 0 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x7 D:0x0 P:0x0 Invalid sense data: 0x73 0x65 0x20.

2019-05-07T17:48:33.038Z cpu9:2097204)NMP: nmp_ThrottleLogForDevice:3727: last error status from device mpx.vmhba1:C0:T0:L0 repeated 2 times

2019-05-07T17:48:33.038Z cpu9:2097204)NMP: nmp_ThrottleLogForDevice:3781: Cmd 0x9e (0x459b98d365c0, 0) to dev "mpx.vmhba1:C0:T0:L0" on path "vmhba1:C0:T0:L0" Failed: H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

2019-05-07T17:48:33.038Z cpu9:2097204)ScsiDeviceIO: 3082: Cmd(0x459b98d365c0) 0x9e, CmdSN 0x16cb from world 0 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x7 D:0x0 P:0x0 Invalid sense data: 0x73 0x65 0x20.

2019-05-07T17:48:33.047Z cpu9:2097204)NMP: nmp_ThrottleLogForDevice:3727: last error status from device mpx.vmhba1:C0:T0:L0 repeated 2 times

2019-05-07T17:48:33.047Z cpu9:2097204)NMP: nmp_ThrottleLogForDevice:3781: Cmd 0x9e (0x459b98d365c0, 0) to dev "mpx.vmhba1:C0:T0:L0" on path "vmhba1:C0:T0:L0" Failed: H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

2019-05-07T17:48:33.047Z cpu9:2097204)ScsiDeviceIO: 3082: Cmd(0x459b98d365c0) 0x9e, CmdSN 0x16cd from world 0 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x7 D:0x0 P:0x0 Invalid sense data: 0x73 0x65 0x20.

2019-05-07T17:48:33.055Z cpu9:2097204)NMP: nmp_ThrottleLogForDevice:3727: last error status from device mpx.vmhba1:C0:T0:L0 repeated 2 times

2019-05-07T17:48:33.059Z cpu4:2247840)FSS: 6092: No FS driver claimed device 'mpx.vmhba1:C0:T0:L0': No filesystem on the device

2019-05-07T17:48:34.438Z cpu1:2097196)ScsiDeviceIO: 3068: Cmd(0x459b98d97dc0) 0x1a, CmdSN 0x1716 from world 0 to dev "t10.SanDisk00Ultra_USB_3.00000004C530000261203102470" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2019-05-07T17:48:34.543Z cpu4:2100460 opID=46e20a1f)VC: 4616: Device rescan time 10157 msec (total number of devices 6)

2019-05-07T17:48:34.543Z cpu4:2100460 opID=46e20a1f)VC: 4619: Filesystem probe time 1628 msec (devices probed 5 of 6)

2019-05-07T17:48:34.543Z cpu4:2100460 opID=46e20a1f)VC: 4621: Refresh open volume time 4 msec

2019-05-07T17:49:04.106Z cpu13:2097835)NMP: nmp_ResetDeviceLogThrottling:3569: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x2 0x3a 0x1 from dev "mpx.vmhba1:C0:T0:L0" occurred 1 times(of 1 commands)

2019-05-07T17:49:19.107Z cpu2:2098823)SunRPC: 1099: Destroying world 0x224cfe

2019-05-07T17:49:34.107Z cpu7:2097507)ScsiVmas: 804: Inquiry for VPD page 00 to device naa.6589cfc000000f7547481ec53d72549e failed with error No connection

2019-05-07T17:49:34.963Z cpu6:2097201)ScsiDeviceIO: 3068: Cmd(0x459b8aafa700) 0x1a, CmdSN 0x1722 from world 0 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2019-05-07T17:50:22.108Z cpu1:2098823)SunRPC: 1099: Destroying world 0x224dfd

2019-05-07T17:51:17.038Z cpu17:2097726)iscsi_vmk: iscsivmk_ConnTxNotifyFailure:1197: vmhba65:CH:0 T:1 CN:0: Connection tx notifying failure: Network Send Error. State=Online

2019-05-07T17:51:17.038Z cpu17:2097726)iscsi_vmk: iscsivmk_ConnTxNotifyFailure:1198: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:all-esxi TPGT: 1 TSIH: 0]

2019-05-07T17:51:17.038Z cpu17:2097726)iscsi_vmk: iscsivmk_ConnTxNotifyFailure:1199: Conn [CID: 0 L: 10.39.130.35:10848 R: 10.39.130.222:3260]

2019-05-07T17:51:17.038Z cpu16:2097688)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba65:CH:0 T:1 CN:0: iSCSI connection is being marked "OFFLINE" (Event:6)

2019-05-07T17:51:17.038Z cpu16:2097688)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:all-esxi TPGT: 1 TSIH: 0]

2019-05-07T17:51:17.038Z cpu16:2097688)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 10.39.130.35:10848 R: 10.39.130.222:3260]

2019-05-07T17:51:17.041Z cpu18:2097547)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue:633: vmhba65:CH:0 T:1 L:70 : Task mgmt "Abort Task" with itt=0xa47 (refITT=0xa3e) timed out.

2019-05-07T17:51:17.041Z cpu18:2097547)NMP: nmp_ThrottleLogForDevice:3781: Cmd 0x89 (0x459a40c1d280, 2097185) to dev "naa.6589cfc0000002975052862c22403cca" on path "vmhba65:C0:T1:L70" Failed: H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

2019-05-07T17:51:17.041Z cpu18:2097547)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6589cfc0000002975052862c22403cca" state in doubt; requested fast path state update...

2019-05-07T17:51:17.041Z cpu16:2097705)NMP: nmp_ThrottleLogForDevice:3781: Cmd 0x28 (0x45a37d996b40, 2098280) to dev "naa.6589cfc0000002975052862c22403cca" on path "vmhba65:C0:T1:L70" Failed: H:0x2 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

2019-05-07T17:51:17.041Z cpu18:2097547)ScsiDeviceIO: 3082: Cmd(0x459a40c1d280) 0x89, CmdSN 0x6d from world 2097185 to dev "naa.6589cfc0000002975052862c22403cca" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x35 0x20.

2019-05-07T17:51:17.041Z cpu16:2097705)ScsiDeviceIO: 3047: Cmd(0x45a37d996b40) 0x28, CmdSN 0x6c from world 2098280 to dev "naa.6589cfc0000002975052862c22403cca" failed H:0x2 D:0x0 P:0x0 Invalid sense data: 0x78 0x32 0x20.

2019-05-07T17:51:17.042Z cpu16:2097211)ScsiDeviceIO: 3047: Cmd(0x459b98c86580) 0x28, CmdSN 0x6e from world 2100138 to dev "naa.6589cfc0000002975052862c22403cca" failed H:0x2 D:0x0 P:0x0 Invalid sense data: 0x9 0x1a 0x45.

2019-05-07T17:51:17.106Z cpu16:2097705)NMP: nmp_ThrottleLogForDevice:3710: last error status from device naa.6589cfc0000002975052862c22403cca repeated 10 times

2019-05-07T17:51:17.110Z cpu8:2097505)LVM: 5748: Received APD EventType: APD_START (3) for device <naa.6589cfc0000002975052862c22403cca:3> (gen 17)

2019-05-07T17:51:17.110Z cpu8:2097505)LVM: 5356: Handling APD EventType: APD_START (3) for device <naa.6589cfc0000002975052862c22403cca:3> (unlocked, gen 17, cur apd state UNKNOWN, cur dev state 1)

2019-05-07T17:51:17.110Z cpu8:2097505)HBX: 6287: APD EventType: APD_START (3) for vol 'SAN-PRODUCTION'

2019-05-07T17:51:17.110Z cpu8:2097505)ScsiDevice: 5978: Device state of naa.6589cfc0000002975052862c22403cca set to APD_START; token num:1

2019-05-07T17:51:17.110Z cpu8:2097505)StorageApdHandler: 1203: APD start for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:51:17.110Z cpu0:2097490)StorageApdHandler: 419: APD start event for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:51:17.110Z cpu0:2097490)StorageApdHandlerEv: 110: Device or filesystem with identifier [naa.6589cfc0000002975052862c22403cca] has entered the All Paths Down state.

2019-05-07T17:51:17.112Z cpu2:2098234)WARNING: NMP: nmp_IssueCommandToDevice:5082: I/O could not be issued to device "naa.6589cfc0000002975052862c22403cca" due to Not found

2019-05-07T17:51:17.112Z cpu2:2098234)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.6589cfc0000002975052862c22403cca": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.

2019-05-07T17:51:17.112Z cpu2:2098234)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6589cfc0000002975052862c22403cca" is blocked. Not starting I/O from device.

2019-05-07T17:51:18.040Z cpu12:2097547)ScsiDeviceIO: 3082: Cmd(0x459b98ce9f40) 0x89, CmdSN 0x6f from world 2097185 to dev "naa.6589cfc0000002975052862c22403cca" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x18 0x4 0x43.

2019-05-07T17:51:18.040Z cpu12:2097547)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6589cfc0000002975052862c22403cca" is blocked. Not starting I/O from device.

2019-05-07T17:51:18.040Z cpu9:2097204)ScsiDeviceIO: 3032: Cmd(0x45a37d996b40) 0x28, CmdSN 0x6c from world 2098280 to dev "naa.6589cfc0000002975052862c22403cca" failed H:0x8 D:0x0 P:0x0

2019-05-07T17:51:18.041Z cpu9:2097204)ScsiDeviceIO: 3032: Cmd(0x459b98c86580) 0x28, CmdSN 0x6e from world 2100138 to dev "naa.6589cfc0000002975052862c22403cca" failed H:0x8 D:0x0 P:0x0

2019-05-07T17:51:18.041Z cpu12:2098281)HBX: 3033: 'SAN-PRODUCTION': HB at offset 3407872 - Waiting for timed out HB:

2019-05-07T17:51:18.041Z cpu12:2098281)  [HB state abcdef02 offset 3407872 gen 37 stampUS 24703934162 uuid 5cd16571-5dd95f1e-0244-782bcb396c1f jrnl <FB 4> drv 24.82 lockImpl 4 ip 172.16.0.35]

2019-05-07T17:51:18.110Z cpu0:2097836)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world restore device "naa.6589cfc0000002975052862c22403cca" - no more commands to retry

2019-05-07T17:51:19.331Z cpu11:2248652)HBX: 3033: 'SAN-PRODUCTION': HB at offset 3407872 - Waiting for timed out HB:

2019-05-07T17:51:19.331Z cpu11:2248652)  [HB state abcdef02 offset 3407872 gen 37 stampUS 24703934162 uuid 5cd16571-5dd95f1e-0244-782bcb396c1f jrnl <FB 4> drv 24.82 lockImpl 4 ip 172.16.0.35]

2019-05-07T17:51:20.146Z cpu22:2098046)iscsi_vmk: iscsivmk_ConnNetRegister:2170: socket 0x430bd54cc9f0 network resource pool netsched.pools.persist.iscsi associated

2019-05-07T17:51:20.146Z cpu22:2098046)iscsi_vmk: iscsivmk_ConnNetRegister:2198: socket 0x430bd54cc9f0 network tracker id 45045162 tracker.iSCSI.10.39.130.222 associated

2019-05-07T17:51:20.398Z cpu22:2098046)WARNING: iscsi_vmk: iscsivmk_StartConnection:880: vmhba65:CH:0 T:1 CN:0: iSCSI connection is being marked "ONLINE"

2019-05-07T17:51:20.398Z cpu22:2098046)WARNING: iscsi_vmk: iscsivmk_StartConnection:881: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:all-esxi TPGT: 1 TSIH: 0]

2019-05-07T17:51:20.398Z cpu22:2098046)WARNING: iscsi_vmk: iscsivmk_StartConnection:882: Conn [CID: 0 L: 10.39.130.35:56605 R: 10.39.130.222:3260]

2019-05-07T17:51:20.541Z cpu2:2100195)WARNING: NMP: nmp_IssueCommandToDevice:5082: I/O could not be issued to device "naa.6589cfc0000002975052862c22403cca" due to Not found

2019-05-07T17:51:20.541Z cpu2:2100195)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.6589cfc0000002975052862c22403cca": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.

2019-05-07T17:51:20.542Z cpu2:2100195)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6589cfc0000002975052862c22403cca" is blocked. Not starting I/O from device.

2019-05-07T17:51:21.114Z cpu5:2097505)ScsiDevice: 6000: Setting Device naa.6589cfc0000002975052862c22403cca state back to 0x2

2019-05-07T17:51:21.114Z cpu5:2097505)LVM: 5748: Received APD EventType: APD_EXIT (4) for device <naa.6589cfc0000002975052862c22403cca:3> (gen 18)

2019-05-07T17:51:21.114Z cpu5:2097505)LVM: 5356: Handling APD EventType: APD_EXIT (4) for device <naa.6589cfc0000002975052862c22403cca:3> (unlocked, gen 18, cur apd state APD_START, cur dev state 1)

2019-05-07T17:51:21.114Z cpu5:2097505)ScsiDevice: 6021: Device naa.6589cfc0000002975052862c22403cca is Out of APD; token num:1

2019-05-07T17:51:21.114Z cpu0:2097255)LVM: 5356: Handling APD EventType: APD_EXIT (4) for device <naa.6589cfc0000002975052862c22403cca:3> (locked, gen 18, cur apd state APD_START, cur dev state 1)

2019-05-07T17:51:21.114Z cpu1:2097836)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - issuing command 0x459a40d93740

2019-05-07T17:51:24.463Z cpu16:2097211)NMP: nmpCompleteRetryForPath:327: Retry world recovered device "naa.6589cfc0000002975052862c22403cca"

2019-05-07T17:51:25.110Z cpu0:2098823)SunRPC: 3303: Synchronous RPC abort for client 0x4305cc2271c0 IP 10.39.115.222.0.111 proc 3 xid 0x439cd42c attempt 3 of 3

2019-05-07T17:51:25.110Z cpu0:2098823)SunRPC: 1099: Destroying world 0x224fd9

2019-05-07T17:51:29.332Z cpu1:2248652)HBX: 3033: 'SAN-PRODUCTION': HB at offset 3407872 - Waiting for timed out HB:

2019-05-07T17:51:29.332Z cpu1:2248652)  [HB state abcdef02 offset 3407872 gen 37 stampUS 24703934162 uuid 5cd16571-5dd95f1e-0244-782bcb396c1f jrnl <FB 4> drv 24.82 lockImpl 4 ip 172.16.0.35]

2019-05-07T17:51:33.390Z cpu13:2098820)WARNING: LSOM: LSOMVsiGetVirstoInstanceStats:800: Throttled: Attempt to get Virsto stats on unsupported disk52256481-aa44-4783-d2f0-625461b41dfe

2019-05-07T17:51:39.335Z cpu1:2248652)HBX: 3033: 'SAN-PRODUCTION': HB at offset 3407872 - Waiting for timed out HB:

2019-05-07T17:51:39.335Z cpu1:2248652)  [HB state abcdef02 offset 3407872 gen 37 stampUS 24703934162 uuid 5cd16571-5dd95f1e-0244-782bcb396c1f jrnl <FB 4> drv 24.82 lockImpl 4 ip 172.16.0.35]

2019-05-07T17:51:46.272Z cpu20:2097729)iscsi_vmk: iscsivmk_ConnTxNotifyFailure:1197: vmhba65:CH:0 T:1 CN:0: Connection tx notifying failure: Network Send Error. State=Online

2019-05-07T17:51:46.272Z cpu20:2097729)iscsi_vmk: iscsivmk_ConnTxNotifyFailure:1198: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:all-esxi TPGT: 1 TSIH: 0]

2019-05-07T17:51:46.272Z cpu20:2097729)iscsi_vmk: iscsivmk_ConnTxNotifyFailure:1199: Conn [CID: 0 L: 10.39.130.35:56605 R: 10.39.130.222:3260]

2019-05-07T17:51:46.272Z cpu18:2097688)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba65:CH:0 T:1 CN:0: iSCSI connection is being marked "OFFLINE" (Event:6)

2019-05-07T17:51:46.272Z cpu18:2097688)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:all-esxi TPGT: 1 TSIH: 0]

2019-05-07T17:51:46.272Z cpu18:2097688)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 10.39.130.35:56605 R: 10.39.130.222:3260]

2019-05-07T17:51:46.281Z cpu18:2097707)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6589cfc0000002975052862c22403cca" state in doubt; requested fast path state update...

2019-05-07T17:51:46.281Z cpu18:2097707)ScsiDeviceIO: 3047: Cmd(0x459a40cde840) 0x28, CmdSN 0x73 from world 0 to dev "naa.6589cfc0000002975052862c22403cca" failed H:0x2 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x43.

2019-05-07T17:51:46.281Z cpu18:2097707)ScsiDeviceIO: 3047: Cmd(0x459b98d932c0) 0x12, CmdSN 0x1782 from world 0 to dev "naa.6589cfc0000002975052862c22403cca" failed H:0x2 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x43.

2019-05-07T17:51:46.394Z cpu18:2097707)NMP: nmp_ThrottleLogForDevice:3710: last error status from device naa.6589cfc0000002975052862c22403cca repeated 20 times

2019-05-07T17:51:46.645Z cpu18:2097707)NMP: nmp_ThrottleLogForDevice:3710: last error status from device naa.6589cfc0000002975052862c22403cca repeated 40 times

2019-05-07T17:51:47.111Z cpu0:2097505)LVM: 5748: Received APD EventType: APD_START (3) for device <naa.6589cfc0000002975052862c22403cca:3> (gen 19)

2019-05-07T17:51:47.111Z cpu0:2097505)LVM: 5356: Handling APD EventType: APD_START (3) for device <naa.6589cfc0000002975052862c22403cca:3> (unlocked, gen 19, cur apd state APD_START, cur dev state 1)

2019-05-07T17:51:47.111Z cpu0:2097505)WARNING: LVM: 5375: Pre-existing unhandled APD event for device <naa.6589cfc0000002975052862c22403cca:3>: type APD_EXIT (gen 18)

2019-05-07T17:51:47.111Z cpu0:2097505)StorageApdHandler: 1315: APD exit for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:51:47.111Z cpu0:2097505)HBX: 6287: APD EventType: APD_START (3) for vol 'SAN-PRODUCTION'

2019-05-07T17:51:47.111Z cpu0:2097505)ScsiDevice: 5978: Device state of naa.6589cfc0000002975052862c22403cca set to APD_START; token num:2

2019-05-07T17:51:47.111Z cpu0:2097505)StorageApdHandler: 1218: APD bounce-start for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca], no event transition

2019-05-07T17:51:47.118Z cpu18:2098469)WARNING: NMP: nmp_IssueCommandToDevice:5082: I/O could not be issued to device "naa.6589cfc0000002975052862c22403cca" due to Not found

2019-05-07T17:51:47.118Z cpu18:2098469)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.6589cfc0000002975052862c22403cca": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.

2019-05-07T17:51:47.118Z cpu18:2098469)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6589cfc0000002975052862c22403cca" is blocked. Not starting I/O from device.

2019-05-07T17:51:48.111Z cpu0:2097836)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - issuing command 0x459a40cde840

2019-05-07T17:51:48.111Z cpu0:2097836)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - failed to issue command due to Not found (APD), try again...

2019-05-07T17:51:48.111Z cpu0:2097836)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6589cfc0000002975052862c22403cca": awaiting fast path state update...

2019-05-07T17:51:49.044Z cpu22:2098046)iscsi_vmk: iscsivmk_ConnNetRegister:2170: socket 0x430bd5546e80 network resource pool netsched.pools.persist.iscsi associated

2019-05-07T17:51:49.044Z cpu22:2098046)iscsi_vmk: iscsivmk_ConnNetRegister:2198: socket 0x430bd5546e80 network tracker id 45045162 tracker.iSCSI.10.39.130.222 associated

2019-05-07T17:51:49.111Z cpu0:2097836)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - issuing command 0x459a40cde840

2019-05-07T17:51:49.111Z cpu0:2097836)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - failed to issue command due to Not found (APD), try again...

2019-05-07T17:51:49.111Z cpu0:2097836)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6589cfc0000002975052862c22403cca": awaiting fast path state update...

2019-05-07T17:51:49.298Z cpu22:2098046)WARNING: iscsi_vmk: iscsivmk_StartConnection:880: vmhba65:CH:0 T:1 CN:0: iSCSI connection is being marked "ONLINE"

2019-05-07T17:51:49.298Z cpu22:2098046)WARNING: iscsi_vmk: iscsivmk_StartConnection:881: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:all-esxi TPGT: 1 TSIH: 0]

2019-05-07T17:51:49.298Z cpu22:2098046)WARNING: iscsi_vmk: iscsivmk_StartConnection:882: Conn [CID: 0 L: 10.39.130.35:37059 R: 10.39.130.222:3260]

2019-05-07T17:51:49.339Z cpu3:2248652)HBX: 3033: 'SAN-PRODUCTION': HB at offset 3407872 - Waiting for timed out HB:

2019-05-07T17:51:49.339Z cpu3:2248652)  [HB state abcdef02 offset 3407872 gen 37 stampUS 24703934162 uuid 5cd16571-5dd95f1e-0244-782bcb396c1f jrnl <FB 4> drv 24.82 lockImpl 4 ip 172.16.0.35]

2019-05-07T17:51:50.113Z cpu10:2097505)ScsiDevice: 6000: Setting Device naa.6589cfc0000002975052862c22403cca state back to 0x2

2019-05-07T17:51:50.113Z cpu10:2097505)LVM: 5748: Received APD EventType: APD_EXIT (4) for device <naa.6589cfc0000002975052862c22403cca:3> (gen 20)

2019-05-07T17:51:50.113Z cpu10:2097505)LVM: 5356: Handling APD EventType: APD_EXIT (4) for device <naa.6589cfc0000002975052862c22403cca:3> (unlocked, gen 20, cur apd state APD_START, cur dev state 1)

2019-05-07T17:51:50.113Z cpu10:2097505)ScsiDevice: 6021: Device naa.6589cfc0000002975052862c22403cca is Out of APD; token num:1

2019-05-07T17:51:50.114Z cpu16:2098023)NMP: nmpCompleteRetryForPath:327: Retry world recovered device "naa.6589cfc0000002975052862c22403cca"

2019-05-07T17:51:53.614Z cpu4:2097255)LVM: 5425: Async handling of event APD_EXIT for device <naa.6589cfc0000002975052862c22403cca:3> took too long - bailing (my gen 18, cur gen 20, cur state APD_START)

2019-05-07T17:51:53.614Z cpu4:2097255)LVM: 5356: Handling APD EventType: APD_EXIT (4) for device <naa.6589cfc0000002975052862c22403cca:3> (locked, gen 20, cur apd state APD_START, cur dev state 1)

2019-05-07T17:51:59.043Z cpu0:2097255)HBX: 6287: APD EventType: APD_EXIT (4) for vol 'SAN-PRODUCTION'

2019-05-07T17:51:59.043Z cpu0:2097255)LVM: 5487: Device <naa.6589cfc000000f7547481ec53d72549e:1> APD handler with closeOnAPDExit - attempting close.

2019-05-07T17:51:59.045Z cpu8:2098161)HBX: 312: 'SAN-PRODUCTION': HB at offset 3407872 - Reclaimed heartbeat [Timeout]:

2019-05-07T17:51:59.045Z cpu8:2098161)  [HB state abcdef02 offset 3407872 gen 37 stampUS 24744936741 uuid 5cd16571-5dd95f1e-0244-782bcb396c1f jrnl <FB 4> drv 24.82 lockImpl 4 ip 172.16.0.35]

2019-05-07T17:51:59.045Z cpu8:2098161)StorageApdHandler: 1315: APD exit for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:51:59.045Z cpu8:2098161)HBX: 5509: Completed APD/PDL event processing on vol 'SAN-PRODUCTION'.

2019-05-07T17:51:59.045Z cpu0:2097490)StorageApdHandler: 507: APD exit event for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:51:59.045Z cpu0:2097490)StorageApdHandlerEv: 117: Device or filesystem with identifier [naa.6589cfc0000002975052862c22403cca] has exited the All Paths Down state.

2019-05-07T17:52:04.114Z cpu4:2097835)NMP: nmp_ResetDeviceLogThrottling:3569: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x24 0x0 from dev "mpx.vmhba1:C0:T0:L0" occurred 1 times(of 1 commands)

2019-05-07T17:52:26.114Z cpu2:2098823)SunRPC: 1099: Destroying world 0x2250fb

2019-05-07T17:52:34.112Z cpu8:2097835)NMP: nmp_ResetDeviceLogThrottling:3575: last error status from device naa.6589cfc0000002975052862c22403cca repeated 73 times

2019-05-07T17:52:58.437Z cpu14:2097918)ScsiDeviceIO: 3068: Cmd(0x45a24068ac40) 0x1a, CmdSN 0x17b8 from world 0 to dev "naa.6782bcb05bcabf0021f7e31c06bb079d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2019-05-07T17:53:19.373Z cpu0:2248652)HBX: 3033: 'SAN-PRODUCTION': HB at offset 3407872 - Waiting for timed out HB:

2019-05-07T17:53:19.373Z cpu0:2248652)  [HB state abcdef02 offset 3407872 gen 37 stampUS 24811934158 uuid 5cd16571-5dd95f1e-0244-782bcb396c1f jrnl <FB 4> drv 24.82 lockImpl 4 ip 172.16.0.35]

2019-05-07T17:53:24.048Z cpu12:2097547)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue:633: vmhba65:CH:0 T:1 L:70 : Task mgmt "Abort Task" with itt=0xb3c (refITT=0xb3b) timed out.

2019-05-07T17:53:24.048Z cpu17:2097726)iscsi_vmk: iscsivmk_ConnTxNotifyFailure:1197: vmhba65:CH:0 T:1 CN:0: Connection tx notifying failure: Network Send Error. State=Online

2019-05-07T17:53:24.048Z cpu17:2097726)iscsi_vmk: iscsivmk_ConnTxNotifyFailure:1198: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:all-esxi TPGT: 1 TSIH: 0]

2019-05-07T17:53:24.048Z cpu17:2097726)iscsi_vmk: iscsivmk_ConnTxNotifyFailure:1199: Conn [CID: 0 L: 10.39.130.35:37059 R: 10.39.130.222:3260]

2019-05-07T17:53:24.048Z cpu23:2097688)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba65:CH:0 T:1 CN:0: iSCSI connection is being marked "OFFLINE" (Event:6)

2019-05-07T17:53:24.048Z cpu23:2097688)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:all-esxi TPGT: 1 TSIH: 0]

2019-05-07T17:53:24.048Z cpu23:2097688)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 10.39.130.35:37059 R: 10.39.130.222:3260]

2019-05-07T17:53:24.052Z cpu12:2097547)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue:633: vmhba65:CH:0 T:1 L:70 : Task mgmt "Abort Task" with itt=0xb3d (refITT=0xb3b) timed out.

2019-05-07T17:53:24.052Z cpu12:2097547)NMP: nmp_ThrottleLogForDevice:3781: Cmd 0x89 (0x459a40c8cf40, 2097185) to dev "naa.6589cfc0000002975052862c22403cca" on path "vmhba65:C0:T1:L70" Failed: H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

2019-05-07T17:53:24.052Z cpu12:2097547)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6589cfc0000002975052862c22403cca" state in doubt; requested fast path state update...

2019-05-07T17:53:24.052Z cpu23:2097218)NMP: nmp_ThrottleLogForDevice:3781: Cmd 0x28 (0x459b98d37c40, 2100138) to dev "naa.6589cfc0000002975052862c22403cca" on path "vmhba65:C0:T1:L70" Failed: H:0x2 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

2019-05-07T17:53:24.052Z cpu12:2097547)ScsiDeviceIO: 3082: Cmd(0x459a40c8cf40) 0x89, CmdSN 0xae from world 2097185 to dev "naa.6589cfc0000002975052862c22403cca" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x35 0x20.

2019-05-07T17:53:24.052Z cpu23:2097218)ScsiDeviceIO: 3047: Cmd(0x459b98d37c40) 0x28, CmdSN 0xad from world 2100138 to dev "naa.6589cfc0000002975052862c22403cca" failed H:0x2 D:0x0 P:0x0 Invalid sense data: 0x78 0x32 0x20.

2019-05-07T17:53:24.052Z cpu8:2097697)ScsiDeviceIO: 3047: Cmd(0x459b98d76d00) 0x28, CmdSN 0xaf from world 0 to dev "naa.6589cfc0000002975052862c22403cca" failed H:0x2 D:0x0 P:0x0 Invalid sense data: 0x2 0x80 0x41.

2019-05-07T17:53:24.114Z cpu1:2097505)LVM: 5748: Received APD EventType: APD_START (3) for device <naa.6589cfc0000002975052862c22403cca:3> (gen 21)

2019-05-07T17:53:24.114Z cpu1:2097505)LVM: 5356: Handling APD EventType: APD_START (3) for device <naa.6589cfc0000002975052862c22403cca:3> (unlocked, gen 21, cur apd state UNKNOWN, cur dev state 1)

2019-05-07T17:53:24.114Z cpu1:2097505)HBX: 6287: APD EventType: APD_START (3) for vol 'SAN-PRODUCTION'

2019-05-07T17:53:24.114Z cpu1:2097505)ScsiDevice: 5978: Device state of naa.6589cfc0000002975052862c22403cca set to APD_START; token num:1

2019-05-07T17:53:24.114Z cpu1:2097505)StorageApdHandler: 1203: APD start for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:53:24.114Z cpu0:2097490)StorageApdHandler: 419: APD start event for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:53:24.114Z cpu0:2097490)StorageApdHandlerEv: 110: Device or filesystem with identifier [naa.6589cfc0000002975052862c22403cca] has entered the All Paths Down state.

2019-05-07T17:53:24.120Z cpu8:2097183)WARNING: NMP: nmp_IssueCommandToDevice:5082: I/O could not be issued to device "naa.6589cfc0000002975052862c22403cca" due to Not found

2019-05-07T17:53:24.120Z cpu8:2097183)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.6589cfc0000002975052862c22403cca": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.

2019-05-07T17:53:24.120Z cpu8:2097183)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6589cfc0000002975052862c22403cca" is blocked. Not starting I/O from device.

2019-05-07T17:53:25.114Z cpu0:2097836)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - issuing command 0x459b98d76d00

2019-05-07T17:53:25.114Z cpu0:2097836)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - failed to issue command due to Not found (APD), try again...

2019-05-07T17:53:25.114Z cpu0:2097836)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6589cfc0000002975052862c22403cca": awaiting fast path state update...

2019-05-07T17:53:26.113Z cpu0:2097836)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - issuing command 0x459b98d76d00

2019-05-07T17:53:26.114Z cpu0:2097836)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - failed to issue command due to Not found (APD), try again...

2019-05-07T17:53:26.114Z cpu0:2097836)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6589cfc0000002975052862c22403cca": awaiting fast path state update...

2019-05-07T17:53:26.559Z cpu12:2098046)iscsi_vmk: iscsivmk_ConnNetRegister:2170: socket 0x430bd5546e80 network resource pool netsched.pools.persist.iscsi associated

2019-05-07T17:53:26.559Z cpu12:2098046)iscsi_vmk: iscsivmk_ConnNetRegister:2198: socket 0x430bd5546e80 network tracker id 45045162 tracker.iSCSI.10.39.130.222 associated

2019-05-07T17:53:26.811Z cpu12:2098046)WARNING: iscsi_vmk: iscsivmk_StartConnection:880: vmhba65:CH:0 T:1 CN:0: iSCSI connection is being marked "ONLINE"

2019-05-07T17:53:26.811Z cpu12:2098046)WARNING: iscsi_vmk: iscsivmk_StartConnection:881: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:all-esxi TPGT: 1 TSIH: 0]

2019-05-07T17:53:26.811Z cpu12:2098046)WARNING: iscsi_vmk: iscsivmk_StartConnection:882: Conn [CID: 0 L: 10.39.130.35:31611 R: 10.39.130.222:3260]

2019-05-07T17:53:27.117Z cpu3:2097505)ScsiDevice: 6000: Setting Device naa.6589cfc0000002975052862c22403cca state back to 0x2

2019-05-07T17:53:27.117Z cpu3:2097505)LVM: 5748: Received APD EventType: APD_EXIT (4) for device <naa.6589cfc0000002975052862c22403cca:3> (gen 22)

2019-05-07T17:53:27.117Z cpu3:2097505)LVM: 5356: Handling APD EventType: APD_EXIT (4) for device <naa.6589cfc0000002975052862c22403cca:3> (unlocked, gen 22, cur apd state APD_START, cur dev state 1)

2019-05-07T17:53:27.117Z cpu3:2097505)ScsiDevice: 6021: Device naa.6589cfc0000002975052862c22403cca is Out of APD; token num:1

2019-05-07T17:53:27.117Z cpu0:2097255)LVM: 5356: Handling APD EventType: APD_EXIT (4) for device <naa.6589cfc0000002975052862c22403cca:3> (locked, gen 22, cur apd state APD_START, cur dev state 1)

2019-05-07T17:53:27.117Z cpu15:2098023)NMP: nmpCompleteRetryForPath:327: Retry world recovered device "naa.6589cfc0000002975052862c22403cca"

2019-05-07T17:53:27.347Z cpu8:2098161)HBX: 312: 'SAN-PRODUCTION': HB at offset 3407872 - Reclaimed heartbeat [Timeout]:

2019-05-07T17:53:27.347Z cpu8:2098161)  [HB state abcdef02 offset 3407872 gen 37 stampUS 24833007489 uuid 5cd16571-5dd95f1e-0244-782bcb396c1f jrnl <FB 4> drv 24.82 lockImpl 4 ip 172.16.0.35]

2019-05-07T17:53:28.113Z cpu0:2098823)SunRPC: 1099: Destroying world 0x2252a2

2019-05-07T17:53:33.286Z cpu0:2097255)HBX: 6287: APD EventType: APD_EXIT (4) for vol 'SAN-PRODUCTION'

2019-05-07T17:53:33.286Z cpu0:2097255)StorageApdHandler: 1315: APD exit for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:53:33.286Z cpu0:2097255)HBX: 5509: Completed APD/PDL event processing on vol 'SAN-PRODUCTION'.

2019-05-07T17:53:33.286Z cpu0:2097255)LVM: 5487: Device <naa.6589cfc000000f7547481ec53d72549e:1> APD handler with closeOnAPDExit - attempting close.

2019-05-07T17:53:33.286Z cpu2:2097490)StorageApdHandler: 507: APD exit event for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:53:33.286Z cpu2:2097490)StorageApdHandlerEv: 117: Device or filesystem with identifier [naa.6589cfc0000002975052862c22403cca] has exited the All Paths Down state.

2019-05-07T17:54:04.114Z cpu2:2097835)NMP: nmp_ResetDeviceLogThrottling:3575: last error status from device naa.6589cfc0000002975052862c22403cca repeated 5 times

2019-05-07T17:54:08.519Z cpu15:2097918)ScsiDeviceIO: 3068: Cmd(0x45a240703b40) 0x1a, CmdSN 0x1819 from world 0 to dev "naa.6782bcb05bcabf00ff0000380397ca71" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2019-05-07T17:54:30.115Z cpu2:2098823)SunRPC: 1099: Destroying world 0x2253c2

2019-05-07T17:54:34.115Z cpu7:2097507)ScsiVmas: 804: Inquiry for VPD page 00 to device naa.6589cfc000000f7547481ec53d72549e failed with error No connection

2019-05-07T17:55:31.118Z cpu11:2098823)SunRPC: 1099: Destroying world 0x225593

2019-05-07T17:56:32.120Z cpu2:2098823)SunRPC: 1099: Destroying world 0x2256a8

2019-05-07T17:56:33.387Z cpu16:2098820)WARNING: LSOM: LSOMVsiGetVirstoInstanceStats:800: Throttled: Attempt to get Virsto stats on unsupported disk52256481-aa44-4783-d2f0-625461b41dfe

2019-05-07T17:56:41.760Z cpu20:2098023)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:517: vmhba65:CH:0 T:1 CN:0: Failed to receive data: Connection closed by peer

2019-05-07T17:56:41.760Z cpu20:2098023)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:525: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:all-esxi TPGT: 1 TSIH: 0]

2019-05-07T17:56:41.760Z cpu20:2098023)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:526: Conn [CID: 0 L: 10.39.130.35:31611 R: 10.39.130.222:3260]

2019-05-07T17:56:41.760Z cpu20:2098023)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1235: vmhba65:CH:0 T:1 CN:0: Connection rx notifying failure: Failed to Receive. State=Online

2019-05-07T17:56:41.760Z cpu20:2098023)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1236: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:all-esxi TPGT: 1 TSIH: 0]

2019-05-07T17:56:41.760Z cpu20:2098023)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1237: Conn [CID: 0 L: 10.39.130.35:31611 R: 10.39.130.222:3260]

2019-05-07T17:56:41.760Z cpu16:2097688)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba65:CH:0 T:1 CN:0: iSCSI connection is being marked "OFFLINE" (Event:6)

2019-05-07T17:56:41.760Z cpu16:2097688)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:all-esxi TPGT: 1 TSIH: 0]

2019-05-07T17:56:41.760Z cpu16:2097688)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 10.39.130.35:31611 R: 10.39.130.222:3260]

2019-05-07T17:56:41.766Z cpu0:2097195)NMP: nmp_ThrottleLogForDevice:3781: Cmd 0x28 (0x459a40d3e9c0, 2100138) to dev "naa.6589cfc0000002975052862c22403cca" on path "vmhba65:C0:T1:L70" Failed: H:0x2 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

2019-05-07T17:56:41.766Z cpu0:2097195)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6589cfc0000002975052862c22403cca" state in doubt; requested fast path state update...

2019-05-07T17:56:41.766Z cpu0:2097195)ScsiDeviceIO: 3047: Cmd(0x459a40d3e9c0) 0x28, CmdSN 0x6d from world 2100138 to dev "naa.6589cfc0000002975052862c22403cca" failed H:0x2 D:0x0 P:0x0 Invalid sense data: 0x0 0x20 0x44.

2019-05-07T17:56:41.908Z cpu0:2097195)NMP: nmp_ThrottleLogForDevice:3710: last error status from device naa.6589cfc0000002975052862c22403cca repeated 10 times

2019-05-07T17:56:42.049Z cpu0:2097689)ScsiDeviceIO: 3047: Cmd(0x459a40cbc880) 0x89, CmdSN 0x6e from world 2097185 to dev "naa.6589cfc0000002975052862c22403cca" failed H:0x2 D:0x0 P:0x0 Invalid sense data: 0x40 0x80 0x41.

2019-05-07T17:56:42.119Z cpu4:2097505)LVM: 5748: Received APD EventType: APD_START (3) for device <naa.6589cfc0000002975052862c22403cca:3> (gen 23)

2019-05-07T17:56:42.119Z cpu4:2097505)LVM: 5356: Handling APD EventType: APD_START (3) for device <naa.6589cfc0000002975052862c22403cca:3> (unlocked, gen 23, cur apd state UNKNOWN, cur dev state 1)

2019-05-07T17:56:42.119Z cpu4:2097505)HBX: 6287: APD EventType: APD_START (3) for vol 'SAN-PRODUCTION'

2019-05-07T17:56:42.119Z cpu4:2097505)ScsiDevice: 5978: Device state of naa.6589cfc0000002975052862c22403cca set to APD_START; token num:1

2019-05-07T17:56:42.119Z cpu4:2097505)StorageApdHandler: 1203: APD start for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:56:42.119Z cpu1:2097490)StorageApdHandler: 419: APD start event for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:56:42.119Z cpu1:2097490)StorageApdHandlerEv: 110: Device or filesystem with identifier [naa.6589cfc0000002975052862c22403cca] has entered the All Paths Down state.

2019-05-07T17:56:42.121Z cpu0:2101764)WARNING: NMP: nmp_IssueCommandToDevice:5082: I/O could not be issued to device "naa.6589cfc0000002975052862c22403cca" due to Not found

2019-05-07T17:56:42.121Z cpu0:2101764)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.6589cfc0000002975052862c22403cca": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.

2019-05-07T17:56:42.121Z cpu0:2101764)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6589cfc0000002975052862c22403cca" is blocked. Not starting I/O from device.

2019-05-07T17:56:43.119Z cpu5:2097836)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - issuing command 0x459a40cbc880

2019-05-07T17:56:43.119Z cpu5:2097836)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - failed to issue command due to Not found (APD), try again...

2019-05-07T17:56:43.119Z cpu5:2097836)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6589cfc0000002975052862c22403cca": awaiting fast path state update...

2019-05-07T17:56:44.119Z cpu4:2097836)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - issuing command 0x459a40cbc880

2019-05-07T17:56:44.119Z cpu4:2097836)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - failed to issue command due to Not found (APD), try again...

2019-05-07T17:56:44.119Z cpu4:2097836)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6589cfc0000002975052862c22403cca": awaiting fast path state update...

2019-05-07T17:56:44.667Z cpu22:2098046)iscsi_vmk: iscsivmk_ConnNetRegister:2170: socket 0x430bd554eaa0 network resource pool netsched.pools.persist.iscsi associated

2019-05-07T17:56:44.667Z cpu22:2098046)iscsi_vmk: iscsivmk_ConnNetRegister:2198: socket 0x430bd554eaa0 network tracker id 45045162 tracker.iSCSI.10.39.130.222 associated

2019-05-07T17:56:44.921Z cpu22:2098046)WARNING: iscsi_vmk: iscsivmk_StartConnection:880: vmhba65:CH:0 T:1 CN:0: iSCSI connection is being marked "ONLINE"

2019-05-07T17:56:44.921Z cpu22:2098046)WARNING: iscsi_vmk: iscsivmk_StartConnection:881: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:all-esxi TPGT: 1 TSIH: 0]

2019-05-07T17:56:44.921Z cpu22:2098046)WARNING: iscsi_vmk: iscsivmk_StartConnection:882: Conn [CID: 0 L: 10.39.130.35:49504 R: 10.39.130.222:3260]

2019-05-07T17:56:45.122Z cpu9:2097505)ScsiDevice: 6000: Setting Device naa.6589cfc0000002975052862c22403cca state back to 0x2

2019-05-07T17:56:45.122Z cpu9:2097505)LVM: 5748: Received APD EventType: APD_EXIT (4) for device <naa.6589cfc0000002975052862c22403cca:3> (gen 24)

2019-05-07T17:56:45.122Z cpu9:2097505)LVM: 5356: Handling APD EventType: APD_EXIT (4) for device <naa.6589cfc0000002975052862c22403cca:3> (unlocked, gen 24, cur apd state APD_START, cur dev state 1)

2019-05-07T17:56:45.122Z cpu9:2097505)ScsiDevice: 6021: Device naa.6589cfc0000002975052862c22403cca is Out of APD; token num:1

2019-05-07T17:56:45.122Z cpu0:2097255)LVM: 5356: Handling APD EventType: APD_EXIT (4) for device <naa.6589cfc0000002975052862c22403cca:3> (locked, gen 24, cur apd state APD_START, cur dev state 1)

2019-05-07T17:56:45.123Z cpu16:2098023)NMP: nmpCompleteRetryForPath:327: Retry world recovered device "naa.6589cfc0000002975052862c22403cca"

2019-05-07T17:56:51.630Z cpu0:2097255)HBX: 6287: APD EventType: APD_EXIT (4) for vol 'SAN-PRODUCTION'

2019-05-07T17:56:51.630Z cpu0:2097255)StorageApdHandler: 1315: APD exit for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:56:51.630Z cpu0:2097255)HBX: 5509: Completed APD/PDL event processing on vol 'SAN-PRODUCTION'.

2019-05-07T17:56:51.630Z cpu0:2097255)LVM: 5487: Device <naa.6589cfc000000f7547481ec53d72549e:1> APD handler with closeOnAPDExit - attempting close.

2019-05-07T17:56:51.630Z cpu4:2097490)StorageApdHandler: 507: APD exit event for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:56:51.630Z cpu4:2097490)StorageApdHandlerEv: 117: Device or filesystem with identifier [naa.6589cfc0000002975052862c22403cca] has exited the All Paths Down state.

2019-05-07T17:57:09.724Z cpu20:2250615)HBX: 3033: 'SAN-PRODUCTION': HB at offset 3407872 - Waiting for timed out HB:

2019-05-07T17:57:09.724Z cpu20:2250615)  [HB state abcdef02 offset 3407872 gen 41 stampUS 25045933439 uuid 5cd16571-5dd95f1e-0244-782bcb396c1f jrnl <FB 8388614> drv 24.82 lockImpl 4 ip 172.16.0.35]

2019-05-07T17:57:18.052Z cpu14:2097547)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue:633: vmhba65:CH:0 T:1 L:70 : Task mgmt "Abort Task" with itt=0xe98 (refITT=0xe97) timed out.

2019-05-07T17:57:19.727Z cpu14:2250615)HBX: 3033: 'SAN-PRODUCTION': HB at offset 3407872 - Waiting for timed out HB:

2019-05-07T17:57:19.727Z cpu14:2250615)  [HB state abcdef02 offset 3407872 gen 41 stampUS 25045933439 uuid 5cd16571-5dd95f1e-0244-782bcb396c1f jrnl <FB 8388614> drv 24.82 lockImpl 4 ip 172.16.0.35]

2019-05-07T17:57:28.054Z cpu14:2097547)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue:633: vmhba65:CH:0 T:1 L:70 : Task mgmt "Abort Task" with itt=0xe99 (refITT=0xe97) timed out.

2019-05-07T17:57:29.731Z cpu13:2250615)HBX: 3033: 'SAN-PRODUCTION': HB at offset 3407872 - Waiting for timed out HB:

2019-05-07T17:57:29.731Z cpu13:2250615)  [HB state abcdef02 offset 3407872 gen 41 stampUS 25045933439 uuid 5cd16571-5dd95f1e-0244-782bcb396c1f jrnl <FB 8388614> drv 24.82 lockImpl 4 ip 172.16.0.35]

2019-05-07T17:57:33.348Z cpu12:2098046)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba65:CH:0 T:1 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)

2019-05-07T17:57:33.348Z cpu12:2098046)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:all-esxi TPGT: 1 TSIH: 0]

2019-05-07T17:57:33.348Z cpu12:2098046)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 10.39.130.35:49504 R: 10.39.130.222:3260]

2019-05-07T17:57:33.352Z cpu12:2097207)NMP: nmp_ThrottleLogForDevice:3710: last error status from device naa.6589cfc0000002975052862c22403cca repeated 20 times

2019-05-07T17:57:33.352Z cpu12:2097207)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6589cfc0000002975052862c22403cca" state in doubt; requested fast path state update...

2019-05-07T17:57:33.352Z cpu12:2097207)ScsiDeviceIO: 3047: Cmd(0x459b98daa240) 0x28, CmdSN 0x7f from world 2100138 to dev "naa.6589cfc0000002975052862c22403cca" failed H:0x2 D:0x0 P:0x0 Invalid sense data: 0x0 0x1a 0x45.

2019-05-07T17:57:33.352Z cpu14:2097547)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue:633: vmhba65:CH:0 T:1 L:70 : Task mgmt "Abort Task" with itt=0xe9b (refITT=0xe97) timed out.

2019-05-07T17:57:33.352Z cpu14:2097547)NMP: nmp_ThrottleLogForDevice:3781: Cmd 0x89 (0x459a40deb580, 2097185) to dev "naa.6589cfc0000002975052862c22403cca" on path "vmhba65:C0:T1:L70" Failed: H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

2019-05-07T17:57:33.352Z cpu14:2097547)ScsiDeviceIO: 3082: Cmd(0x459a40deb580) 0x89, CmdSN 0x80 from world 2097185 to dev "naa.6589cfc0000002975052862c22403cca" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x78 0x35 0x20.

2019-05-07T17:57:33.352Z cpu8:2097697)NMP: nmp_ThrottleLogForDevice:3781: Cmd 0x28 (0x459b98d0b3c0, 0) to dev "naa.6589cfc0000002975052862c22403cca" on path "vmhba65:C0:T1:L70" Failed: H:0x2 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL

2019-05-07T17:57:33.352Z cpu8:2097697)ScsiDeviceIO: 3047: Cmd(0x459b98d0b3c0) 0x28, CmdSN 0x81 from world 0 to dev "naa.6589cfc0000002975052862c22403cca" failed H:0x2 D:0x0 P:0x0 Invalid sense data: 0x30 0x20 0x50.

2019-05-07T17:57:33.498Z cpu8:2097697)NMP: nmp_ThrottleLogForDevice:3710: last error status from device naa.6589cfc0000002975052862c22403cca repeated 10 times

2019-05-07T17:57:33.786Z cpu8:2097697)NMP: nmp_ThrottleLogForDevice:3710: last error status from device naa.6589cfc0000002975052862c22403cca repeated 20 times

2019-05-07T17:57:34.121Z cpu2:2097505)LVM: 5748: Received APD EventType: APD_START (3) for device <naa.6589cfc0000002975052862c22403cca:3> (gen 25)

2019-05-07T17:57:34.121Z cpu2:2097505)LVM: 5356: Handling APD EventType: APD_START (3) for device <naa.6589cfc0000002975052862c22403cca:3> (unlocked, gen 25, cur apd state UNKNOWN, cur dev state 1)

2019-05-07T17:57:34.121Z cpu2:2097505)HBX: 6287: APD EventType: APD_START (3) for vol 'SAN-PRODUCTION'

2019-05-07T17:57:34.121Z cpu2:2097505)ScsiDevice: 5978: Device state of naa.6589cfc0000002975052862c22403cca set to APD_START; token num:1

2019-05-07T17:57:34.121Z cpu2:2097505)StorageApdHandler: 1203: APD start for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:57:34.121Z cpu4:2097490)StorageApdHandler: 419: APD start event for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:57:34.121Z cpu4:2097490)StorageApdHandlerEv: 110: Device or filesystem with identifier [naa.6589cfc0000002975052862c22403cca] has entered the All Paths Down state.

2019-05-07T17:57:34.121Z cpu8:2098823)SunRPC: 1099: Destroying world 0x22586e

2019-05-07T17:57:34.133Z cpu8:2097900)WARNING: NMP: nmp_IssueCommandToDevice:5082: I/O could not be issued to device "naa.6589cfc0000002975052862c22403cca" due to Not found

2019-05-07T17:57:34.133Z cpu8:2097900)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.6589cfc0000002975052862c22403cca": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.

2019-05-07T17:57:34.133Z cpu8:2097900)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6589cfc0000002975052862c22403cca" is blocked. Not starting I/O from device.

2019-05-07T17:57:35.120Z cpu4:2097836)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - issuing command 0x459b98d0b3c0

2019-05-07T17:57:35.120Z cpu4:2097836)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - failed to issue command due to Not found (APD), try again...

2019-05-07T17:57:35.120Z cpu4:2097836)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6589cfc0000002975052862c22403cca": awaiting fast path state update...

2019-05-07T17:57:36.120Z cpu2:2097836)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - issuing command 0x459b98d0b3c0

2019-05-07T17:57:36.121Z cpu2:2097836)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - failed to issue command due to Not found (APD), try again...

2019-05-07T17:57:36.121Z cpu2:2097836)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6589cfc0000002975052862c22403cca": awaiting fast path state update...

2019-05-07T17:57:36.121Z cpu13:2098046)iscsi_vmk: iscsivmk_ConnNetRegister:2170: socket 0x430bd563c320 network resource pool netsched.pools.persist.iscsi associated

2019-05-07T17:57:36.121Z cpu13:2098046)iscsi_vmk: iscsivmk_ConnNetRegister:2198: socket 0x430bd563c320 network tracker id 45045162 tracker.iSCSI.10.39.130.222 associated

2019-05-07T17:57:36.375Z cpu13:2098046)WARNING: iscsi_vmk: iscsivmk_StartConnection:880: vmhba65:CH:0 T:1 CN:0: iSCSI connection is being marked "ONLINE"

2019-05-07T17:57:36.375Z cpu13:2098046)WARNING: iscsi_vmk: iscsivmk_StartConnection:881: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:all-esxi TPGT: 1 TSIH: 0]

2019-05-07T17:57:36.375Z cpu13:2098046)WARNING: iscsi_vmk: iscsivmk_StartConnection:882: Conn [CID: 0 L: 10.39.130.35:15542 R: 10.39.130.222:3260]

2019-05-07T17:57:37.124Z cpu5:2097505)ScsiDevice: 6000: Setting Device naa.6589cfc0000002975052862c22403cca state back to 0x2

2019-05-07T17:57:37.124Z cpu5:2097505)LVM: 5748: Received APD EventType: APD_EXIT (4) for device <naa.6589cfc0000002975052862c22403cca:3> (gen 26)

2019-05-07T17:57:37.124Z cpu5:2097505)LVM: 5356: Handling APD EventType: APD_EXIT (4) for device <naa.6589cfc0000002975052862c22403cca:3> (unlocked, gen 26, cur apd state APD_START, cur dev state 1)

2019-05-07T17:57:37.124Z cpu5:2097505)ScsiDevice: 6021: Device naa.6589cfc0000002975052862c22403cca is Out of APD; token num:1

2019-05-07T17:57:37.124Z cpu0:2097255)LVM: 5356: Handling APD EventType: APD_EXIT (4) for device <naa.6589cfc0000002975052862c22403cca:3> (locked, gen 26, cur apd state APD_START, cur dev state 1)

2019-05-07T17:57:37.125Z cpu12:2098023)NMP: nmpCompleteRetryForPath:327: Retry world recovered device "naa.6589cfc0000002975052862c22403cca"

2019-05-07T17:57:37.343Z cpu1:2098161)HBX: 312: 'SAN-PRODUCTION': HB at offset 3407872 - Reclaimed heartbeat [Timeout]:

2019-05-07T17:57:37.343Z cpu1:2098161)  [HB state abcdef02 offset 3407872 gen 41 stampUS 25083008142 uuid 5cd16571-5dd95f1e-0244-782bcb396c1f jrnl <FB 8388614> drv 24.82 lockImpl 4 ip 172.16.0.35]

2019-05-07T17:57:42.573Z cpu7:2097255)HBX: 6287: APD EventType: APD_EXIT (4) for vol 'SAN-PRODUCTION'

2019-05-07T17:57:42.573Z cpu7:2097255)StorageApdHandler: 1315: APD exit for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:57:42.573Z cpu7:2097255)HBX: 5509: Completed APD/PDL event processing on vol 'SAN-PRODUCTION'.

2019-05-07T17:57:42.573Z cpu7:2097255)LVM: 5487: Device <naa.6589cfc000000f7547481ec53d72549e:1> APD handler with closeOnAPDExit - attempting close.

2019-05-07T17:57:42.573Z cpu4:2097490)StorageApdHandler: 507: APD exit event for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:57:42.573Z cpu4:2097490)StorageApdHandlerEv: 117: Device or filesystem with identifier [naa.6589cfc0000002975052862c22403cca] has exited the All Paths Down state.

2019-05-07T17:57:55.833Z cpu22:2098023)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:517: vmhba65:CH:0 T:1 CN:0: Failed to receive data: Connection closed by peer

2019-05-07T17:57:55.833Z cpu22:2098023)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:525: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:all-esxi TPGT: 1 TSIH: 0]

2019-05-07T17:57:55.833Z cpu22:2098023)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:526: Conn [CID: 0 L: 10.39.130.35:15542 R: 10.39.130.222:3260]

2019-05-07T17:57:55.833Z cpu22:2098023)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1235: vmhba65:CH:0 T:1 CN:0: Connection rx notifying failure: Failed to Receive. State=Online

2019-05-07T17:57:55.833Z cpu22:2098023)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1236: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:all-esxi TPGT: 1 TSIH: 0]

2019-05-07T17:57:55.833Z cpu22:2098023)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1237: Conn [CID: 0 L: 10.39.130.35:15542 R: 10.39.130.222:3260]

2019-05-07T17:57:55.833Z cpu22:2097688)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba65:CH:0 T:1 CN:0: iSCSI connection is being marked "OFFLINE" (Event:6)

2019-05-07T17:57:55.833Z cpu22:2097688)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:all-esxi TPGT: 1 TSIH: 0]

2019-05-07T17:57:55.833Z cpu22:2097688)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 10.39.130.35:15542 R: 10.39.130.222:3260]

2019-05-07T17:57:55.838Z cpu2:2097197)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6589cfc0000002975052862c22403cca" state in doubt; requested fast path state update...

2019-05-07T17:57:55.838Z cpu2:2097197)ScsiDeviceIO: 3047: Cmd(0x459b98dd7d80) 0x28, CmdSN 0x95 from world 2100138 to dev "naa.6589cfc0000002975052862c22403cca" failed H:0x2 D:0x0 P:0x0 Invalid sense data: 0x0 0x1a 0x45.

2019-05-07T17:57:56.065Z cpu2:2097197)NMP: nmp_ThrottleLogForDevice:3710: last error status from device naa.6589cfc0000002975052862c22403cca repeated 40 times

2019-05-07T17:57:56.121Z cpu2:2097197)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6589cfc0000002975052862c22403cca" state in doubt; requested fast path state update...

2019-05-07T17:57:56.121Z cpu5:2097505)LVM: 5748: Received APD EventType: APD_START (3) for device <naa.6589cfc0000002975052862c22403cca:3> (gen 27)

2019-05-07T17:57:56.121Z cpu5:2097505)LVM: 5356: Handling APD EventType: APD_START (3) for device <naa.6589cfc0000002975052862c22403cca:3> (unlocked, gen 27, cur apd state UNKNOWN, cur dev state 1)

2019-05-07T17:57:56.121Z cpu5:2097505)HBX: 6287: APD EventType: APD_START (3) for vol 'SAN-PRODUCTION'

2019-05-07T17:57:56.121Z cpu5:2097505)ScsiDevice: 5978: Device state of naa.6589cfc0000002975052862c22403cca set to APD_START; token num:1

2019-05-07T17:57:56.121Z cpu5:2097505)StorageApdHandler: 1203: APD start for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:57:56.121Z cpu6:2097490)StorageApdHandler: 419: APD start event for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:57:56.121Z cpu6:2097490)StorageApdHandlerEv: 110: Device or filesystem with identifier [naa.6589cfc0000002975052862c22403cca] has entered the All Paths Down state.

2019-05-07T17:57:56.135Z cpu2:2099067)WARNING: NMP: nmp_IssueCommandToDevice:5082: I/O could not be issued to device "naa.6589cfc0000002975052862c22403cca" due to Not found

2019-05-07T17:57:56.135Z cpu2:2099067)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.6589cfc0000002975052862c22403cca": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.

2019-05-07T17:57:56.135Z cpu2:2099067)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6589cfc0000002975052862c22403cca" is blocked. Not starting I/O from device.

2019-05-07T17:57:57.122Z cpu5:2097836)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - issuing command 0x459b98dd7d80

2019-05-07T17:57:57.122Z cpu5:2097836)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - failed to issue command due to Not found (APD), try again...

2019-05-07T17:57:57.122Z cpu5:2097836)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6589cfc0000002975052862c22403cca": awaiting fast path state update...

2019-05-07T17:57:58.123Z cpu5:2097836)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - issuing command 0x459b98dd7d80

2019-05-07T17:57:58.123Z cpu5:2097836)WARNING: NMP: nmpDeviceAttemptFailover:715: Retry world failover device "naa.6589cfc0000002975052862c22403cca" - failed to issue command due to Not found (APD), try again...

2019-05-07T17:57:58.123Z cpu5:2097836)WARNING: NMP: nmpDeviceAttemptFailover:765: Logical device "naa.6589cfc0000002975052862c22403cca": awaiting fast path state update...

2019-05-07T17:57:58.851Z cpu12:2098046)iscsi_vmk: iscsivmk_ConnNetRegister:2170: socket 0x430bd5399020 network resource pool netsched.pools.persist.iscsi associated

2019-05-07T17:57:58.851Z cpu12:2098046)iscsi_vmk: iscsivmk_ConnNetRegister:2198: socket 0x430bd5399020 network tracker id 45045162 tracker.iSCSI.10.39.130.222 associated

2019-05-07T17:57:59.104Z cpu12:2098046)WARNING: iscsi_vmk: iscsivmk_StartConnection:880: vmhba65:CH:0 T:1 CN:0: iSCSI connection is being marked "ONLINE"

2019-05-07T17:57:59.104Z cpu12:2098046)WARNING: iscsi_vmk: iscsivmk_StartConnection:881: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:all-esxi TPGT: 1 TSIH: 0]

2019-05-07T17:57:59.104Z cpu12:2098046)WARNING: iscsi_vmk: iscsivmk_StartConnection:882: Conn [CID: 0 L: 10.39.130.35:47878 R: 10.39.130.222:3260]

2019-05-07T17:57:59.124Z cpu0:2097505)ScsiDevice: 6000: Setting Device naa.6589cfc0000002975052862c22403cca state back to 0x2

2019-05-07T17:57:59.124Z cpu0:2097505)LVM: 5748: Received APD EventType: APD_EXIT (4) for device <naa.6589cfc0000002975052862c22403cca:3> (gen 28)

2019-05-07T17:57:59.124Z cpu0:2097505)LVM: 5356: Handling APD EventType: APD_EXIT (4) for device <naa.6589cfc0000002975052862c22403cca:3> (unlocked, gen 28, cur apd state APD_START, cur dev state 1)

2019-05-07T17:57:59.124Z cpu0:2097505)ScsiDevice: 6021: Device naa.6589cfc0000002975052862c22403cca is Out of APD; token num:1

2019-05-07T17:57:59.124Z cpu7:2097255)LVM: 5356: Handling APD EventType: APD_EXIT (4) for device <naa.6589cfc0000002975052862c22403cca:3> (locked, gen 28, cur apd state APD_START, cur dev state 1)

2019-05-07T17:58:02.946Z cpu17:2097212)NMP: nmpCompleteRetryForPath:327: Retry world recovered device "naa.6589cfc0000002975052862c22403cca"

2019-05-07T17:58:09.744Z cpu14:2250615)HBX: 3033: 'SAN-PRODUCTION': HB at offset 3407872 - Waiting for timed out HB:

2019-05-07T17:58:09.744Z cpu14:2250615)  [HB state abcdef02 offset 3407872 gen 41 stampUS 25108932940 uuid 5cd16571-5dd95f1e-0244-782bcb396c1f jrnl <FB 8388614> drv 24.82 lockImpl 4 ip 172.16.0.35]

2019-05-07T17:58:11.157Z cpu7:2097255)HBX: 6287: APD EventType: APD_EXIT (4) for vol 'SAN-PRODUCTION'

2019-05-07T17:58:11.157Z cpu7:2097255)LVM: 5487: Device <naa.6589cfc000000f7547481ec53d72549e:1> APD handler with closeOnAPDExit - attempting close.

2019-05-07T17:58:11.157Z cpu8:2098234)ScsiDeviceIO: 4536: Command 0x89 (CmdSN 0xa3, World 2097185) to device naa.6589cfc0000002975052862c22403cca timed out: expiry time occurs 2105ms in the past

2019-05-07T17:58:11.157Z cpu8:2098234)WARNING: ScsiVmas: 3021: Failed to issue command on device "naa.6589cfc0000002975052862c22403cca": Timeout

2019-05-07T17:58:11.476Z cpu1:2098161)HBX: 312: 'SAN-PRODUCTION': HB at offset 3407872 - Reclaimed heartbeat [Timeout]:

2019-05-07T17:58:11.476Z cpu1:2098161)  [HB state abcdef02 offset 3407872 gen 41 stampUS 25117052859 uuid 5cd16571-5dd95f1e-0244-782bcb396c1f jrnl <FB 8388614> drv 24.82 lockImpl 4 ip 172.16.0.35]

2019-05-07T17:58:11.476Z cpu1:2098161)StorageApdHandler: 1315: APD exit for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:58:11.477Z cpu1:2098161)HBX: 5509: Completed APD/PDL event processing on vol 'SAN-PRODUCTION'.

2019-05-07T17:58:11.477Z cpu6:2097490)StorageApdHandler: 507: APD exit event for 0x4304183eeea0 [naa.6589cfc0000002975052862c22403cca]

2019-05-07T17:58:11.477Z cpu6:2097490)StorageApdHandlerEv: 117: Device or filesystem with identifier [naa.6589cfc0000002975052862c22403cca] has exited the All Paths Down state.

2019-05-07T17:58:34.122Z cpu10:2097835)NMP: nmp_ResetDeviceLogThrottling:3575: last error status from device naa.6589cfc0000002975052862c22403cca repeated 4 times

2019-05-07T17:58:36.123Z cpu8:2098823)SunRPC: 1099: Destroying world 0x225a09

[root@esxi1:~]

iqn.2005-10.org.freenas.ctl:all-esxi

Running  6.5.0 (Build 4887370)

Hey guys quite new to the server world, I wanted to turn it into a VM machine so I went to install VMware.  Seemed to installed fine, but it's not picking up the RAIDed hard drives.  Is there any way I can get these recognised and installed to work correctly?

Does anyone know how to add cpus to a sql server which hosts the vcenter database

Environment

vCenter server

SQL hosting the vCenter database

If I shut down the SQL server to add cpus - I lose connectivity to the vCenter console so I can only add CPU's via the local esxi console.

As this server is managed by the vCenter , what will happen if I add CPUS this way? Will this screw things up once the  vCenter comes back online

I don't have hot add cpu enabled on the server and cannot change this setting until switched off - thus the chicken and egg scenario above.

Thanks ahead

Arthur

I am trying to enable my windows 7 VM 64bit to be able to have virtualisation in the BIOS,

The scenario I have is that I have installed Oracle Virtual Box on the VM but it will not recognise any 64 bit operating options to install a vm

I have checked under the Windows 7 VM settings and it has options to change the CPU/ MMU Virtualization but what option I choose you can still not see it in the BIOS and Virtual Box still will see any 64 bit options

Any help would be greatly appreciated

Hi everyone,

So I recently installed ESXi Hypervisor 6.5 on a Dell EMC Server PowerEdge T440.

We then procured an nVidia Quadro P4000. Upon scouring the forums and the internet, I do not see anywhere that this GPU can be used in a shared-VM set up. The VIB drivers page for ESXi 6.5 say they are only applicable to GRID GPUs.

I was however able to set it in dedicated GPU mode by using PCI-passthrough. But this is the least favored situation. It would be best if I were able to share the GPU across multiple VMs. Is that not possible?

I have an Adaptec RAID 71605Q and 8 hdds: 4x 2TB in a raid5 array, and 4x 3TB in a raid10 array. (Two connections on the card, and two of those 4-into-1 cables; one cable per array, if that makes sense.)

I'm running a standalone esxi 6.0.0 Update 2 (Build 3620759) host (I run esxi on a usb stick).

I have (had!) 2 datastores; one per array.

About a year ago, 2 drives in the raid10 array failed...I've been doing this for 29 years and have never had 2 drives in an array fail. I know others have, but I never have. Guess it's my turn!

Just for grins, I bought 2 more 3TB drives and plugged them in. It wasn't readily apparent how to recover, so I literally set it aside until the other day.

So now, I have 3 of the 4 original drives plus a new drive plugged in, and have an online / degraded array.

In esxi web client, I don't see that datastore (I can still see and use the raid5 datastore). I'd like to recover it if I can; I've spent a few hours combing the interwebs and still have had no luck, so maybe someone here can help!

Here are some commands I've run via putty, and the results:

esxcli storage vmfs snapshot list:

   Volume Name:

   VMFS UUID:

   Can mount: false

   Reason for un-mountability: some extents missing

   Can resignature: false

   Reason for non-resignaturability: some extents missing

   Unresolved Extent Count: 1

(It also shows my other datastore with no issues.)

esxcli storage core device list:

eui.2e73998e00d00000

   Display Name: Local ASR7160 Disk (eui.2e73998e00d00000)

   Has Settable Display Name: true

   Size: 5611518

   Device Type: Direct-Access

   Multipath Plugin: NMP

   Devfs Path: /vmfs/devices/disks/eui.2e73998e00d00000

   Vendor: ASR7160

   Model: datastore1

   Revision: V1.0

   SCSI Level: 2

   Is Pseudo: false

   Status: on

   Is RDM Capable: false

   Is Local: true

   Is Removable: false

   Is SSD: false

   Is VVOL PE: false

   Is Offline: false

   Is Perennially Reserved: false

   Queue Full Sample Size: 0

   Queue Full Threshold: 0

   Thin Provisioning Status: unknown

   Attached Filters:

   VAAI Status: unsupported

   Other UIDs: vml.01000000003865393937333265646174617374

   Is Shared Clusterwide: false

   Is Local SAS Device: false

   Is SAS: false

   Is USB: false

   Is Boot USB Device: false

   Is Boot Device: false

   Device Max Queue Depth: 256

   No of outstanding IOs with competing worlds: 32

   Drive Type: unknown

   RAID Level: unknown

   Number of Physical Drives: unknown

   Protection Enabled: false

   PI Activated: false

   PI Type: 0

   PI Protection Mask: NO PROTECTION

   Supported Guard Types: NO GUARD SUPPORT

   DIX Enabled: false

   DIX Guard Type: NO GUARD SUPPORT

   Emulated DIX/DIF Enabled: false

esxcfg-volume -l

UnresolvedVmfsVolume: Unable to find device in unresolved list:0#eui.2e73998e00d00000:1VMFS UUID/label: n.a./n.a.

Can mount: No (some extents missing)

Can resignature: No (some extents missing)

Extent name: eui.2e73998e00d00000:1     range: 962072674048 - 962072674303 (MB)

esxcli storage vmfs snapshot extent list

Volume Name  VMFS UUID  Extent Number  Device Name           Partition         Start           End

-----------  ---------  -------------  --------------------  ---------  ------------  ------------

                                    0  eui.2e73998e00d00000          1  962072674048  962072674303

partedUtil getptbl /vmfs/devices/disks/eui.2e73998e00d00000

gpt

715368 255 63 11492388864

1 128 11492388824 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

partedUtil getptbl /vmfs/devices/disks/eui.2e73998e00d00000:1

unknown

715368 255 63 11492388697

esxcli storage core device partition list

Device                Partition  Start Sector   End Sector  Type           Size

--------------------  ---------  ------------  -----------  ----  -------------

mpx.vmhba32:C0:T0:L0          0             0     15826944     0     8103395328

mpx.vmhba32:C0:T0:L0          1            64         8192     0        4161536

mpx.vmhba32:C0:T0:L0          5          8224       520192     6      262127616

mpx.vmhba32:C0:T0:L0          6        520224      1032192     6      262127616

mpx.vmhba32:C0:T0:L0          7       1032224      1257472    fc      115326976

mpx.vmhba32:C0:T0:L0          8       1257504      1843200     6      299876352

mpx.vmhba32:C0:T0:L0          9       1843200      7086080    fc     2684354560

eui.2e73998e00d00000          0             0  11492388864     0  5884103098368

eui.2e73998e00d00000          1           128  11492388825    fb  5884103012864

eui.ae00911700d00000          0             0  11702087680     0  5991468892160

eui.ae00911700d00000          1           128  11702087641    fb  5991468806656

df -h

Filesystem   Size   Used Available Use% Mounted on

VMFS-5       5.4T   4.2T      1.3T  77% /vmfs/volumes/Datastore2-3Gbps

vfat       285.8M 202.6M     83.2M  71% /vmfs/volumes/5898d802-31df4d42-c887-001b214c3280

vfat       249.7M 168.7M     81.0M  68% /vmfs/volumes/80d144e6-f6071fe9-239e-81751ede2f2f

vfat       249.7M 168.7M     81.0M  68% /vmfs/volumes/cb65cbe6-73ef9335-efb9-ecd97551455f

How can I span a Virtual Machine across two physical servers?

Each server is a Poweredge R230 with CPU(4 cores, 8 threads), 64GB RAM, a dual 10Gb/s PCIe Ethernet adapter.

These two servers will be connected together by dual 10Gb/s Ethernet (20Gb/s aggregated throughput ethernet connection)

The virtualization Hypervisor used will only be one of the following:

1.  Oracle Virtualbox

2.  Microsoft Hyper V

3.  VMware vSphere ESXi

Thank you.

Hi , Im New here.

After load image from iso it was coming a black mirror ...Then on the display

Shutting down firmware service ...

Relocating modules and starting up the kernel.

What is the problem ?

We have requirements to change passwords periodically.  I understand that the root password, if ever lost, is not recoverable other than through re-installation of the hypervisor.  However, all I'm looking to do is change the password since we know it.  I can log in to the ESXi host, see the warning that it's being managed by the vCenter Server, and I can get to the user config and edit the 'root' account where it shows the password fields, but I'm wondering if it's that simple to change it since I had an issue a while back that was strange, and since I'm not sure if changing it here also impacts the SSH and DCUI access for the host.  Is it that simple?  (I believe I had changed it this way once and tried to log into the DCUI as root but it used an older password, which I still knew, so I'm skeptical)

Hello,

Running Esxi 6.7 and I've been having HD noise chatter (oscillating head movement) when the VM is performing no disk IO. This issue also exists on Esxi 6.0.

If I shut the VM down the noise chatter (rapid head movement) stops. Restart the VM in 10 to 30 minutes the noise chatter starts up again and doesn't stop until I shut the VM down again. The chatter goes in cycles, chatter for 5 seconds, then quite for 2, chatter for 5 seconds, then quite for 2, etc.

Anyone have any idea what is causing this unnecessary head movement (chatter) and how to stop it?

The VM is Opensuse 14.3

The drives are mounted as follows:

/dev/sdb1        /mnt/diskp      ext4    defaults 0 0

/dev/sdc1        /mnt/disk1      ext4    defaults 0 0

/dev/sdd1        /mnt/disk2      ext4    defaults 0 0

/dev/sde1        /mnt/media1     ext4    defaults 0 0

Using command line tool: dstat shows no significant read/write operations taking place.

Host Image:

• 6.7.0 Update 2 (Build 13006603) / ESXi-6.7.0-20190402001-standard (VMware, Inc.)

Guest Settings:

• Guest OS - Microsoft Windows 10 (64-bit) VBS Enabled
• Compatibility - ESXi 6.7 U2 virtual machine
• VMware Tools - Yes/Current
• USB Controller - USB 2.0

Our domain has Smart Card enforcement enabled for all workstations through group policy. Deploying our Windows 10 enterprise image to local VMware Workstation Pro (licensed), I can connect Shared Dell Smart Card Reader Keyboard to the VM to be able to log in through the console. We have now purchased a machine to have as a stand-alone host for Hypervisor. Whether I use browser console, VMRC, or VMware Workstation Pro remotely connected to the server, I am unable to connect the Dell keyboard for smart card logon. Using RDP I am able to log in normally, but I would prefer having users log into the web console and just give them the VM name to use instead of having to fish the IP address each time. Would a remote connection to the host be preventing a shared keyboard (technical limitation), or is it a limitation of being not connected to a vCenter server? Is there an advanced setting I am missing that I have not been able to find any documentation on so far?