Wednesday, 25 January 2017

NETWORK TROUBLSHOOTING USING ESXCLI AND VSISH COMMAND


The objective of this document to cover troubleshooting network performance issues using ESXCLI and VSHISH commands

       ESXCLI

 You can use vCLI network commands to view network statistics and troubleshoot your networking setup. The nested hierarchy of commands allows you to drill down to potential trouble spots.

1. List all virtual machine networks on a host.

esxcli network vm list

The command returns for each virtual machine the World ID, name, number of ports, and networks, as in the following example.

World ID              Name                                       Num Ports             Networks
--------         -------------------------------------  ---------  ---------------------------
 646632      VM NAME 1                                            2            Wintel Network
3749030    VM NAME 2                                             1             Wintel Network
 6521771    VM NAME 3                                            1             Wintel Network
20714376   VM NAME 4                                            1             Wintel Network


2. List the ports for one of the VMs by specifying its World ID.

esxcli network vm port list -w 646632

The command returns port information, as in the following example.

   Port ID: 50331682
   vSwitch: vSwitch1
   Portgroup: Wintel Network
   DVPort ID:
   MAC Address: 00:50:56:91:24:27
   IP Address: 0.0.0.0
   Team Uplink: vmnic3
   Uplink Port ID: 50331652
   Active Filters:

   Port ID: 100663302
   vSwitch: vSwitch4
   Portgroup: ILO Network
   DVPort ID:
   MAC Address: 00:50:56:91:0a:f5
   IP Address: 0.0.0.0
   Team Uplink: vmnic6
   Uplink Port ID: 100663298
   Active Filters:

3. Retrieve the switch statistics for a port.

 esxcli network port stats get -p 50331682

The command returns detailed statistics, as in the following example. This will help you to find out a drop in network.

  Packet statistics for port 50331682
   Packets received: 1237624170
   Packets sent: 30179514
   Bytes received: 113310026172
   Bytes sent: 6341761418
   Broadcast packets received: 708707216
   Broadcast packets sent: 2037956
   Multicast packets received: 508976173
   Multicast packets sent: 9430448
   Unicast packets received: 19940781
   Unicast packets sent: 18711110
   Receive packets dropped: 1202750738
   Transmit packets dropped: 0


4. Retrieve the filter information for the port. The command returns detailed statistics, as in the following example.

esxcli network port filter stats get -p 50331682

Filter statistics for dvfilter-test:
Filter direction: Receive
Packets in: 202080
Packets out: 202080
Packets dropped: 0
Packets filtered: 0
Packets faulted: 0
Packets queued: 0
Packets injected: 0
Packet errors: 0

5. Retrieve complete statistics for a NIC.

esxcli network nic stats get -n vmnic3

   NIC statistics for vmnic3
   Packets received: 72994042944
   Packets sent: 52891670321
   Bytes received: 68186453587443
   Bytes sent: 50171818016986
   Receive packets dropped: 0
   Transmit packets dropped: 0
   Multicast packets received: 2844327960
   Broadcast packets received: 0
   Multicast packets sent: 0
   Broadcast packets sent: 0
   Total receive errors: 0
   Receive length errors: 0
   Receive over errors: 0
   Receive CRC errors: 0
   Receive frame errors: 0
   Receive FIFO errors: 0
   Receive missed errors: 719
   Total transmit errors: 0
   Transmit aborted errors: 0
   Transmit carrier errors: 0
   Transmit FIFO errors: 0
   Transmit heartbeat errors: 0
   Transmit window errors: 0

6.  Get a perVLAN packed breakdown on a NIC.

Let’s take a look at statistics on a per port group or more specifically on a per VLAN basis using the new “vlan” namespace. By default VLAN statistics is disabled and you need to enable it on each vmnic that you want to pull statistics for.

We will enable VLAN stats collection on vmnic3 by running the following command (replace with the vmnic of choice)


Enabled VLAN statistic on vmnic3


esxcli network nic vlan stats set -e true -n vmnic3


Get VLAN stats


esxcli network nic vlan stats get -n vmnic0

Above command returns the number of packets sent and received for the VLAN you specified.

VLAN 0
   Packets received: 32749862
   Packets sent: 24

VLAN 4
   Packets received: 14334203
   Packets sent: 0

VLAN 5
   Packets received: 3330475941
   Packets sent: 0

VLAN 8
   Packets received: 538680670
   Packets sent: 0

VLAN 130
   Packets received: 21116900071
   Packets sent: 13272199149

VLAN 519
   Packets received: 797082327
   Packets sent: 0

VLAN 526
   Packets received: 756335360
   Packets sent: 0

VLAN 527
   Packets received: 141830192
   Packets sent: 0

VLAN 528
   Packets received: 662613091
   Packets sent: 0

VLAN 536
   Packets received: 101178020
   Packets sent: 0


·        VSISH


VMkernel system information shell (known as vsish) is a great tool for troubleshooting network performance issue.  vsish runs in the ESXi shell and let you look at in advanced system parameters and performance information of the ESXi host and associated objects like the VMs.

Will look at the performance counters of a VM's vmxnet3 network adapter:  As a prerequisite we need to find out the name of the port group that the VM is attached to and its port ID. An easy way to get this information is to launch the esxtop utility and look at its network display (Press n to get there). Second option to get port id, you can try command 1 and 2 in ESXCLI section.


Locate the name of the VM in the USED-BY field and take a note of the port ID displayed in the PORT-ID column (in this case: 33554833)

Run vsish at ESXI CLI and look at this specific port: ->net/portsets/vSwitch0/ports/33554833/> (Note: - Target VM is mapped to vSwitch0)

Type cat status command



NoteCommand vsish is like using UNIX filesystem tree that is full of text files. You use cd to change to different folders or branches, ls to display nodes and sub-branches and cat to display the contents of nodes.

Cat status e.g. display some details about the port's configuration. As we know the existing port is using VMXNET3 adapter type, let’s use cat stats command within same PORT ID to check network statistics



In my case, the stats shows that the vNIC is not dropping a TX/RX packages. The issue comes in notice when TX/RX value seen at higher level and then you can dig further to understand the cause for VM performance issue.


Let's switch to the vNIC type specific sub-branch vmxnet3 (use cd vmxnet3 command) and look at the rxSummary node there:



If you see high value in running out of buffer field and 1st ring that means vmxnet3 driver is running out of buffers. In my case it is showing zero, so it looks good.

You can dig down more and check rxqueue if you see high buffer value.



It turns out that there is only one receive queue - that means RSS is not enabled on this adapter. RSS stands for Receive Side Scaling and allows having multiple receive queues for a network adapter which are handled by different CPUs.

We can also see that the ring sizes of the receive queue are 512, resp. 32 (which are the default values).

The ring sizes, the number of buffers and the RSS features of the vmxnet3 adapter are parameters that can be configured from inside the VM's guest OS (Windows in this case):



VMware article about RSS settings:

RSS for Windows server:
 RSS for Linux server:





5 comments:

  1. Very helpful artical about how to debug ESXi networking!

    ReplyDelete
  2. This was extremely helpful and gave me some experience with a toolbox I had yet to utilize. Thank you!

    ReplyDelete
  3. yes while im mailing this sandbox info my email got queued and i thought what does queued email means in gmail

    ReplyDelete
  4. As a networking student i too research a lot on computer networks and stuff and here is a good general on networks.you can read different types of network's in computer networks in general

    ReplyDelete