NETWORK TROUBLSHOOTING USING ESXCLI AND VSISH COMMAND

The objective of this document to cover troubleshooting network performance issues using ESXCLI and VSHISH commands

ESXCLI

You can use vCLI network commands to view network statistics and troubleshoot your networking setup. The nested hierarchy of commands allows you to drill down to potential trouble spots.

1. List all virtual machine networks on a host.

esxcli network vm list

The command returns for each virtual machine the World ID, name, number of ports, and networks, as in the following example.

World ID Name Num Ports Networks

-------- ------------------------------------- --------- ---------------------------

646632 VM NAME 1 2 Wintel Network

3749030 VM NAME 2 1 Wintel Network

6521771 VM NAME 3 1 Wintel Network

20714376 VM NAME 4 1 Wintel Network

2. List the ports for one of the VMs by specifying its World ID.

esxcli network vm port list -w 646632

The command returns port information, as in the following example.

Port ID: 50331682

vSwitch: vSwitch1

Portgroup: Wintel Network

DVPort ID:

MAC Address: 00:50:56:91:24:27

IP Address: 0.0.0.0

Team Uplink: vmnic3

Uplink Port ID: 50331652

Active Filters:

Port ID: 100663302

vSwitch: vSwitch4

Portgroup: ILO Network

DVPort ID:

MAC Address: 00:50:56:91:0a:f5

IP Address: 0.0.0.0

Team Uplink: vmnic6

Uplink Port ID: 100663298

Active Filters:

3. Retrieve the switch statistics for a port.

esxcli network port stats get -p 50331682

The command returns detailed statistics, as in the following example. This will help you to find out a drop in network.

Packet statistics for port 50331682

Packets received: 1237624170

Packets sent: 30179514

Bytes received: 113310026172

Bytes sent: 6341761418

Broadcast packets received: 708707216

Broadcast packets sent: 2037956

Multicast packets received: 508976173

Multicast packets sent: 9430448

Unicast packets received: 19940781

Unicast packets sent: 18711110

Receive packets dropped: 1202750738

Transmit packets dropped: 0

4. Retrieve the filter information for the port. The command returns detailed statistics, as in the following example.

esxcli network port filter stats get -p 50331682

Filter statistics for dvfilter-test:

Filter direction: Receive

Packets in: 202080

Packets out: 202080

Packets dropped: 0

Packets filtered: 0

Packets faulted: 0

Packets queued: 0

Packets injected: 0

Packet errors: 0

5. Retrieve complete statistics for a NIC.

esxcli network nic stats get -n vmnic3

NIC statistics for vmnic3

Packets received: 72994042944

Packets sent: 52891670321

Bytes received: 68186453587443

Bytes sent: 50171818016986

Receive packets dropped: 0

Transmit packets dropped: 0

Multicast packets received: 2844327960

Broadcast packets received: 0

Multicast packets sent: 0

Broadcast packets sent: 0

Total receive errors: 0

Receive length errors: 0

Receive over errors: 0

Receive CRC errors: 0

Receive frame errors: 0

Receive FIFO errors: 0

Receive missed errors: 719

Total transmit errors: 0

Transmit aborted errors: 0

Transmit carrier errors: 0

Transmit FIFO errors: 0

Transmit heartbeat errors: 0

Transmit window errors: 0

6. Get a per‐VLAN packed breakdown on a NIC.

Let’s take a look at statistics on a per port group or more specifically on a per VLAN basis using the new “vlan” namespace. By default VLAN statistics is disabled and you need to enable it on each vmnic that you want to pull statistics for.

We will enable VLAN stats collection on vmnic3 by running the following command (replace with the vmnic of choice)

Enabled VLAN statistic on vmnic3

esxcli network nic vlan stats set -e true -n vmnic3

Get VLAN stats

esxcli network nic vlan stats get -n vmnic0

Above command returns the number of packets sent and received for the VLAN you specified.

VLAN 0

Packets received: 32749862

Packets sent: 24

VLAN 4

Packets received: 14334203

Packets sent: 0

VLAN 5

Packets received: 3330475941

Packets sent: 0

VLAN 8

Packets received: 538680670

Packets sent: 0

VLAN 130

Packets received: 21116900071

Packets sent: 13272199149

VLAN 519

Packets received: 797082327

Packets sent: 0

VLAN 526

Packets received: 756335360

Packets sent: 0

VLAN 527

Packets received: 141830192

Packets sent: 0

VLAN 528

Packets received: 662613091

Packets sent: 0

VLAN 536

Packets received: 101178020

Packets sent: 0

· VSISH

VMkernel system information shell (known as vsish) is a great tool for troubleshooting network performance issue. vsish runs in the ESXi shell and let you look at in advanced system parameters and performance information of the ESXi host and associated objects like the VMs.

Will look at the performance counters of a VM's vmxnet3 network adapter: As a prerequisite we need to find out the name of the port group that the VM is attached to and its port ID. An easy way to get this information is to launch the esxtop utility and look at its network display (Press n to get there). Second option to get port id, you can try command 1 and 2 in ESXCLI section.

Locate the name of the VM in the USED-BY field and take a note of the port ID displayed in the PORT-ID column (in this case: 33554833)

Run vsish at ESXI CLI and look at this specific port: ->net/portsets/vSwitch0/ports/33554833/> (Note: - Target VM is mapped to vSwitch0)

Type cat status command

Note: Command vsish is like using UNIX filesystem tree that is full of text files. You use cd to change to different folders or branches, ls to display nodes and sub-branches and cat to display the contents of nodes.

Cat status e.g. display some details about the port's configuration. As we know the existing port is using VMXNET3 adapter type, let’s use cat stats command within same PORT ID to check network statistics

In my case, the stats shows that the vNIC is not dropping a TX/RX packages. The issue comes in notice when TX/RX value seen at higher level and then you can dig further to understand the cause for VM performance issue.

Let's switch to the vNIC type specific sub-branch vmxnet3 (use cd vmxnet3 command) and look at the rxSummary node there:

If you see high value in running out of buffer field and 1st ring that means vmxnet3 driver is running out of buffers. In my case it is showing zero, so it looks good.

You can dig down more and check rxqueue if you see high buffer value.

It turns out that there is only one receive queue - that means RSS is not enabled on this adapter. RSS stands for Receive Side Scaling and allows having multiple receive queues for a network adapter which are handled by different CPUs.

We can also see that the ring sizes of the receive queue are 512, resp. 32 (which are the default values).

The ring sizes, the number of buffers and the RSS features of the vmxnet3 adapter are parameters that can be configured from inside the VM's guest OS (Windows in this case):