The
objective of this document to cover troubleshooting network performance issues
using ESXCLI and VSHISH commands
ESXCLI
You can use vCLI network
commands to view network statistics and troubleshoot your networking setup. The
nested hierarchy of commands allows you to drill down to potential trouble
spots.
1. List all virtual machine
networks on a host.
esxcli network vm list
The command returns for each
virtual machine the World ID, name, number of ports, and networks, as in the
following example.
World
ID
Name
Num
Ports
Networks
--------
-------------------------------------
--------- ---------------------------
646632 VM NAME 1
2 Wintel
Network
3749030 VM
NAME
2
1 Wintel
Network
6521771
VM NAME
3
1
Wintel Network
20714376 VM NAME
4
1
Wintel Network
2. List the ports for one of
the VMs by specifying its World ID.
esxcli network vm port list
-w 646632
The command returns port
information, as in the following example.
Port ID: 50331682
vSwitch: vSwitch1
Portgroup:
Wintel Network
DVPort ID:
MAC Address:
00:50:56:91:24:27
IP Address:
0.0.0.0
Team Uplink:
vmnic3
Uplink Port ID:
50331652
Active Filters:
Port ID:
100663302
vSwitch: vSwitch4
Portgroup: ILO
Network
DVPort ID:
MAC Address:
00:50:56:91:0a:f5
IP Address:
0.0.0.0
Team Uplink:
vmnic6
Uplink Port ID:
100663298
Active Filters:
3. Retrieve the switch
statistics for a port.
esxcli network port stats get -p 50331682
The command returns detailed
statistics, as in the following example. This will help you to find out a drop
in network.
Packet statistics for
port 50331682
Packets
received: 1237624170
Packets sent:
30179514
Bytes received:
113310026172
Bytes sent:
6341761418
Broadcast
packets received: 708707216
Broadcast
packets sent: 2037956
Multicast
packets received: 508976173
Multicast
packets sent: 9430448
Unicast packets
received: 19940781
Unicast packets
sent: 18711110
Receive packets
dropped: 1202750738
Transmit packets
dropped: 0
4. Retrieve the filter
information for the port. The command returns detailed statistics, as in the
following example.
esxcli network port filter
stats get -p 50331682
Filter statistics for
dvfilter-test:
Filter direction: Receive
Packets in: 202080
Packets out: 202080
Packets dropped: 0
Packets filtered: 0
Packets faulted: 0
Packets queued: 0
Packets injected: 0
Packet errors: 0
5. Retrieve complete statistics for a NIC.
esxcli network nic stats get
-n vmnic3
NIC statistics for vmnic3
Packets
received: 72994042944
Packets sent:
52891670321
Bytes received:
68186453587443
Bytes sent:
50171818016986
Receive packets
dropped: 0
Transmit packets
dropped: 0
Multicast
packets received: 2844327960
Broadcast
packets received: 0
Multicast
packets sent: 0
Broadcast
packets sent: 0
Total receive
errors: 0
Receive length
errors: 0
Receive over
errors: 0
Receive CRC
errors: 0
Receive frame
errors: 0
Receive FIFO
errors: 0
Receive missed errors: 719
Total transmit
errors: 0
Transmit aborted
errors: 0
Transmit carrier
errors: 0
Transmit FIFO
errors: 0
Transmit
heartbeat errors: 0
Transmit window
errors: 0
6. Get a per‐VLAN packed breakdown on a NIC.
Let’s take a look at statistics on a per port group or more
specifically on a per VLAN basis using the new “vlan” namespace. By default
VLAN statistics is disabled and you need to enable it on each vmnic that you
want to pull statistics for.
We will enable VLAN stats collection on vmnic3 by running the
following command (replace with the vmnic of choice)
Enabled VLAN statistic on vmnic3
esxcli network nic vlan stats set -e true -n vmnic3
Get VLAN stats
esxcli network nic vlan stats
get -n vmnic0
Above command returns the
number of packets sent and received for the VLAN you specified.
VLAN 0
Packets
received: 32749862
Packets sent: 24
VLAN 4
Packets
received: 14334203
Packets sent: 0
VLAN 5
Packets
received: 3330475941
Packets sent: 0
VLAN 8
Packets
received: 538680670
Packets sent: 0
VLAN 130
Packets
received: 21116900071
Packets sent:
13272199149
VLAN 519
Packets
received: 797082327
Packets sent: 0
VLAN 526
Packets
received: 756335360
Packets sent: 0
VLAN 527
Packets
received: 141830192
Packets sent: 0
VLAN 528
Packets
received: 662613091
Packets sent: 0
VLAN 536
Packets
received: 101178020
Packets sent: 0
· VSISH
VMkernel
system information shell (known as vsish) is a great tool for
troubleshooting network performance issue.
vsish runs in the ESXi shell and let you look at
in advanced system parameters and performance information of the ESXi host and
associated objects like the VMs.
Will
look at the performance counters of a VM's vmxnet3 network adapter: As a
prerequisite we need to find out the name of the port group that the VM is
attached to and its port ID. An easy way to get this information is to launch
the esxtop utility and look at its network display (Press n to
get there). Second option to get port id, you can try command 1 and 2 in ESXCLI
section.
Locate
the name of the VM in the USED-BY field and take a note of the port
ID displayed in the PORT-ID column (in this case: 33554833)
Run
vsish at ESXI CLI and look at this specific port:
->net/portsets/vSwitch0/ports/33554833/> (Note: - Target VM is mapped to
vSwitch0)
Type cat
status command
Note: Command
vsish is like using UNIX filesystem tree that is full of text files. You
use cd to change to different folders or branches, ls to
display nodes and sub-branches and cat to display the contents of
nodes.
Cat status e.g. display some details about the port's configuration. As we know the existing port is using VMXNET3 adapter type, let’s use cat stats command within same PORT ID to check network statistics
In my case, the stats shows that the vNIC is not dropping a
TX/RX packages. The issue comes in notice when TX/RX value seen at higher level
and then you can dig further to understand the cause for VM performance issue.
Let's
switch to the vNIC type specific sub-branch vmxnet3 (use cd vmxnet3 command)
and look at the rxSummary node there:
If you see high value in running out of buffer field and 1st
ring that means vmxnet3 driver is running out of buffers. In my case it is
showing zero, so it looks good.
You
can dig down more and check rxqueue if you see high buffer value.
It
turns out that there is only one receive queue - that means RSS is not enabled
on this adapter. RSS stands for Receive Side Scaling and allows having multiple
receive queues for a network adapter which are handled by different CPUs.
We
can also see that the ring sizes of the receive queue are 512, resp. 32 (which
are the default values).
The
ring sizes, the number of buffers and the RSS features of the vmxnet3 adapter
are parameters that can be configured from inside the VM's guest OS (Windows in
this case):
VMware article about RSS settings:
RSS for Windows server:
RSS for Linux server:
Very helpful artical about how to debug ESXi networking!
ReplyDeleteThis was extremely helpful and gave me some experience with a toolbox I had yet to utilize. Thank you!
ReplyDeleteyes while im mailing this sandbox info my email got queued and i thought what does queued email means in gmail
ReplyDeleteAs a networking student i too research a lot on computer networks and stuff and here is a good general on networks.you can read different types of network's in computer networks in general
ReplyDeletevery nice… i really like your blog…
ReplyDeleteDevOps Training
DevOps Online Training