Tuesday 10 July 2018

Best Practices - Virtualizing Hadoop on VMware vSphere

The best practices for setting up each of the main compute resources—disk storage, I/O, CPUs, and memory—when preparing them to run virtualized Hadoop-based workloads. This data should be read as an introduction to these best-practice areas. 

  • The sum of all the memory size configured in the VMs on a server should not exceed the size of physical memory on the host server. Reserve about 5-6% of total server memory for ESXi; use the remainder for the virtual machines.
  • The physical CPUs on the vSphere host should not be overcommitted. One viable approach here is that the total number of vCPUs configured across all VMs on a host server is equal to the physical core count on that server. This more conservative approach ensures that no vCPU is waiting for a physical CPU to be available before it can execute. If that type of waiting were to occur, the administrator would see a sustained increase in %Ready time as measured by the vSphere performance tools. 
  • When hyperthreading is enabled at the BIOS level, as is recommended, the total number of vCPUs in all VMs on a host server can be set up to be equal to twice the number of physical cores—that is, equal to the number of logical cores on the server. This “exactly committed” approach is used in demanding situations where the best performance is a requirement. Both the conservative method and the match-to-logical-core method are viable approaches, with the latter being seen as the more aggressive of the two in achieving performance results.
  • VMs whose vCPU count fits within the number of cores in a CPU socket, and that exclusively use the associated NUMA memory for that socket, have been shown to perform better than larger VMs that span multiple sockets. The recommendation is to limit the vCPUs in any VM to a number that is less than or equal to the number of cores in a CPU socket on the target hardware. This prevents the VM from being spread across multiple CPU sockets and can help it perform more efficiently
  • Create 1 or more virtual machines per NUMA node.
  • Limit the number of disks per DataNode to maximize the utilization of each disk – 4 to 6 is a good starting point.
  • Use eager-zeroed thick Disk VMDKs along with the ext4 filesystem inside the guest.
  • Use the VMware Paravirtual SCSI (pvscsi) adapter for disk controllers; use all 4 virtual SCSI controllers available in vSphere 6.0.
  • Use dedicated network switches for the Hadoop cluster if possible and ensure that all physical servers are connected to a ToR switch. Use the vmxnet3 network driver; configure virtual switches with MTU=9000 for jumbo frames.
  • Use a network that has the bandwidth of at least 10 GB per second to connect servers running virtualized Hadoop workloads.
  • When configuring ESXi host networking, consider the traffic and loading requirements of the following consumers, each port should be connected to a separate switch for optimizing network usage.
  1. The management network
  2. VM port groups
  3. IP storage (NFS, iSCSI, FCoE)
  4. vSphere vMotion
  5. Fault tolerance
  • Configure the guest operating system for Hadoop performance including enabling jumbo IP frames, reducing swappiness, and disabling transparent hugepage compaction.
  • Place Hadoop master roles, ZooKeeper, and journal nodes on three virtual machines for optimum performance and to enable high availability.
  • Dedicate the worker nodes to run only the HDFS DataNode, YARN NodeManager, and Spark Executor roles.
  • Use the Hadoop rack awareness feature to place virtual machines belonging to the same physical host in the same rack for optimized HDFS block placement.
  • Run the Hive Metastore in a separate MySQL database.
  • Set the Yarn cluster container memory and vcores to slightly overcommit both resources.
  • Adjust the task memory and vcore requirement to optimize the number of maps and reduces for each application. 
Above details are provided in more details in below articles.


https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/bigdata-perf-vsphere6.pdf

3 comments: