Analysis the esxtop data for performance issue (part1)

When you meet some performance issue on esxi platform , you will need the esxtop tools to trouble shooting .

Now this article just discussed , how to find useful data from the esxtop perf data .

Get the performance data

How to collect data , you can read the KB 

Gathering esxtop performance data at specific times using crontab (1033346)

Or  write a simple bash script to collect 24 hours performance data . Depend on your real environment , you change the esxtop parameter. 

!-- HTML generated using hilite.me -->

#!/bin/bash
for i in $(seq 1 24)
do
 echo $i >> esxtop.log
 esxcli system time get >> esxtop.log 
 esxtop -lab -d 2 -n 1800 >$i.csv
 esxcli system time get >> esxtop.log 
done

Sometimes , I usually get a log files, but the issue time just in one csv . Use following command 

1 get interesting csv file 

 for i in $(ls *.csv); do echo ======$i====== ;head -2 $i |awk -F , '{print $1}'; done

2 generate the headers( this part ,I referenceed http://virtuallyhyper.com/）

$ head -1 esxtop-b.csv | sed 's/\,/\n/g' > headers

Now checking out the ‘headers’ file, I saw the following:

$ head headers
"(PDH-CSV 4.0) (UTC)(0)"
"\\local\Memory\Memory Overcommit (1 Minute Avg)"
"\\local\Memory\Memory Overcommit (5 Minute Avg)"
"\\local\Memory\Memory Overcommit (15 Minute Avg)"
"\\local\Physical Cpu Load\Cpu Load (1 Minute Avg)"
"\\local\Physical Cpu Load\Cpu Load (5 Minute Avg)"
"\\local\Physical Cpu Load\Cpu Load (15 Minute Avg)"
"\\local\Physical Cpu(0)\% Processor Time"
"\\local\Physical Cpu(1)\% Processor Time"
"\\local\Physical Cpu(2)\% Processor Time" 

The columns are now separated by new lines, that's I want 

Searching keyword , and get the number 

 grep -n 'SCSI.*Average Driver MilliSec/Command' heades

27183:"\\localhost\Physical Disk SCSI Device(naa.6005076801828732d800000000000151)\Average Driver MilliSec/Command"

29286:"\\localhost\Physical Disk SCSI Device(naa.6234567890abcde01e30fcc911b435f5)\Average Driver MilliSec/Command"

You can use awk  >> to generate new file "column" and store the column number which you want . 

grep -n 'SCSI.*Average Driver MilliSec/Command' heades | awk -F :  '{print $1}' >> column

check the content 

$ head column
27183
29286

If you just need pick up 2 or 3 column , you can use following command

1	awk -F , '{print $1","$27183","$29286}' esxtop-b.csv > latency.csv

But If you need pick up ten or more column , you need a python script.Thanks my friend Haojie help on that.

import pandas as pd

df = pd.read_csv('esxtop-b.csv', header=None)
df_new = pd.DataFrame(df[[0]])

with open('coloumn') as f:
    for line in f:
        df_new = df_new.join(df[[int(line)]])

df_new.to_csv('out.csv')

Here is summary :
1 Create header files from CSV
2 Generate your custom column number to a column files
3 Get your interesting data.

How to prepare PowerCLI environment on windows 7 and prepare PowerShell ISE

I am  a newbie for Powershell and PowerCli , so just write a notes .

Step one ： Install Powershell on Windows 7 

 1.1， Install .NET Framework 4.0 or later

 I download .NET Framework 4.5.2 offline installer ,

 1.2  Install the Windows Management Framework 3.0 

 Windows Remote Management Framework v3.0

• If you have Windows 7 64-bit, you want the file named: Windows6.1-KB2506143-x64.msu
• If you have Windows 7 32-bit, you want the file named: Windows6.1-KB2506143-x86.msu

   Step Two: Install PowerCLI
   Go to VMware PowerCLI Site,download the latest version .


Also I need a development environment  for PowerShell , the first option  is 

Integrated Scripting Environment (ISE)

If you want to use ISE , just need to type command "Powershell_ise" in vsphere powershell. 

After that, I got a new issue " ISE didn't load vsphere module". 

Here is my simply way , Write a Powershell_ISE profile .

"C:\Users\shensong\Documents\WindowsPowershell\Microsoft.PowerShellISE_profile"

the content:

"& 'C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI\Scripts\Initialize-PowerCLIEnvironment.ps1'"

You can type "echo $profile" to check your current powershell profile.Note that, Powershell ISE load different profile than Powershell