Trouble shooting inventory service issue , case 1

What is inventory service ?

1. All virtual infrastructure objects defined by VMware
2. All physical objects like hosts, disks, arrays that are under Visualization management
3. Partner defined virtual objects
4. Physical objects like blades, enclosures, power systems that live outside a host
5. Meta data information such as privileges

Inventory service actually is a tomcat server , we can config the heap size or max memory for JVM

Case description :

Customer search function abnormal 

Customer said , he added two new esxi host, but can not search the new vms by search function. 

Here is my trouble shooting flow .

First  step ,please check the inv-svc.log . The location , please reference KB 

location of VMware vCenter Server 6.0 log files (2110014)

check the issue time , there is no usful information . Alos use keyword "xdb" 

xdb is a file to store object information , eg: vm object

Ha~~~,got that 

2016-03-18T09:37:17.187+08:00 [provider-manager-task-66  ERROR com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl  opId=] Caught unexpected excep     tion:^M

 382 java.lang.OutOfMemoryError: Java heap space^M

  at java.util.Arrays.copyOf(Arrays.java:2367)^M

 384         at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)^M

 385         at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)^M

 386         at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:535)^M

 387         at java.lang.StringBuilder.append(StringBuilder.java:204)^M

 388         at com.xhive.xDB_10_2_0.h.b(xdb:159)^M

 389         at com.xhive.xDB_10_2_0.f.a(xdb:201)^M

 390         at com.xhive.xDB_10_2_0.f.parseWithContext(xdb:47)^M

 391         at com.vmware.vim.query.server.store.impl.AtomFeedProcessor$EntryUtil.create(AtomFeedProcessor.java:756)^M

 392         at com.vmware.vim.query.server.store.impl.AtomFeedProcessor$PullEntryProcessor.processEntries(AtomFeedProcessor.java:329)^M

 393         at com.vmware.vim.query.server.store.impl.AtomFeedProcessor$EntryProcessor.call(AtomFeedProcessor.java:117)^M

 394         at com.vmware.vim.query.server.store.impl.StoreImpl.updateVmomiPullAtomFeed(StoreImpl.java:2539)^M

 395         at com.vmware.vim.query.server.store.impl.StoreImpl.update(StoreImpl.java:2412)^M

 396         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)^M

 397         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)^M

 398         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)^M

 399         at java.lang.reflect.Method.invoke(Method.java:606)^M

 400         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)^M

 401         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)^M

 402         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)^M

 403         at org.springframework.aop.interceptor.JamonPerformanceMonitorInterceptor.invokeUnderTrace(JamonPerformanceMonitorInterceptor.java:108)^M

 404         at org.springframework.aop.interceptor.AbstractTraceInterceptor.invoke(AbstractTraceInterceptor.java:111)^M

 405         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)^M

 406         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)^M

 407         at com.sun.proxy.$Proxy46.update(Unknown Source)^M

 408         at com.vmware.vim.query.server.provider.AbstractAtomPullProviderBase.checkFeed(AbstractAtomPullProviderBase.java:150)^M

 409         at com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl$FeedPump.run(ProviderManagerServiceImpl.java:1107)^M

 410         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)^M

It looks like some meory issue on JVM ,

Second steps ,check the wrapper.log what is warpper.log please see following KB

Deleting or rotating the wrapper.log file in vFabric tc Server (2010299)

I found from the 1/21 , the inventory serivce works abnormal 

16190 INFO   | jvm 1    | 2016/01/21 10:48:32 | java.lang.OutOfMemoryError: GC overhead limit exceeded

 16191 INFO   | jvm 1    | 2016/01/21 10:48:32 | Dumping heap to D:\ProgramData\VMware\vCenterServer\logs\invsvc\java_pid3320.hprof ...

Java process crash , and dumpping heap a lot of times 

Suspecting memory leak on inveroy service 6.0 ,check the knowissue at VMware KB .  The answer is no 

Continue check ,customer enviorment .  Customer using windows VC, Windows OS with 8192 MB memory 

Now ,ask other questions , how many VMs and how many esxi hosts under this VC control  ?About over 100 vms now. 

Oh....

The third steps 

We need check “vCenter Server for Windows Hardware Requirements” 

 We have to add window OS  memory to 16G. 

Please consider the JVM setting , 

D:\ProgramData\VMware\vCenterServer\cfg/java/vmware-invsvc_jvm.conf

change 

-Xmx950m -XX:MaxPermSize=256m -XX:ThreadStackSize=256 -XX:ParallelGCThreads=1

to

-Xmx1506m -XX:MaxPermSize=256m -XX:ThreadStackSize=256 -XX:ParallelGCThreads=1

After change above settings , the issue is gone . 

KB Links :

http://kb.vmware.com/kb/2110014

Location of VMware vCenter Server 6.0 log files (2110014)

http://kb.vmware.com/kb/2010299

Deleting or rotating the wrapper.log file in vFabric tc Server (2010299)

Besides , I didn't suggest resetting DB firstly , because  I didn't see any xdb file io  error or read error at inv-svc.log