What is inventory service ?
- All virtual infrastructure objects defined by VMware
- All physical objects like hosts, disks, arrays that are under Visualization management
- Partner defined virtual objects
- Physical objects like blades, enclosures, power systems that live outside a host
- Meta data information such as privileges
Inventory service actually is a tomcat server , we can config the heap size or max memory for JVM
Case description :
Customer search function abnormal
Customer said , he added two new esxi host, but can not search the new vms by search function.
Here is my trouble shooting flow .
First step ,please check the inv-svc.log . The location , please reference KB
check the issue time , there is no usful information . Alos use keyword "xdb"
xdb is a file to store object information , eg: vm object
Ha~~~,got that
2016-03-18T09:37:17.187+08:00 [provider-manager-task-66 ERROR com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl opId=] Caught unexpected excep tion:^M
382 java.lang.OutOfMemoryError: Java heap space^M
at java.util.Arrays.copyOf(Arrays.java:2367)^M
384 at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)^M
385 at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)^M
386 at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:535)^M
387 at java.lang.StringBuilder.append(StringBuilder.java:204)^M
388 at com.xhive.xDB_10_2_0.h.b(xdb:159)^M
389 at com.xhive.xDB_10_2_0.f.a(xdb:201)^M
390 at com.xhive.xDB_10_2_0.f.parseWithContext(xdb:47)^M
391 at com.vmware.vim.query.server.store.impl.AtomFeedProcessor$EntryUtil.create(AtomFeedProcessor.java:756)^M
392 at com.vmware.vim.query.server.store.impl.AtomFeedProcessor$PullEntryProcessor.processEntries(AtomFeedProcessor.java:329)^M
393 at com.vmware.vim.query.server.store.impl.AtomFeedProcessor$EntryProcessor.call(AtomFeedProcessor.java:117)^M
394 at com.vmware.vim.query.server.store.impl.StoreImpl.updateVmomiPullAtomFeed(StoreImpl.java:2539)^M
395 at com.vmware.vim.query.server.store.impl.StoreImpl.update(StoreImpl.java:2412)^M
396 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)^M
397 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)^M
398 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)^M
399 at java.lang.reflect.Method.invoke(Method.java:606)^M
400 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)^M
401 at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)^M
402 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)^M
403 at org.springframework.aop.interceptor.JamonPerformanceMonitorInterceptor.invokeUnderTrace(JamonPerformanceMonitorInterceptor.java:108)^M
404 at org.springframework.aop.interceptor.AbstractTraceInterceptor.invoke(AbstractTraceInterceptor.java:111)^M
405 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)^M
406 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)^M
407 at com.sun.proxy.$Proxy46.update(Unknown Source)^M
408 at com.vmware.vim.query.server.provider.AbstractAtomPullProviderBase.checkFeed(AbstractAtomPullProviderBase.java:150)^M
409 at com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl$FeedPump.run(ProviderManagerServiceImpl.java:1107)^M
410 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)^M
It looks like some meory issue on JVM ,
Second steps ,check the wrapper.log what is warpper.log please see following KB
Deleting or rotating the wrapper.log file in vFabric tc Server (2010299)
I found from the 1/21 , the inventory serivce works abnormal
16190 INFO | jvm 1 | 2016/01/21 10:48:32 | java.lang.OutOfMemoryError: GC overhead limit exceeded
16191 INFO | jvm 1 | 2016/01/21 10:48:32 | Dumping heap to D:\ProgramData\VMware\vCenterServer\logs\invsvc\java_pid3320.hprof ...
Java process crash , and dumpping heap a lot of times
Suspecting memory leak on inveroy service 6.0 ,check the knowissue at VMware KB . The answer is no
Continue check ,customer enviorment . Customer using windows VC, Windows OS with 8192 MB memory
Now ,ask other questions , how many VMs and how many esxi hosts under this VC control ?About over 100 vms now.
Oh....
The third steps
We need check “vCenter Server for Windows Hardware Requirements”
We have to add window OS memory to 16G.
Please consider the JVM setting ,
D:\ProgramData\VMware\vCenterServer\cfg/java/vmware-invsvc_jvm.conf
change
-Xmx950m -XX:MaxPermSize=256m -XX:ThreadStackSize=256 -XX:ParallelGCThreads=1
to
-Xmx1506m -XX:MaxPermSize=256m -XX:ThreadStackSize=256 -XX:ParallelGCThreads=1
After change above settings , the issue is gone .
KB Links :
http://kb.vmware.com/kb/2110014
Location of VMware vCenter Server 6.0 log files (2110014)
http://kb.vmware.com/kb/2010299
Deleting or rotating the wrapper.log file in vFabric tc Server (2010299)
Besides , I didn't suggest resetting DB firstly , because I didn't see any xdb file io error or read error at inv-svc.log
