Investigate 'GC overhead limit exceeded' error

[Twan Goosen]

In a recent VLO importer log (2014-09-04), the following error occurs (full stacktrace below)

java.lang.OutOfMemoryError: GC overhead limit exceeded

This might be caused by a memory leak or other issue in the importer, the solrj library or (more likely) the solr server. See ​this topic on stackoverflow for more information.

It did not break the entire import but may have caused some omissions in the index. The same error occurred on 2014-08-30.

2014-09-04 04:32:25,182 ERROR [eu.clarin.cmdi.vlo.importer.MetadataImporter#startImport:162] - error updating files:

org.apache.solr.client.solrj.SolrServerException: java.lang.OutOfMemoryError: GC overhead limit exceeded
        at eu.clarin.cmdi.vlo.importer.MetadataImporter.sendDocs(MetadataImporter.java:417)
        at eu.clarin.cmdi.vlo.importer.MetadataImporter.updateDocument(MetadataImporter.java:359)
        at eu.clarin.cmdi.vlo.importer.MetadataImporter.processCmdi(MetadataImporter.java:291)
        at eu.clarin.cmdi.vlo.importer.MetadataImporter.startImport(MetadataImporter.java:146)
        at eu.clarin.cmdi.vlo.importer.MetadataImporter.main(MetadataImporter.java:514)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.nio.CharBuffer.wrap(CharBuffer.java:369)
        at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:265)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:113)
        at java.io.OutputStreamWriter.write(OutputStreamWriter.java:194)
        at org.apache.solr.common.util.XML.writeXML(XML.java:151)
        at org.apache.solr.client.solrj.util.ClientUtils.writeVal(ClientUtils.java:161)
        at org.apache.solr.client.solrj.util.ClientUtils.writeXML(ClientUtils.java:129)
        at org.apache.solr.client.solrj.request.UpdateRequest.writeXML(UpdateRequest.java:355)
        at org.apache.solr.client.solrj.request.RequestWriter.write(RequestWriter.java:73)
        at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner$1.writeTo(ConcurrentUpdateSolrServer.java:185)
        at org.apache.http.entity.EntityTemplate.writeTo(EntityTemplate.java:72)
        at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98)
        at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
        at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:122)
        at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:271)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:197)
        at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:257)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
        at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:232)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

[Twan Goosen]

issue with solr server more likely cause

[Twan Goosen]

Update: has occurred after initial report but not in past ~3 weeks:

vlouser@catalog:/lat/webapps/vlo/current/bin> grep java.lang.OutOfMemoryError vlo-importer*log.2014*
vlo-importer.log.2014-08-30:java.lang.OutOfMemoryError: GC overhead limit exceeded
vlo-importer.log.2014-08-30:org.apache.solr.client.solrj.SolrServerException: java.lang.OutOfMemoryError: GC overhead limit exceeded
vlo-importer.log.2014-08-30:Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
vlo-importer.log.2014-09-04:java.lang.OutOfMemoryError: GC overhead limit exceeded
vlo-importer.log.2014-09-04:org.apache.solr.client.solrj.SolrServerException: java.lang.OutOfMemoryError: GC overhead limit exceeded
vlo-importer.log.2014-09-04:Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
vlo-importer.log.2014-09-22:java.lang.OutOfMemoryError: GC overhead limit exceeded
vlo-importer.log.2014-09-22:org.apache.solr.client.solrj.SolrServerException: java.lang.OutOfMemoryError: GC overhead limit exceeded
vlo-importer.log.2014-09-22:Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
vlo-importer.log.2014-09-25:java.lang.OutOfMemoryError: GC overhead limit exceeded
vlo-importer.log.2014-09-25:org.apache.solr.client.solrj.SolrServerException: java.lang.OutOfMemoryError: GC overhead limit exceeded
vlo-importer.log.2014-09-25:Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

[Twan Goosen]

Not happening anymore, maybe because of increased heapspace