Details
-
Type:
Task
-
Status: In Progress
-
Resolution: Unresolved
-
Affects Version/s: BIGDATA_RELEASE_1_1_0
-
Fix Version/s: None
-
Component/s: CI, Project Management
-
Labels:None
Description
There are a lot of temporary journals showing up in CI. Track down tests which are creating these journals but failing to destroy them.
Activity
I have disabled the embedded federation test suite for the sail again. Per the comment above, it is leaking threads which the JVM is unable to recover without a full GC. In fact, this is probably the root cause for [1]. I only identified this interaction by forcing a full GC on the JVM using an external utility (the profiler). It is not enough to request a GC from within the JVM as it is free to ignore that request.
com.bigdata.rdf.sail.TestAll:
// suite.addTest(TestBigdataSailEmbeddedFederationWithQuads.suite());
[1] https://sourceforge.net/apps/trac/bigdata/ticket/196 (Journal leaks memory)
Committed revision r6200.
Per [1], it appears that the root cause may be the failure of the JVM to force a full GC when there are active threads which could be collected. This needs to be investigated further. If so, it is a JVM bug/feature.
[1] https://sourceforge.net/apps/trac/bigdata/ticket/524 (Temporary Journals in CI)
The above comment was for [1].
[1] https://sourceforge.net/apps/trac/bigdata/ticket/196 (Journal leaks memory)
Restored test suites per [1,2]. Now that the journal memory leak has been addressed, I want to see if we can enable the embedded federation test suite in CI.
com.bigdata.rdf.sail.TestAll: suite.addTest(TestBigdataSailEmbeddedFederationWithQuads.suite());
Committed revision r6824. (Dev branch)
[1] http://sourceforge.net/apps/trac/bigdata/ticket/196 (Journal leaks memory).
[2] http://sourceforge.net/apps/trac/bigdata/ticket/523 (Temporary journals in CI : blocking embedded federation CI)
Re-disabling in CI. It looks like the embedded federation is still leaking resources. It gets very slow during the embedded federation test suite run and I observe a LOT of threads when requesting a thread dump.
com.bigdata.rdf.sail.TestAll: suite.addTest(TestBigdataSailEmbeddedFederationWithQuads.suite());
Committed revision r6828.
CI has been failing repeatedly due to "Too many open files" with 10000 open file handles.
I have temporarily disabled two of the major test suites in an attempt to reduce the #of open files pending resolution of this issue.
Changes are to: com.bigdata.rdf.store.TestAll:
and com.bigdata.rdf.sail.TestAll:
The lines indicated above MUST be restored once this issue is resolved.