VisualVM to the rescue

Since a few days I frequently got a java.lang.OutOfMemoryError: Metaspace after a few tests in the SBT shell. I attached VisualVM to the SBT process and indeed Metaspace didn’t go down after test had finished:

metaspace up to 250MB and not going down

By the way with many SBT processes running the following SBT alias comes handy: It shows the PID of the running process which helps you pick the right SBT process to attach to in VisualVM:

addCommandAlias(
  "runtimeName",
  "eval java.lang.management.ManagementFactory.getRuntimeMXBean().getName()"
)

I had a suspicion: The application runs on top of Akka, and perhaps I had forgotten to shutdown some actor systems in my tests? And indeed in the “Threads” tab I saw a couple of threads lingering even after test had finished:

lingering threads after test finished

Akka conveniently names threads after the Actor system they belong to, and since luckily I had named my Actor systems after the test suites that created them I immediately knew which test suites to look at.

Indeed these suites didn’t have the trait that implemented actor system shutdown mixed in. I fixed these test suites, and the OutOfMemory error was gone. The whole thing took less than half an hour. Never underestimate the power of monitoring tools!