Live reload classloader memory leak when using hibernate-envers with a custom revision entity

[jin-harmoney]

Describe the bug

I'm not sure if this issue belongs here or on the Hibernate issue tracker.

We're observing ever increasing metaspace memory consumption after each live reload.

The increasing metaspace memory consumption is because the classloaders are not garbage collected.

We're creating revision entries for all entities using Hibernate envers. In order to store some additional information on each revision, we use a custom revision entity. Removing the custom revision entity seems to allow garbage collection to occur, hence resolving the increasing memory consumption.

Expected behavior

Metaspace memory is stable over time, even when many live-reloads occur.

Classloaders are garbage collected even though a custom revision entity is used.

Actual behavior

When running quarkus dev, metaspace memory keeps increasing.

This is almost not noticeable in the reproducer I provided, because there are only 3 classes loaded. In our application, each live reload adds about 50MB to metaspace, so eventually this leads to a large amount of memory that is not released.

The screenshot below (not super clear, sorry) shows the classloaders when I'm doing live-reloads. Sometimes, the classloaders are garbage collected, sometimes they are not. That corresponds to when I remove/restore the CustomerRevisionEntity class.

How to Reproduce?

Reproducer: https://github.com/jin-harmoney/quarkus-hibernate-envers-classloader-leak

• Run quarkus dev
• Attach your favorite profiler
• Do a couple changes to trigger multiple live reloads and observe that classloaders are not garbage collected
• Remove the class CustomRevisionEntity
• Do a couple changes to trigger multiple live reloads and observe that classloaders are not garbage collected

Output of uname -a or ver

No response

Output of java -version

OpenJDK 64-Bit Server VM Temurin-21.0.3+9 (build 21.0.3+9-LTS, mixed mode, sharing)

Quarkus version or git rev

3.18.1

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.9.9

Additional information

No response

[quarkus-bot]

/cc @gsmet (hibernate-orm)

[gsmet]

Thanks for the reproducer, I'll have a look soon.

[gsmet]

I can confirm I can reproduce the issue. Now I need to understand what's going on.

[gsmet]

hibernate/hibernate-orm#9699 should fix it.

If you want, you can try to apply it locally to Envers and report back with your project.

• Clone https://github.com/hibernate/hibernate-orm
• Get the 6.6 branch
• Apply the patch (it applies cleanly in both main and 6.6 branches) - you can get the patch directly with https://patch-diff.githubusercontent.com/raw/hibernate/hibernate-orm/pull/9699.diff
• Build Hibernate ORM locally with ./gradlew publishToMavenLocal -x test
• Update your dependency:

        <dependency>
            <groupId>org.hibernate.orm</groupId>
            <artifactId>hibernate-envers</artifactId>
            <version>6.6.7-SNAPSHOT</version>
        </dependency>

[jin-harmoney]

@gsmet amazing, thank you!

I can confirm the classloaders are garbage collected both in the reproducer and in our application!

Small anecdote: we had issues with classloader memory leaks in Wildfly when redeploying our WAR. I tried to resolve it multiple times over the past 3 years, but I never figured it out. It was actually one of the reasons I pushed the migration to Quarkus :) It's a small fix, but to me (and my team), finally resolving this is a huge deal! So thank you again!

[gsmet]

Thanks for the confirmation and having a simple reproducer made a huge difference so thanks for taking the time to assemble one!

I created a PR to get it backported to Hibernate ORM 6.6 so I hope it will land in Quarkus in a not too distant future.

[gsmet]

Small anecdote: we had issues with classloader memory leaks in Wildfly when redeploying our WAR

FWIW, I'm not very surprised. Quarkus dev mode allowed us to find many issues in various components not being very nice with reloading - and you would end up with the same problem in any app server when redeploying.

[geoand]

FWIW, I'm not very surprised. Quarkus dev mode allowed us to find many issues in various components not being very nice with reloading - and you would end up with the same problem in any app server when redeploying.

So so true. Lot's of software out there is not created with reloading in mind

[jin-harmoney]

Fix was merged on hibernate-orm 6.6.7 👍

[gsmet]

@jin-harmoney the fix was released in 3.18.3.

Thanks again for the nice reproducer!