Associated collection deleted after loading in byte-enhanced entities

I use load-time weaving to enhance my entities and in the latest version of 5.1 (5.1.5) I noticed that some data was being lost due to hibernate issuing a collection deletion after a simple collection getter usage. Problematic in this specific case as it was user permissions.
I encountered the same problem some versions ago but it got fixed in the meantime, sadly I don't remember which one exactly though.

I tried to distill my use case as much as I could and attached a test case for it. The test still uses spring boot but hope it helps nevertheless.

edit - tested with hibernate 5.2, last working version was 5.2.7 (updated affected versions)