I think @OneToMany annotation does not work at all with @JoinFormula annotation.
I found some old issue https://hibernate.atlassian.net/browse/HHH-9897, but I'm not sure how to interpret it - that is whether it indeed was never supported, or that the specific case in that issue wasn't supported (i.e. with @JoinColumnsOrFormulas).
Having the following entity:
The above mapping does not work. During validation we get exception
Adding reverse side of the relationship (@ManyToOne with @JoinFormula) and adding "mappedBy" to @OneToMany does not change anything - still same error. Also it does not matter that in the attached example we have relation from table A to the same table A - I tested it with different tables with same result.