Latest Gradle versions come with support for "incremental annotation processing", which means that re-running processors on unrelated files can be avoided as long as annotation processors specify on which source files their work (validation, file generation etc.) is based.
See https://github.com/gradle/gradle/issues/5277.
It should be quite easy to suppor that with the HV AP. The ORM entity model generation one might be more tricky (separate issue warranted anyways), as IIRC it also must read persistence.xml.