The idea is to have one "builder factory" instance for each field and each DSL (predicate/sort/projection). See org.hibernate.search.backend.lucene.document.model.impl.LuceneIndexSchemaFieldNode#predicateBuilderFactory for example.
In Guillaume is working on taking this path for projections as well, and I think it's the more future-proof choice.
Currently we use this pattern rather inconsistently; for Lucene it's mostly used everywhere, but not for Elasticsearch.
Also, we have some issues with the "factory" interfaces:
I think the equals/hashCode on the LuceneFieldPredicateBuilderFactory interface doesn't serve any purpose. In fact, it's only implemented in a single factory, and according to Coveralls it's never called in our tests. I think we can safely remove these methods from both the interface and implementations, I must have forgotten to do it when I introduced isDSLCompatibleWith.
Ultimately we should do the same for SortContributors (declare an isXXXCompatible method instead of equals/hashCode).