Move testing of non-default environments to a separate job
Description
relates to
Activity
Yoann Rodière August 28, 2023 at 3:20 PM
Closing after discussing this with .
Since we now have incremental builds thanks to , and since nowadays Jenkins PR builds report the status of each build stage on the PR as soon as the stage completes, the changes suggested in this ticket would bring close to nothing to PRs: their builds are both faster to run and more quickly reported.
For main builds, TBH the problem is not so big, and the main advantage of this move would be to convert more of our pipelines to Jenkins' “declarative” syntax instead of the confusing “script” syntax. So we’ll work on converting the syntax of the main Jenkinsfile instead, though not now since it’s not urgent.
Yoann Rodière August 24, 2023 at 12:27 PM
To be considered as it seems relevant to this issue:
I believe this will lead to rebuilding modules we already built in the previous stage, e.g. the
backend/elasticsearchmodule...Couldn't you just split the stages all the time, even for non-incremental builds? You'd rename this stage
Default environment ITs, remove theif ( incrementalBuild )just above in this stage, and useif ( enableDefaultBuildIT )just above in this stage instead ofif ( enableDefaultBuild ).Then you can use
-f integrationtestshere and remove the${enableDefaultBuildIT ? '' : '-DskipITs'}.
Oh, but that would require running unit tests in the first stage and not here, and then they wouldn't be included in Sonar test coverage reports... That would be annoying.
I wonder what happens if we just execute Sonar in every single stage... Would Sonar aggregate the results? 🤔 If so that would be really great as we would also be able to execute sonar in (BEFORE_MERGE) non-default environements and would get test coverage numbers that more closely reflect reality.
Yoann Rodière January 16, 2023 at 10:39 AM
It might also be a good idea to split the job into multiple jobs, one for each of the following categories: Compilers/JDKs, DBs, Elasticsearch/Opensearch. So that when a particular environment fails, we only need to re-run the build for the related environments (no need to re-test OpenSearch 1.3 if the build failed only against DB2, for instance).
Details
Assignee
Yoann RodièreYoann RodièreReporter
Yoann RodièreYoann RodièreComponents
Priority
Major
Details
Details
Assignee
Yoann RodièreReporter
Yoann RodièreComponents
Priority
Major
The number of environments we have to test is getting wild, and it’s taking a lot of time.
I think we should consider only testing the default environment in the main build: latest LTS JDK, latest Elasticsearch, no Opensearch, a single DB (H2 or maybe postgresql).
We would move testing of other environments to a separate Jenkins job (
hibernate-search-matrix?hibernate-search-environments?).Ideally, we’d take inspiration from
ci/dependency-update/Jenkinsfilefor that job, because it’s using the declarative syntax for Jenkinsfiles, and believe me that’s way easier to debug.As to the specifics of that job:
It would have to cover everything from the “non-default environments” part of the current main Jenkins job.
We’d forbid concurrent builds, similarly to other jobs, but probably wouldn’t cancel running builds.
When running against main branches (not PRs), it would have to be rate-limited (throttled), similarly to other
ci/dependency-update/Jenkinsfile.It would probably need some form of parameters, to be able to run tests against a particular platform easily. I’d argue that using only some regexp condition would be fine. We can definitely simplify what we’re currently doing in the main Jenkins job.
It would run against branches, but also against PRs. At the very least it should test PRs against a subset of all environments (currently I think we do Eclipse compiler, Oracle, PostgreSQL and that’s it; maybe we’d want to add the latest early-access JDK).
Ideally we would add some code to detect when changes in a given PR have no impact on a given environment and thus skip some environments. E.g. if there are no changes in
hibernate-search-backend-elasticsearchor any of its dependencies (engine, …), then there’s no need to run tests against all flavors of OpenSearch. That would allow us to expand a bit the subset of environments we test against PRs. But maybe that should be a separate Jira issue?