Set up tasks and CI job to test API/SPI compatibility on an ongoing basis.
These should be 2 (or 3) separate tasks/jobs:
Any release within a major version should have its designated API contracts tested against that major version's initial Final ("GA") release. E.g., API in 5.x versions (after 5.0.0.Final) should be tested against API in 5.0.0.Final.
Any release within a release family should have its designated SPI contracts tested against that family's initial Final ("GA") release. E.g. SPI in any 5.x.y ought to have its SPI contracts tested against 5.x.0.Final.
Ideally we'd also run an initial set of API compatibility checks when we work toward a new major release (its Alphas, Betas, CRs, Final) against the previous major release. E.g. check the API of 5.0.0 against the API of 4.0.0.
Any deviations should be either fixed or added to the release's migration guide.