BOBX-9: Investigation - BoB "major" API-version handling

Introduction

When the BoB specifications (API's & MTS's) needs changes that are not backwards compatible we need a "major" version upgrade according to the Semantic versioning rules decided in the Change management process.
Today all API's follows the same major version (v1) and there has not, until now with reference to BOBX-5 - Getting issue details... STATUS , been any request to upgrade the major version and handling of different versions between the different API's within BoB, which is not possible today.

The request is that for a decided final solution (agreed upon within BoB-tech) after this investigation that is to be able to manage different (new and old) versions in parallel. Refer to internal JIRA-issue BOB-70 - Getting issue details... STATUS

Q: Same major version for all specifications?

Would it be possible to have a mix of different versions amongst the different BoB specifications?
Would, for example a Sales Channel, be able to request a ProductAPI version 1 and use its responding manifest in a second request towards a TicketAPI version 2? It might be the case, that it is not possible due to that anything could have been changed between the two versions making the communication impossible.  

What about if the Sales Channel takes responsibility for adopting the request towards version 2 of the TicketAPI, would it then be possible?
That is probably also dependent on the type of change between the versions. The manifest used from version 1 could potentially not even be interpreted in version 2 of the TicketAPI.

Major version update frequency

A recommendation would be to keep the frequency of major version upgrades on a low level - once a year could be reasonable where, if applicable, several issues demanding a major version update are bundled into the same release. It is important to have an agreement between the stakeholders within BoB around this where a release cycle somehow is clarified, documented and managed through the BoB-documentation.

Major version support principles and scenario problems

After a new major version has been released, the implementors of that version should still support the previous version, see the already documented principles in Change Management Process.
Eventhough given the above followed principle it could potentially still lead to problematic cases when additionally a 3rd major version is released: 

  1. Suppose that changes to one specific BoB API is released with a new major version 2 .
    Participant A chooses to update directly to version 2 still supporting the old version 1.
    Participant B chooses to remain with version 1 for the time being.
    At this stage participants A and B can still communicate with each others implementations.
  2. At a later stage the same API as in previous scenario is again updated with changes to a new major version 3.
    Participant A chooses to remain on version 2 (supporting the old version 1).
    Participant B chooses to upgrade directly to version 3 (skipping version 2).
    The participants can no longer communicate with each other.

The above cases may be avoided by human communication and agreements before upgrading.

Solution alternatives

Rejected* Alternative 1: Separate ParticipantMetadata setups for different major versions - all other API's follows the version.

* Rejected as not feasible during discussions on BoB-Tech Workshop 2018-10-16 - all API-endpoints in back-end need new versions for each major update eventhough they are unaffected by the change generating waste in resource and time when updating. 

 Click here to expand...

If one would like that all the BoB-API's follows the same common major version, then one possible solution could be that it is managed through which version of the Participant Metadata that is used.

  • Each version of the Participant Metadata is seen as its own "universe".
  • In this case each endpoint would be registered separately in all the different Participant Metadata "universes".
  • You also need to decide how keys would be handled in the different Participant Metadata versions.
    • One alternative is to have them completely separated so that keys must be registered separately for each Participant Metadata version.
    • Another alternative is that keys are automatically contained in all versions of the Participant Metadata - giving that only endpoints are registered separately in each version.

Pros:

  • Since all API's are updated to a new major version when a major change occurs gives that it is clear which version of the BoB-standard that is used.

Cons:

  • The API's that has not been changed are still affected due to the versioning rule of all API's. It generates unnecessary work both for the implementor and the users of the API's.
  • All API's that are supposed to be supported in the new major version have to register its endpoints in the new Participant Metadata version.
  • All API-users that are to be using the new major version needs to request the new Participant Metadata version.
  • API-users requesting several different API's might potentially be forced to request several different Participant Metadata versions.

Example of how a few endpoint URLs for the BoB-API's could look like and be stored in the ParticipantMetadata: 

BoB-standard version 1 - current version:

Participant Metadata v1: https://bobmetadata.samtrafiken.se/api/v1

Device v1https://bob.vasttrafik.se/api/v1

Validation v1https://bob.vasttrafik.se/api/v1

Product v1https://bob.vasttrafik.se/api/v1

Ticket v1https://bob.vasttrafik.se/api/v1

Authentication v1https://bobapi.ridango.com/bob/4096/api/v1

BoB-standard version 2 - Major version changes: BOBX-5 Device API, BOBX-6 Ticket API)

Participant Metadata v2: https://bobmetadata.samtrafiken.se/api/v2

Device v2https://bob.vasttrafik.se/api/v2

Validation v2https://bob.vasttrafik.se/api/v2

Product v2https://bob.vasttrafik.se/api/v2

Ticket v2https://bob.vasttrafik.se/api/v2

Authentication v2https://bobapi.ridango.com/bob/4096/api/v2

Principles - Alternative 1:

The diagram describes the principals for solution alternative 1.

BoB-participant "Blue" has made an agreement with the Sales Channel that v2 is to be used for all API's.

BoB-participant "Blue" should still in parallell be able to support v1 since all the other BoB-participants could not be forced to upgrade to a new release at the same time.

BoB-participant "Yellow" has not yet updated to v2. "Yellow" can still remain on v1 as long as the Participant Metadata service exists in v1.
When version 3 is released participant "Yellow" needs to upgrade all services to BoB version 3. 

In the cases where the Sales Channel is doing requests towards both "Blue" and "Yellow" participants they will adressed to both v2 and v1 versions.


Alternative 2: Separate ParticipantMetadata setups for different major versions - all other API's according to specification

 Click here to expand...

The major version of the ParticipantMetadata gives the major version of the BoB-standard in total. In this case when a major change occurs it is only the affected API's that is given a new version, the rest of the non affected API's remains with the previous released version. The ParticipantMetadata version is always given a new version when a major change occurs representing a 1:1 relationship with the version of the released BoB-standard in total.

Endpoints used are registered separately in the different ParticipantMetadata versions.  

Pros:

  • Given the 1:1 relationship between the versions of the BoB-standard and the ParticipantMetadata it is clear which version of the BoB-standard that is used.
  • The non affected API's of a major change does not require a new major version.

Cons:

  • All API's that are supposed to support the new major version needs to register its Endpoints in the new ParticipantMetadata version. All API endpoints are registered in a new Major version of the Participant Metadata service - including the ones that are not changed. 
  • All users that decides to use the new major version must change their requests towards the new ParticipantMetadata version.
  • Participants doing requests towards several API's might be forced to do requests towards different ParticipantMetadata versions.

Example of how a few endpoint URLs for the BoB-API's could look like and be stored in the ParticipantMetadata: 

BoB-standard version 1 - current version:

Participant Metadata v1: https://bobmetadata.samtrafiken.se/api/v1

Device v1https://bob.vasttrafik.se/api/v1

Validation v1https://bob.vasttrafik.se/api/v1

Product v1https://bob.vasttrafik.se/api/v1

Ticket v1https://bob.vasttrafik.se/api/v1

Authentication v1https://bobapi.ridango.com/bob/4096/api/v1

BoB-standard version 2 - Major version changes: BOBX-5 Device API, BOBX-6 Ticket API)

Participant Metadata v2: https://bobmetadata.samtrafiken.se/api/v2 (version 2 is created 

Device v2https://bob.vasttrafik.se/api/v2

Validation v1https://bob.vasttrafik.se/api/v1

Product v1https://bob.vasttrafik.se/api/v1

Ticket v2https://bob.vasttrafik.se/api/v2

Authentication v1https://bobapi.ridango.com/bob/4096/api/v1

Principles - alternative 2:

The diagram describes the principals for solution alternative 2.

A new definition "BoB-release" is added to the BoB-standard which gives the "parenting" relationship for all the different API-versions that are current and included in a certain release of the BoB-standard. Read more about it below.

BoB-participant "Blue" has agreed with the Sales Channel that Release 2 of the ParticipantMetadata service is to be used. Requests is done towards ParticipantMetadata service within Release 2 which responds with the API-endpoints included in BoB release 2 - Ticket v2, Product v1 and Device v2.

BoB-participant "Blue" should still in parallell be able to have support for BoB release 1 since all the other BoB-participants could not be forced to upgrade to a new release at the same time.

BoB-participant "Yellow" has not yet upgraded to Release 2. Participant "Yellow" can still use Release 1 as long as the ParticipantMetadata service of version 1 is available. By Release 3 participant "Yellow" needs to upgrade all services to BoB Release 3.  

In the case where the Sales Channel is doing requests towards both the participants "Blue" & "Yellow" it will then be done towards "Blue" Release 2 and "Yellow" Release 1.

"BoB Release" - new definition to BoB-standard

The new definition BoB-release becomes part of the BoB-standard and the change management process and is maintained by Samtrafiken.
The purpose of the addition is to: 

  • a clearer communication of the differences between version numbers of the BoB-standard in total och the version numbers of the included API's
  • in a centralised way being able to combine different versions of API's in the same standard and by that avoiding versioning of API's that are not affected by a non backwards compatible change.

The image below illustrates three different BoB Releases over time and how its consisting API versions are changed. Red circle symbol indicates new major version within each specific major BoB Release

How the URL would be to the ParticipantMetadata service version 1:

https://bobmetadata-pp.samtrafiken.se/api/v1/participantMetadata/{pid} - v1 corresponds to BoB-release 1.

And with BoB-release 2 new URL as follows:

https://bobmetadata-pp.samtrafiken.se/api/v2/participantMetadata/{pid} - v2 corresponds to BoB-release 2.

By this URL-indication the 1:1 relationship between the version of ParticipantMetadata service and the BoB-release is more clear.

Further investigations for "BoB-release" needed?

In the design for this proposal with BoB-Release further investigations such as below could be good:

  • if subprojects should be created in BoB repository https://bitbucket.org/samtrafiken/bob/src/master/ to support individual tagging of the respective API and its corresponding version number.
  • whether the BoB release version numbering should be according to Major.Minor or simply just Major.
    • Could be good to indicate the BoB-release with a new minor numbering when bigger additions to the standard are made such as "ID based travelling" 
  • if the version numbers included in the yaml-files for each API should be according to Major.Minor or just Major
  • if there could be consequences given that the version numbering of the ParticipantMetadata service https://bobmetadata-pp.samtrafiken.se/api/v1/participantMetadata/{pid} differs from the other API's giving a mix between BoB-release and API version number. This is a deviation from the semantic versioning rules currently applied for the BoB-standard since the service major version is updated independent of a non backwards compatible change or not to the service itself. A common alternatively used way to filter resources in REST services is the use of query parameters: https://bobmetadata-pp.samtrafiken.se/api/v1/participantMetadata/{pid}?rel=1 - for BoB Release 1 and https://bobmetadata-pp.samtrafiken.se/api/v1/participantMetadata/{pid}?rel=2 - for BoB Release 2.
    • This approach gives initial impact on the existing consumers of the participantMetadata service to enable support of parallell releases of BoB. On the other hand the benefit would be that Samtrafiken is not forced to release a new participantMetadata service each time when some other service has updated its major version.


Alternative 3: Individual versioning of each BoB-API - ParticipantMetadata includes information of all endpoint versions.

 Click here to expand...

In this alternative the major version for each BoB-API is handled individually when a major change is triggered. It is then not necessary to handle a specific "parent" version for the BoB-standard

  • The Participant Metadata service does not need a new version when a major version change of some other API is triggered.
    • Endpoints are registered in one single "version" of the Participant Metadata service
      • If a major change to the Participant Metadata service is triggering a new major version of itself - the consisting participant data (endpoints) it handles remains.  
    • The Participant Metadata service needs to have the addition of several versions of each API-endpoints used for each specific participant
    • A new version attribute to the endpoint objects needs to be added.  

Pros:

  • The non affected API's of a major change does not require a new major version.
  • Only new versions of API's needs to be registered in the Participant Metadata service since that version remains. 

Cons:

  • It is not clear which versions of different API's that can be combined.
  • The API's and its users needs, as a first step, to be modified so that they can manage the new version of the Participant Metadata service along with the new version attribute added to the endpoint objects. 
  • Cases might occur where data/keys must be changed in a way that it is not possible to combine these towards several versions anymore. In this case you would probably need to create a completely new Participant Metadata anyhow.

Example of how a few endpoint URLs for the BoB-API's could look like and be stored in the ParticipantMetadata:

BoB-standard version 1 - current version:

Participant Metadata v1: https://bobmetadata.samtrafiken.se/api/v1

Device v1https://bob.vasttrafik.se/api/v1

Validation v1https://bob.vasttrafik.se/api/v1

Product v1https://bob.vasttrafik.se/api/v1

Ticket v1https://bob.vasttrafik.se/api/v1

Authentication v1https://bobapi.ridango.com/bob/4096/api/v1

BoB-standard version 2 - Major version changes: BOBX-5 Device API, BOBX-6 Ticket API, ParticipantMetadata API (endpoint)

Participant Metadata v1: https://bobmetadata.samtrafiken.se/api/v1

Participant Metadata v2: https://bobmetadata.samtrafiken.se/api/v2   (version 2 needs to be created due to the addition of endpoint version attribute in ParticipantMetadata)

Device v1https://bob.vasttrafik.se/api/v1

Device v2https://bob.vasttrafik.se/api/v2

Validation v1https://bob.vasttrafik.se/api/v1

Product v1https://bob.vasttrafik.se/api/v1

Ticket v1https://bob.vasttrafik.se/api/v1

Ticket v2https://bob.vasttrafik.se/api/v2

Authentication v1https://bobapi.ridango.com/bob/4096/api/v1

BoB-standard version 3 - Major version changes: ex. Product API och Ticket API

Participant Metadata v1: https://bobmetadata.samtrafiken.se/api/v1

Participant Metadata v2: https://bobmetadata.samtrafiken.se/api/v2 (version 2 remains with added endpoint versions from Product v2 and Ticket v3)

Device v1https://bob.vasttrafik.se/api/v1

Device v2: https://bob.vasttrafik.se/api/v2

Validation v1https://bob.vasttrafik.se/api/v1

Product v1https://bob.vasttrafik.se/api/v1

Product v2: https://bob.vasttrafik.se/api/v2

Ticket v1https://bob.vasttrafik.se/api/v1

Ticket v2: https://bob.vasttrafik.se/api/v2

Ticket v3https://bob.vasttrafik.se/api/v3

Authentication v1https://bobapi.ridango.com/bob/4096/api/v1

Principles - alternative 3:

The diagram describes the principals for solution alternative 3.

BoB participant "Blue" has agreed with the sales channel that version 1 of the Participant Metadata service is to be used.
Requests are made towards the Participant Metadata service version 1 which responds with all API- endpoint major versions registered for the participant: Ticket (v1, v3) , Product (v1) and Device (v1, v2).

Participant "Blue" needs still to support version 1 of its API's/endpoints since all the other BoB-participants could not be forced to upgrade to a new release at the same time.  

BoB-Participant "Yellow" is still running version 1 of its API's and can remain as long as the Participant Metadata service returns version 1 API-endpoint responses.

As one can see in this alternative, the Participant Metadata service is not versioned when new major versions are created for other API's.
The information of which API's/endpoints that can be used to communicate with a specific participant is completely managed by the information published for each participant in the Participant Metadata service version.
This differs from Alternative 2 in a way that the management of which version of an API/endpoint that is actually to be used is put outside of the central maintanance of the standard.


The effects on the Participant Metadata service

Both alternative 2 and 3 demands an adaptation of the Participant Metadata service managed by Samtrafiken.
The amount of effort regarding the adaptation is estimated to be equivalent in both cases - however an analysis and time estimate needs to be done.

There may as well be other unforeseen changes needed.

NOTE: For alternative 3 the existing participant metadata clients must be updated to support multiple endpoints for the same service.