Messaging at its best – IBM MQ with REST – From the Depth of Integration

Introduction

Many MQ developers and administrators have noted the REST Messaging abilities provided with IBM MQ 9.0.4+ CD and IBM MQ 9.1 LTS. MQ Web Service (REST/SOAP…) Messaging is not something new. Some time ago the IBM MQ Bridge for HTTP – as a component of IBM MQ Transport for SOAP – met this requirement. It is now deprecated however. The details can be found at https://www.ibm.com/docs/en/ibm-mq/9.0?topic=900-deprecated-stabilized-removed-features#q127140___trans4soap. Additional means making it possible to connect to IBM MQ using REST can be found at: https://www.ibm.com/docs/en/ibm-mq/9.2?topic=applications-developing-rest-mq.

It should however be noted, that even though a number of approaches are mentioned on the aforementioned link, the only available means of actually sending HTTP messages directly to the IBM MQ product is by using the previously mentioned two techniques: IBM MQ Bridge for HTTP was implemented as a Java EE application in an application server – and the IBM MQ REST interface – together with the MQ Web GUI, introduced in MQv9 – are implemented as web applications in a bundled IBM WebSphere Liberty Profile Instance. This feature is known as the Web Administration feature during installation and it must be installed together with the Extended Messaging feature.

It should be mentioned that IBM states that: “From IBM MQ Version 8.0, the IBM MQ Bridge for HTTP is deprecated. The IBM messaging REST API provided from IBM MQ Version 9.0.4 should be used as an alternative.”

This blog entry will focus on what to think about when utilizing the MQ REST APIs. The foundation for this task is to acquire a Swagger definition for the relevant API subset. This Swagger definition can be used both to access MQ directly via test tools like SoapUI, but also by creating IBM MQ REST messaging applications. The areas illustrated will be:

Acquiring the Swagger details of the IBM MQ Messaging REST interfaces.
Call API’s from tools like cURL and SoapUI
Create a small application, which will login to IBM MQ and use the resulting Light Third Party Authentication token in subsequent send/read calls. To alleviate the effort we will make use of the Eclipse WDT Swagger->Java wizard in order to minimize any manual effort. In addition it will be highlighted that it is necessary to add extra functionality to the resulting code in order to accommodate all authentication mechanisms.

REST and Messaging

One of the advantages of bundling the IBM MQ REST functionality in IBM Liberty is that it is possible to make use of the Liberty apiDiscovery feature. This features enables discovery of HTTP REST APIs in deployed Liberty Artifacts. There are different approaches on how to assist the discovery features to find IBM WebSphere Liberty API’s to be used by client applications:

One of the approaches to notify the discovery features about available REST APIs is to use standard Swagger and jax-rs annotations when writing new functionality. Common examples of such annotation could be @POST, @ApiOperation, @Consumes, @Produces, @ApiResponses and @ApiResponse.
Another approach is to for example expose Servlet functionality for REST calls, without having to change any Java-code at all, is to create a swagger file in the META-INF directory with the proper interface.

From Liberty 17.0.0.3 and later it is possible to use the openapi-feature instead, which is suggested as the successor to the apiDiscovery feature. The Liberty version provided in IBM MQ 9.0.4 is 17.0.0.2 and thus one would need to settle with apiDiscovery.

It could be mentioned that at the time of writing, the APIs available in z/OS Connect (which also is able to use an MQ Service provider, thus enabling HTTP APIs to access IBM MQ) do not make use of the apiDiscovery functionality, despite of them running in a bundled Liberty. Instead they are available using a separate interface. IBM MQ on the other hand makes full use of the discovery features and if one is to be a serious IBM MQ REST Messaging user, it is highly recommended to update the mqwebuser.xml file and add this feature to the others:

After starting and verifying the proper REST URL, the Explorer can be opened using the standard Context root used by discovery options. The default Context root can be replaced by other values should it be required. In order to ascertain user name and password – to gain access to the IBM API Discovery APIfunctionality (/ibm/api/docs|explorer) – it is recommended to look at the mqwebuser.xml configuration file.

In the Explorer all exposed REST API’s in the Liberty Server will be visible. By omitting the initial “/ibm” in the Explorer URL, the API Discovery API will not be visible in the Explorer.

Acquiring Swagger for MQ Operations

What we want to do is to:

call the MQ login API
use the two MQ Messaging APIs: DELETE for a destructive read and POST to put a message on a queue.

In order to use the API Discovery API to acquire the proper Swagger representation of just these three interfaces, we would typically go into API Discovery (default URL: https://localhost:9443/ibm/api/explorer/#!/API_Discovery/getDocs), pick the /ibm/api/docs/ API GET operation and specify the two context roots that would be of interest:

A resulting URL will be provided when trying out the API – and if applied in a web browser the proper Swagger representation for logging in to Liberty and to send/read messages would be acquired. An example URL could possibly be: https://localhost:9443/ibm/api/docs/?root=%2Fibmmq%2Frest%2Fv1%2Flogin&root=%2Fibmmq%2Frest%2Fv1%2Fmessaging%2Fqmgr%2F%7BqmgrName%7D%2Fqueue%2F%7BqName%7D%2Fmessage&compact=false.

The Swagger can be saved to the operating system. It will only contain the interfaces for logging in and for messaging operations, since these are context roots we specified. Thus it is possible to limit the size and amount of information to the smallest amount necessary. The easy way to direct the Swagger into a file is to just copy and paste from the web browser window into an editor of choice. An alternative could be to run a command line cURL and redirect the output directly to a file.

Using SoapUI to Call the REST Service Based on the Generated Swagger

There are of course a number of ways to demonstrate and test the MQ REST functionality. The easiest way is of course to run them directly from the API Explorer in the MQ Liberty instance. This environment is perhaps not ideal when it comes to managing LTPA-tokens and other headers. Alternatives could involve using curl directly – like for example using the below command to send a message to queue qDest on queue manager qm1 using Basic Authentication.

Other alternatives could involve using SoapUI – which typically is particularly good at SOAP based Web Services, but is able also to manage REST requests and Swagger files. By saving the generated Swagger information to a file – it is possible to import the Swagger into an empty project – and the result using the previously created Swagger definition would look something like in the picture below.

Obviously only the login and messaging operations have been included. In the example below we will use the SoapUI getLogin operation with Basic Authentication (default mqadmin/mqadmin) to ascertain our login status. An alternative would have been to send a login json POST message without basic authentication and use the resulting LTPA-token in a subsequent GET request, but the basic authentication mechanism will prove the point.

Note: – in order to allow Soap UI to work well with the MQweb Liberty instance – it is a good idea to a) import the Liberty certificate to the SoapUI truststore and b) make certain that the URL indeed is based on HTTPs, instead of the default HTTP.

The resulting json assures us that we are using basic authentication and that we are logged in as mqadmin. It also tells us that we are a member of the MQWebAdmin and MQWebUser roles. Role information can be altered using the mqwebuser xml file. Information about what groups a user belongs to can be valuable for example when ascertaining that a profile making use of the Messaging API indeed belongs to MQWebUser.

If making use of the token based API login (postLogin) operation and sending a login body:

{
  "username":"jah",
  "password":"xxxxx"
}

… it should be noted that the API returns an LTPA-token as a part of the Set-Cookie-header. This token is to be saved and reused for future token based requests. There could be instances when it might be difficult to save the LtpaToken2 cookie, where statelessness is of essence. For those instances alternative authentication methods should be considered.

Set-Cookie: LtpaToken2=pJZIF/XNrcjjIkTef6MeHKIPPY9/TIdGTm06cVevFFeRI0aoQkxiorysKInjFI6q4zs9dJqc8ChkJC+LDbrOxCpz3DoOElxE8besSmOCX7p5tlXiTRottIM4qJy1kglKlpXI423cwveCEoyY4u9hkqY3ZzoQORPh3Bu0h5HwWG0PpRCTlp5VLeshUV0mWxB0yZ6kDsUL3P9kZ9X8M4bi0NN8oLF7Bm3vVyED0Q6OPBh7crahFPmz7ArqoqNzxG3RbuLuso0pWXBYAdOXIh+zV+99wn37LqJCE0ovs6+rvE+71MkUBoC2ixjf0RuY/PP5; Path=/; HttpOnly

These Cookies can – and should be used – on subsequent token based Messaging operations such as POSTing a message – again using SoapUI – by specifying the “Cookie” header to the aforementioned retrieved Light Weight Third Party Authentication token value. In the example below everything went well and we received a 201 successful HTTP return code.

Generating an MQ Java REST Client Based on the Generated Swagger

So – what are the possibilities to use the resulting Swagger to easily create an MQ Java REST Client making use of the published APIs

If we used IBM WebSphere Liberty, WDT and created a new Eclipse Project, it would be possible to select the option to generate a JAX-RS Client from the saved file. This will result in a file structure looking similar to below:

As can be seen in the api package, one source file has been created for using the LoginAPI and another one for using the MessagingApi. What one needs to do from here is to create a small sourcefile making use of the generated functionality – and the interface to appropriate jax-rs classes. It should be mentioned here and now, that the generated code is not complete for IBM MQ Messaging authentication usage. As discussed later, we will need to access cookies generated by the login API – and this is not supported by the generated code. In addition the generated code is not sufficient to access messages read using the DELETE operation (for MQ destructive reads). However – it should be mentioned – that the code is well structured and highly modularised and thus not very complicated to maintain.

Authorization and Authentication

There are a number of ways to authenticate to the REST (and IBM GUI) mqweb functionality. This has been discussed in a number of KC-items as well as blogs. What is important is that just because an application has properly authenticated using a certificate, basic authentication or a token based approach, does not mean that it can freely start submitting and reading messages. The connection to IBM MQ will use the user associated by the login – and if authorization is lacking – well it will simply not work. In order to verify the examples in this blog entry, I a) created an OS user, b) added that same user to the Liberty Basic Registry and c) applied setmqaut to give the proper authorizations:

 <user name="mquser" password="xxxxx"/>

Using the token-based approach is a bit interesting. When performing a logon to mqweb, in IBM MQ 9.0.1 – 9.0.4 two cookies will be returned:

*** Name: csrfToken ***
*** Value: xxx ***
*** Name: LtpaToken2 ***
*** Value: yyy ***

The LtpaToken is to be submitted in subsequent messaging requests to prove that the application is authenticated. Up to IBM MQ 9.0.4, a valid csfrToken was to be provided as a header field to all messaging operations as well as a means to confirm that the credentials that are being used to authenticate the request are being used by the owner of the credentials. That is, the token is used to prevent cross-site request forgery attacks.

The functionality of the csrfToken is removed from IBM MQ 9.0.5 (and LTS 9.1). It must still be specified, but the value will not be checked or validated.

Thus: what an application should do, is to save the LTPA token resulting from the login request and supply it in subsequent requests as a cookie and – before IBM MQ 9.0.5 – supply the retrieved csrfToken in all non-GET operations.

This is where the generated Eclipse code in the previous chapter is a little bit at loss. Functionality needs to be added both to acquire received cookies and to set the LtpaToken properly. A suggestion how to do this could involve the following changes in in APIClient.java:

import javax.ws.rs.core.NewCookie;
private Map<String, NewCookie> cookieMap;
private Map<String, NewCookie> requestCookieMap = new HashMap<String, NewCookie>();

/**
   * Gets the response cookies of the previous request
   */
  public  Map<String, NewCookie> getResponseCookies() {
    return cookieMap;
  }

  /**
   * Sets the request cookie
   */
  public ApiClient  setRequestCookie(String sKey, String sValue) {
           requestCookieMap.put(sKey, new NewCookie(sKey, sValue));
    return this;
  }
// Before invoking API:
for (String key : requestCookieMap.keySet()) {
         String value = requestCookieMap.get(key).getValue();
         if (value != null) {
                 invocationBuilder = invocationBuilder.cookie(key, value);
         }
    }
// After invoking API:
cookieMap = response.getCookies();

Using Basic Authentication is a fair amount easier using the generated code – since there are already place holders for this functionality, where using certificates for authentication purposes would require some work.

Messaging

The messaging APIs however work mostly well – and using this approach to create Java MQ Rest Clients based on Swagger specifications is quite convenient. Methods will be created in the APIClient java file and once the authentication is done with typically the code can look like:

try {
         lapi.postLogin(cpo, authNames);
         if (apic.getStatusCode() < 300) {
                  MessagingApi mapi = new MessagingApi(apic);
                  manageResponse(apic);
                  System.out.println("*** Setting Request Cookie: " + LTPATOKEN + " : " + ltpaCookie.getValue());
                  apic.setRequestCookie(LTPATOKEN, ltpaCookie.getValue());
                  System.out.println("*** About to send message! ***");
                  mapi.sendMessage("qm1", "qDest", "this is a test message", csrfToken, null, null, null, null, authNames);           
                  manageResponse(apic);
                  System.out.println("*** About to receive message! ***");
                  String sData=mapi.receiveMessage("qm1", "qDest", null, null, "0", csrfToken, authNames);
                  System.out.println("*** Read Data is: " + sData);
                  manageResponse(apic);
}
} catch (Throwable t) {

         t.printStackTrace();
}

One should remember that HTTP is not transactional – and thus the same QOS as the native MQ protocol would provide is not applicable in this scenario. A client can from each operation verify the HTTP return code and ascertain what the next step would be. The possible return codes are themselves documented in the Swagger and based on that the client knows what to expect.

In order to make certain that messages are not lost when during a message read if a system failure occurs, The REST APIs are however based on a syncpoint operation and is managed as follows:

Receive the REST GET request from the REST-client
Get the message with syncpoint GMO-option
Return the message to the REST client
Perform an MQCMIT (Commit) or equivalent

It is also important to know that the use of MQ Web Roles differ between using administrative APIs and when using the messaging APIs. Using the messaging API, the logged in user needs to belong to the MQWebUser group and it does not matter what user runs the MQ REST API Client or who has started the MQWeb Liberty instance – the logged in user will accompany the message all the way to the queue.

This allows for any user logging into MQWeb, using either of the three approaches, to maintain a separate set of authorizations and it would also allow a user granularity on audit trails, properly reflecting the logged in user’s activities.

Summary

There is plenty of information available about the 9.0.4+ MQ REST interface. There is also plenty of information available about how to use the apidiscovery/openapi features of IBM WebSphere Liberty Profile (WLP).

This blog entry focuses more on how to use the Liberty API Explorer functionality to acquire the Swagger specification of two of the available MQ APIs: Login and Messaging.

The acquired Swagger information is:

imported into SoapUI and examples show how to authenticate (either using Basic Authentication – or token based authentication before actually submitting a message using token based authentication, specifically adding the received LTPA token cookie to the request header
imported into the Eclipse WDT Rest Client Wizard and used as a foundation for creating a Java MQ REST client (typically a Servlet) to be deployed on WLP. Typically, additional cookie based token functionality needs to be added for authentication. Basic authentication support is to a high extent already implemented in the generated Java code, whereas support for certificates was missing.