SMF 120-11 Websphere Liberty (z/OS Connect)

Introduction

There is an IBM document that illustrates the importance of Liberty SMF records (120-11) and their history https://www.ibm.com/support/pages/system/files/inline-files/WP102655_Liberty_SMF.pdf

A Brief History of SMF Records in WebSphere Liberty

For quite a while after the initial release of Liberty the server didn't cut any SMF records itself at all. If you needed visibility to activity inside the server via SMF you needed to enable the z/OS WLM feature and assign transaction class names to incoming HTTP requests. That would cause a WLM enclave to be associated with the dispatch of the HTTP request. Ultimately that would be reflected in the SMF 72 records written by RMF and often made readable using the RMF Workload Activity Report (or similar vendor product).

That's enough to get you things like request counts and average response time and CPU time. But it doesn't tell you a lot of specifics about what actually ran. Even if you got a little carried away putting different URIs into different report classes, it is still more of a summary of what happened. That's often enough, but sometimes, especially when things are going badly, you'd like some more detail.

The first hint of SMF support in Liberty came as part of the z/OS Connect Liberty feature. When it first appeared, z/OS Connect was just another feature available as part of Liberty (and again, if you aren't familiar with z/OS Connect, this isn't the place to explain it..there's lots of material out there to explain how it fits into API Management and so forth1 ).

One of the core capabilities of z/OS Connect is the option to have interceptors get control around the dispatch of a z/OS Connect REST request. The interceptor API is documented and so you can write your own, but IBM thought it would be a good idea to provide a few basic ones to get things started. One of the interceptors that shipped with the z/OS Connect feature was called the Audit Interceptor.

The z/OS Connect Audit interceptor got control before and after dispatch and grabbed some useful information about the request and wrote an SMF record for it. It wrote a Type 120, subtype 11 record. The Type 120 record is the WebSphere SMF record and WebSphere traditional used subtypes up through 10, so that seemed the right thing to do.

The 120-11 record written by the Audit Interceptor had just two sections. A server identification section and a user-data section. The server identification section obviously had stuff in it to help you identify which server wrote the record. The information about the REST request went in the user-data section. But why would it be user data? The WebSphere traditional 120-9 record has a user data section that applications can use for their own information. But the Audit Interceptor is part of z/OS Connect which is part of Liberty..why would it be user data?

Well, user data comes with an integer tag to identify what user data it is (to help formatters figure out what to do). And the user data in the 120-9 record reserved a range of tag values for IBMs own use. To an application server, any application, even one written by IBM, is still an application and could have user data. The thought was that IBM-written applications might want to put their own information into the WebSphere SMF records and this gave us a way to do it without colliding with any customer/vendor usage of the user data API.

But still...z/OS Connect is part of Liberty, so it isn't an application, is it? Well, it was packaged as part of Liberty so kind of no, but yet it was really a servlet processing REST requests so also kind of yes. The hope was that eventually Liberty would write a real SMF record for HTTP requests with all sorts of good things in it and the Audit Interceptor could be updated to just call the user data API like any other application and add its specific stuff into the record. That's why the Audit Interceptor didn't include a lot of things in the record that you'd probably really want for a general SMF record for an HTTP request. More was coming...we thought.

As it turned out, the z/OS Connect feature in Liberty evolved into a product called z/OS Connect Enterprise Edition Version 2.0. And that product has all sorts of cool things in it way beyond what the z/OS Connect feature in Liberty has. And along the way the Audit Interceptor got rewritten to write its own SMF record that wasn't tied to the WebSphere record type (since z/OS Connect EE was its own product and not part of Websphere). The z/OS Connect EE V2.0 SMF record is the Type 123 and we're not going to talk about that anymore here.

But we still needed an SMF record for requests processed by Liberty that weren't related to z/OS Connect. And in Liberty 16.0.0.2 we finally got it done. We made a lot of enhancements to the record and, because of that, we changed the version number. The SMF 120-11 Version 1 is the record written by the original z/OS Connect V1.0 Audit Interceptor. The SMF 120-11 Version 2 is the new record written by Liberty for any HTTP request. In summary:

And that's what we're going to talk about here.

Record Enabling Liberty SMF Recording

In WebSphere traditional to enable SMF recording you needed to set some configuration options for the server and make sure the SMFPRMxx parmlib member you were using allowed SMF 120 records to be recorded (and that you didn't have an SMF exit suppressing them...).

WebSphere Liberty is the same (SMFPRMxx allows SMF 120, and no SMF exit suppressing them, and is also a bit different. Like pretty much everything in Liberty, SMF recording is an optional feature. If you add the zosRequestLogging-1.0 feature to the featureManager clause of your server.xml the server will try to write an SMF record for every HTTP request.

But that's not all you need to do. Writing SMF records requires the calling program to be APF authorized. However, a Liberty server is generally not running as an APF authorized program. We could have made use of the Angel to get authorized (as Liberty does for other system functions). But we decided we didn't want to require the Angel to get SMF records. WebSphere traditional didn't have this concern since SMF records were written by the controller which always ran APF authorized.

Fortunately for Liberty, LE/USS provides an smf_record( ) API that can be called by unauthorized programs to write SMF records. To make this work the identity of the server has to be permitted READ access to the BPX.SMF entity in the FACILITY class:

Therefore you also need to ensure that whatever userid (or group) the server is running under is granted this access. If the server is a started task then the started task identity is the one you want. If the server is started from the shell using the 'server' command then it will be the userid of the shell.

IBM® WebSphere® Liberty is a Java EE application server with a low-overhead Java runtime environment for cloud-native apps and microservices. WebSphere Liberty was created to be highly composable, start fast, use less memory, and scale easily. Using the open-sourced IBM Open Liberty (link resides outside IBM) code base provides low-cost experimentation, customization, and seamless migration from open source to production.

zOS Connect is a product used to make easier the interface between user applications and mainframe, and auditing is made through SMF record 123, but zOS Connect is also a software function written by IBM that runs inside an instance of WebSphere Liberty Profile z/OS and uses existing connector technology to get to the backend systems.

Since Liberty is an essential part of zOS Connect, the SMF data it produces is relevant to better understand zOS Connect behavior and performance.

Fields

Stored in wspLibData