The new Messaging (Streaming Agent) interfaces and functionalities

Introduction to the new interfaces

Why new interfaces?

Whether your infrastructure is composed of 5 LPARs or over 100, we offer a more comprehensive presentation through tables. Additionally, you can customize and filter these tables. We have also distinguished between parsers and connectors tables for a clearer view.

The new interfaces provided an opportunity to include additional functionalities and services: agent monitoring, autostart, automatic restart, and tags (see the tags dedicated article here). Everything is described below.

Furthermore, the new design offers a preview of what the future Zetaly suite will look like.

How to access the new products ?

 By clicking on the menu Streaming Agent / Messaging (1), you will access the new screens and functionalities (2). From there, you will be able to navigate exclusively within the new interfaces.

To return to the general and classic navigation, simply click the "Back to PORTAL" button (3). The default page is "My Bookmarks".

How to deal with both versions?

Use the classic version to continue operating as before, while enhancing your experience with the functionnalities of the latest version.

You can navigate between versions without any restrictions or loss of information because they are synchronized.

In addition to the services you already know, you can use the new interfaces for example to add the autostart function to your agents. 

Full description of the new products

The navigation

Between the 2 versions

The new interfaces are only accessible through the menu Streaming Agent / Messaging on the current Messaging interface.

Once you have completed your operations, you can return to the regular interface to continue and access the standard version of your functionalities. To do so, click the "Back to PORTAL" button, which is present on every new interface.

All your operations are synchronized.

Within the new version

Access is restricted within the new version: you can navigate to the three new Messaging interfaces (monitoring, connectors table and parsers table) as well as the tags table (more details in the dedicated article here).

New on the Messaging interface: the top navigation bar. 

Instead of one interface mixing all your agents, they are now separated into distinct pages to offer a clearer view.

Additionally, you now have access to a monitoring page focused on the queue system and the agents status. This page will soon be supplemented with additional monitoring widgets for more detailed information.

The monitoring

Dashboard

This description is a simplified version of the detailed article (here).

The main purpose of the dashboard is to warn you in case of errors so that you can take action. 

The dashboard is updated every 5 minutes and can be refreshed at any time (unlike the charts described below which are in real time).

The queue system monitoring allows you to see if your processing is operating well. When it isn't, the ratio turns orange or red, indicating that more messages are stored than processed.

The status widgets display the statuses of all your agents, with a focus on error statuses so that you can quickly identify which agents need to be restored. To avoid unnecessary manual actions, you can now activate both the autostart and retry functionalities (see below).

Connectors Charts

Two graphs show the activity of the selected connector: 

• (1) The "Usage" graph displays the average activity percentage of all instances of the connector. If the usage value exceeds 90% for more than 5 minutes and autoscaling is enabled, a new instance will be automatically created.
• (2) The "Queue" graph represents the number of messages retrieved in the last 30 seconds (enqueue) and the number of messages currently stored in the queue (current).

Parsers Charts - CSV

Four graphs help you track the performance of the CSV parser:

• (1)The "Parsed records" graph shows the number of parsed records in the last 15 seconds, and also the ignored records (due to unknown types, which is rare) and error records.
• (2) The "Inserted records" graph displays the number of records waiting for insertion and those that have been inserted.
• (3) The "Usage" graph displays the average activity percentage of all instances of the parser. If the usage value exceeds 90% for more than 5 minutes and autoscaling is enabled, a new instance will be automatically created.
• (4) The "Queue" graph displays the number of processed messages (dequeue) and the number of stored messages (current).

Parsers Charts - RAW

Five graphs help you track the performance of the RAW parser:

• (1) The "Parsed records" graph shows the number of parsed SMF in the last 15 seconds, and the errors too.
• (2) The "Queue" graph displays the number of processed messages (dequeue) and the number of stored messages (current).
• (3) The "Inserted records" graph displays the records inserted into the database, and error records and also the records without a parser or records with multiple parsers.
• (4) The "Usage" graph displays the average activity percentage of all the instances of the parser. If the usage value exceeds 90% for more than 5 minutes and autoscaling is enabled, a new instance will be automatically created.
• (5) The "RAW records bunches" graph displays the number of processed groups of messages (dequeue) and the number of stored groups of messages (current).

Functions common to all charts

You can zoom in on the charts to focus on a specific timeframe and better visualize the data.

Additionally, you can also enlarge the chart to full screen using the dedicated icon at the top left of the chart.

Finally, you can select or deselect the elements to be included in the graph by clicking in the legend.

The tables

Connectors and parsers are now separated into two dinstinct tables. All the information is still present but displayed differently. Each row is expandable and contains both specific statistics (described above) and logs.

The main information is displayed in the different columns. 

  (1) You can customize these columns using the dedicated functionality:  choose whether to display a column and in       what order, so that the table meets your needs.

  (2) If too many agents are displayed, we have added an expandable search bar that allows you to quickly find your agent in the   table. You can search for the name of the connector, its host, or the tag if one or more is assigned.

  (3) If the search bar is not sufficient, you can also apply filters.

Filter the tables and the monitoring dashboard

Dedicated filters help you select relevant information so that the table only displays the related information (data, connector or parser). 

You can make your selection by choosing the Host, CSV or RAW type, a specific status, or even by tag specification.

The same filters are available in all three Messaging interfaces.

Tag assignment

The tag system has been implemented to help you further customize the identification of your connectors. This allows you to be more specific in qualifying your connectors. More information about how to manage tags can be found in the dedicated article here.

Once created and active, tags are available to be assigned. 

You can assign tags either at the creation or during the editing of a connector. There is a dedicated tab in the forms.

A tag can be assigned to several connectors and one connector can have multiple tags. Every tag you create appears in the filters if it is active.

Mass actions

To help you manage a potentially large number of connectors, we implemented the mass action functionality.

First, you need to select the connectors in the table: checkboxes appear when hovering over the rows, allowing you to select them. You can select all by checking the box in the table header.

The possible actions are available in the top bar of the interface through icons:

• (1) Start all selected connectors
  • If some of the selected connectors are already running or ready, this action won't affect them but will start all the stopped ones.
• (2) Stop all selected connectors
  • If some of the selected connectors are already stopped, this action won't affect them but will stop all the running and ready ones.
• (3) Assign Tags to all the selected connectors

• Assign tags: 

In the dialog, the left box lists all available tags or the tags not yet assigned to all (a) the selected connectors. 

This is a picklist, so you can select one or more tags to assign them to the selected connectors by clicking on the corresponding arrow (b).

Confirm by clicking on Edit.

• Deassign tags: 

All tags assigned to every single selected connector appear in the right box of the dialog (a). 

To deassign from all connectors, click on the tag(s) to select them and then click on the corresponding arrow to deassign (b).

Click "Edit" to confirm.

Click "Reset" to revert to the original state.

You can also undo or redo your last action.

• (4) Delete all the selected connectors
  • A deleted connector is first stopped. Its data remains available in the Data Explorer but won't generate any new data.

Autostart, retry and autoscale functionalities

When creating (and also editing) a connector you can now:

• (1) Activate the autostart function
  • This functionality automatically launches the connector at creation and in the event of machine outages.
  • Note: If you manually stop a connector in the Messaging interface, the autostart function won't be effective. It only works when it is active and in case of external interruptions.

• (2) For both connectors and parsers you can also set up a number of retries and their delay (in seconds) in case of failure.
  • Note1: This won't apply in case of shutdown performed by the user. 
  • Note2: If you don't want to activate this functionality, simply enter "0" in the number of retries field.

• (3) Autoscale Functionality for Connectors and Parsers

• (3) Both connectors and parsers benefit from the autoscale functionality. It is deactivated by default but can be activated in the Properties section of the Edition dialog.
• This functionality automatically creates a new instance when the Usage Value (4) reaches 90%.
  • Meaning that all instances of your connectors and parsers collectively reach an average of 90% usage during the last minute.
    • Instances are checked every 30 seconds.
  • When the Usage Value drops to an average of 50%, an instance is automatically removed.
  • The number of instances ranges between 1 and 200 (maximum for all connectors and parsers).
  • The number of instances cannot exceed 10 per connector or 10 per parser.

Reminder: Status

• Running: the item is currently working
• Ready: the item has been started but there are no messages to process
• Starting: a transitional state lasting a few seconds, indicating that the item has been launched (the next state will be either "running", "ready" or "error")
• Stopping: this a transitional state lasting a few seconds, indicating that the item has been stopped but is still processing the last messages in the queue (the next state will be "stopped")
• Stopped: the item is inactive
• Error: the item has encountered an error - check the logs for more information
• Degraded: at least one instance of the connector is in "error", but not all the instances