Zetaly Streaming Agent Parser

The ZSA Parser was created to offer full customization autonomy to the users who want to collect, analyze and save records of data in any format, by configuring themselves how these are supposed to be transformed and saved.

The parsing configuration interface simplifies the process of configuration, by allowing you to decide how each value of your record should be transformed, either by itself or according to other values within the record, and if they are to be saved in storage or not.

It is accessible in a tab of ZSA if you have the user rights to do so (see User management).

Parser main page with two configurations already created

 

What does the parser do ?

What we define as "Parser" is a collection of configurations, each one defining how a single type of record should be parsed in order to output sensible data. In a configuration, you will detail how the record will be interpreted by defining "Fields". A field is the result of a series of "transformations" applied to the record, which will eventually produce a value that can either be saved in cold storage or reused during the parsing.

Any configuration created will be used by ZSA to parse the related records. 

Creating a basic configuration

Creating the configuration

To create a new configuration, press the "CREATE _ PARSER CONFIGURATION".

At first, only two spaces are available :

• On the left side, the description of the configuration, that will be displayed on the main table
• On the right side, the Fields space is where you will be defining how any data is to be parsed. Press "add field" to start configuring one

 First screen when creating a new configuration

 Configuring a field

A field finality is to define a data to be produced from the record. After filling the name and its description, you can start selecting the series of transformations to go from the raw record data to the value you desire.

Transformations, as their name suggest, transform a value into another, with all the transformations in a field being applied sequentially, one after the other from top to bottom. Usually starting with a selector ("Byte Range" for example), from which result is passed to the next transformation, and so on.

A field with a single transformation, selecting a range of bytes from the record

The result of the range selection will then be passed to a byte to integer converter. Our field produces an integer

 

 Once all the transformations are completely filled(No warning before their label), the resulting value format is displayed on the right of the field's name.

Field in error because a transformation is missing a parameter. The field is marked as "to be kept"

   

Once all fields are fixed, the field is considered valid and its final value format is displayed right to its name

Testing your configuration

Configuring a whole parser is not an trivial task, and you might want to test it as you progress.

To do so, you first need to effectively create the configuration, using the "CREATE" button on the upper-right (a name is required). Once it is done, a new window will be visible under the configuration's description that allows you to upload a reference file. 

This reference will be used to test the parsing and display the result as a preview of your configuration result, by clicking on the "refresh preview" button that appears once one is uploaded. This file will be saved in ZSA and doesn't need to be reuploaded every time.

Your modifications to the configuration are only saved when you click on the "UPDATE" button on the upper-right corner

This means that you can modify your configuration and test it using the preview without modifying the one currently saved, but any change will be lost when you leave the page.

Sub-window allowing to select a reference file used to test the configuration

Preview of a finished configuration, tested on a reference file

 Once you are satisfied with the preview, you can save your work with the "UPDATE" button, and start using the configuration in your collect.

Saving a field and/or reusing it

 The "keep" toggle on the upper-right of a field marks it for persistence : indeed, some data on your record might be necessary to compute other ones, but are not required to be saved in cold storage. By toggling it off, the field will be computed, but its result will only be conserved during the parsing and discarded afterwards.
You can use any field result in two ways.

The first one, by selecting the "Field from record mapper" transformation. This tells the parser to get to result from the field transformations, on to which you can chain any transformation as usual.
The second, by selecting as a transformation parameter. Every transformation offers to switch parameters origin : either "raw" (entered manually like we did until now) or "field" that expects a name of another field.
Either way, the used field must be filled and valid to be reused elsewhere.

The first field produces a value that is reused in the second one in the dedicated "Field from record mapper" transformation ...

 

 

... Or as a parameter of another transformation, using the "field" mode