Load Views: Aggregate data (deprecated)

Prev Next

The LoadViews is a batch process to load imported data into the historical tables. This process is usually done once a day and can be automated as well. The following diagram gives an overview of the process:

 

The LoadViews is done through the Zetaly Portal. However, you can save a Load configuration on the ZetalyDB and automate its execution through the API.

ZSA Portal and Linux version
The ZSA portal is still under development. All the Load views functionnality are not accessible from the Zetaly Portal and needs the former Zetaly Explorer client (compatible with Windows only). See the legacy documentation if needed.

Creating a LoadViews configuration

If you can execute database scripts, you may execute the AddLoadViews.sql script located into the LoadViews.zip folder into the Scripts folder: 

 

Testing the Load Configuration 

Zetaly Portal Load Views
All the Load Views should be launched through the Zetaly Portal. This is safer than the Data Loader execution.

The Zetaly Portal gives you access to the Load Views process and their status:

Only one can be launched at a time.

  1.  LoadViews information. When started, it displays the current task.
  2.  LoadViews status:
    • STOPPED: The LoadViews is not running and did not terminate with an error.
    • ERROR: The LoadViews is not running and terminated with an error.
    • STARTING: The LoadViews initialize everything it needs before starting to aggregate data
    • STARTED: The LoadViews is running and aggregating data
    • STOPPING: The LoadViews is stopping
  3. LoadViews action:
    1. Start the LoadViews
    2. LoadViews logs
Zetaly Portal does not allow to stop a LoadViews process. Stopping a Loadviews can lead to corrupted data

LoadViews Process

The LoadViews process can be summarized in the four steps below: 

PhaseDescription
Stop messagingsStop all messagings to pause the importing of mainframe data on the import tables
Load viewsLoad history tables based on the data present on the importation tables
Truncate dataClear the importation tables because the data has already loaded on the history tables
Start messagingsResume the import of mainframe data

When you launch a LoadViews process through the Zetaly portal, the messagings are automatically stopped and started before and after the process.

Enable the swap option

The ZETALY messaging needs to stop loading data on 'xxxData' table to load the data to a historical table. This process may take a few minutes and during this time, there will not be real time processes in the table. 

To minimize downtime, you can now enable the swap option (disabled by default). It is located in the configuration of each load view (cog icon on the card) and also when creating a new load views. Simply tick the box to activate this option and save.

What the Swap data does is rename the 'xxxData' table to 'xxxData1' and the existent 'xxxData1' table to 'xxxData'. It allows the ZETALY messaging to keep loading new data on the table. All this process is very fast, and the sequence is:

  1. Stop importing from xxxData
  2. Rename xxxData to xxxData1
  3. Rename xxxData1 to xxxData
  4. Start importing to xxxData

Now, you can execute loadviews from xxxData1.

Loader & Compress fields Configuration

The default configuration for aggregating your dataset includes all fields. You can customize this in the global load views configuration (cog icon at the top of the load views page).

Once you've chosen your product and target (1), you can select the relevant fields based on your specific needs by checking or unchecking them according to your preferences (2).

Once you're done, click the save button.

Scheduling a LoadViews Task 

Please note that...
... launching multiple LoadViews in parallel is not recommended. When a LoadView finishes, all agents are automatically restarted, which can lead to inconsistencies in the data of LoadViews that are still running.

Scheduling feature on the LoadViews interface

Schedule your LoadView by accessing the scheduling feature through the clock icon on the LoadView card. You have two choices:

  • Standard setting: Schedule the LoadView to run every day at a chosen time you have to set (and remember to save).

  • Alternatively, set a specific cycle using the advanced setting:



Define your cycle using the CRON language.

If you are unfamiliar with it, refer to the provided sentence that translates your choices into a comprehensive statement. Remember to save your configuration.

If you've already established an advanced scheduling, you can:

  • Edit it by modifying the fields and save your updated choices.
  • Delete it (denoted by a red trash icon [a]).
  • Add a new schedule rule to be cumulated with the existing one (indicated by a "+" symbol [b]).





It is also possible to call the ZSA API to launch the LoadViews process.

Windows scheduler

You can use Windows Task Scheduler to automate your LoadView task. All you need is the possibility to call the ZSA url.

For this example, we are going to use the curl library.

 1) Create a new script called "loadviews.cmd" with the following line

curl -k -X POST https://ZETALY_HOST/zsa/api/v1/loadviews/LOAD_VIEWS_NAME/start

Replace ZETAY_HOST by your zetaly host, and LOAD_VIEWS_NAME by the LoadViews name to launch.

2) Create the task

Run the following command in an Adminstrative console: 

schtasks /CREATE /TN ZetalyLoad /RU "SYSTEM" /SC daily /ST 05:00 /TR "PATH_TO_SCRIPT\loadviews.cmd"

Replace PATH_TO_SCRIPT with the folder used to create the script at 1)

You can replace the /ST and /SC parameters by the hour and periodicity you want. See windows documentation for more details.

Docker (crontab-ui)

Get the image

To be able to schedule loadviews using a docker container, Zetaly recommand to use crontab-ui

If needed, Zetaly provides a s390x image for ZCX : 


Load the tar file:

docker load -i crontab-ui.tar.gz

Run the container

Once the image is available on your docker, you can run a container with crontab-ui:

docker run -d -p 8000:8000 alseambusher/crontab-ui

Scheduling

At the top of the ui, use "New":


Create a job with the desired sheduling. This job should run first following command line:

token=$(curl -k -H "Accept: application/json" -H "Content-type: application/json" -X POST https://ZETALY_HOST/zhb/api/v1/users/public/login -d '{"username":"USERNAME","password":"PASSWORD"}' | python3 -c "import sys, json; print(json.load(sys.stdin)['token'])");

Replace ZETAY_HOST by your zetaly host, and USERNAME/PASSWORD by your user credentials.

Technical User
It is recommand to create a specific user to execute this script. If a generic one is used, it will deconnect it.

After the first line, add the following command line:

curl -ki -H "token: $token" -X POST https://ZETALY_HOST/zsa/api/v1/loadviews/LOAD_VIEWS_NAME/start

Replace ZETAY_HOST by your zetaly host, and LOAD_VIEWS_NAME by the LoadViews name to launch.

Host ip
Docker provides the host named "host.docker.internal" if you want to refers to the docker host inside a container.

Example, every day at 02am:

Once the job is created, you can try it:

When the job is working, you can save it to cron:


Scope

In the historical tables resulting from the LoadViews process, a "scope" column is added. It identifies the type of aggregation performed. Below are the different possible values:

ValueScopeDescriptionPériodicityScope Event
10MinuteEndedAll events completed at tbe same minuteMinuteFinishedAt
9MinuteFullAll events start and ended at the same minuteMinuteOnThePeriod
55MinuteIntervalInterval events byminuteMinuteInterval
12TenMinEndedAll events completed at tbe same 10-minute intervalTenMinuteFinishedAt
11TenMinFullAll events start and ended at the same 10-minute intervalTenMinuteOnThePeriod
56TenMinIntervalInterval events by ten minutesTenMinuteInterval
2HourEndedAll events ended in the hour intervalHourlyFinishedAt
1HourFullStarted and ended at the same hourHourlyOnThePeriod
51HourIntervalInterval events by hourHourlyInterval
4DailyEndedAll events ended at a specific dayDailyFinishedAt
3DailyFullAll events started and ended at the dayDailyOnThePeriod
52DailyIntervalInterval events bydayDailyInterval
6WeekEndedAll events ended at the same weekWeeklyFinishedAt
5WeekFullStarted and ended at the same weekWeeklyOnThePeriod
53WeekIntervalInterval events byweekWeeklyInterval
8MonthEndedAll events completed at tbe same monthMonthlyFinishedAt
7MonthFullAll events start and ended at the same monthMonthlyOnThePeriod
54MonthIntervalInterval events bymonthMonthlyInterval
0AnytimeAny time record (such as RMF interval)OtherOther