Load Views: Aggregate data

The LoadViews is a batch process to load imported data into the historical tables. This process is usually done once a day and can be automated as well. The following diagram gives an overview of the process:

The LoadViews is done through the Zetaly Portal. However, you can save a Load configuration on the ZetalyDB and automate its execution through the API.

Creating a LoadViews configuration

If you can execute database scripts, you may execute the AddLoadViews.sql script located into the LoadViews.zip folder into the Scripts folder: 

 

Testing the Load Configuration 

The Zetaly Portal gives you access to the Load Views process and their status:

Only one can be launched at a time.

1.  LoadViews information. When started, it displays the current task.
2.  LoadViews status:
   • STOPPED: The LoadViews is not running and did not terminate with an error.
   • ERROR: The LoadViews is not running and terminated with an error.
   • STARTING: The LoadViews initialize everything it needs before starting to aggregate data
   • STARTED: The LoadViews is running and aggregating data
   • STOPPING: The LoadViews is stopping
3. LoadViews action:
   1. Start the LoadViews
   2. LoadViews logs

LoadViews Process

The LoadViews process can be summarized in the four steps below: 

When you launch a LoadViews process through the Zetaly portal, the messagings are automatically stopped and started before and after the process.

Enable the swap option

The ZETALY messaging needs to stop loading data on 'xxxData' table to load the data to a historical table. This process may take a few minutes and during this time, there will not be real time processes in the table. 

To minimize downtime, you can now enable the swap option (disabled by default). It is located in the configuration of each load view (cog icon on the card) and also when creating a new load views. Simply tick the box to activate this option and save.

What the Swap data does is rename the 'xxxData' table to 'xxxData1' and the existent 'xxxData1' table to 'xxxData'. It allows the ZETALY messaging to keep loading new data on the table. All this process is very fast, and the sequence is:

1. Stop importing from xxxData
2. Rename xxxData to xxxData1
3. Rename xxxData1 to xxxData
4. Start importing to xxxData

Now, you can execute loadviews from xxxData1.

Loader & Compress fields Configuration

The default configuration for aggregating your dataset includes all fields. You can customize this in the global load views configuration (cog icon at the top of the load views page).

Once you've chosen your product and target (1), you can select the relevant fields based on your specific needs by checking or unchecking them according to your preferences (2).

Once you're done, click the save button.

Scheduling a LoadViews Task 

Scheduling feature on the LoadViews interface

Schedule your LoadView by accessing the scheduling feature through the clock icon on the LoadView card. You have two choices:

• Standard setting: Schedule the LoadView to run every day at a chosen time you have to set (and remember to save).

• Alternatively, set a specific cycle using the advanced setting:

Define your cycle using the CRON language.

If you are unfamiliar with it, refer to the provided sentence that translates your choices into a comprehensive statement. Remember to save your configuration.

If you've already established an advanced scheduling, you can:

• Edit it by modifying the fields and save your updated choices.
• Delete it (denoted by a red trash icon [a]).
• Add a new schedule rule to be cumulated with the existing one (indicated by a "+" symbol [b]).

It is also possible to call the ZSA API to launch the LoadViews process.

Windows scheduler

You can use Windows Task Scheduler to automate your LoadView task. All you need is the possibility to call the ZSA url.

For this example, we are going to use the curl library.

 1) Create a new script called "loadviews.cmd" with the following line

curl -k -X POST https://ZETALY_HOST/zsa/api/v1/loadviews/LOAD_VIEWS_NAME/start

Replace ZETAY_HOST by your zetaly host, and LOAD_VIEWS_NAME by the LoadViews name to launch.

2) Create the task

Run the following command in an Adminstrative console: 

schtasks /CREATE /TN ZetalyLoad /RU "SYSTEM" /SC daily /ST 05:00 /TR "PATH_TO_SCRIPT\loadviews.cmd"

Replace PATH_TO_SCRIPT with the folder used to create the script at 1)

You can replace the /ST and /SC parameters by the hour and periodicity you want. See windows documentation for more details.

Docker (crontab-ui)

Get the image

To be able to schedule loadviews using a docker container, Zetaly recommand to use crontab-ui. 

If needed, Zetaly provides a s390x image for ZCX : 

Load the tar file:

docker load -i crontab-ui.tar.gz

Run the container

Once the image is available on your docker, you can run a container with crontab-ui:

docker run -d -p 8000:8000 alseambusher/crontab-ui

Scheduling

At the top of the ui, use "New":

Create a job with the desired sheduling. This job should run first following command line:

token=$(curl -k -H "Accept: application/json" -H "Content-type: application/json" -X POST https://ZETALY_HOST/zhb/api/v1/users/public/login -d '{"username":"USERNAME","password":"PASSWORD"}' | python3 -c "import sys, json; print(json.load(sys.stdin)['token'])");

Replace ZETAY_HOST by your zetaly host, and USERNAME/PASSWORD by your user credentials.

After the first line, add the following command line:

curl -ki -H "token: $token" -X POST https://ZETALY_HOST/zsa/api/v1/loadviews/LOAD_VIEWS_NAME/start

Replace ZETAY_HOST by your zetaly host, and LOAD_VIEWS_NAME by the LoadViews name to launch.

Example, every day at 02am:

Once the job is created, you can try it:

When the job is working, you can save it to cron:

Scope

In the historical tables resulting from the LoadViews process, a "scope" column is added. It identifies the type of aggregation performed. Below are the different possible values: