Engine

In "Engine, you can manage Engines, add external storages, and configure jobs.

If you don't have an Engine yet, you can request one by clicking the Request access now button.

How to access

  1. There are a few ways to access this feature:

  • Click "Transform your data" on the Studio Home screen

  • Or go directly to the Data Lake menu

  1. Then, click on "Engine" under the Data Management section.

  2. Once you access the menu, the following screen will be displayed:

Where:

Basic Information

In the first section, you can view the Engine’s name and its description (if available).

Metrics

Monitor the Engine’s performance through the following metrics:

  • Memory: percentage of memory usage;

  • Disk: percentage of disk space used;

  • CPU: percentage of CPU usage. This value may fluctuate;

  • Total memory: amount of memory allocated;

  • Total disk: amount of disk space allocated;

  • CPU cores: number of cores allocated;

  • View processes: If any processes are running (e.g., SQL), they will be displayed in a list. You can also cancel active processes from this view.

Armazenamentos externos

You can view all external storages registered in the Data Engine. Additionally, you can create, edit, and delete a storage.

Create external storage

To create an external storage, follow these steps:

  1. Click "Criar novo armazenamento externo".

  2. Choose the type of storage and fill in the fields according to the selected type. The types of storage are:

AWS S3

To create a new external storage of type AWS S3, fill in the following fields:

  • Storage name: Enter a name to identify the storage.

  • Access Key ID: Insert your AWS Access Key ID.

  • Secret access key: Insert your AWS Secret Access Key.

  • Storage region: Select the region where your S3 bucket is located.

  • Endpoint (optional): Custom endpoint for reading the file within Data Studio.

  • Storage URL (optional): Custom URL for connecting to S3.

Google Cloud Storage

To create a new external storage of type Google Cloud Storage, fill in the following fields:

  • Storage name: Enter a name to identify the storage.

  • Access Key ID: Access ID generated by the HMAC Key, used for reading files.

  • Secret access key: Access key generated by the HMAC Key, used for reading files.

  • Service Account: Enter the associated service account.

Azure

To create a new external storage of type Azure, fill in the following fields:

  • Storage name: Enter a name to identify the storage.

  • Service type: Choose the type of service—Blob Storage or Data Lake Storage.

  • Connection string: Enter the Azure connection string.

Custom Storage

To create a new external storage of type Custom Storage, fill in the following fields:

  • Storage name: Enter a name to identify the storage.

  • Access Key ID: Enter the Access Key ID.

  • Secret access key: Enter the Secret Access Key.

  • Storage region: Enter the region where the storage is located.

  • Endpoint: Endpoint used for reading files within Data Studio.

  • Storage URL: URL of the storage used for authentication.

ℹ️ Tested external storages: localstack and clouds2africa.

  1. To finalize, click "Create".

Edit external storage

To edit external storage, follow these steps:

  1. In the storage list, click "Edit" (pencil icon) on the item you want to modify.

  2. Make the desired changes.

  1. To finish, click "Save changes".

Delete external storage

To delete external storage, follow these steps:

  1. In the storage list, click "Delete" (trash can icon) on the item you want to remove.

  2. In the modal window that appears, confirm the deletion of the external storage by typing the storage name and then clicking "Delete".

You can also delete an external storage via the "Edit" option.

Data Consumers

When creating a “file template, you need to configure which users will have access to the endpoint. These users are called "Data Consumers".

Follow the steps below to create, edit, and delete a data consumer:

Create Consumer

To create a data consumer, follow these steps:

  1. Click "Create consumer".

  2. Note that the user will be named user@engine_name. Add the user’s name.

  3. By default, the authentication type is "Basic".

  4. Add a password.

  5. To finish, click "Save user".

Done! The data consumer has been created!

Edit Consumer

To edit a data consumer, follow these steps:

  1. In "Data Consumers", click "Edit" (pencil icon) on the name you want to modify.

  2. Make the desired changes.

You cannot change the user’s name.

  1. To finish, click "Save changes".

Delete Consumer

To delete a data consumer, follow these steps:

  1. In "Data Consumers", click "Delete" (trash can icon) on the name you want to remove.

  2. In the modal window that appears, confirm the deletion of the user by typing the user’s name and then clicking "Delete".

You can also delete a user via the "Edit" option.

ODBC

ODBC (Open Database Connectivity) is a standard for accessing database management systems (DBMS). It allows applications to access different databases in a consistent and standardized manner.

Prerequisites for ODBC Connection

To establish an ODBC connection, ensure that the following requirements are met:

  1. The database information must be available. If the database does not yet exist, it can be created using Skyone Studio.

  2. The user must belong to the same group as the database they wish to access.

  3. The IP used for the connection must be allowed on the Whitelist.

There are two areas for accessing ODBC: Users and Whitelist.

Below, we detail each of them.

Users

The “Users” section allows you to manage connections between the database and Skyone Studio. When accessing this area, you will find:

  • User Listing: View all registered users.

  • Search: Use the search bar to find a specific data consumer.

  • Selection: Select one or more users for simultaneous deletion.

  • Editing and Deletion: Edit or delete each user individually.

Creating a consumer

Click "Create consumer” and fill in the following fields:

  • User: Choose the username.

  • Password: Set a password.

  • Groups: Select one or more groups. By default, the user has access to the "Default" group.

User Addition Screen

To finalize, click "Save user”.

The user must belong to the same group as the database they wish to access. Learn more about Groups.

Whitelist

The “Whitelist” section manages access permissions based on IP addresses. When accessing this area, you will find:

  • Whitelist Listing: View all registered entries, including name, IP address, and status.

  • Activation/Deactivation: Enable or disable individual whitelist connections.

  • Editing and Deletion: Edit or delete each whitelist entry independently.

Creating a whitelist

To create a whitelist, fill in the following fields:

  • Name: Enter the whitelist name.

  • IP Address: Provide the IP address that will be granted access.

  • Status: Set the connection status.

Tela para criação de uma whitelist

To finalize, click "Save”.

The IP address can be found in the error message. Learn more.

Using ODBC with Power BI

Requirements:

Before using ODBC with Power BI, follow these steps:

  1. Install the PostgreSQL driver.

  2. Configure the driver, filling in the required fields.

Steps to connect:

  1. Click “Get Data” in Power BI.

  2. Search for “ODBC”.

  3. Select the previously configured database and click “OK”.

  4. On the first access, enter the password.

  5. Open the Power BI navigator and connect using the appropriate option.

Alternative option: Using SQL

  1. In Power BI, click “Get Data

  2. Search for “ODBC”.

  3. Select the previously configured database.

  4. Click “Advanced options” and enter the SQL command to retrieve the data.

  5. Click “OK” to view the retrieved information.

Managing IP Connection Errors

1. Connection error due to IP

If a connection error occurs due to an IP restriction, follow these steps:

  1. Check the error message displayed, which will indicate the IP used in the connection attempt.

  2. Copy the IP provided.

  3. Add the IP to the Whitelist to grant access..

2. How to Find the IP

If you are unsure of the IP address, follow these steps:

  1. Attempt to establish the ODBC connection.

  2. In the error message, locate the IP used in the connection attempt.

  3. Add the identified IP to the Whitelist.

Last updated