---
title: 'Abstract data layer'
section: 'Concepts'
weight: 2
description: The abstract data layer, the unified system data model, equipment abstraction, and data normalization that decouple applications from physical storage.
related:
  - basic-modules/modules
  - data-representation/components
  - architecture/data-collection-layer
  - rest-api
---

The abstract data layer sits between the physical storage of data and the applications that use it. It abstracts away the physical implementation and hardware specifics, providing a single interface for working with data. It offers:

- **Hardware independence**: Data is stored on different devices (servers, the cloud, IoT devices), but the application interacts with it through a single interface.
- **Simpler development**: Developers work with abstract entities without worrying about the details of physical storage.
- **Flexibility**: Changes to the physical structure of the data (for example, migration to the cloud) do not affect applications.

## Unified data model

The system uses a unified data model that is adapted to different types of equipment and lets you work with them in a uniform way, regardless of their source or equipment type. It includes common data structures, uniform processing rules, and semantic consistency of data across all systems and subsystems.

For example, a unified system data model includes:

- **Entities**: Equipment, channels, sessions, units, and reference parameters.
- **Relationships**: Equipment has channels, channels are related to data, and data is related to sessions.
- **Attributes**: Attributes are defined for each entity (for example, `equipment.serial_number` and `channel_data.value`).

### Structure of the unified system data model

The unified system data model is a structure for storing and managing data related to equipment, channels, sessions, and measurements. Let's look at each table and its purpose, as well as the relationships between them.

#### Table `channel_data`

This table stores the data received from the equipment channels within specific sessions.

| Field             | Type                             | Description                                        |
| ----------------- | -------------------------------- | -------------------------------------------------- |
| `id`              | `bigint`                         | Unique record identifier (primary key).            |
| `equipment_id`    | `bigint`                         | Reference to equipment (`equipment.id`).           |
| `seance_id`       | `bigint`                         | Reference to session (`seances.id`).               |
| `channel_id`      | `bigint`                         | Reference to channel (`channels.id`).              |
| `unit_id`         | `bigint`                         | Reference to unit of measurement (`units.id`).     |
| `archive_type_id` | `integer`                        | Archive type reference (`referenceparameters.id`). |
| `event_time`      | `integer`                        | The time of the event (eg timestamp).              |
| `value`           | `character varying(100)`         | The value received from the channel.               |
| `created_at`      | `timestamp(6) without time zone` | Time when the record was created.                  |
| `updated_at`      | `timestamp(6) without time zone` | Time when the record was last updated.             |

Relationships:

- The `unit_id` foreign key references the `units` table.
- The `archive_type_id` foreign key references the `referenceparameters` table.
- The `equipment_id` foreign key references the `equipment` table.
- The `channel_id` foreign key references the `channels` table.
- The `seance_id` foreign key refers to the `seances` table.

#### Table `units`

This table stores the units of measurement used for the channel data.

| Field               | Type                             | Description                                      |
| ------------------- | -------------------------------- | ------------------------------------------------ |
| `id`                | `bigint`                         | Unique record identifier (primary key).          |
| `name`              | `character varying(30)`          | Name of the unit of measurement.                 |
| `varname`           | `character varying(30)`          | Short variable name for the unit of measurement. |
| `description`       | `character varying(100)`         | Description of the unit of measurement.          |
| `conversion_factor` | `double precision`               | Conversion factor for a unit of measurement.     |
| `rounding`          | `smallint`                       | Number of decimal places to round to.            |
| `synonyms`          | `character varying[]`            | Array of synonyms for the unit of measurement.   |
| `created_at`        | `timestamp(6) without time zone` | Time when the record was created.                |
| `updated_at`        | `timestamp(6) without time zone` | Time when the record was last updated.           |

#### Table `referenceparameters`

This table stores reference parameters such as archive types or other qualifiers.

| Field               | Type                             | Description                                      |
| ------------------- | -------------------------------- | ------------------------------------------------ |
| `id`                | `bigint`                         | Unique identifier of the record (primary key).   |
| `name`              | `character varying(30)`          | The name of the unit of measure.                 |
| `varname`           | `character varying(30)`          | Short variable name for the unit of measure.     |
| `description`       | `character varying(100)`         | Description of the unit of measurement.          |
| `parent_id`         | `integer`                        | A reference to the parent parameter (hierarchy). |
| `referencemodel_id` | `integer`                        | Reference to the directory model.                |
| `created_at`        | `timestamp(6) without time zone` | Time when the record was created.                |
| `updated_at`        | `timestamp(6) without time zone` | Time when the record was last updated.           |
| `deleted_at`        | `timestamp(6) without time zone` | Time when the record was deleted (soft delete).  |

Relationships:

- The foreign key `parent_id` references the same table (`referenceparameters.id`), allowing hierarchies to be built.

#### Table `equipment`

| Field               | Type                             | Description                                    |
| ------------------- | -------------------------------- | ---------------------------------------------- |
| `id`                | `bigint`                         | Unique identifier of the record (primary key). |
| `equipment_type_id` | `bigint`                         | Reference to the equipment type.               |
| `serial_number`     | `character varying(25)`          | Serial number of the equipment.                |
| `manufacture_date`  | `timestamp(6) without time zone` | The date of manufacture of the equipment.      |
| `installation_date` | `timestamp(6) without time zone` | Installation date of the equipment.            |
| `program_version`   | `character varying(100)`         | Software version of the equipment.             |
| `created_at`        | `timestamp(6) without time zone` | Time when the record was created.              |
| `updated_at`        | `timestamp(6) without time zone` | Time when the record was last updated.         |

#### Table `channels`

This table stores information about the equipment channels.

| Field               | Type                             | Description                                            |
| ------------------- | -------------------------------- | ------------------------------------------------------ |
| `id`                | `bigint`                         | Unique identifier of the record (primary key).         |
| `equipment_type_id` | `bigint`                         | Reference to the equipment type (`equipment_type.id`). |
| `unit_id`           | `bigint`                         | Reference to the unit of measurement (`units.id`).     |
| `name`              | `character varying(100)`         | Channel name.                                          |
| `varname`           | `character varying(20)`          | Short variable name for the channel.                   |
| `created_at`        | `timestamp(6) without time zone` | Time when the record was created.                      |
| `updated_at`        | `timestamp(6) without time zone` | Time when the record was last updated.                 |

Relationships:

- The `equipment_type_id` foreign key references the `equipment_type` table.
- The `unit_id` foreign key refers to the `units` table.

#### Table `seances`

This table stores information about the equipment's communication sessions.

| Field          | Type                             | Description                                    |
| -------------- | -------------------------------- | ---------------------------------------------- |
| `id`           | `bigint`                         | Unique identifier of the record (primary key). |
| `telemetry_id` | `bigint`                         | Reference to telemetry (`telemetry.id`).       |
| `event_time`   | `integer`                        | The time of the event.                         |
| `evtid`        | `smallint`                       | The identifier of the event.                   |
| `trycnt`       | `smallint`                       | The number of attempts.                        |
| `tryfl`        | `character varying(12)`          | The flag of the attempt.                       |
| `state`        | `smallint`                       | Session status.                                |
| `btm`          | `integer`                        | Battery charge.                                |
| `rssi`         | `integer`                        | Signal strength (if used).                     |
| `created_at`   | `timestamp(6) without time zone` | Time when the record was created.              |
| `updated_at`   | `timestamp(6) without time zone` | Time when the record was last updated.         |

#### General structure and relationships

**Main table** — `channel_data`, which links channel data to equipment, sessions, and units.

**Relationship tables**:

- `units` — units of measurement.
- `referenceparameters` — archive types and other qualifiers.

**Equipment tables**:

- `equipment` — equipment information.
- `channels` — equipment channels.

**Sessions table** — `seances`, which stores information about communication sessions.

**Usage example**:

- Channel data (`channel_data`) comes from equipment (`equipment`) within specific sessions (`seances`).
- Each channel (`channels`) has its own unit of measurement (`units`).
- The archive type (`referenceparameters`) determines how data should be stored or processed.

### Benefits of a unified model

This data model provides the flexibility and scalability to store and analyze data from different equipment.

Some of the benefits of a unified data model are:

- **Consistency**: Data has the same structure and semantics across all systems.
- **Scalability**: It is easy to add new data sources or equipment types.
- **Data protection**: Unified authentication, authorization, and encryption mechanisms.
- **Simpler analysis**: Data can be analyzed with unified tools.

## Equipment abstraction

Equipment abstraction is a key system design principle that separates the logic of data handling from the physical characteristics of the equipment. This is especially important in systems that use heterogeneous devices.

Equipment abstraction means that the system handles data at a logical level that is independent of the physical devices on which the data is stored or processed. This is achieved by:

- **Unification of interfaces**: A single way of accessing data, regardless of the equipment.
- **Hiding implementation details**: The physical location of data, transmission protocols, and other technical aspects are hidden from applications.
- **Adapters**: Conversion of data from an equipment-specific format to a unified format.

### Implementing equipment abstraction

The following example shows how this approach is implemented in our system. It traces data from an IoT device through an adapter, into the unified data model, and out through the API.

Consider an IoT device that sends gas consumption data. The implemented algorithms allow us to:

- Receive the data from the device through an adapter.
- Convert it into a `channel_data` structure.
- Store it in a database.
- Provide access to the data via an API.

For each type of equipment, adapters are created that handle device-specific protocols and convert the data into a unified format.

**Example: data transformation**

For an IoT device, data can arrive in JSON format:

```json title="example.json"
{
  "device_id": "sensor-123",
  "timestamp": 1739950861,
  "value": 42.5
}
```

The adapter converts the data into a `channel_data` structure:

```sql title="example.sql"
INSERT INTO channel_data (equipment_id, seance_id, channel_id,
 archive_type_id, value, event_time)
VALUES (1, 2, 3, 4, '42.5', 1739950861);
```

Data request:

```bash title="Request example"
GET /api/v1/channel_data?equipment_id=1&archive_type=daily&channel_id=3
```

Response:

```json title="Response example"
[
  {
    "id": 123,
    "equipment_id": 1,
    "seance_id": 2,
    "channel_id": 3,
    "archive_type_id": 4,
    "value": 42.5,
    "event_time": 1739950861
  }
]
```

### Advantages of equipment abstraction

- **Hardware independence**: Applications work with data without knowing where it physically resides.
- **Flexibility**: Easily add new devices or modify existing ones.
- **Simpler development**: Developers work with abstract entities without worrying about hardware details.
- **Scalability**: Data can be stored on different devices, but the system continues to operate in a uniform manner.

## Data normalization

Data normalization is the process of organizing data in a database in a way that minimizes redundancy and improves data integrity. In the context of the abstract data layer, normalization plays a key role in creating a unified, efficient data model that can be used to handle different types of equipment and data sources.

Data normalization divides data into logical tables and establishes relationships between them in order to:

- Eliminate duplicate data.
- Simplify data maintenance and updates.
- Ensure data integrity.
- Improve query performance.

The system's unified data model is reduced to (third) normal Boyce-Codd form in terms of atomic (scalar) values. Some data model entities use composite structures to optimize the handling of rare, non-standard attributes.

### Benefits of normalization

- **Removes redundancy**: Data is stored in one place, reducing redundancy.
- **Data consistency**: Maintaining data consistency is simplified.
- **Flexibility**: It is easy to make changes to the data structure.
- **Performance**: Query performance is improved (in most cases).

### Normalization in the context of the abstract data layer

Data normalization is an important step in the design of the abstract data layer.

It allows you to:

- Create a unified data model that can be used to handle different types of equipment.
- Eliminate redundancy and duplication of data.
- Ensure data integrity and consistency, regardless of its source.
- Simplify the integration of new devices and systems.