Abstract data layer

The abstract data layer sits between the physical storage of data and the applications that use it. It abstracts away the physical implementation and hardware specifics, providing a single interface for working with data. It offers:

• Hardware independence: Data is stored on different devices (servers, the cloud, IoT devices), but the application interacts with it through a single interface.
• Simpler development: Developers work with abstract entities without worrying about the details of physical storage.
• Flexibility: Changes to the physical structure of the data (for example, migration to the cloud) do not affect applications.

Unified data model

The system uses a unified data model that is adapted to different types of equipment and lets you work with them in a uniform way, regardless of their source or equipment type. It includes common data structures, uniform processing rules, and semantic consistency of data across all systems and subsystems.

For example, a unified system data model includes:

• Entities: Equipment, channels, sessions, units, and reference parameters.
• Relationships: Equipment has channels, channels are related to data, and data is related to sessions.
• Attributes: Attributes are defined for each entity (for example, equipment.serial_number and channel_data.value).

Structure of the unified system data model

The unified system data model is a structure for storing and managing data related to equipment, channels, sessions, and measurements. Let’s look at each table and its purpose, as well as the relationships between them.

Table channel_data

This table stores the data received from the equipment channels within specific sessions.

Relationships:

• The unit_id foreign key references the units table.
• The archive_type_id foreign key references the referenceparameters table.
• The equipment_id foreign key references the equipment table.
• The channel_id foreign key references the channels table.
• The seance_id foreign key refers to the seances table.

Table units

This table stores the units of measurement used for the channel data.

Table referenceparameters

This table stores reference parameters such as archive types or other qualifiers.

Relationships:

• The foreign key parent_id references the same table (referenceparameters.id), allowing hierarchies to be built.

Table equipment

Table channels

This table stores information about the equipment channels.

Relationships:

• The equipment_type_id foreign key references the equipment_type table.
• The unit_id foreign key refers to the units table.

Table seances

This table stores information about the equipment’s communication sessions.

General structure and relationships

Main table — channel_data, which links channel data to equipment, sessions, and units.

Relationship tables:

• units — units of measurement.
• referenceparameters — archive types and other qualifiers.

Equipment tables:

• equipment — equipment information.
• channels — equipment channels.

Sessions table — seances, which stores information about communication sessions.

Usage example:

• Channel data (channel_data) comes from equipment (equipment) within specific sessions (seances).
• Each channel (channels) has its own unit of measurement (units).
• The archive type (referenceparameters) determines how data should be stored or processed.

Benefits of a unified model

This data model provides the flexibility and scalability to store and analyze data from different equipment.

Some of the benefits of a unified data model are:

• Consistency: Data has the same structure and semantics across all systems.
• Scalability: It is easy to add new data sources or equipment types.
• Data protection: Unified authentication, authorization, and encryption mechanisms.
• Simpler analysis: Data can be analyzed with unified tools.

Equipment abstraction

Equipment abstraction is a key system design principle that separates the logic of data handling from the physical characteristics of the equipment. This is especially important in systems that use heterogeneous devices.

Equipment abstraction means that the system handles data at a logical level that is independent of the physical devices on which the data is stored or processed. This is achieved by:

• Unification of interfaces: A single way of accessing data, regardless of the equipment.
• Hiding implementation details: The physical location of data, transmission protocols, and other technical aspects are hidden from applications.
• Adapters: Conversion of data from an equipment-specific format to a unified format.

Implementing equipment abstraction

The following example shows how this approach is implemented in our system. It traces data from an IoT device through an adapter, into the unified data model, and out through the API.

Consider an IoT device that sends gas consumption data. The implemented algorithms allow us to:

• Receive the data from the device through an adapter.
• Convert it into a channel_data structure.
• Store it in a database.
• Provide access to the data via an API.

For each type of equipment, adapters are created that handle device-specific protocols and convert the data into a unified format.

Example: data transformation

For an IoT device, data can arrive in JSON format:

{
  "device_id": "sensor-123",
  "timestamp": 1739950861,
  "value": 42.5
}

The adapter converts the data into a channel_data structure:

INSERT INTO channel_data (equipment_id, seance_id, channel_id,
 archive_type_id, value, event_time)
VALUES (1, 2, 3, 4, '42.5', 1739950861);

Data request:

GET /api/v1/channel_data?equipment_id=1&archive_type=daily&channel_id=3

Response:

[
  {
    "id": 123,
    "equipment_id": 1,
    "seance_id": 2,
    "channel_id": 3,
    "archive_type_id": 4,
    "value": 42.5,
    "event_time": 1739950861
  }
]

Advantages of equipment abstraction

• Hardware independence: Applications work with data without knowing where it physically resides.
• Flexibility: Easily add new devices or modify existing ones.
• Simpler development: Developers work with abstract entities without worrying about hardware details.
• Scalability: Data can be stored on different devices, but the system continues to operate in a uniform manner.

Data normalization

Data normalization is the process of organizing data in a database in a way that minimizes redundancy and improves data integrity. In the context of the abstract data layer, normalization plays a key role in creating a unified, efficient data model that can be used to handle different types of equipment and data sources.

Data normalization divides data into logical tables and establishes relationships between them in order to:

• Eliminate duplicate data.
• Simplify data maintenance and updates.
• Ensure data integrity.
• Improve query performance.

The system’s unified data model is reduced to (third) normal Boyce-Codd form in terms of atomic (scalar) values. Some data model entities use composite structures to optimize the handling of rare, non-standard attributes.

Benefits of normalization

• Removes redundancy: Data is stored in one place, reducing redundancy.
• Data consistency: Maintaining data consistency is simplified.
• Flexibility: It is easy to make changes to the data structure.
• Performance: Query performance is improved (in most cases).

Normalization in the context of the abstract data layer

Data normalization is an important step in the design of the abstract data layer.

It allows you to:

• Create a unified data model that can be used to handle different types of equipment.
• Eliminate redundancy and duplication of data.
• Ensure data integrity and consistency, regardless of its source.
• Simplify the integration of new devices and systems.