---
title: 'Data collection layer'
section: 'Concepts'
weight: 1
description: Structure of the IIoT Platform's data collection layer, its daemons, data-exchange principles, and interaction with the management layer.
related:
  - architecture
  - architecture/abstract-data-layer
  - protocols
  - equipment
---

import Alert from '@/components/docs/Alert.astro';

## Structure and composition of the data collection layer

**The data collection layer** is a key component of the IIoT Platform that provides integration with devices. It includes a set of specialized applications, each designed to interact with a specific type of IoT device (meters, sensors, and so on). These applications function as network services, handling incoming connections through dedicated ports and managing data transfer between devices and the platform. This architecture guarantees scalability and flexibility when working with heterogeneous equipment.

<Alert type="note">
  For added security, resilience, and ease of installation and upgrades, applications are packaged
  in Docker containers. The container set is managed by [Docker Compose
  technology](https://docs.docker.com/compose/), which lets linked containers be combined into a
  single control panel.
</Alert>

The structure and composition of the data collection layer applications are summarized below:

```mermaid
flowchart TD
    A(Household energy meters) <-->|Data transfer| B("Data collection daemon (Type-J)")
    C(Industrial energy meters) <-->|Data transfer| D("Data collection daemon (Type-T)")
    E(Smart sensors) <-->|Data transfer| F("Data collection daemon (DDT)")
    subgraph " "
    B("Data collection daemon (Type-J)") <--> I{"Balancer (Pgbouncer)"}
    D("Data collection daemon (Type-T)") <--> I{"Balancer (Pgbouncer)"}
    F("Data collection daemon (DDT)") <--> I{"Balancer (Pgbouncer)"}
    X@{ shape: braces, label: "Data Polling Subsystem" }
    end
    I{"Balancer (Pgbouncer)"} <--> J[(DBMS PostgreSQL)]

```

Exchanges with end devices are performed over a TCP connection, with each collection daemon implementing its own application exchange protocol to serve a particular type of IoT device.

These daemons share several common architectural traits, described below.

### Architecture features

- Each container includes an application specialized for a particular protocol.
- Lightweight base images are used (`alpine` in the base configuration) to minimize overhead. However, it is possible to build images for the customer's OS.
- To manage dependencies between services (such as database access), `depends_on` and `health-check` sections are specified in Docker Compose.
- The compose daemons run in multiple instances (replicas) to distribute the load. In this case, each collection daemon can serve tens of thousands of connections simultaneously.
- A balancer (HAProxy, Nginx Stream) directs traffic to available daemons using Round Robin or Least Connections algorithms.
- In cloud environments, you can add automatic scaling based on metrics (CPU, number of connections).
- All communication with devices is logged in detail to files, so it can be analyzed if problems and failures are detected.

### Data exchange principles

Below is a diagram of a communication session between an IoT device and the collection daemon:

```mermaid
sequenceDiagram
    autonumber
    participant D as IoT device
    participant I as Data collection <br/>daemon
    Note over I,D: Stages of interaction
    D->>I: Session start, authentication
    I-->>D: Sending request parameters
    D->>I: Data acquisition
    I-->>D: Sending the next session time
```

Principles of data exchange organization:

- **Initiation of a communication session**.  
   A communication session is always initiated by the IoT device. After establishing a connection to the data collection daemon, the device transmits an identity packet.

- **Authentication procedure**  
   The data collection daemon authenticates the device. If the authentication is successful, it performs:
  - Retrieving the saved device configuration from the database.
  - Sending the current interface settings to the device.

- **Data transfer**  
   Based on the received request, the device transfers to the collection server:
  - Current device readings.
  - Archived records (if any).

- **Scheduling the next session**  
   After data reception is complete, the collection daemon:
  - Generates the next session timestamp.
  - Transmits the next session timestamp to the device.
  - Initiates connection termination.

- **Data processing and storage**  
   Received information is:
  - Aggregated into structured JSON objects.
  - Stored in an intermediate PostgreSQL database.
  - Formatted according to the protocol specification of a particular daemon.

- **Control system integration**  
   Data becomes available to the upper level of the system (management subsystem) for:
  - Further analytical processing.
  - Visualization in management interfaces.
  - Generation of automated reports.

## Interaction with the management layer of the platform

### Subsystem interconnection architecture

The data collection daemons integrate with the top-level system (management layer) through an intermediate database. To enable this interaction, a Docker container with a deployed PostgreSQL DBMS is used.

<Alert type="note">
  The management subsystem stores all the necessary configuration of the polled IoT devices in this
  database. Data collection daemons use these settings to establish communication with the devices
  and perform data exchange operations.
</Alert>

The results of successful interaction with devices are automatically saved by daemons to a specialized table in the same database, providing transparency of data transfer between the control level and devices.

The subsystem interaction scheme is shown below:

```mermaid
flowchart LR
A@{ shape: procs, label: "Data Polling Subsystem"} -->|Readings data| B[(DBMS PostgreSQL)]
B[(DBMS PostgreSQL)] -->|Configuration| A@{ shape: procs, label: "Data Polling Subsystem"}
B[(DBMS PostgreSQL)] -->|Readings data| C@{ shape: procs, label: "Control Subsystem"}
C@{ shape: procs, label: "Control Subsystem"} -->|Configuration| B[(DBMS PostgreSQL)]
```

### Device configuration

The device configuration includes the following information:

- **Dates of last readings by archive**: hourly, daily, monthly. These parameters indicate the date from which the data of the corresponding archive should be read out.
- **Metering device code** (for IoT devices using the Type-T protocol). This parameter is required only for the collection daemons serving the Type-T protocol, and it tells the collection server which algorithm to use to work with the device.
- **Serial port speed** (for IoT devices running the Type-T protocol). This parameter tells the IoT device how fast it should communicate with the meter. The collection daemon sends it at the beginning of an exchange session.
- **Communication schedule**. The management subsystem saves all schedules to an intermediate database, and the collection daemon calculates the nearest date from the received schedules and sends it to the device.
- **Device control commands**. Depending on the type of IoT device and its capabilities, the management commands include:
  - A command to update the software of the telemetry part of the IoT device. On receiving this command, the device downloads new firmware from the server and performs the update.
  - A command to read back the configuration of the metering device.
  - A command to close or open the valve (if a valve is present and supported by the IoT device software).
  - A command to set the operating parameters of the metering device (depends on the model).
  - A command to reboot the IoT device.

The main part of the device configuration is stored in an intermediate database in JSON format.

An example of the configuration is shown below:

```json
{
  "settings": "1",
  "day_event": 1706140800,
  "hour_event": 1706140800,
  "month_event": 1673857740,
  "net_address": 58
}
```

Here `day_event`, `hour_event`, and `month_event` are timestamps in `unixtime` format of the most recently saved records of the daily, hourly, and monthly archives, respectively.

Schedules are stored separately, also in JSON format. The schedule string itself is formed and processed in CRON format. This approach provides flexibility in setting up and processing any schedules.

Below is an example of storing several schedules for a device:

```json
[
  { "crontab": "10 * * * *", "schedule_id": 6 },
  { "crontab": "40 * * * *", "schedule_id": 7 }
]
```

According to these schedules, the device should connect at 10 and 40 minutes past each hour. The daemon itself determines the most suitable schedule at the moment of device communication and transmits the nearest one.