IoT Mirroring - Edge VM to Cloud
InfinyOn IoT edge is a ~14 Mb binary that runs on ARMv7 chips on less than 256 MB memory. We are working with teams building the future of monitoring dynamic assets to push the boundaries of edge data stream processing.
-
If connected, InfinyOn IoT edge sends telemetry and events to the InfinyOn Cloud in real-time using mirroring.
-
If disconnected, the InfinyOn IoT edge stream processor caches events locally. When the connection resumes, the InfinyOn IoT edge stream processor brings InfinyOn Cloud up to date and continues mirroring until the subsequent connection loss.
The benefits of the InfinyOn solution are as follows:
- Reliable edge-to-cloud synchronization:
- Real-time publishing when connected.
- Automatic synchronization after reconnect.
- Edge devices can be offline for extended periods (days).
- Edge collection without downtime when disconnected.
- Reliable local caching for gigabytes of data.
- Simplified logic for edge clients.
- Edge cluster provides a reliable connection to the local clients.
- Intelligent processing at the edge with InfnyOn Smartmodules
- filter
- transform
- enrich
- Hierarchical processing, where you decide where to apply the transformations.
- Built-in cloud connectors to push events to databases and other core products.
In this toutorial we’ll use VM emulator to create and edge endpoint and mirror traffic to InfinyOn Cloud.
Let’s get started.
Mirroring is an experimental feature using a development cluster. Please get in touch with us on Discord to request access for your organization. Upon approval, please continue as follows:
Using your web browser, navigate to https://dev.infinyon.cloud/signup, where this experimental feature is available.
After the account is created, you will be placed in the Dashboard. You may choose to create a cluster in the GUI. In this tutorial, we’ll create a cluster using the CLI later.
Download and install mirroring binary.
Use curl
to download and install:
curl -fsS https://hub.infinyon.cloud/install/install.sh | VERSION='0.10.15-dev-2+mirroring-9961bdb' bash
Make sure to add .fluvio/bin
to the $PATH
as specified in the installation script.
Login to InfinyOn Cloud (dev):
fluvio cloud login --remote=https://dev.infinyon.cloud --use-oauth2
Leave out --use-oauth2
if you prefer username/password method.
Let’s provision a new cluster in AWS eu-central
using the experimental fluvio version:
fluvio cloud cluster create --region aws-eu-central-1 --version 0.10.15-dev-2+mirroring-b4f07fc
Check cluster status:
fluvio cluster status
Next, we’ll configure the cluster to receive traffic from the edge clusters.
Each edge cluster mirror connects a partion of a topic, where each partition has a 1-to-1
relationship with the edge cluster.
Create a partition assignment file with an array of edge mirros we expect to connect this cluster:
echo '[
"edge1", "edge2"
]' > assignment_file.json
Apply the configuration file to create the topic:
fluvio topic create edge-topic --mirror-assignment assignment_file.json
List partitions to check the assignment:
fluvio partition list
It should display all partitions:
TOPIC PARTITION LEADER MIRROR REPLICAS RESOLUTION SIZE HW LEO LRS FOLLOWER OFFSETS
edge-topic 0 5001 edge1 [] Online 0 B 0 0 0 0 []
edge-topic 1 5001 edge2 [] Online 0 B 0 0 0 0 []
We created 2 partitions, but we’ll only use one in this tutorial.
Let’s register the edge cluster edge1
to inform our Cloud cluster to accept connection requests from the remote device:
fluvio cluster remote-cluster register --type mirror-edge edge1
List remote clusters to check their status:
fluvio cluster remote-cluster list
It should show the following:
RemoteCluster RemoteType Paired Status Last Seen
edge1 mirror-edge - - -
In the next step, we’ll create a configuration file that we’ll need to pass along to the edge device. It’s easier if we make a clean directory and pass it along to the VM emulator:
mkdir -p ~/local/projects/mirror; cd ~/local/projects/mirror
Each edge cluster requires a unique metadata file that informs the edge cluster how to connect with the target cluster. Create the config file by running the following command:
fluvio cluster remote-cluster metadata export --topic edge-topic --mirror edge1 --file edge1.json
The Cloud cluster configuration is now complete. Next, we’ll create an edge cluster and configure a mirror topic that synchronizes data to the Cloud.
We’ll start an edge cluster on our local computer in a VM using OrbStack.
We’ll use OrbStack for the VM management:
-
Start Ubuntu VM machine.
-
Click the VM to open a terminal.
-
Using the terminal, navigate to your data directory:
cd local/projects/mirror
All files we’ve generated on the local machines should be visible here.
Download binaries:
curl -fsS https://hub.infinyon.cloud/install/install.sh | VERSION='0.10.15-dev-2+mirroring-9961bdb' bash
Add to path:
echo 'export PATH="${HOME}/.fluvio/bin:${PATH}"' >> ~/.bashrc
source ~/.bashrc
Run the following command to double check:
fluvio version
We’ll use the metadata edge1
to start the edge cluster:
fluvio cluster start --read-only edge1.json
Let’s check the partitions:
fluvio partition list
The edge device should show the following partition:
TOPIC PARTITION LEADER MIRROR REPLICAS RESOLUTION SIZE HW LEO LRS FOLLOWER OFFSETS
edge-topic 0 5001 upstream:0 [] Online 0 B 0 0 0 0 []
Let’s produce on the Edge VM cluster and consume from the Cloud cluster.
Produce on the edge
terminal:
fluvio produce edge-topic
> 1
Ok!
> 2
Ok!
Consume on the cloud
terminal:
fluvio consume edge-topic --mirror edge1 -B
1
2
Mirror test is successful.
To simulate a disconnect, we’ll perform the following steps:
-
Turn off the network connection to the internet.
-
Produce records on the
edge
terminal.
fluvio produce edge-topic
> 3
Ok!
> 4
Ok!
> 5
Ok!
- Turn on the network connection and check that the data is synced.
The topic on the Cloud cluster should automatically synchronize with the edge cluster.
- Consume from the
cloud
terminal:
Wait for the connection retry interval to trigger for the new records to arrive, then consume:
fluvio consume edge-topic --mirror edge1 -B
1
2
3
4
5
The disconnect test was successful.
This test ensures that the edge cluster will preserve all cached data following a power loss.
On the edge
terminal, shutdown the cluster:
fluvio cluster shutdown --local
Restart the cluster:
fluvio cluster upgrade --read-only edge1.json
On the edge
terminal, consume from the cluster:
fluvio consume edge-topic -B
1
2
3
4
5
Produce records and observe that the mirror will resume the synchronization.
🎉 Congratulations! These tests confirm that the synchronization from Edge to Cloud works as expected. It is now time to roll it out in your environment.
Join us on Discord if you have questions, or would like to suggest new improvements.