This tutorial expects you to already have the Fluvio CLI installed, and InfinyOn
Cloud set up. If neither of these is the case, please follow the instructions here!
There are four main steps for this tutorial:
Installing the SmartModule Development Kit CLI, smdk
Generating a SmartModule project
Testing the SmartModule behavior with inputs
Loading the SmartModule into a cluster
SmartModules
SmartModules are user defined functions set to run on and modify the inputs/outputs to
a Fluvio database.
In this tutorial, we will create a configurable SmartModule that takes in a regular expression rule, and filters input based on whether it matches the rule.
Installing SmartModule Development Kit CLI
This first example will demonstrate creating, loading, and testing a SmartModule that filters based on whether the content matches a regular expression rule.
We will need a custom map SmartModule project, which we can generate with the SmartModule Development Kit CLI, smdk.
You can install smdk with the Fluvio CLI
$ fluvio install smdk
Generate a new SmartModule project
With smdk, you can generate a new SmartModule project with the guided wizard. We’re going to create a project named regex_filter in our current directory by running smdk generate regex_filter
$ smdk generate regex_filter
Generating new SmartModule project: regex_filter
fluvio-smartmodule-cargo-dependency => '0.3.0'⚠️ Renaming project called `regex_filter` to `regex-filter`...
✔ 🤷 Which type of SmartModule would you like? · filter
🤷 Please set a group name : my-group
✔ 🤷 Will your SmartModule use init parameters? · true✔ 🤷 Will your SmartModule be public? · falseIgnoring: /var/folders/r8/4x6_d2rn283946frzd1gc1pr0000gn/T/.tmpQXKwnh/smartmodule/cargo_template/cargo-generate.toml
[1/5] Done: Cargo.toml
[2/5] Done: README.md
[3/5] Done: SmartModule.toml
[4/5] Done: src/lib.rs
[5/5] Done: src
🔧 Moving generated files into: `/home/user/project/regex_filter`...
💡 Initializing a fresh Git repository
✨ Done! New project created /home/user/project/regex_filter
hub: hubid my-group is sethubid set to my-group
Navigate to the project directory and take a look at the Cargo.toml file:
When you list the SmartModules in your cluster, you’ll see that the regex-filter results match the name, version and group from the SmartModule.toml file.
$ fluvio sm list
SMARTMODULE SIZE
my-group/[email protected] 316.4 KB
With the SmartModule loaded in the cluster, we will test that our filter works with data from a topic.
This basic example we’ll create a new topic, and load it with values.
Create a file values.txt with the following
contents:
$ cat values.txt
42
abc
abc123
Create a topic filter-test:
$ fluvio topic create filter-test
Load values.txt file to filter-test topic:
$ fluvio produce filter-test -f values.txt
$ fluvio consume filter-test -dB --smartmodule my-group/[email protected] -e regex="^[0-9]*$"Consuming records from the beginning of topic 'filter-test'42
Or you can just use the SmartModule by name, if it is unique
$ fluvio consume filter-test -dB --smartmodule regex-filter -e regex="^[0-9]*$"Consuming records from the beginning of topic 'filter-test'42
You now know the development workflow for SmartModules with smdk. You can now generate your own project that processes data. For additional information on how to publish and share SmartModules checkout SmartModule Hub.