iobeam library for common DeviceOps operations.
Prerequisites for using the DeviceOps library:
-
Generate the framework for an iobeam Spark app (includes DeviceOps sample app)
iobeam provides library support for common operations associated with DeviceOps, including stateful series processing with filtering and triggers.
The input to a DeviceOps application is a stream of measurements (e.g., cpu, battery levels, memory consumption, etc.), while the output will be a derived stream (the result of processing and filtering the incoming measurement stream) and a stream of events to be triggered.
This library provides a DeviceOpsConfigBuilder that allows one to perform simple stream processing in just a few lines of configuration, including common stateful transformations (e.g., computing exponentially-weighted moving averages, derivations, etc.), filters (e.g., simple thresholds, time-based stateful conditions, etc.) and event generation.
By now, you should have first followed the setup and installation instructions for generating an iobeam Spark app. Once you've done that, you can modify and/or deploy the DeviceOps app.
At this point, you should see the sample device ops app DeviceOpsApp.scala in a path that looks something like this:
/myApp/src/main/scala/com/mycompany/apps/examples/
This section break downs the DeviceOpsApp.scala example app generated by the mvn command.
In general, the core functionality of a device ops application can be
realized using iobeam's DeviceOpsConfigBuilder, which allows one
to chain a series of filters and transformations to generate derived data
streams and trigger events. The full list of supported filters and triggers
can be found in the Spark Device Ops Library Scaladocs,
with a few examples at the end of this document.
The example app, also shown below, first takes an input stream that includes
a time series of raw cpu readings, then smoothes the series using an
exponentially-weighted moving average.
The app also generates various trigger events named:
- cpu_over_90 when the smoothed series goes above 90%
- cpu_below_70 when a smoothed series that had been above 90% then drops below 70%
- cpu_over_85_for_5_s when the smoothed series remains over 85% for more than 5 seconds
- cpu_restored when the series that had been above 85% for 5 seconds drops below 70%
- device_offline when no data is received from the device in a 5 minute period
/**
* Builds a Device Ops app using iobeam library.
*/
@SparkRun("deviceOps")
object DeviceOpsApp extends SparkApp {
override def main(appContext: AppContext):
OutputStreams = {
// Get raw input data
val stream = appContext.getData("input")
// Build device ops config
val config = new DeviceOpsConfig()
// smooth the input series cpu with a EWMA filter with alpha 0.2
.addFieldTransform("cpu", "cpu_smoothed", new Ewma(0.2))
// Add a threshold trigger to the series
// where device_ops is the namespace and cpu_smoothed is the field to output.
// Write the trigger events to the same namespace in the field cpu_triggers
.addFieldTransform("device_ops", "cpu_smoothed", "device_ops", "cpu_triggers",
new ThresholdTrigger(90.0, "cpu_over_90", 70, "cpu_below_70"))
// Add a threshold timeout trigger
.addFieldTransform("cpu", "cpu_triggers",
new ThresholdTimeoutTrigger(85.0, 70.0, Seconds(5), "cpu_over_85_for_5_s",
"cpu_restored"))
// Add check of when devices go offline
.addNamespaceTransform("input", "device_status", "timeouts", new DeviceTimeoutTrigger
(Minutes(5), "device_offline"))
val outStream = DeviceOps.getDeviceOpsOutput(stream, config)
stream.foreachRDD(rdd => {
println("Batch in")
rdd.foreach(tr => println(s"TR in: $tr"))
})
OutputStreams(outStream)
}
}To build a device ops app, run mvn package from your top-level directory,
as detailed in the README. This will create a
JAR file that can be uploaded to iobeam using our command-line interface.
E.g., to upload and run the device ops example app:
iobeam app create -name deviceOps -path target/myApp-1.0-SNAPSHOT.jar
New app created.
App ID: [your_app_id]
App Name: deviceOps
Once your app is running on iobeam, you can register triggers to get notified when an event occurs. Please see our trigger documentation for more detailed info.
E.g., to send an email to yourself when the example app detects the CPU above 90% (event cpu_over_90), use the following command:
iobeam trigger create email -name "cpu_over_90" \
-payload "The CPU of device {{ .device_id }} is above 90%" \
-fireWhen "{{ cpu_trigger }} == \"cpu_over_90\"" -subject "High CPU" -to
-to [your_email_address]
And then to test that your trigger works:
iobeam import -namespace device_ops -fields cpu_triggers -values \"cpu_over_90\" -label device_id=\"CLI\"
For a full end-to-end test, we'll need to send some test data:
iobeam device create -id DeviceOps-Test-1
for v in 85 95 99 99 50 ; do
iobeam import -label deviceId=\"DeviceOps-Test-1\" -fields cpu -values $v ;
done
You can verify that this data was received:
iobeam query
In fact, you can even query the system to determine when some condition occurs:
iobeam query -where "eq(deviceId,DeviceOps-Test-1)" -series cpu_over_90
{
"result": [
{
"data": [
{
"time": 1460137113326,
"value": 1.0
}
],
"device_id": "DeviceOps-Test-1",
"name": "cpu_over_90",
"project_id": [your_project_id]
}
],
"timefmt": "msec"
}
For an overall view of the raw data received:
iobeam query -output csv
For an overall view of the derived data:
iobeam query -namespace device_ops -output csv
Because the stock example generates an event whenever the device is offline for more than 5 minutes (which can also be tied to a trigger), you should stop your app when you are done testing it:
iobeam app stop -id [your_app_id]
Requested status: STOPPED. Waiting for current status to change.
Checking app status...RUNNING
Checking app status...STOPPED
Success!
If you don't remember your app id, just run:
iobeam app list
App ID : [your_app_id]
App Name: deviceOps
Created : 2016-04-08T14:37:15Z
Requested Status: RUNNING
Current Status : RUNNING
BUNDLE INFO
URI : file://telemetryApp-1.0-SNAPSHOT.jar
Type : JAR
Checksum: 2889f37315fcfbd8ba5e780e2d9097fa48e64162a1f0c924f499516548598976 (SHA256)
Now that we've deployed the stock device ops app, you can now extend the
program according to your needs. Either edit the example in-place, or
copy the main function to MyApp.scala.
To redeploy your app, run mvn clean package and then run the update command.
Include the -name parameter if you modified the main function in MyApp.scala.
mvn clean package
iobeam app update -id [your_app_id] -name myApp -path target/myApp-1.0-SNAPSHOT.jar
There are two main ways to process data streams using the device ops library:
-
Filters: Filters work on individual series and output a new value on each sample. They can be attached to the output from other filters but will then operate on the data from the previous batch.
-
Triggers: Triggers are evaluated on each sample and output a trigger name or None. Triggers can be attached to derived series.
A common use of the trigger system is to generate events when a certain metric passes a threshold. For example when the battery level of a device is below a defined level. To setup a trigger for when the series named "battery" is below 10%, you set up a ThresholdTrigger:
new DeviceOpsConfigBuilder()
.addSeriesTrigger("battery", new ThresholdTrigger(10.0, "battery_below_10", 15.0))
.buildwhere 10.0 is the threshold level (or "low watermark") and 15.0 is the trigger release level (or "high watermark").
This hysteresis means that the trigger will not create multiple events if the battery readings oscillate around the
trigger level (in this case, system will not generate multiple triggers if the battery level rises above/below 10%
but above 15%).
When monitoring a metric where you are interested both when it enters and leaves a problematic area, such as high CPU, you configure the threshold trigger:
new DeviceOpsConfigBuilder()
.addSeriesTrigger("cpu", new ThresholdTrigger(90.0, "cpu_above_90", 70.0, "cpu_below_70"))
.buildThis will create a trigger event when the CPU readings go above 90% and another event when the series
go below 70%.
To set a threshold trigger that detects when a series (e.g. cpu) is above a threshold longer than a time limit:
new DeviceOpsConfigBuilder()
.addSeriesTrigger("cpu", new ThresholdTimeoutTrigger(70.0, 50.0, Seconds(30), "above_threshold_30s"))
.buildThis will create a trigger when CPU readings remove above 70% without dipping below 50% for over 30secs.
Detecting quick changes in series can be done by connecting a threshold trigger to a derivative filter.
new DeviceOpsConfigBuilder()
.addSeriesFilter("cpu", "cpu_derived", new DerivativeFilter)
.addSeriesTrigger("cpu_derived", ThresholdTrigger(1.0, "cpu_increase_high", 0.0, "cpu_leveled_out"))
.buildAs noise on a series can make the derivative very jumpy, a smoothing filter can be applied before
new DeviceOpsConfigBuilder()
.addSeriesFilter("noisy_series", "smoothed_series", new Ewma(0.1)))
.addSeriesFilter("smoothed_series", "series_derived", new DerivativeFilter)
.addSeriesTrigger("series_derived", new ThresholdTrigger(1.0, "series_increase_high", 0.0, "series_leveled_out"))
.buildFor irregularly sampled series, EwmaIrregular can be used instead.
Detecting when a device has gone offline, i.e., last sent data to iobeam longer than a specific timeout threshold.
new DeviceOpsConfigBuilder()
.addDeviceTrigger(new DeviceTimeoutTrigger(Minutes(5), "device_offline"))
.buildQuestions? Please reach out to us at [email protected].