{"id":2874,"date":"2018-12-01T22:56:30","date_gmt":"2018-12-01T17:26:30","guid":{"rendered":"http:\/\/navveenbalani.dev\/?p=2874"},"modified":"2019-12-18T23:01:46","modified_gmt":"2019-12-18T17:31:46","slug":"building-application-with-open-source-iot-stack","status":"publish","type":"post","link":"https:\/\/navveenbalani.dev\/index.php\/articles\/building-application-with-open-source-iot-stack\/","title":{"rendered":"Building Application with Open Source IoT stack"},"content":{"rendered":"\n<p>This article is part of IoT Architecture Series &#8211; <a href=\"https:\/\/navveenbalani.dev\/index.php\/articles\/internet-of-things-architecture-components-and-stack-view\/\">https:\/\/navveenbalani.dev\/index.php\/articles\/internet-of-things-architecture-components-and-stack-view\/<\/a><\/p>\n\n\n\n<p>In this article, we will build an IoT stack using open source software. Many vendors are building their own platform for realizing their end to end use cases. Our assessment shows there are at least 100 plus providers providing IoT stack in some form or the other.<\/p>\n\n\n\n<p>Well, the first question that should come\nto your mind is why you need to build your own\nIoT stack when there are multiple platform providers providing the same\nfunctionality. We list some of the motivations below:<\/p>\n\n\n\n<ul><li>Start-up providers \u2013 The\nstart-up ventures eyes IoT as the hot cake in the market today and are at the\nforefront of building a cloud platform providing all kinds of cloud services,\nexploiting various gaps to solve industry-specific\nuse cases.<\/li><li>Hardware\/Embedded software\nmanufacturers \u2013 Many embedded software manufacturers,\nprovides a cloud platform to store the data from the devices, like various OBD vehicle\ndevice manufacturers. The data from devices are aggregated in the cloud and\nanalyzed using various tools and techniques.<\/li><li>Internal evaluation and demos \u2013\nSoftware service providers and System Integrators starting their IoT consulting\njourney or having an embedded software\npractice and looking at ways to analyze continuous streams of data and build\npredictive models as service offerings.<\/li><li>Regulatory compliance &#8211; Industries\nthat can\u2019t use cloud services due to compliance are building their own in-house platforms for monitoring and\ncondition maintenance activities.<\/li><\/ul>\n\n\n\n<p>Let\u2019s look at our open source IoT stack. We have envisioned this using a popular set of integrated open source software providing high performance, high scalability, and low throughput capabilities.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" width=\"1024\" height=\"771\" src=\"https:\/\/navveenbalani.dev\/wp-content\/uploads\/2019\/12\/opensource-iot-solution1-1024x771.jpg\" alt=\"\" class=\"wp-image-2875\" srcset=\"https:\/\/navveenbalani.dev\/wp-content\/uploads\/2019\/12\/opensource-iot-solution1-1024x771.jpg 1024w, https:\/\/navveenbalani.dev\/wp-content\/uploads\/2019\/12\/opensource-iot-solution1-300x226.jpg 300w, https:\/\/navveenbalani.dev\/wp-content\/uploads\/2019\/12\/opensource-iot-solution1-768x578.jpg 768w, https:\/\/navveenbalani.dev\/wp-content\/uploads\/2019\/12\/opensource-iot-solution1.jpg 1418w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p><strong>Open\nSource Device SDKs<\/strong><\/p>\n\n\n\n<p>The open source IoT platform provides support for wide variety of protocols like MQTT, AXMP, and HTTP protocol. For MQTT, we can leverage the Eclipse-based Paho library for connecting any device to the core platform or open source library like cyclon.js, which makes it easier to connect various devices using Node.js. We really liked the cyclone.js library and the extensions provided by the library to support various devices.&nbsp; The following is a snapshot of some of the devices supported by cyclon.js<\/p>\n\n\n\n<p><\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" width=\"1024\" height=\"835\" src=\"https:\/\/navveenbalani.dev\/wp-content\/uploads\/2019\/12\/opensource-iot-solution2-1024x835.jpg\" alt=\"\" class=\"wp-image-2876\" srcset=\"https:\/\/navveenbalani.dev\/wp-content\/uploads\/2019\/12\/opensource-iot-solution2-1024x835.jpg 1024w, https:\/\/navveenbalani.dev\/wp-content\/uploads\/2019\/12\/opensource-iot-solution2-300x245.jpg 300w, https:\/\/navveenbalani.dev\/wp-content\/uploads\/2019\/12\/opensource-iot-solution2-768x626.jpg 768w, https:\/\/navveenbalani.dev\/wp-content\/uploads\/2019\/12\/opensource-iot-solution2.jpg 1482w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Below is the sample code to connect to an analog sensor attached to pin \u201c0\u201d of a device, read the values and transmit the data to the cloud platform using MQTT protocol. It is just 20 lines of code.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>var Cylon = require('cylon');\nCylon.robot({\n  connections:{\n    mqtt:{ adaptor:'mqtt', host:'mqtt:\/\/ourserver:port'},\n    arduino:{ adaptor:'firmata', port:'\/dev\/ttyACM0'}\n},\n  devices:{\n    allimit:{ driver:'mqtt', topic:'llimit', adaptor:'mqtt'},\n    aulimit:{ driver:'mqtt', topic:'ulimit', adaptor:'mqtt'},\n    sensor:{ driver:\"analogSensor\", pin:0, upperLimit:900, lowerLimit:100}\n},\n  work:function(my)\n    my.sensor.on(\"upperLimit\",function(val){\n       my.aulimit.publish(val);\n});\n    my.sensor.on(\"lowerLimit\",function(val){\n      my.allimit.publish(val);\n});\n}\n}).start();\n<\/code><\/pre>\n\n\n\n<p><strong>Protocol\nBridge<\/strong><\/p>\n\n\n\n<p>This service acts as a router or gateway\nfor converting incoming protocols to the protocol supported by the core IoT\nplatform. For instance, if the core platform has Apache Kafka setup to process messages and the client\ncommunicates using MQTT, then we could write a\nMQTT server that will receive the data and convert it into the format\nthat Apache Kafka understands. The protocol bridge takes care of sending\nmessages to Apache Kafka over specified topics and determines which partition\nthe message is to be sent. <\/p>\n\n\n\n<p><strong>Apache\nKafka<\/strong><\/p>\n\n\n\n<p>Apache Kafka service provides us with a\nhighly scalable, low latency, fast and distributed publish-subscribe messaging\nsystem.<\/p>\n\n\n\n<p>Apache Kafka was developed at LinkedIn.&nbsp;\nThe following link is an excellent blog which talks about how Kafka was used and what problems it solved at\nLinkedIn &#8211; http:\/\/www.confluent.io\/blog\/stream-data-platform-1\/. Though Kafka was\nnot designed as a typical messaging system.\nit found its roots in many applications, which required high throughput,\ndistributed messaging framework.<\/p>\n\n\n\n<p>Similar to any publish-subscribe messaging\nsystem; Kafka maintains feeds of messages in categories called topics.\nProducers publish data to topics and consumers subscribe to topics to read\nmessages. In our case, the publisher is the protocol bridge that posts the\nmessages (data from devices) on specified topics. Topics are further\npartitioned and replicated across nodes. For the connected car use case, we use\nthe main controller device id (the telematics device id or the hardware device\nid connected to OBD port) as the partition id to ensure all messages from the\nsame device id ends up in the same partition.<\/p>\n\n\n\n<p>Kafka can retain messages after the\nspecified time interval has elapsed, unlike other messaging systems that delete\nmessages as soon as they are consumed.&nbsp; Going through the entire capabilities of\nKafka and its working would require a book in itself. We suggest going through\nthe document http:\/\/kafka.apache.org\/documentation.html, which provides an\nexcellent source of information on Apache Kafka.<\/p>\n\n\n\n<p><strong>Cassandra<\/strong><\/p>\n\n\n\n<p>We use Apache Cassandra for storing the\ncontinuous stream of data coming from devices. We create a Kafka consumer that listens to a specified topic, consumes the\nmessage and stores the message in one of the Cassandra tables. <\/p>\n\n\n\n<p>We use Cassandra for historical data\nanalysis to gain insights on various usages, aggregations,\nand computations, build correlations, and to develop our machine learning\nmodels iteratively for anomaly detection and predictive analytics. For developing\nmachine learning models, we use Spark Core libraries for connecting to our\nCassandra tables and perform data transformation and Spark MLlib libraries for\ndeveloping machine learning models.<\/p>\n\n\n\n<p><strong>Apache\nSpark Streaming<\/strong><\/p>\n\n\n\n<p>Apache Spark streaming component from\nApache Spark project adds real-time data stream processing and data\ntransformation for further processing by systems. We chose Apache Spark as it\u2019s\nan ultra-fast in-memory data processing framework. It provides a stack of\nlibraries including SQL and DataFrames, MLlib for machine learning, GraphX, and\nSpark Streaming. Apache Spark streaming supports real-time processing as well\nas batch updates on top of Spark engine, which makes it the perfect choice for\napplications which requires responding to real-time events and batch processing\nvia Hadoop jobs for complex data analysis. Apache Spark streaming component\nprovides first class integration with Apache Kafka using the Spark Stream Kafka\nmodule (spark-streaming-kafka_2.10 version was the latest when we tested the\nintegration). Details of integration are available at this link &#8211;\nhttp:\/\/spark.apache.org\/docs\/latest\/streaming-kafka-integration.html. There are\ntwo approaches \u2013 Receiver based approach, which uses Kafka high-level consumer APIs\nto consume the messages from a specified topic. This\nis the old way and requires some workaround to ensure there is no data\nloss during failures. The other approach is using a direct approach which\nperiodically queries Kafka for the latest offsets in each topic + partition and\nuses the offset ranges to process data in batches. We have used direct approach\nfor integration and recommend using the same.&nbsp;\nOnce integrated, the streams of data start flowing in Apache Spark\nSteaming layer, where you can do real-time\ntransformation, filtering and store the data for further analysis or invoke\nmachine learning models (like streaming linear regression).<\/p>\n\n\n\n<p><strong>Apache\nSpark MLlib<\/strong><\/p>\n\n\n\n<p>Apache Spark MLlib service is used to build\nmachine learning models or combine multiple machine learning models using a\nstandardized API. Building a machine learning models requires a series of step\nas discussed earlier \u2013like cleansing and transformation, creating feature\nvectors, correlation, splitting up data in training and test sets, selecting\nalgorithms for building up required models (prediction, classification,\nregression, etc.) and iteratively training the model for required accuracy. &nbsp;Typically multiple tools are required to carry\nout the tasks describe earlier and using Spark APIs and ML libraries everything\ncan be developed iteratively in a single environment.<\/p>\n\n\n\n<p><strong>Custom Rules and Events<\/strong><\/p>\n\n\n\n<p>For custom rules and events, we create\ncustom application code which is executed\nas part of the spark streaming flow and\ntriggers the required action based on the incoming data. Apache Zeppelin\nproject can be used to quickly build an\ninteractive data analytics dashboard. Zeppelin has built-in support for\nSpark integration and SparkSQL. The project is under incubation and evolving. <\/p>\n\n\n\n<h3><a>Implementation Overview<\/a><\/h3>\n\n\n\n<p>The implementation steps of realizing the connected car or elevator use case using the open source IoT stack is pretty much the same that we described earlier for IoT stacks from Microsoft, Amazon, and IBM, and it\u2019s just a matter of replacing services provided by these vendors with open source offerings described above. Though we did not design a complete IoT stack, as we left out the device management capabilities, it still provides a decent perspective of building an end-to-end IoT solution using open source products. The device management and security aspect require developing custom components to meet the requirements. We leave this as an exercise for you to bring out the best of open source platforms and envision a complete IoT stack.<\/p>\n\n\n\n<p>This completes the article series.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This article is part of IoT Architecture Series &#8211; https:\/\/navveenbalani.dev\/index.php\/articles\/internet-of-things-architecture-components-and-stack-view\/ In this article, we will build an IoT stack using open source software. Many vendors are building their own platform for realizing their end to end use cases. Our assessment shows there are at least 100 plus providers providing IoT stack in some form or [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":2876,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[3,156],"tags":[287],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v16.0.2 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Building Application with Open Source IoT stack - Current and Future Technology Trends by Navveen Balani<\/title>\n<meta name=\"description\" content=\"Building Application with Open Source IoT stack - Articles\" \/>\n<link rel=\"canonical\" href=\"https:\/\/navveenbalani.dev\/index.php\/articles\/building-application-with-open-source-iot-stack\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Building Application with Open Source IoT stack - Current and Future Technology Trends by Navveen Balani\" \/>\n<meta property=\"og:description\" content=\"Building Application with Open Source IoT stack - Articles\" \/>\n<meta property=\"og:url\" content=\"https:\/\/navveenbalani.dev\/index.php\/articles\/building-application-with-open-source-iot-stack\/\" \/>\n<meta property=\"og:site_name\" content=\"Current and Future Technology Trends by Navveen Balani\" \/>\n<meta property=\"article:published_time\" content=\"2018-12-01T17:26:30+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-12-18T17:31:46+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/navveenbalani.dev\/wp-content\/uploads\/2019\/12\/opensource-iot-solution2.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1482\" \/>\n\t<meta property=\"og:image:height\" content=\"1208\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\">\n\t<meta name=\"twitter:data1\" content=\"7 minutes\">\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/navveenbalani.dev\/#website\",\"url\":\"https:\/\/navveenbalani.dev\/\",\"name\":\"Current and Future Technology Trends by Navveen Balani\",\"description\":\"Current and Future Technology Trends by Navveen Balani\",\"publisher\":{\"@id\":\"https:\/\/navveenbalani.dev\/#\/schema\/person\/51f7ab14b20611d95e3c7fd4ea0950bf\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/navveenbalani.dev\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/building-application-with-open-source-iot-stack\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/navveenbalani.dev\/wp-content\/uploads\/2019\/12\/opensource-iot-solution2.jpg\",\"width\":1482,\"height\":1208},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/building-application-with-open-source-iot-stack\/#webpage\",\"url\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/building-application-with-open-source-iot-stack\/\",\"name\":\"Building Application with Open Source IoT stack - Current and Future Technology Trends by Navveen Balani\",\"isPartOf\":{\"@id\":\"https:\/\/navveenbalani.dev\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/building-application-with-open-source-iot-stack\/#primaryimage\"},\"datePublished\":\"2018-12-01T17:26:30+00:00\",\"dateModified\":\"2019-12-18T17:31:46+00:00\",\"description\":\"Building Application with Open Source IoT stack - Articles\",\"breadcrumb\":{\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/building-application-with-open-source-iot-stack\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/navveenbalani.dev\/index.php\/articles\/building-application-with-open-source-iot-stack\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/building-application-with-open-source-iot-stack\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/navveenbalani.dev\/\",\"url\":\"https:\/\/navveenbalani.dev\/\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"position\":2,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/building-application-with-open-source-iot-stack\/\",\"url\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/building-application-with-open-source-iot-stack\/\",\"name\":\"Building Application with Open Source IoT stack\"}}]},{\"@type\":\"Article\",\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/building-application-with-open-source-iot-stack\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/building-application-with-open-source-iot-stack\/#webpage\"},\"author\":{\"@id\":\"https:\/\/navveenbalani.dev\/#\/schema\/person\/51f7ab14b20611d95e3c7fd4ea0950bf\"},\"headline\":\"Building Application with Open Source IoT stack\",\"datePublished\":\"2018-12-01T17:26:30+00:00\",\"dateModified\":\"2019-12-18T17:31:46+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/building-application-with-open-source-iot-stack\/#webpage\"},\"publisher\":{\"@id\":\"https:\/\/navveenbalani.dev\/#\/schema\/person\/51f7ab14b20611d95e3c7fd4ea0950bf\"},\"image\":{\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/building-application-with-open-source-iot-stack\/#primaryimage\"},\"keywords\":\"iot-guide\",\"articleSection\":\"Articles,IOT\",\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/navveenbalani.dev\/#\/schema\/person\/51f7ab14b20611d95e3c7fd4ea0950bf\",\"name\":\"Navveen\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/navveenbalani.dev\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/navveenbalani.dev\/wp-content\/uploads\/2019\/07\/navveen_balani.jpeg\",\"width\":200,\"height\":200,\"caption\":\"Navveen\"},\"logo\":{\"@id\":\"https:\/\/navveenbalani.dev\/#personlogo\"},\"sameAs\":[\"http:\/\/naveenbalani.com\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/posts\/2874"}],"collection":[{"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/comments?post=2874"}],"version-history":[{"count":2,"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/posts\/2874\/revisions"}],"predecessor-version":[{"id":2878,"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/posts\/2874\/revisions\/2878"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/media\/2876"}],"wp:attachment":[{"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/media?parent=2874"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/categories?post=2874"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/tags?post=2874"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}