{"id":2789,"date":"2018-12-01T18:59:02","date_gmt":"2018-12-01T13:29:02","guid":{"rendered":"http:\/\/navveenbalani.dev\/?p=2789"},"modified":"2019-12-18T19:22:36","modified_gmt":"2019-12-18T13:52:36","slug":"internet-of-things-core-platform-layer","status":"publish","type":"post","link":"https:\/\/navveenbalani.dev\/index.php\/articles\/internet-of-things-core-platform-layer\/","title":{"rendered":"Internet of Things – Core Platform Layer"},"content":{"rendered":"\n

This article is part of IoT Architecture Series – https:\/\/navveenbalani.dev\/index.php\/articles\/internet-of-things-architecture-components-and-stack-view\/.<\/a><\/p>\n\n\n\n

Core Platform Layer<\/a><\/h3>\n\n\n\n

The core\nplatform layer provides a set of capabilities to connect, collect,\nmonitor and control millions of devices. Let\u2019s look at each of the components\nof the core platform layer.<\/p>\n\n\n\n

Protocol Gateway<\/a><\/h4>\n\n\n\n

An Enterprise IoT platform typically supports one protocol end to end like AMQP or MQTT as part of the overall stack.  However in an IoT landscape, where there are no standardized protocols where all vendors can converge on, an Enterprise IoT stack needs to provide support for commonly used protocols, industry protocols, and support for evolving standards in future. <\/p>\n\n\n\n

One option is to use a device gateway that\nwe had discussed earlier to convert device proprietary protocols to the\nprotocol supported by the platform for communication, but that may not always\nbe possible as devices may connect directly or the device gateway may not\nsupport the protocol supported by the IoT stack. In order to support handling multiple protocols, a protocol gateway\nis used, which does the conversion of the protocol supported by your IoT stack.\nBuilding an abstraction like protocol gateway would make it easier to support\ndifferent protocols in future.<\/p>\n\n\n\n

The protocol\ngateway layer provides connectivity to the devices over protocols\nsupported by the IoT stack. Typically the communication is channelized to the\nmessaging platform or middleware like MQTT or AMQP. The protocol gateway layer can also act as a facade for supporting\ndifferent protocols, performing conversions across protocols and handing off\nthe implementation to corresponding IoT messaging platform.<\/p>\n\n\n\n

IoT Messaging Middleware<\/a><\/h4>\n\n\n\n

Messaging middleware is a software or an\nappliance that allows senders (publishers) and receivers (consumer\u2019s consuming\nthe information) to distribute messages in a loosely coupled manner without\nphysically connected to each other. A message middleware allows multiple\nconsumers to receive messages from a single publisher,\nor a single consumer can send messages to multiple senders. <\/p>\n\n\n\n

Messaging middleware is not a new concept; it has been used as a backbone for message communication between enterprise systems, as an integration pattern with various distributed systems in a unified way or inbuilt as part of the application server. In the context of IoT, the messaging middleware becomes a key capability providing a highly scalable, high-performance middleware to accommodate a vast number of ever-growing connected devices. Gartner, Inc. forecasts that 4.9 billion connected things will be in use in 2015 and will reach 25 billion by 2020. <\/p>\n\n\n\n

The IoT messaging middleware platform needs\nto provide various device management aspects like registering the devices,\nenabling storage of device meta-model, secure connectivity for devices, storage\nof device data for specified interval and dashboards to view connected devices.\nThe storage requirements imposed by an IoT platform is quite different from a\ntraditional messaging platform as we are looking at terabytes of data from the\nconnected devices and at the same time ensuring high performance and fault\ntolerance guarantees. The IoT messaging middleware platform typically holds the device data for the specified\ninterval and in-turn a dedicated storage service is used to scale, compute and\nanalyze information. <\/p>\n\n\n\n

The device meta-model that we mentioned\nearlier is one of the key aspects for an\nEnterprise IoT application. A device meta-model can be visualized as a set of metadata about the device, parameters\n(input and output) and functions (send, receive) that a device may emit or\nconsume. By designing the device meta-model as part of the IoT solution, you\ncould create a generalized solution for each industry\/verticals which abstracts\nout the dependency between data send by the devices and data used by the IoT\nplatform and services that work on that\ndata. You can even work on a virtual device using the device model and build\nand test the entire application, without physically connected the device. We would\ntalk about the device meta-model in detail in the solution section.<\/p>\n\n\n\n

All the capabilities of the IoT stack,\nstarting from the messaging platform right up to the cognitive platform are\ntypically made available as services over the cloud platform, which can be\nreadily consumed to build IoT applications.<\/p>\n\n\n\n

We would revisit the topic in detail in\nChapter 3 when we talk about the IoT\ncapabilities offered by popular cloud vendors and using open source software.<\/p>\n\n\n\n

Data Storage<\/a><\/h4>\n\n\n\n

The data storage component deals with storage of continuous stream of data from devices. As mentioned earlier we are looking at a highly scalable storage service which can storage\nterabytes of data and enable faster retrieval of data. The data needs to be\nreplicated across servers to ensure high availability and no single point of\nfailure.<\/p>\n\n\n\n

Typically a NoSQL database or high\nperformance optimized storage is used for storing data. The design of data\nmodel and schema becomes a key to enable faster retrieval, perform computations\nand makes it easier for down processing systems, which use the data for analysis. For\ninstance, if you are storing data from a connected car every minute, your data\ncan be broken down in ids and values, the id field would not change for a\nconnected car instance, but values would keep on changing \u2013 for instance like\nspeed =60 km\/hour, speed =65 km\/hour, etc. In that case instead of\nstoring key=value, you can store key=value1, value2 and key=timeint1, timeint2\nand so on. The data represents a sequence of values for a specific attribute\nover a period of time. This concept is referred to as time series, and database, which supports these requirements, is called as Time Series database. You could\nuse NoSQL databases like MongoDB or Cassandra and design your schema in a way\nthat it is optimized for storing time\nseries data and doing statistical computations. \nNot all use cases may require the use of time series database, but it is an important concept to\nkeep in mind while designing IoT applications.<\/p>\n\n\n\n

In the context of an enterprise IoT application, the data storage layer should also support storage and handling of unstructured and semi-structured information. The data from these sources (structured, unstructured and semi-structured) can be used to correlate and derive insights. For instance, information data from equipment manual (unstructured text of information) can be fed into the system, and sensor data from the connected equipment can be correlated with the equipment manuals for raising critical functional alerts and suggesting corrective measures. The IoT messaging middleware service usually provides a set of configurations to store the incoming data from the devices automatically into the specified storage service. <\/p>\n\n\n\n

In a future article , I would talk about various storage service options provided by cloud providers for storing a massive amount of data from connected devices.<\/p>\n\n\n\n

Data aggregation and filter<\/a><\/h4>\n\n\n\n

The IoT core platform deals with raw data\ncoming from multiple devices, and not all\ndata needs to be consumed and treated equally by your application.  We talked about device gateway pattern earlier which can filter data before sending\nit over to the cloud platform. Your device gateway may not have the luxury and\ncomputation power to store and filter out volumes of data or be able to filter\nout all scenarios. In some solutions, the devices can directly connect to the\nplatform without a device gateway. As part of your IoT application, you need to\ndesign this carefully as what data needs to be\nconsumed and what data might not be relevant in that context. The data\nfilter component could provide simple rules to complex conditions based on your\ndata dependency graph to filter out the incoming data. The data mapper\ncomponent is also used to convert raw data from the devices into an abstract\ndata model which is used by rest of the components.<\/p>\n\n\n\n

In some cases, you need to contextualize\nthe device data with more information, like aggregating the current data from\ndevices with existing asset management software to retrieve warranty\ninformation of physical devices or from a weather station for further analysis.\nThat\u2019s where the data\naggregation components come in which\nallows to aggregate and enrich the incoming data.  The aggregation component can also be\npart of the messaging stream processing framework that we will discuss in the\nnext section, but instead of using complex flows for just aggregating information,\nthis requirement could be easily handled without much overhead using simplified\nflows and custom coding, without using a stream processing infrastructure. <\/p>\n\n\n\n

Analytics Platform Layer<\/a><\/h3>\n\n\n\n

The Analytics Platform layer provides a set of key capabilities to analyze large volumes of information, derive insights and enable applications to take required action.<\/p>\n\n\n\n

Stream Processing<\/a><\/h4>\n\n\n\n

Real-time stream processing is about processing streams of data from devices (or\nany source) in real-time, analyze the information, do computations and trigger\nevents for required actions.  The stream\nprocessing infrastructure directly interacts with the IoT Messaging Middleware component\nby listening to specified topics. The stream processing infrastructure acts as\na subscriber, which consumes messages (data from devices) arriving continuously\nat the IoT Messaging Middleware layer.   <\/p>\n\n\n\n

A typical requirement for stream processing\nsoftware includes high scalability, handling a large volume of continuous data,\nprovide fault tolerance and support for interactive queries in some form like\nusing SQL queries which can act on the stream of data and trigger alerts if\nconditions are not met.<\/p>\n\n\n\n

Most of the big data implementations\nstarted with Hadoop which supported only batch processing, but with the advent\nof various real-time streaming technologies and changing the requirement of\ndealing with massive amount of data in real-time, applications are now\nmigrating to stream based technology that can handle data processing in real-time.\nStream processing platform like Apache Spark Streaming enables you to write\nstream jobs to process streaming data using Spark API. It enables you to\ncombine streams with batch and interactive queries. Projects already using\nexisting Hadoop-based batch processing\nsystem can still take the benefit of real-time stream processing by combining\nthe batch processing queries with stream based interactive queries offered by\nSpark Streaming. <\/p>\n\n\n\n

The stream processing engine acts as a data processing backbone which can hand off streams of data to multiple other services simultaneously for parallel execution or to your own custom application to process the data. For instance, a stream processing instance can invoke a set of custom applications, one which can execute complex rules and other invoking a machine learning service.<\/p>\n\n\n\n

Machine Learning<\/a><\/h4>\n\n\n\n

Following is the Wikipedia definition of\nMachine Learning \u2013<\/p>\n\n\n\n

\u201cMachine\nlearning explores the study and construction of algorithms that can learn from\nand make predictions on data.\u201d<\/em><\/p>\n\n\n\n

In simple terms, machine learning is how we\nmake computers learn from data using various algorithms without explicitly\nprogramming it so that it can provide the\nrequired outcome \u2013 like classifying an email as spam or not spam or predicting\na real estate price based on historical values and other environmental factors.<\/p>\n\n\n\n

Machine learning types are typically classified into three broad categories <\/p>\n\n\n\n