{"id":2298,"date":"2016-11-11T17:43:12","date_gmt":"2016-11-11T12:13:12","guid":{"rendered":"http:\/\/naveenbalani.com\/?p=2298"},"modified":"2016-11-11T18:10:06","modified_gmt":"2016-11-11T12:40:06","slug":"applying-deep-learning-and-computer-vision-on-the-edge-gateway","status":"publish","type":"post","link":"https:\/\/navveenbalani.dev\/index.php\/articles\/applying-deep-learning-and-computer-vision-on-the-edge-gateway\/","title":{"rendered":"Applying deep learning and computer vision on the Edge Gateway"},"content":{"rendered":"<p>Edge gateway is about providing real-time decision making at the edge. In this article, I will talk about how to apply computer vision at the edge for various security and surveillance activities. For instance, you would like to detect suspicious activities at the ATM, simple cases like people wearing helmets and entering the ATM to unusual movements or would like to capture and identify objects in remote areas using Drones like crops length to mining activities. In such scenario, you would need real-time decision making at the edge as it might not make sense to transfer the data over the cloud for processing, be it for latency issues, bandwidth, cost issues or there might be none or very\u00a0limited connectivity. In this case you employ intelligence at the edge of the devices.<\/p>\n<p><img loading=\"lazy\" class=\"alignleft size-medium wp-image-2302\" src=\"http:\/\/naveenbalani.com\/wp-content\/uploads\/2016\/09\/EDGEGATEWAY-USE-300x134.png\" alt=\"edgegateway-use\" width=\"300\" height=\"134\" srcset=\"https:\/\/navveenbalani.dev\/wp-content\/uploads\/2016\/09\/EDGEGATEWAY-USE-300x134.png 300w, https:\/\/navveenbalani.dev\/wp-content\/uploads\/2016\/09\/EDGEGATEWAY-USE-768x344.png 768w, https:\/\/navveenbalani.dev\/wp-content\/uploads\/2016\/09\/EDGEGATEWAY-USE.png 792w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/>In order to build out the solution, you need to employ computer vision algorithms on the edge. You can build this using commercial available API\u2019s or using various open source deep learning framework like Theano, TensorFlow, Caf\u00e9 etc. \u00a0Deep learning is a branch of machine learning for learning multiple levels of representation through neural networks.\u00a0 Neural network in image processing, automatically infer rules for recognizing images instead of you extracting thousands of features to detect images.<\/p>\n<p>Various deep learning architectures are available like convolutional neural network, recurring neural network to solve specific problems like computer vision, natural language processing, speech recognition and achieves state of the art results.<\/p>\n<p>For Computer Vision, we specially use a CNN network to identify images or use of the pre-trained instance (like Inception model released by Google which is a 22 layers\u2019 deep network which achieves start of the art results for classification and detection for images). For a computer, images are nothing but a vector or set of numbers\/pixels. Convolutional Neural Networks learns features automatically using small frames of equal size images and each network gets the same input and interesting pieces are extracted (for more details, please refer to how CNN works). \u00a0The extract of interesting pieces (which are again vector of numbers) is the heart of how CNN works. For instance, in case of helmet, some network would learn a round edge, some may learn a glass pane in the front and so on. The idea is irrespective of where the object is in the frame, you should be able to identify to image.<\/p>\n<p>Prior to CNN for image detection, you would need to crop the images and provide area of interest. For instance, if you are detecting various categories of birds, you would usually crop the bird image and remove any surrounding context, like trees, bushes, sea, sky etc and provide the bird image only. With CNN, the idea is to train with those images and let network figure this out (though cropping may help in some cases for increasing accuracy). Based on my experiments, the CNN is able to predict object most of the times which have surrounding context, but with lower accuracy. As mentioned, the idea is to identify objects irrespective of where they are found in the image and I am sure lot of research is going on to improve the CNN networks. Having the right training data (images and label) is a must for training networks with such variations.<\/p>\n<p><img loading=\"lazy\" class=\"alignleft size-medium wp-image-2301\" src=\"http:\/\/naveenbalani.com\/wp-content\/uploads\/2016\/09\/edgeo-207x300.png\" alt=\"edgeo\" width=\"207\" height=\"300\" srcset=\"https:\/\/navveenbalani.dev\/wp-content\/uploads\/2016\/09\/edgeo-207x300.png 207w, https:\/\/navveenbalani.dev\/wp-content\/uploads\/2016\/09\/edgeo.png 285w\" sizes=\"(max-width: 207px) 100vw, 207px\" \/><\/p>\n<p>&nbsp;<\/p>\n<p>On the left hand side, is the stack view of employing deep learning API on the edge gateway. It consists of API, the learning model and the classification service which\u00a0is used for classification of objects. There are lot of innovations happening to optimize the use of deep learning libraries on the edge.<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>So how do you go about implementing\u00a0this. I will talk about one approach of building this out &#8211;<\/p>\n<ol>\n<li>Build your own CNN or Start with a pertained CNN network (like an Inception Model)<\/li>\n<li>Get the training and test data (images and labels)<\/li>\n<li>Train or\u00a0retrain (i.e transfer learning) the network.<\/li>\n<li>Optimize with learning rates, batch size etc. for desired accuracy. Get the trained model<\/li>\n<li>Use the trained model for classification.<\/li>\n<li>Deploy TensorFlow API and trained model on the edge \u00a0(you can package this in docker)<\/li>\n<li>Use the classification code on the edge gateway to detect\u00a0objects.<\/li>\n<\/ol>\n<p><em>The following is part of my presentation that I delivered at IoTNext. I will update the article with the youtube video once available.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Edge gateway is about providing real-time decision making at the edge. In this article, I will talk about how to apply computer vision at the edge for various security and surveillance activities. For instance, you would like to detect suspicious activities at the ATM, simple cases like people wearing helmets and entering the ATM to [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":2301,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[66,3,267,269,156],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v16.0.2 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Applying deep learning and computer vision on the Edge Gateway - Current and Future Technology Trends by Navveen Balani<\/title>\n<meta name=\"description\" content=\"Applying deep learning and computer vision on the Edge Gateway -\" \/>\n<link rel=\"canonical\" href=\"https:\/\/navveenbalani.dev\/index.php\/articles\/applying-deep-learning-and-computer-vision-on-the-edge-gateway\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Applying deep learning and computer vision on the Edge Gateway - Current and Future Technology Trends by Navveen Balani\" \/>\n<meta property=\"og:description\" content=\"Applying deep learning and computer vision on the Edge Gateway -\" \/>\n<meta property=\"og:url\" content=\"https:\/\/navveenbalani.dev\/index.php\/articles\/applying-deep-learning-and-computer-vision-on-the-edge-gateway\/\" \/>\n<meta property=\"og:site_name\" content=\"Current and Future Technology Trends by Navveen Balani\" \/>\n<meta property=\"article:published_time\" content=\"2016-11-11T12:13:12+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2016-11-11T12:40:06+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/navveenbalani.dev\/wp-content\/uploads\/2016\/09\/edgeo.png\" \/>\n\t<meta property=\"og:image:width\" content=\"285\" \/>\n\t<meta property=\"og:image:height\" content=\"413\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\">\n\t<meta name=\"twitter:data1\" content=\"4 minutes\">\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/navveenbalani.dev\/#website\",\"url\":\"https:\/\/navveenbalani.dev\/\",\"name\":\"Current and Future Technology Trends by Navveen Balani\",\"description\":\"Current and Future Technology Trends by Navveen Balani\",\"publisher\":{\"@id\":\"https:\/\/navveenbalani.dev\/#\/schema\/person\/51f7ab14b20611d95e3c7fd4ea0950bf\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/navveenbalani.dev\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/applying-deep-learning-and-computer-vision-on-the-edge-gateway\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/navveenbalani.dev\/wp-content\/uploads\/2016\/09\/edgeo.png\",\"width\":285,\"height\":413},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/applying-deep-learning-and-computer-vision-on-the-edge-gateway\/#webpage\",\"url\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/applying-deep-learning-and-computer-vision-on-the-edge-gateway\/\",\"name\":\"Applying deep learning and computer vision on the Edge Gateway - Current and Future Technology Trends by Navveen Balani\",\"isPartOf\":{\"@id\":\"https:\/\/navveenbalani.dev\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/applying-deep-learning-and-computer-vision-on-the-edge-gateway\/#primaryimage\"},\"datePublished\":\"2016-11-11T12:13:12+00:00\",\"dateModified\":\"2016-11-11T12:40:06+00:00\",\"description\":\"Applying deep learning and computer vision on the Edge Gateway -\",\"breadcrumb\":{\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/applying-deep-learning-and-computer-vision-on-the-edge-gateway\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/navveenbalani.dev\/index.php\/articles\/applying-deep-learning-and-computer-vision-on-the-edge-gateway\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/applying-deep-learning-and-computer-vision-on-the-edge-gateway\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/navveenbalani.dev\/\",\"url\":\"https:\/\/navveenbalani.dev\/\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"position\":2,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/applying-deep-learning-and-computer-vision-on-the-edge-gateway\/\",\"url\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/applying-deep-learning-and-computer-vision-on-the-edge-gateway\/\",\"name\":\"Applying deep learning and computer vision on the Edge Gateway\"}}]},{\"@type\":\"Article\",\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/applying-deep-learning-and-computer-vision-on-the-edge-gateway\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/applying-deep-learning-and-computer-vision-on-the-edge-gateway\/#webpage\"},\"author\":{\"@id\":\"https:\/\/navveenbalani.dev\/#\/schema\/person\/51f7ab14b20611d95e3c7fd4ea0950bf\"},\"headline\":\"Applying deep learning and computer vision on the Edge Gateway\",\"datePublished\":\"2016-11-11T12:13:12+00:00\",\"dateModified\":\"2016-11-11T12:40:06+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/applying-deep-learning-and-computer-vision-on-the-edge-gateway\/#webpage\"},\"publisher\":{\"@id\":\"https:\/\/navveenbalani.dev\/#\/schema\/person\/51f7ab14b20611d95e3c7fd4ea0950bf\"},\"image\":{\"@id\":\"https:\/\/navveenbalani.dev\/index.php\/articles\/applying-deep-learning-and-computer-vision-on-the-edge-gateway\/#primaryimage\"},\"articleSection\":\"Architecture Patterns,Articles,Artificial Intelligence,Deep Learning,IOT\",\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/navveenbalani.dev\/#\/schema\/person\/51f7ab14b20611d95e3c7fd4ea0950bf\",\"name\":\"Navveen\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/navveenbalani.dev\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/navveenbalani.dev\/wp-content\/uploads\/2019\/07\/navveen_balani.jpeg\",\"width\":200,\"height\":200,\"caption\":\"Navveen\"},\"logo\":{\"@id\":\"https:\/\/navveenbalani.dev\/#personlogo\"},\"sameAs\":[\"http:\/\/naveenbalani.com\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/posts\/2298"}],"collection":[{"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/comments?post=2298"}],"version-history":[{"count":6,"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/posts\/2298\/revisions"}],"predecessor-version":[{"id":2308,"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/posts\/2298\/revisions\/2308"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/media\/2301"}],"wp:attachment":[{"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/media?parent=2298"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/categories?post=2298"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/navveenbalani.dev\/index.php\/wp-json\/wp\/v2\/tags?post=2298"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}