Distilling knowledge from a Teacher to a Student in a Deep Neural Network

Image for post
Image for post
Photo by Mikael Kristenson on Unsplash

In this article, you will learn.

  • An easy to understand explanation of Teacher-Student knowledge distillation neural networks

Overview of Knowledge Distillation

Large deep neural networks or an ensemble of deep neural networks built for better accuracy are computationally expensive, resulting in longer inference times which may not be ideal for near-real-time inference required at the Edge.

The training requirements for a model are different from requirements at the inference. …

A simplistic explanation of what is Blockchain and how it works

You are in a grocery store and scan the QR code on your favorite coffee. You can view the details of where the coffee beans grew, cleaned, triaged, processed, and then roasted. what-if you can trace the supply and distribution to know the means of transport and how long it took to arrive at the store and so forth

Is there a technology that can help record information at each step and make that information available to everyone transparently?

Welcome to the world of Blockchain

Image for post
Image for post
image by author

Blockchain is an immutable, distributed, decentralized; peer-to-peer ledger replicated across multiple nodes connected in a…

A Stochastic Optimization Technique Inspired by Nature

Particle Swarm Optimization(PSO) is a meta-heuristic stochastic nonlinear optimization algorithm proposed by Kennedy and Eberhart in 1995 to solve computationally challenging optimization problems.

Image for post
Image for post
Photo by zhan zhang on Unsplash


Particle Swarm Optimization(PSO) is inspired by nature. Flocks of bird and fish schools adjust their physical movement to avoid predators, seek food or mates, or optimize environmental parameters such as temperature.

Birds and fishes use swarm intelligence, social behavior, and movement dynamics to share information among their group for a greater survival advantage.


The PSO optimization objective is to find the optimum function that either maximizes the accuracy or precision or minimizes the loss.

The objective of PSO is to find a global optimum over a search space. A swarm in the algorithm is represented by a set of particles…

Explore and Understand Visual Transformers to perform Image Recognition

You will explore

  • Strength and weaknesses of the current Convolutional Neural Network

Strength and Weaknesses with Convolutional Neural Network(CNN)


  • CNN has inductive biases like translation invariance and locally restricted receptive field. Translational invariance recognizes an object even when its appearance varies, like changing orientation or zoom in or zoom out.


  • Specifically designed for images.

Implement Facial Matching Algorithm using Siamese Network in TensorFlow

My previous post was an Intuitive explanation of the Siamese Network, and in this post, it is the implementation of the Siamese network for Facial Recognition in TensorFlow.

Facial Recognition

Facial Recognition is to identify or verify people in the images or videos. It is the task to match faces to a given face or finding similar faces from a given facial dataset.

Image for post
Image for post
Image source:https://en.wikipedia.org/wiki/Dalai_Lama

Siamese Network is a One-shot classifier with two mirror image subnetworks to rank similarity or dissimilarity between the two inputs using Similarity score.

The Density-based Unsupervised Clustering algorithm robust to outliers

You will learn what DBSCAN is, how it works, the pros and cons of DBSCAN, and finally, implementation.

What is DBSCAN?

DBSCAN is Density-Based Spatial Clustering for Applications with Noise, an unsupervised clustering algorithm which finds core sample of high density and expands cluster from them

  • Clusters data into a high-density area separated by a low-density area.

An intuitive explanation of Siamese Network

You have a new account holder in your bank and would like to set his signature for verification. You have only one sample signature from the member. How can you use a neural network to perform the signature verification?

To find the answer, read on…

Siamese network inspired by Siamese twins has a unique architecture to naturally rank similarity or dissimilarity between inputs.

Key features of Siamese Network

  • Siamese network takes two different inputs passed through two similar subnetworks with the same architecture, parameters, and weights.

Perform fraud detection using Autoencoders in TensorFlow

Learn what are AutoEncoders, how they work, their usage, and finally implement Autoencoders for anomaly detection.

AutoEncoder is a generative unsupervised deep learning algorithm used for reconstructing high-dimensional input data using a neural network with a narrow bottleneck layer in the middle which contains the latent representation of the input data.

Image for post
Image for post

Autoencoder consists of an Encoder and a Decoder.

  • Encoder network: Accepts high-dimensional input data and translates it to latent low-dimensional data. The input size to an Encoder network is larger than its output size.

Explore One-Class SVM for Anomaly detection

This post is the second in the series; here you will explore One-Class SVM, a semi-supervised technique for anomaly detection.

Anomaly Detection Techniques: Part 1- Understand Inter-Quartile Range, Elliptic Envelope, and Isolated Forest

Anomaly detection using Machine Learning can be divided into Supervised, Semi-Supervised, or Unsupervised algorithms.

  • Supervised Anomaly Detection: A labeled dataset with inliers and outlier data points where learning happens based on the labeled dataset used for training.

Explore Interquartile Range, Elliptic Envelope, and Isolated Forest for Anomaly or Outlier Detection

In this post, you will explore supervised, semi-supervised, and unsupervised techniques for Anomaly detection like Interquartile range, Isolated forest, and Elliptic envelope for identifying anomalies in data.

You might have played “Find the odd one out” as a kid where you were given a few data points like Rose, Lily, Tulip, Daffodils, and Egg Plant, and then based on the data pattern, figured out that Egg Plant was the odd one out as all the remaining items are the name of flowers and Egg Plant is a vegetable.

Anomaly detection is very similar to finding the odd one out where…

Renu Khandelwal

Loves learning, sharing, and discovering myself. Passionate about Machine Learning and Deep Learning

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store