The coronavirus disease 2019 (COVID-19) has made it mandatory for people all over the world to wear facial masks to prevent the spread of the virus. As the mask hides most of the face, even humans find it difficult to identify a known face. Removing the mask can risk the health of people as it can lead to the transmission of the virus. The conventional face recognition systems used for security purposes have become incapable in the current situation because, the face mask covers most of the important facial features such as nose, mouth, etc. making it very difficult to recognize the person. The efficacy of a face recognition system is affected when trying to identify a masked face. We have proposed a system that uses the deep metric learning technique using our own FaceMaskNet-21 deep learning network to produce 128-d encodings that help in the face recognition process from static images, live video streams, as well as, video files. The HOG technique is used to enable faster recognition of faces with a mask. We achieved an accuracy of 88.92\% with an execution time of less than 10 milliseconds. The ability of the system to perform in real-time makes it suitable to recognize people in CCTV footage in malls, banks, ATMs, etc. Our system, due to its fast performance the system can be used in schools and colleges for attendance as well as in banks and other high-security zones to grant access to only authorized ones.