Siamese Neural Networks for Detecting Brand Impersonation

The presentation discusses a project on bread identification at different junction points in a food production company. The speaker shares insights on how to approach a business problem, choose the right architecture, and automate data generation and annotation.

• Understanding the problem statement and the data generation process is crucial
• Automating boring tasks like data annotation can save time and effort
• Choosing the right augmentation techniques is important for business use cases
• Using pre-trained models can be advantageous
• The speaker shares a story on how they automated the data annotation process for bread identification

The speaker shares how they used awk transform and control logic to split images into sub-images and avoid false positives while automating the data annotation process. They also used a pre-trained model, Mask R-CNN, to segment the bread images and extract data. This reduced manual annotation by 60-70% and made the process more efficient.

Brand impersonation is a key attack strategy in which a malicious user crafts content to look like a known brand to deceive a user into entering sensitive information, such as account passwords or credit card details. To address this issue, we developed and trained a Siamese Neural Network on labeled images to detect brand impersonation. Specifically, our dataset consists of over 50,000 screenshots of known malicious log-in pages encompassing over 1000 brand impersonations. Our Siamese network learns to embed images of the same brand relatively close together in a low dimensional space while images of different brands are embedded further apart. We then perform a nearest neighbor classification in the embedded space. To present the results and fully characterize the performance of our Siamese network, we developed metrics that capture how well the Siamese network performs on known as well as previously unseen brands and show how the network outperforms a baseline image hashing algorithm on a held-out training set. We will then discuss further applications and planned enhancements to the baseline machine learning model.