Turing in a Box: Applying Artificial Intelligence as a Service to Targeted Phishing and Defending Against AI Generated Attacks

The presentation discusses the use of AI in red teaming operations and the challenges of detecting synthetic text. It also explores potential solutions for defense against AI phishing pipelines.

• The collaboration of MIT IBM Watson AI Lab and Harvard NLP resulted in the development of GLTR, a project that uses a matrix and GPT to detect synthetic text.
• The team faced challenges in detecting synthetic text due to limited access to GPT language models.
• AI phishing pipelines can significantly outperform human workflows in mass phishing stages.
• Integrating AI into red teaming operations can streamline and standardize operations.
• Potential solutions for defense against AI phishing pipelines include using zero-shot detection and incorporating values, risks, and responsibilities relating to algorithmic decision making.

The team found that integrating AI into their red teaming operations helped to streamline and standardize their operations. They were able to tweak various quantitative factors such as the temperature or randomness of their GPT3 API, effectively encoding their workflow and allowing them to iterate in a methodological and testable manner. Additionally, the portability of the API meant that they could easily integrate it into existing tools such as the Goldfish open-source phishing framework.

With recent advances in next-generation language models such as OpenAI's GPT-3, AI generated text has reached a level of sophistication that matches or even exceeds human generated output. The proliferation of Artificial Intelligence as a Service (AIaaS) products places these capabilities in the hands of a global market, bypassing the need to independently train models or rely on open-source pre-trained models. By greatly reducing the barriers to entry, AIaaS gives consumers access to state-of-the-art AI capabilities at a fraction of the cost through user-friendly APIs.In our research, we present a novel approach that uses AIaaS to improve the delivery of Red Team operations - in particular, the conduct of phishing campaigns. We developed a targeted phishing pipeline that uses OpenAI and Personality Analysis AIaaS products to generate persuasive phishing emails. Our pipeline automatically personalizes the content based on the target's background and personality. We observed that AI generated phishing content outperformed those that were manually created by Red Team operators. Furthermore, the pipeline freed up Red Team resources to focus on higher-value work such as context building and intelligence gathering.In addition, we present an AIaaS-powered phishing defense framework to detect such attacks. Compared to traditional classification-based email filters, our framework adapts deep learning language models such as OpenAI's GPT-3 to accurately distinguish between AI and human generated text. This allows security teams to mount a credible defense against advanced AI text generators without requiring significant AI expertise or resources.Our research provides actionable takeaways for both red and blue teams to prepare for the current reality of advanced AI proliferation. We discuss the long-term implications of this trend and recommend high-level strategies such as AI governance frameworks to safeguard against the abuse of AIaaS products.