You're just complaining because you're guilty: A DEF CON Guide to Adversarial Testing of Software Used In the Criminal Justice System

The talk discusses the need for transparency, third-party review, adversarial testing, and true accountability in the criminal justice system's use of software. It highlights the challenges in achieving oversight and the importance of algorithmic accountability and transparency.

• Software is increasingly used to make important decisions about people's lives in various areas, including the criminal justice system.
• The criminal justice system uses proprietary software, and the trade secrets of software vendors are often deemed more important than the rights of the accused to understand and challenge decisions made by these complex systems.
• The lack of transparency and accountability in the criminal justice system's use of software can lead to serious problems, such as sending innocent people to prison or accusing them of crimes they did not commit.
• Third-party review, adversarial testing, and true accountability are essential to ensure the accuracy and fairness of software-based decisions in the criminal justice system.
• The Association for Computing Machinery's Tech policy groups have come up with a set of principles for algorithmic accountability and transparency that could be a starting point for building awareness, access, and redress, accountability, explanation, data provenance, validation, and testing.
• The software justice community could make a big difference in improving the criminal justice system's use of software by providing evidence to improve systems for all stakeholders.

The talk provides real examples of problems that have arisen due to the lack of transparency and accountability in the criminal justice system's use of software, such as the Loomis vs. Wisconsin case, where a defendant was sent to prison based on proprietary software's decision that they were likely to commit another crime, but they could not question how the software made that decision.

Software is increasingly used to make huge decisions about people's lives and often these decisions are made with little transparency or accountability to individuals. If there is any place where transparency, third-party review, adversarial testing and true accountability is essential, it is the criminal justice system. Nevertheless, proprietary software is used throughout the system, and the trade secrets of software vendors are regularly deemed more important than the rights of the accused to understand and challenge decisions made by these complex systems. In this talk, we will lay out the map of software in this space from DNA testing to facial recognition to estimating the likelihood that someone will commit a future crime. We will detail the substantial hurdles that prevent oversight and stunning examples of real problems found when hard won third-party review is finally achieved. Finally, we will outline what you as a concerned citizen/hacker can do. Nathan Adams will demo his findings from reviewing NYC's FST source code, which was finally made public by a federal judge after years of the city's lab fighting disclosure or even review. Jerome Greco will provide his insight into the wider world of software used in the criminal justice system—from technology that law enforcement admits to using but expects the public to trust without question to technology that law enforcement denies when the evidence says otherwise. Jeanna Matthews will talk about the wider space of algorithmic accountability and transparency and why even open source software is not enough.