Written in 2002, "Lessons Learned in Implementing and Deploying Crypto Software" ( https://www.cs.auckland.ac.nz/~pgut001/pubs/usenix02.pdf ) combines real-world experience implementing and supporting a popular cryptography library with broader observations on programming and security which have aged fairly well:
"In the last five years or so the basic tools for strong encryption have become fairly widespread, gradually displacing the snake oil products that they had shared the environment with until then. As a result, it’s now fairly easy to obtain software that contains well-established, strong algorithms such as triple DES and RSA instead of pseudo one time-pads. Unfortunately, this hasn’t solved the snake oil problem, but has merely relocated it elsewhere.
"The determined programmer can produce snake oil using any crypto tools."
There is a "CliffsNotes" overview at Adrian Colyer's blog: https://blog.acolyer.org/2015/09/17/lessons-learned-in-implementing-and-deploying-crypto-software/
We will discuss the paper informally over lunch. Come hungry for sandwiches and geek talk. I will also try to recruit you to submit talk proposals for CodeMash.