Security and cryptography are notoriously hard, and I am not claiming to be an expert on these. But on several projects I have used my software engineering experience, a basic understanding of the appropriate maths, and the ability to engage others and work towards a consensus, to adapt security-related software for particular uses.
The most complete case involved extending OpenSSH to work with X509 certificates and Spyrus hardware - keys (and certificates) are stored on a USB dongle and the user does not need to enter a password.
OpenSSH includes support for OpenSSL engines (I can’t find a good link to describe these; they are dynamic modules with a standard API that can be used to delegate cryptographic operations, like signing, to a hardware device). I extended an incomplete Spyrus engine to inter-operate with OpenSSH, both alone and using the X509 patch.
Modified this patch to co-exist with the X509 work;
Extended both to use DSA keys (note that this does not require detailed crypto knowledge since all operations are abstracted to the level of OpenSSL key methods);
Wrote a minimal adapter to provide a PKCS11 interface to the Spyrus hardware library (technically, this was particularly interesting: I used (the excellent) pyparser to autogenerate C stubs from the header files provided by RSA; each stub printed its name and returned an error; running sshd against the compiled stubs allowed me to quickly “fill in” the minimal implementation required).
I combined multiple components into a single, functional, solution. This required merging conflicting patches, extending functionality at the level of OpenSSL key methods, and adapting libraries to standard interfaces.