Some of the components that are shipped with Mac OS X are open sourced this is the case for the Security framework , CMS (Cryptographic Message Syntax) is the way to go to sign & cypher mail messages in Mac OS X but until Leopard, all those API are kept private, and no public version was made. Fortunately, you can still access them if you decide to build the Security framework by yourself. The best way to do it right now is to use darwinbuild and follow exactyy what is said in the ReadMe file, compiling the security framework should be rather straight forward after that. Apple also ships a binary called ’security’ (The real project name is ’SecurityTool’ and is also open sourced] that enable you to access CMS and sign/cypher a message.
The integretaion of a modified version of the ’SecurityTool’ in XMail to allow the encryption & signature of mails should not really be a real problem, and XMail might well integrate this before the 4.0 release.