We currently use the CSL 1.0 (citation style language) specification for the citation styles we use in Mendeley Desktop. The format specifications and some instructions on how to modify, or create your own citation styles can be found here.
A repository of currently available styles can also be found here.
The citation style files used by Mendeley Desktop are located in the citationStyles-1.0 sub-directory of the folder where Mendeley Desktop is installed. Adding new formatted .csl citation style files, or modifying existing ones will make them available in Mendeley Desktop.
As of Mendeley 1.7 and later, you can also create your own citation style based on an existing one, more information can be found here: http://support.mendeley.com/customer/portal/articles/895771-getting-started-with-the-visual-csl-editor