Four of the most respected algorithms.
Y-SSIM
Structural Similarity algorithm [1] applied to luma channel only.
RGB-SSIM
Average of Structural Similarity algorithm [1] applied to R, G, and B channels.
IW-SSIM
Information Content Weighted Structural Similarity algorithm [2] applied to luma channel only.
PSNR-HVS-M
Peak Signal to Noise Ratio taking into account Contrast Sensitivity Function (CSF) and between-coefficient contrast masking of DCT basis functions [3].
All of these algorithms compare two images and return a number indicating the degree to which the second image is similar to the first. In all cases, no matter what the scale, higher numbers indicate a higher degree of similarity.
It's unclear which algorithm is best in terms of human visual perception.
An implementation of these algorithms can be found on github.
In October of 2013, Mozilla Corporation published a study entitled "Lossy Compressed Image Formats Study" where they tested the aforementioned algorithms.
The SSIM Index for Image Quality Assessment.
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh and Eero P. Simoncelli.
IW-SSIM: Information Content Weighted Structural Similarity Index for Image Quality Assessment.
Zhou Wang and Qiang Li, "Information Content Weighting for Perceptual Image Quality Assessment," IEEE Transactions on Image Processing, vol. 20, no. 5, pp. 1185-1198, May 2011.
Nikolay Ponomarenko homepage - PSNR-HVS-M download page.
Nikolay Ponomarenko, Flavia Silvestri, Karen Egiazarian, Marco Carli, Jaakko Astola, Vladimir Lukin, On between-coefficient contrast masking of DCT basis functions, CD-ROM Proceedings of the Third International Workshop on Video Processing and Quality Metrics for Consumer Electronics VPQM-07, Scottsdale, Arizona, USA, 25-26 January, 2007, 4 p.