| Andrew Cooke | Contents | Latest | RSS | Twitter | Previous | Next


Welcome to my blog, which was once a mailing list of the same name and is still generated by mail. Please reply via the "comment" links.

Always interested in offers/projects/new ideas. Eclectic experience in fields like: numerical computing; Python web; Java enterprise; functional languages; GPGPU; SQL databases; etc. Based in Santiago, Chile; telecommute worldwide. CV; email.

Personal Projects

Lepl parser for Python.

Colorless Green.

Photography around Santiago.

SVG experiment.

Professional Portfolio

Calibration of seismometers.

Data access via web services.

Cache rewrite.

Extending OpenSSH.

C-ORM: docs, API.

Last 100 entries

Consciousness From Max Entropy; Democrats; Harvard Will Fix Black Poverty; Modelling Bicycle Wheels; Amusing Polling Outlier; If Labour keeps telling working class people...; Populism and Choice; Books on Defeat; Enrique Ferrari - Argentine Author; Transcript of German Scientists on Learning of Hiroshima; Calvert Journal; Telephone System Quotes for Cat Soft LLC; Owen Jones on Twitter; Telephone System Quotes for Cat Soft LLC; Possible Japanese Authors; Complex American Literature; Chutney v5; Weird Componentized Virus; Interesting Argentinian Author - Antonio Di Benedetto; Useful Thread on MetaPhysics; RAND on fighting online anarchy (2001); Now Is Cat Soft LLC's Chance To Save Up To 32% On Mail; NSA Hacked; Call Center Services for Cat Soft LLC; Very Good LRB Article on Brexit; Nussbaum on Anger; Credit Card Processing for Cat Soft LLC; Discover new movies on demand in our online cinema; Tasting; Credit Card Processing for Cat Soft LLC; Apple + Kiwi Jam; Hit Me; Increase Efficiency with GPS Vehicle Tracking for Cat Soft LLC; Sudoku - CSP + Chaos; Recycling Electronics In Santiago; Vector Displays in OpenGL; Call Center Services for Cat Soft LLC; And Anti-Aliased; OpenGL - Render via Intermediate Texture; And Garmin Connect; Using Garmin Forerunner 230 With Linux; Payroll Service Quotes for Cat Soft LLC; (Beating Dead Horse) StackOverflow; Current State of Justice in China; Now Is Cat Soft LLC's Chance To Save Up To 32% On Mail; Axiom of Determinacy; Ewww; Fee Chaos Book; Course on Differential Geometry; Increase Efficiency with GPS Vehicle Tracking for Cat Soft LLC; Okay, but...; Sparse Matrices, Deep Learning; Sounds Bad; Applebaum Rape; Tomato Chutney v4; Have to add...; Culturally Liberal and Nothing More; Weird Finite / Infinite Result; Your diamond is a beaten up mess; Maths Books; Good Bike Route from Providencia / Las Condes to Panul\; Iain Pears (Author of Complex Plots); Plum Jam; Excellent; More Recently; For a moment I forgot StackOverflow sucked; A Few Weeks On...; Chilean Book Recommendations; How To Write Shared Libraries; Jenny Erpenbeck (Author); Dijkstra, Coins, Tables; Python libraries error on OpenSuse; Deserving Trump; And Smugness; McCloskey Economics Trilogy; cmocka - Mocks for C; Concept Creep (Americans); Futhark - OpenCL Language; Moved / Gone; Fan and USB issues; Burgers in Santiago; The Origin of Icosahedral Symmetry in Viruses; autoenum on PyPI; Jars Explains; Tomato Chutney v3; REST; US Elections and Gender: 24 Point Swing; PPPoE on OpenSuse Leap 42.1; SuperMicro X10SDV-TLN4F/F with Opensuse Leap 42.1; Big Data AI Could Be Very Bad Indeed....; Cornering; Postcapitalism (Paul Mason); Black Science Fiction; Git is not a CDN; Mining of Massive Data Sets; Rachel Kaadzi Ghansah; How great republics meet their end; Raspberry, Strawberry and Banana Jam; Interesting Dead Areas of Math; Later Taste; For Sale

© 2006-2015 Andrew Cooke (site) / post authors (content).

Finding Matches in Graphical Hashes

From: andrew cooke <andrew@...>

Date: Sun, 20 May 2012 20:43:21 -0400

I'm currently working on some code that generates graphical representations of
hashes.  The idea is that you might use them to check downloaded files, much
like numerical hashes (in a perfect world you would use both, for details that
I won't go into here).

Now it's all fun + games developing an algorithm, but I am now wondering how
best to quantify the accuracy of the results.  In particular, how unique is
each image?

This is a hard problem - it involves psycho-optics (asusming I've not made
that word up) - but is simplified a little by the approach I have used.

First, the image is "quantised" as a mosaic (it is not a smooth image, but
built from squares of colour).  This removes issues about things like "feature

Second, each mosaic is generated from a base colour and an array of float
values in the range [-1 1].  There is one float per tile in the mosaic, which
represents the "distance" from the base colour (this is translated into a
change in hue and lightness, which are correlated so that the results can be
distinguished even by colourblind users).

So to a first approximation we can ignore a lot of the hard parts and focus on
"how closely" arrays of float values match.  The rest of the email describes
how I will do this.

To find useful matches I will need (I hope!) quite a large data set.  So the
most expensive part of the processing is likely the generation of many hashes
(as you might expect, generating a hash involves quite complex calculations
since it relies on cryptographic primitives).

The first step in my analysis is, therefore, to generate a large set of data.
I will convert each [-1 1] range to a byte, and write the data to a file.
Since the value can be the "line" number, this could be a simple binary file -
that would support fast random access, although I am not sure I need it.

Next, two filters that operate on that data.  One selecting random (but fixed
per run) "pattern" of bytes and another reducing the byte by discarding least
significant bits.

And finally, a program that buckets the filtered data, looking for matches.

The idea is that the selected, reduced data form simple locality-sensitive
hashes, and that the sensitivity of the hashes can be tuned by hand (the
filters and buckets being fairly fast to re-run, and with easy-to-understand

In this way I hope to be able to calculate how frequent collisions are for
different resolutions (bits per float).  Even if the bit resolution at which I
can detect collisions is so low that the "real" images look different I may be
able to extrapolate to higher resolutions.


Re: Finding Matches in Graphical Hashes

From: Michiel Buddingh' <michiel@...>

Date: Mon, 21 May 2012 06:03:36 +0200

From what I've heard, one of the more current metrics to evaluate
image or video compression quality is the Structural Similarity Index

You might also want to incorporate something as described here:
http://stevehanov.ca/blog/index.php?id=62 , mapping the bit values to
a L*u*v colour space rather than to a RGB colour space, presuming you
don't already do this.

Good luck!  I've noticed that ssh-keygen had started outputting teeny
ascii-art visual fingerprints for newly minted keys, but in the
post-teletype age, what you describe makes a lot more sense.


Re: Finding Matches in Graphical Hashes

From: andrew cooke <andrew@...>

Date: Mon, 21 May 2012 20:24:52 -0400

Thanks for the pointers.  SSIM looks interesting - a lot simpler than I
expected.  I am in the middle of generating 10M hashes and will try that on

As for Luv - I am actually using HSL which is a clunky approximation that is
much easier to deal with but not as "physiological".  However, the work
described here is actually on an earlier form - just an array of float values
between -1 and 1 (the HSL is generated from those - basically they are used to
select a hue and lightness).


Comment on this post