| Andrew Cooke | Contents | Latest | RSS | Twitter | Previous | Next

C[omp]ute

Welcome to my blog, which was once a mailing list of the same name and is still generated by mail. Please reply via the "comment" links.

Always interested in offers/projects/new ideas. Eclectic experience in fields like: numerical computing; Python web; Java enterprise; functional languages; GPGPU; SQL databases; etc. Based in Santiago, Chile; telecommute worldwide. CV; email.

Personal Projects

Lepl parser for Python.

Colorless Green.

Photography around Santiago.

SVG experiment.

Professional Portfolio

Calibration of seismometers.

Data access via web services.

Cache rewrite.

Extending OpenSSH.

C-ORM: docs, API.

Last 100 entries

Also: Go, Blake's 7; Delusions of Gender (book); Crypto AG DID work with NSA / GCHQ; UNUMS (Universal Number Format); MOOCs (Massive Open Online Courses); Interesting Looking Game; Euler's Theorem for Polynomials; Weeks 3-6; Reddit Comment; Differential Cryptanalysis For Dummies; Japanese Graphic Design; Books To Be Re-Read; And Today I Learned Bugs Need Clear Examples; Factoring a 67 bit prime in your head; Islamic Geometric Art; Useful Julia Backtraces from Tasks; Nothing, however, is lost with less discomfort than that which, when lost, cannot be missed; Article on Didion; Cost of Living by City; British Slavery; Derrida on Metaphor; African SciFi; Traits in Julia; Alternative Japanese Lit; Pulic Key as Address (Snow); Why Information Grows; The Blindness Of The Chilean Elite; Some Victoriagate Links; This Is Why I Left StackOverflow; New TLS Implementation; Maths for Physicists; How I Am 8; 1000 Word Philosophy; Cyberpunk Reading List; Detailed Discussion of Message Dispatch in ParserCombinator Library for Julia; FizzBuzz in Julia w Dependent Types; kokko - Design Shop in Osaka; Summary of Greece, Currently; LLVM and GPUs; See Also; Schoolgirl Groyps (Maths); Japanese Lit; Another Example - Modular Arithmetic; Music from United; Python 2 and 3 compatible alternative.; Read Agatha Christie for the Plot; A Constructive Look at TempleOS; Music Thread w Many Recommendations; Fixed Version; A Useful Julia Macro To Define Equality And Hash; k3b cdrom access, OpenSuse 13.1; Week 2; From outside, the UK looks less than stellar; Huge Fonts in VirtualBox; Keen - Complex Emergencies; The Fallen of World War II; Some Spanish Fiction; Calling C From Fortran 95; Bjork DJ Set; Z3 Example With Python; Week 1; Useful Guide To Starting With IJulia; UK Election + Media; Review: Reinventing Organizations; Inline Assembly With Julia / LLVM; Against the definition of types; Dumb Crypto Paper; The Search For Quasi-Periodicity...; Is There An Alternative To Processing?; CARDIAC (CARDboard Illustrative Aid to Computation); The Bolivian Case Against Chile At The Hague; Clear, Cogent Economic Arguments For Immigration; A Program To Say If I Am Working; Decent Cards For Ill People; New Photo; Luksic And Barrick Gold; President Bachelet's Speech; Baltimore Primer; libxml2 Parsing Stream; configure.ac Recipe For Library Path; The Davalos Affair For Idiots; Not The Onion: Google Fireside Chat w Kissinger; Bicycle Wheels, Inertia, and Energy; Another Tax Fraud; Google's Borg; A Verion That Redirects To Local HTTP Server; Spanish Accents For Idiots; Aluminium Cans; Advice on Spray Painting; Female View of Online Chat From a Male; UX Reading List; S4 Subgroups - Geometric Interpretation; Fucking Email; The SQM Affair For Idiots; Using Kolmogorov Complexity; Oblique Strategies in bash; Curses Tools; Markov Chain Monte Carlo Without all the Bullshit; Email Para Matias Godoy Mercado; The Penta Affair For Idiots; Example Code To Create numpy Array in C

© 2006-2015 Andrew Cooke (site) / post authors (content).

Finding Matches in Graphical Hashes

From: andrew cooke <andrew@...>

Date: Sun, 20 May 2012 20:43:21 -0400

I'm currently working on some code that generates graphical representations of
hashes.  The idea is that you might use them to check downloaded files, much
like numerical hashes (in a perfect world you would use both, for details that
I won't go into here).

Now it's all fun + games developing an algorithm, but I am now wondering how
best to quantify the accuracy of the results.  In particular, how unique is
each image?

This is a hard problem - it involves psycho-optics (asusming I've not made
that word up) - but is simplified a little by the approach I have used.

First, the image is "quantised" as a mosaic (it is not a smooth image, but
built from squares of colour).  This removes issues about things like "feature
size".

Second, each mosaic is generated from a base colour and an array of float
values in the range [-1 1].  There is one float per tile in the mosaic, which
represents the "distance" from the base colour (this is translated into a
change in hue and lightness, which are correlated so that the results can be
distinguished even by colourblind users).

So to a first approximation we can ignore a lot of the hard parts and focus on
"how closely" arrays of float values match.  The rest of the email describes
how I will do this.

To find useful matches I will need (I hope!) quite a large data set.  So the
most expensive part of the processing is likely the generation of many hashes
(as you might expect, generating a hash involves quite complex calculations
since it relies on cryptographic primitives).

The first step in my analysis is, therefore, to generate a large set of data.
I will convert each [-1 1] range to a byte, and write the data to a file.
Since the value can be the "line" number, this could be a simple binary file -
that would support fast random access, although I am not sure I need it.

Next, two filters that operate on that data.  One selecting random (but fixed
per run) "pattern" of bytes and another reducing the byte by discarding least
significant bits.

And finally, a program that buckets the filtered data, looking for matches.

The idea is that the selected, reduced data form simple locality-sensitive
hashes, and that the sensitivity of the hashes can be tuned by hand (the
filters and buckets being fairly fast to re-run, and with easy-to-understand
parameters.

In this way I hope to be able to calculate how frequent collisions are for
different resolutions (bits per float).  Even if the bit resolution at which I
can detect collisions is so low that the "real" images look different I may be
able to extrapolate to higher resolutions.

Andrew

Re: Finding Matches in Graphical Hashes

From: Michiel Buddingh' <michiel@...>

Date: Mon, 21 May 2012 06:03:36 +0200

From what I've heard, one of the more current metrics to evaluate
image or video compression quality is the Structural Similarity Index
(SSIM).

You might also want to incorporate something as described here:
http://stevehanov.ca/blog/index.php?id=62 , mapping the bit values to
a L*u*v colour space rather than to a RGB colour space, presuming you
don't already do this.

Good luck!  I've noticed that ssh-keygen had started outputting teeny
ascii-art visual fingerprints for newly minted keys, but in the
post-teletype age, what you describe makes a lot more sense.

Michiel

Re: Finding Matches in Graphical Hashes

From: andrew cooke <andrew@...>

Date: Mon, 21 May 2012 20:24:52 -0400

Thanks for the pointers.  SSIM looks interesting - a lot simpler than I
expected.  I am in the middle of generating 10M hashes and will try that on
those.

As for Luv - I am actually using HSL which is a clunky approximation that is
much easier to deal with but not as "physiological".  However, the work
described here is actually on an earlier form - just an array of float values
between -1 and 1 (the HSL is generated from those - basically they are used to
select a hue and lightness).

Cheers,
Andrew

Comment on this post