| Andrew Cooke | Contents | Latest | RSS | Twitter | Previous | Next

C[omp]ute

Welcome to my blog, which was once a mailing list of the same name and is still generated by mail. Please reply via the "comment" links.

Always interested in offers/projects/new ideas. Eclectic experience in fields like: numerical computing; Python web; Java enterprise; functional languages; GPGPU; SQL databases; etc. Based in Santiago, Chile; telecommute worldwide. CV; email.

Personal Projects

Lepl parser for Python.

Colorless Green.

Photography around Santiago.

SVG experiment.

Professional Portfolio

Calibration of seismometers.

Data access via web services.

Cache rewrite.

Extending OpenSSH.

Last 100 entries

British Words; Chinese Govt Intercepts External Web To DDOS github; Numbering Permutations; Teenage Engineering - Low Price Synths; GCHQ Can Do Whatever It Wants; Dublinesque; A Cryptographic SAT Solver; Security Challenges; Word Lists for Crosswords; 3D Printing and Speaker Design; Searchable Snowden Archive; XCode Backdoored; Derived Apps Have Malware (CIA); Rowhammer - Hacking Software Via Hardware (DRAM) Bugs; Immutable SQL Database (Kinda); Tor GPS Tracker; That PyCon Dongle Mess...; ASCII Fluid Dynamics; Brandalism; Table of Shifter, Cassette and Derailleur Compatability; Lenovo Demonstrates How Bad HTTPS Is; Telegraph Owned by HSBC; Smaptop - Sunrise (Music); Equation Group (NSA); UK Torture in NI; And - A Natural Extension To Regexps; This Is The Future Of Religion; The Shazam (Music Matching) Algorithm; Tributes To Lesbian Community From AIDS Survivors; Nice Rust Summary; List of Good Fiction Books; Constructing JSON From Postgres (Part 2); Constructing JSON From Postgres (Part 1); Postgres in Docker; Why Poor Places Are More Diverse; Smart Writing on Graceland; Satire in France; Free Speech in France; MTB Cornering - Where Should We Point Our Thrusters?; Secure Secure Shell; Java Generics over Primitives; 2014 (Charlie Brooker); How I am 7; Neural Nets Applied to Go; Programming, Business, Social Contracts; Distributed Systems for Fun and Profit; XML and Scheme; Internet Radio Stations (Curated List); Solid Data About Placebos; Half of Americans Think Climate Change Is a Sign of the Apocalypse; Saturday Surf Sessions With Juvenile Delinquents; Ssh, tty, stdout and stderr; Feathers falling in a vacuum; Santiago 30m Bike Route; Mapa de Ciclovias en Santiago; How Unreliable is UDP?; SE Santiago 20m Bike Route; Cameron's Rap; Configuring libxml with Eclipse; Reducing Combinatorial Complexity With Occam - AI; Sentidos Comunes (Chilean Online Magazine); Hilary Mantel: The Assassination of Margaret Thatcher - August 6th 1983; NSA Interceptng Gmail During Delivery; General IIR Filters; What's happening with Scala?; Interesting (But Largely Illegible) Typeface; Retiring Essentialism; Poorest in UK, Poorest in N Europe; I Want To Be A Redneck!; Reverse Racism; The Lost Art Of Nomography; IBM Data Center (Photo); Interesting Account Of Gamma Hack; The Most Interesting Audiophile In The World; How did the first world war actually end?; Ky - Restaurant Santiago; The Black Dork Lives!; The UN Requires Unaninmous Decisions; LPIR - Steganography in Practice; How I Am 6; Clear Explanation of Verizon / Level 3 / Netflix; Teenage Girls; Formalising NSA Attacks; Switching Brakes (Tektro Hydraulic); Naim NAP 100 (Power Amp); AKG 550 First Impressions; Facebook manipulates emotions (no really); Map Reduce "No Longer Used" At Google; Removing RAID metadata; New Bike (Good Bike Shop, Santiago Chile); Removing APE Tags in Linux; Compiling Python 3.0 With GCC 4.8; Maven is Amazing; Generating Docs from a GitHub Wiki; Modular Shelves; Bash Best Practices; Good Emergency Gasfiter (Santiago, Chile); Readings in Recent Architecture; Roger Casement; Integrated Information Theory (Or Not); Possibly undefined macro AC_ENABLE_SHARED; Update on Charges

© 2006-2013 Andrew Cooke (site) / post authors (content).

(Not So) Random Walks on Graphs

From: andrew cooke <andrew@...>

Date: Tue, 27 Oct 2009 20:14:42 -0300

So, UKYFD - http://uykfd.googlecode.com/ - is working, sort of.  The
trouble is, that it seems to get stuck in a rather popular group of
artists.  It's quite possible this is a bug in the implementation, and
I should really add better logging and some kind of graphical
visualisation to understand better what's going on.

But it's cheaper to think.

And I think that what might be happening is that my rather jumbled and
ad-hoc exploration algorithm tends to guide the walk into this
cluster, then keep it there.  I don't have any proof, but this seems
quite a reasonable if that cluster has many highly-weighted links.

At this point I should record what the current algorithm is:

1 - Lastfm artist tags are used to construct "weights" between related
artists.  To guarantee a completely connected graph a very low weight
link is added between all artists (so the graph is not just connected,
but complete)

2 - From the weights, a table of "distances" for the complete graph is
calculated (these are just weights again, but take into account
"friend of a friend".

The above two steps are done once, the following steps are done for
each track in a playlist, starting with a track selected at random:

3 - Tracks for the K closest artists to the artist of the current
track are selected and given a score of 1.

4 - Tracks for artists directly connected (as 1 above, ignoring the
low weight link) have their score multiplied by 10.

5 - Tracks scores are modified according to distance from the previous
artist.  The idea here is that by penalising or rewarding distance we
can control the form of the walk (2nd order Markov process)

6 - Tracks that are from an artist played in the last L tracks have
scores lowered by a factor of 10.

7 Tracks that have been played in the last M tracks have scores
lowered by a factor of 10.

8 A track is chose at random, with likelihood proportional to the score.

And looking at that, I think it's clear that possibilities for the
kind of clustering described above exist.  So what would a better
process look like?

Steps 6-8 seem fine to me - they just push us towards a self-avoiding
walk, which is not a problem.  Any bias seems to come from the earlier
steps.  Here's a possible alternative:

A - Before generating a playlist (ie for each playlist) a new graph is
constructed over the artists in which all have the same number of
neighbours (some fairly small number like 5 or 10).  For some isolated
artists this will mean choosing at random from equi-distant "very
distant" artists (via the low weight tag added to keep the graph
connected).  This random choice should be different for each playlist
(hence the need to reconstruct this graph each time).

B - An artist is chosen, using a non-parametric (ie link count, not
weight) version of the second order Markov scoring.  Perhaps at first
this is simply random.  Perhaps avoidance of previously played artists
comes here.

C - Once the artist is chose, a track is chosen, avoiding repetition
if possible.

By keeping making all nodes the same order, and removing weights, I
hope the above addresses any clustering issues.  I don't think this is
certain though - it seems to me that there could still be
topological(?)/clustering effects.

Andrew

Comment on this post