| Andrew Cooke | Contents | Latest | RSS | Twitter | Previous | Next


Welcome to my blog, which was once a mailing list of the same name and is still generated by mail. Please reply via the "comment" links.

Always interested in offers/projects/new ideas. Eclectic experience in fields like: numerical computing; Python web; Java enterprise; functional languages; GPGPU; SQL databases; etc. Based in Santiago, Chile; telecommute worldwide. CV; email.

Personal Projects

Lepl parser for Python.

Colorless Green.

Photography around Santiago.

SVG experiment.

Professional Portfolio

Calibration of seismometers.

Data access via web services.

Cache rewrite.

Extending OpenSSH.

C-ORM: docs, API.

Last 100 entries

Re: Quick message - This link is broken; Adding Reverb To The Echo Chamber; Sox Audio Tools; Would This Have Been OK?; Honesty only important economically before institutions develop; Stegangraphy via PS4; OpenCL Mess; More Book Recommendations; Good Explanation of Difference Between Majority + Minority; Musical Chairs - Who's The Privileged White Guy; I can see straight men watching this conversation and laffing; When it's Actually a Source of Indignation and Disgust; Meta Thread Defending POC Causes POC To Close Account; Indigenous People Of Chile; Curry Recipe; Interesting Link On Marginality; A Nuclear Launch Ordered, 1962; More Book Recs (Better Person); It's Nuanced, And I Tried, So Back Off; Marx; The Negative Of Positive; Jenny Holzer Rocks; Huge Article on Cultural Evolution and More; "Ignoring language theory"; Negative Finger Counting; Week 12; Communication Via Telecomm Bids; Finding Suspects Via Relatives' DNA From Non-Crime Databases; Statistics and Information Theory; Ice OK in USA; On The Other Hand; (Current Understanding Of) Chilean Taxes / Contributions; M John Harrison; Playing Games on a Cloud GPU; China Gamifies Real Life; Can't Help Thinking It's Thoughtcrime; Mefi Quotes; Spray Painting Bike Frame; Weeks 10 + 11; Change: No Longer Possible To Merge Metadata; Books on Old Age; Health Tree Maps; MRA - Men's Rights Activists; Writing Good C++14; Risk Assessment - Fukushima; The Future of Advertising and Surveillance; Travelling With Betaferon; I think I know what I dislike so much about Metafilter; Weeks 8 + 9; More; Pastamore - Bad Italian in Vitacura; History Books; Iraq + The (UK) Governing Elite; Answering Some Hard Questions; Pinochet: The Dictator's Shadow; An Outsider's Guide To Julia Packages; Nobody gives a shit; Lepton Decay Irregularity; An Easier Way; Julia's BinDeps (aka How To Install Cairo); Good Example Of Good Police Work (And Anonymity Being Hard); Best Santiago Burgers; Also; Michael Emmerich (Vibrator Translator) Interview (Japanese Books); Clarice Lispector (Brazillian Writer); Books On Evolution; Looks like Ara (Modular Phone) is dead; Index - Translations From Chile; More Emotion in Chilean Wines; Week 7; Aeon Magazine (Science-ish); QM, Deutsch, Constructor Theory; Interesting Talk Transcripts; Interesting Suggestion Of Election Fraud; "Hard" Books; Articles or Papers on depolarizing the US; Textbook for "QM as complex probabilities"; SFO Get Libor Trader (14 years); Why Are There Still So Many Jobs?; Navier Stokes Incomplete; More on Benford; FBI Claimed Vandalism; Architectural Tessellation; Also: Go, Blake's 7; Delusions of Gender (book); Crypto AG DID work with NSA / GCHQ; UNUMS (Universal Number Format); MOOCs (Massive Open Online Courses); Interesting Looking Game; Euler's Theorem for Polynomials; Weeks 3-6; Reddit Comment; Differential Cryptanalysis For Dummies; Japanese Graphic Design; Books To Be Re-Read; And Today I Learned Bugs Need Clear Examples; Factoring a 67 bit prime in your head; Islamic Geometric Art; Useful Julia Backtraces from Tasks; Nothing, however, is lost with less discomfort than that which, when lost, cannot be missed; Article on Didion

© 2006-2015 Andrew Cooke (site) / post authors (content).

(Not So) Random Walks on Graphs

From: andrew cooke <andrew@...>

Date: Tue, 27 Oct 2009 20:14:42 -0300

So, UKYFD - http://uykfd.googlecode.com/ - is working, sort of.  The
trouble is, that it seems to get stuck in a rather popular group of
artists.  It's quite possible this is a bug in the implementation, and
I should really add better logging and some kind of graphical
visualisation to understand better what's going on.

But it's cheaper to think.

And I think that what might be happening is that my rather jumbled and
ad-hoc exploration algorithm tends to guide the walk into this
cluster, then keep it there.  I don't have any proof, but this seems
quite a reasonable if that cluster has many highly-weighted links.

At this point I should record what the current algorithm is:

1 - Lastfm artist tags are used to construct "weights" between related
artists.  To guarantee a completely connected graph a very low weight
link is added between all artists (so the graph is not just connected,
but complete)

2 - From the weights, a table of "distances" for the complete graph is
calculated (these are just weights again, but take into account
"friend of a friend".

The above two steps are done once, the following steps are done for
each track in a playlist, starting with a track selected at random:

3 - Tracks for the K closest artists to the artist of the current
track are selected and given a score of 1.

4 - Tracks for artists directly connected (as 1 above, ignoring the
low weight link) have their score multiplied by 10.

5 - Tracks scores are modified according to distance from the previous
artist.  The idea here is that by penalising or rewarding distance we
can control the form of the walk (2nd order Markov process)

6 - Tracks that are from an artist played in the last L tracks have
scores lowered by a factor of 10.

7 Tracks that have been played in the last M tracks have scores
lowered by a factor of 10.

8 A track is chose at random, with likelihood proportional to the score.

And looking at that, I think it's clear that possibilities for the
kind of clustering described above exist.  So what would a better
process look like?

Steps 6-8 seem fine to me - they just push us towards a self-avoiding
walk, which is not a problem.  Any bias seems to come from the earlier
steps.  Here's a possible alternative:

A - Before generating a playlist (ie for each playlist) a new graph is
constructed over the artists in which all have the same number of
neighbours (some fairly small number like 5 or 10).  For some isolated
artists this will mean choosing at random from equi-distant "very
distant" artists (via the low weight tag added to keep the graph
connected).  This random choice should be different for each playlist
(hence the need to reconstruct this graph each time).

B - An artist is chosen, using a non-parametric (ie link count, not
weight) version of the second order Markov scoring.  Perhaps at first
this is simply random.  Perhaps avoidance of previously played artists
comes here.

C - Once the artist is chose, a track is chosen, avoiding repetition
if possible.

By keeping making all nodes the same order, and removing weights, I
hope the above addresses any clustering issues.  I don't think this is
certain though - it seems to me that there could still be
topological(?)/clustering effects.


Comment on this post