Andrew Cooke | Contents | Latest | RSS | Previous | Next

C[omp]ute

Welcome to my blog, which was once a mailing list of the same name and is still generated by mail. Please reply via the "comment" links.

Always interested in offers/projects/new ideas. Eclectic experience in fields like: numerical computing; Python web; Java enterprise; functional languages; GPGPU; SQL databases; etc. Based in Santiago, Chile; telecommute worldwide. CV; email.

Personal Projects

Choochoo Training Diary

Last 100 entries

[Programming] React Leaflet; AliExpress Independent Sellers; Applebaum - Twilight of Democracy; [Politics] Back + US Elections; [Programming,Exercise] Simple Timer Script; [News] 2019: The year revolt went global; [Politics] The world's most-surveilled cities; [Bike] Hope Freehub; [Restaurant] Mama Chau's (Chinese, Providencia); [Politics] Brexit Podcast; [Diary] Pneumonia; [Politics] Britain's Reichstag Fire moment; install cairo; [Programming] GCC Sanitizer Flags; [GPU, Programming] Per-Thread Program Counters; My Bike Accident - Looking Back One Year; [Python] Geographic heights are incredibly easy!; [Cooking] Cookie Recipe; Efficient, Simple, Directed Maximisation of Noisy Function; And for argparse; Bash Completion in Python; [Computing] Configuring Github Jekyll Locally; [Maths, Link] The Napkin Project; You can Masquerade in Firewalld; [Bike] Servicing Budget (Spring) Forks; [Crypto] CIA Internet Comms Failure; [Python] Cute Rate Limiting API; [Causality] Judea Pearl Lecture; [Security, Computing] Chinese Hardware Hack Of Supermicro Boards; SQLAlchemy Joined Table Inheritance and Delete Cascade; [Translation] The Club; [Computing] Super Potato Bruh; [Computing] Extending Jupyter; Further HRM Details; [Computing, Bike] Activities in ch2; [Books, Link] Modern Japanese Lit; What ended up there; [Link, Book] Logic Book; Update - Garmin Express / Connect; Garmin Forerunner 35 v 230; [Link, Politics, Internet] Government Trolls; [Link, Politics] Why identity politics benefits the right more than the left; SSH Forwarding; A Specification For Repeating Events; A Fight for the Soul of Science; [Science, Book, Link] Lost In Math; OpenSuse Leap 15 Network Fixes; Update; [Book] Galileo's Middle Finger; [Bike] Chinese Carbon Rims; [Bike] Servicing Shimano XT Front Hub HB-M8010; [Bike] Aliexpress Cycling Tops; [Computing] Change to ssh handling of multiple identities?; [Bike] Endura Hummvee Lite II; [Computing] Marble Based Logic; [Link, Politics] Sanity Check For Nuclear Launch; [Link, Science] Entropy and Life; [Link, Bike] Cheap Cycling Jerseys; [Link, Music] Music To Steal 2017; [Link, Future] Simulated Brain Drives Robot; [Link, Computing] Learned Index Structures; Solo Air Equalization; Update: Higher Pressures; Psychology; [Bike] Exercise And Fuel; Continental Race King 2.2; Removing Lowers; Mnesiacs; [Maths, Link] Dividing By Zero; [Book, Review] Ray Monk - Ludwig Wittgenstein: The Duty Of Genius; [Link, Bike, Computing] Evolving Lacing Patterns; [Jam] Strawberry and Orange Jam; [Chile, Privacy] Biometric Check During Mail Delivery; [Link, Chile, Spanish] Article on the Chilean Drought; [Bike] Extended Gear Ratios, Shimano XT M8000 (24/36 Chainring); [Link, Politics, USA] The Future Of American Democracy; Mass Hysteria; [Review, Books, Links] Kazuo Ishiguro - Never Let Me Go; [Link, Books] David Mitchell's Favourite Japanese Fiction; [Link, Bike] Rear Suspension Geometry; [Link, Cycling, Art] Strava Artwork; [Link, Computing] Useful gcc flags; [Link] Voynich Manuscript Decoded; [Bike] Notes on Servicing Suspension Forks; [Links, Computing] Snap, Flatpack, Appimage; [Link, Computing] Oracle is leaving Java (to die); [Link, Politics] Cubans + Ultrasonics; [Book, Link] Laurent Binet; VirtualBox; [Book, Link] No One's Ways; [Link] The Biggest Problem For Cyclists Is Bad Driving; [Computing] Doxygen, Sphinx, Breathe; [Admin] Brokw Recent Permalinks; [Bike, Chile] Buying Bearings in Santiago; [Computing, Opensuse] Upgrading to 42.3; [Link, Physics] First Support for a Physics Theory of Life; [Link, Bike] Peruvian Frame Maker; [Link] Awesome Game Theory Tit-For-Tat Thing; [Food, Review] La Fabbrica - Good Italian Food In Santiago; [Link, Programming] MySQL UTF8 Broken; [Link, Books] Latin American Authors

© 2006-2017 Andrew Cooke (site) / post authors (content).

(Not So) Random Walks on Graphs

From: andrew cooke <andrew@...>

Date: Tue, 27 Oct 2009 20:14:42 -0300

So, UKYFD - http://uykfd.googlecode.com/ - is working, sort of.  The
trouble is, that it seems to get stuck in a rather popular group of
artists.  It's quite possible this is a bug in the implementation, and
I should really add better logging and some kind of graphical
visualisation to understand better what's going on.

But it's cheaper to think.

And I think that what might be happening is that my rather jumbled and
ad-hoc exploration algorithm tends to guide the walk into this
cluster, then keep it there.  I don't have any proof, but this seems
quite a reasonable if that cluster has many highly-weighted links.

At this point I should record what the current algorithm is:

1 - Lastfm artist tags are used to construct "weights" between related
artists.  To guarantee a completely connected graph a very low weight
link is added between all artists (so the graph is not just connected,
but complete)

2 - From the weights, a table of "distances" for the complete graph is
calculated (these are just weights again, but take into account
"friend of a friend".

The above two steps are done once, the following steps are done for
each track in a playlist, starting with a track selected at random:

3 - Tracks for the K closest artists to the artist of the current
track are selected and given a score of 1.

4 - Tracks for artists directly connected (as 1 above, ignoring the
low weight link) have their score multiplied by 10.

5 - Tracks scores are modified according to distance from the previous
artist.  The idea here is that by penalising or rewarding distance we
can control the form of the walk (2nd order Markov process)

6 - Tracks that are from an artist played in the last L tracks have
scores lowered by a factor of 10.

7 Tracks that have been played in the last M tracks have scores
lowered by a factor of 10.

8 A track is chose at random, with likelihood proportional to the score.

And looking at that, I think it's clear that possibilities for the
kind of clustering described above exist.  So what would a better
process look like?

Steps 6-8 seem fine to me - they just push us towards a self-avoiding
walk, which is not a problem.  Any bias seems to come from the earlier
steps.  Here's a possible alternative:

A - Before generating a playlist (ie for each playlist) a new graph is
constructed over the artists in which all have the same number of
neighbours (some fairly small number like 5 or 10).  For some isolated
artists this will mean choosing at random from equi-distant "very
distant" artists (via the low weight tag added to keep the graph
connected).  This random choice should be different for each playlist
(hence the need to reconstruct this graph each time).

B - An artist is chosen, using a non-parametric (ie link count, not
weight) version of the second order Markov scoring.  Perhaps at first
this is simply random.  Perhaps avoidance of previously played artists
comes here.

C - Once the artist is chose, a track is chosen, avoiding repetition
if possible.

By keeping making all nodes the same order, and removing weights, I
hope the above addresses any clustering issues.  I don't think this is
certain though - it seems to me that there could still be
topological(?)/clustering effects.

Andrew

Comment on this post