| Andrew Cooke | Contents | Latest | RSS | Twitter | Previous | Next

C[omp]ute

Welcome to my blog, which was once a mailing list of the same name and is still generated by mail. Please reply via the "comment" links.

Always interested in offers/projects/new ideas. Eclectic experience in fields like: numerical computing; Python web; Java enterprise; functional languages; GPGPU; SQL databases; etc. Based in Santiago, Chile; telecommute worldwide. CV; email.

Personal Projects

Lepl parser for Python.

Colorless Green.

Photography around Santiago.

SVG experiment.

Professional Portfolio

Calibration of seismometers.

Data access via web services.

Cache rewrite.

Extending OpenSSH.

Last 100 entries

Chinese Govt Intercepts External Web To DDOS github; Numbering Permutations; Teenage Engineering - Low Price Synths; GCHQ Can Do Whatever It Wants; Dublinesque; A Cryptographic SAT Solver; Security Challenges; Word Lists for Crosswords; 3D Printing and Speaker Design; Searchable Snowden Archive; XCode Backdoored; Derived Apps Have Malware (CIA); Rowhammer - Hacking Software Via Hardware (DRAM) Bugs; Immutable SQL Database (Kinda); Tor GPS Tracker; That PyCon Dongle Mess...; ASCII Fluid Dynamics; Brandalism; Table of Shifter, Cassette and Derailleur Compatability; Lenovo Demonstrates How Bad HTTPS Is; Telegraph Owned by HSBC; Smaptop - Sunrise (Music); Equation Group (NSA); UK Torture in NI; And - A Natural Extension To Regexps; This Is The Future Of Religion; The Shazam (Music Matching) Algorithm; Tributes To Lesbian Community From AIDS Survivors; Nice Rust Summary; List of Good Fiction Books; Constructing JSON From Postgres (Part 2); Constructing JSON From Postgres (Part 1); Postgres in Docker; Why Poor Places Are More Diverse; Smart Writing on Graceland; Satire in France; Free Speech in France; MTB Cornering - Where Should We Point Our Thrusters?; Secure Secure Shell; Java Generics over Primitives; 2014 (Charlie Brooker); How I am 7; Neural Nets Applied to Go; Programming, Business, Social Contracts; Distributed Systems for Fun and Profit; XML and Scheme; Internet Radio Stations (Curated List); Solid Data About Placebos; Half of Americans Think Climate Change Is a Sign of the Apocalypse; Saturday Surf Sessions With Juvenile Delinquents; Ssh, tty, stdout and stderr; Feathers falling in a vacuum; Santiago 30m Bike Route; Mapa de Ciclovias en Santiago; How Unreliable is UDP?; SE Santiago 20m Bike Route; Cameron's Rap; Configuring libxml with Eclipse; Reducing Combinatorial Complexity With Occam - AI; Sentidos Comunes (Chilean Online Magazine); Hilary Mantel: The Assassination of Margaret Thatcher - August 6th 1983; NSA Interceptng Gmail During Delivery; General IIR Filters; What's happening with Scala?; Interesting (But Largely Illegible) Typeface; Retiring Essentialism; Poorest in UK, Poorest in N Europe; I Want To Be A Redneck!; Reverse Racism; The Lost Art Of Nomography; IBM Data Center (Photo); Interesting Account Of Gamma Hack; The Most Interesting Audiophile In The World; How did the first world war actually end?; Ky - Restaurant Santiago; The Black Dork Lives!; The UN Requires Unaninmous Decisions; LPIR - Steganography in Practice; How I Am 6; Clear Explanation of Verizon / Level 3 / Netflix; Teenage Girls; Formalising NSA Attacks; Switching Brakes (Tektro Hydraulic); Naim NAP 100 (Power Amp); AKG 550 First Impressions; Facebook manipulates emotions (no really); Map Reduce "No Longer Used" At Google; Removing RAID metadata; New Bike (Good Bike Shop, Santiago Chile); Removing APE Tags in Linux; Compiling Python 3.0 With GCC 4.8; Maven is Amazing; Generating Docs from a GitHub Wiki; Modular Shelves; Bash Best Practices; Good Emergency Gasfiter (Santiago, Chile); Readings in Recent Architecture; Roger Casement; Integrated Information Theory (Or Not); Possibly undefined macro AC_ENABLE_SHARED; Update on Charges; Sunburst Visualisation

© 2006-2013 Andrew Cooke (site) / post authors (content).

Clojure macro example - dopar

From: andrew cooke <andrew@...>

Date: Sun, 10 Jun 2012 14:28:20 -0400

I just wrote my first "worthwhile" macro in Clojure (by which I mean, does
something non-trivial I expected the language to do for me).

I have some code I want to run multiple times.  The only things that changes
each time is an integer, but each run takes 10 hours.  So I had a loop that
took 10 hours per iteration.  If you know Clojure then you'll know that my
code looked like this:

  (doseq [i (range 4)]
    (do-stuff-with i))

which is basically a "for loop" that changes "i".

To save time I wanted to run those things in parallel.  So I wanted something
like Python's multiprocessing - a simple way to make "do-stuff-with i" run on
multiple cores.

Solving this went through three phases:

 - Looking for an obvious existing solution.  Couldn't find one.

 - Trying to write an implementation using basic concepts from Java like
   semaphores or ThreadPoolExecutor.  This was complicated and "felt wrong".

 - Working out how to do it the "Clojure way".  Understanding that I needed
   agents, and then writing a macro to manage the agents.

At first I was confused by agents - I thought they would be something like
processes.  But really they are just values that a process can access.  So in
my code above.  So my original idea to put "do-stuff-with" in an agent was
wrong - it is the "i" that "goes inside" the agent.

So the basic way to solve this problem in Clojure is to:

 - Create an agent for each "i"

 - Use "send" to send the function "do-stuff-with" to each agent.

 - Wait for all the agents to finish.

Behind the scenes, Clojure is careful to only let a few agents run at a time
(this is what "send" takes care of).

So a solution would look something like:

  (apply await
    (for [i (range 4)]
      (let [a (agent i)]
        (send a do-stuff-with)
        a)))

which:

 - Generates an agent for each i

 - Sends the work to the agent (which is queued to run in a thread from a 
   thread pool)

 - Creates a list of agents (the result of the "for")

 - Waits for all the agents to finish.

OK, so where does the macro come into this?  Well, this code:

  (defmacro dopar [seq-expr & body]
    (assert (= 2 (count seq-expr)) "single pair of forms in sequence
    expression")
    (let [[k v] seq-expr]
      `(apply await
	 (for [k# ~v]
	   (let [a# (agent k#)]
	     (send a# (fn [~k] ~@body))
	     a#)))))

lets me write the (apply....) above as:

  (dopar [i (range 4)]
    (do-stuff-with i))

which is, I think, pretty awesome.

Andrew

Comment on this post