C[omp]ute

From: andrew cooke <andrew@...>

Date: Sun, 5 Feb 2012 11:23:10 -0300

These are a bunch of random notes I wrote for work when someone asked what was
"new".  I realise some of this is quite old - that's partly me not knowing
stuff, and partly because I wasn't sure what was required.  Anyway, feedback
is welcomed.

The original was a wiki page; I've had to reformat it here.


Cloud Computing

 Various ideas / acronyms related to "the cloud" are worth understanding:

  IAAS - Infrastructure As A Service - This is where we are heading with our
  VPS.  The idea is that infrastricture is something that you buy as a service
  (obviously). VPS is still tied to the idea of individual machines; IAAS is
  pushing that more towards have a network of machines that grows on demand.
  AWS http://aws.amazon.com/ are dominant, but there are also a bunch of other
  players like RackSpace http://www.rackspace.com/, etc.

  PAAS - Platform As A Service - GAE http://code.google.com/appengine/ (Google
  App Engine) is the prime example.  They have layered software over IAAS to
  make everything simpler to use.  You develop programs according to the
  platform's API and end up with a system that automatically scales with
  demand.

  SAAS - Software As A Service - Google docs https://docs.google.com
  etc. Software that comes from the cloud and stores data there, using only a
  "thin client".

 Writing a web service for GAE in Python is a very productive way to get a
 site running quickly that will handle loads well (you can use much of Django
 - the only new thing to learn is that way that data storage works - see "big
 data" elsewhere).

Big Data

 Related to the above, and driven by companies like Facebook, Twitter and
 Google, are various ideas related to "big data".  The most visible of these
 are a whole slew of "NoSQL" databases.  These are just starting to die off -
 many didn't do much that SQL can't do anyway, like Object and XML databases
 before - but various niches remain:

  Memcaced http://memcached.org/ and Redis http://redis.io/ are proving useful
  for caching data in-memory;

  HBase http://hbase.apache.org/ and Google's Big Table
  http://research.google.com/archive/bigtable.html work on problems so large
  that SQL really does fail (closely related are Hadoop
  http://hadoop.apache.org/ and Map reduce
  http://code.google.com/edu/parallel/mapreduce-tutorial.html, which allow
  data stored in those databases to be processed and collected) (Update:
  Amazon just released their database, Dynamo
  http://www.allthingsdistributed.com/2012/01/amazon-dynamodb.html);

  Neo4j http://neo4j.org/ is focussed on storing graphs.

 Associated with NoSQL http://en.wikipedia.org/wiki/NoSQL are various spin-off
 technologies - libraries for serialisation of data like Avro
 http://avro.apache.org/ and Thrift http://thrift.apache.org/ - and the rising
 importance of data visualisation.

 ZeroMQ http://www.zeromq.org/ feels like it fits here somehow, if only
 because it follows the trend of reinventing stuff that already worked - it's
 the most fashionable messaging library at the moment (simple and easy to use,
 but without the reliability guarantees of "enterprise" messaging - think of
 it more like TCP on steroids).

Web

 Web programming is changing rapidly. Web applications are becoming more and
 more dynamic. This is supported on the client by increasing use of Javascript
 (the latest library is Backbone.js http://documentcloud.github.com/backbone/
 which helps assemble web pages on the client from fragments of data).  This
 requires the server, increasingly, to "push" data (a typical HTTP page used
 to be "pulled" by the client; now the server sends data even when the client
 doesn't "know" that an update is needed).  Push used to imply Comet
 http://en.wikipedia.org/wiki/Comet_(programming) (a technology hack using
 stalled HTTP requests), but HTML 5 http://www.html5rocks.com/en/ (which is
 the future of HTML - XHTML is a largely unused dead-end) is introducing web
 sockets http://www.html5rocks.com/en/tutorials/websockets/basics/ that allow
 bi-directional transfer (like TCP sockets).

 Kind-of related to server push is the idea that servers should be
 asynchronous and event driven. Instead of having a thread for each
 connection, there's just one thread in the server, which reacts to incoming
 client requests on demand.  For this to work the thread cannot "block" (eg
 while waiting for data to be read from the database) so everything is handled
 via callbacks (eg so a database read is handled by a library function (which
 is a separate thread or process); once the data have been read from the
 database a callback is used to transfer the data to the main thread to
 continue processing).  That is very like the way events are handled in
 Javascript on the client, which is where Node.js http://nodejs.org/ and
 server-side Javascript come in (if you know how traditional GUIs work, with a
 single render thread, this is very similar).  Related: JSON
 http://people.planetpostgresql.org/andrew/index.php?/archives/244-Under-the-wire.html
 as a database type http://en.wikipedia.org/wiki/Document-oriented_database;
 Javscript http://code.google.com/p/plv8js/wiki/PLV8 in the database.

 A common theme here is communication; another aspect of that is SPDY
 http://dev.chromium.org/spdy, which is intended to replace HTTP, offering
 more efficient mutliplexing of sessions.  See this article
 http://www.igvita.com/2012/01/18/building-a-modern-web-stack-for-the-realtime-web/
 for an overview of how this affects the technology stack.

 Back on the client, new technologies include WebGL
 http://www.chromeexperiments.com/webgl (this lets you do graphics on the GPU,
 so you can have 3D effects etc), Canvas
 http://www.html5rocks.com/en/tutorials/canvas/integrating/ and SVG
 http://www.html5rocks.com/en/tutorials/raphael/intro/ (for displaying data),
 liquid layouts
 http://designfestival.com/resizing-fixed-fluid-or-responsive-layouts/ (very
 fluid, handling everything from phones to tables to desktop brownsers in a
 single page) layouts, web storage http://en.wikipedia.org/wiki/Web_Storage
 (on the client), much better support for fonts
 http://designmodo.com/custom-web-fonts-fontface, and CSS3 (or processors like
 Sass http://sass-lang.com/) (Update: possibly relevant book
 http://shop.oreilly.com/product/0636920018421.do).

 A big thread running through much of current web work is the idea of social
 cues and interaction - sharing, feedback, progressing, belonging, etc.

Small Computers

 Many of the web developments are being driven by mobile phones. And phone
 hardware technology is now starting to spread - we're starting to see the
 "internet of things" (where famously, even your fridge and toaster have
 internet connections).  One example of this is the Nest Labs thermostat
 http://www.nest.com/; it is "just" a house heating thermostat, but "done like
 Apple".  Another example is Raspberry PI http://www.raspberrypi.org/ - a
 computer capable of running Linux for just $25. Small computers are going to
 be everywhere, very soon.

 Connected to this, there's the idea of a general "dumbing down" of consumer
 computing.  You can see this in the iPad, for example (one take is that it's
 intended to help people consume, not create).  The general idea is of
 "computing appliances" (with restricted choice of software from "app stores")
 replacing general computing.  Another driver for this is the crackdown on
 "pirating".

Programming Languages

 Java is looking increasingly outdated, but there's still no obvious
 replacement. The next big language is going to have to combine two sets of
 ideas: functional programming and better types.  Functional programming, when
 it works, tends to lead to simpler, more compact code, that is easier to make
 parallel across multiple cores.  Better types means that people want
 programming in Java to be as quick and easy as programming in Python, while
 keeping Java's speed and ability to detect errors at compile time.

 For a long time it looked like Scala http://www.scala-lang.org/ would be the
 future, but increasingly the consensus seems to be that it's just too damn
 complicated.  Clojure http://clojure.org/ is a more recent contender, but it
 probably doesn't have enough ideas from the "better types" to win out. More
 obscure (and less ambitious) possibilities include Kotlin
 http://blog.jetbrains.com/kotlin/, Xtend http://www.eclipse.org/Xtext/xtend/
 and Rust http://www.rust-lang.org/.

 For system level programming Go http://golang.org/ is actually pretty good. I
 don't think it's going to be huge, but it really is a better C (and way, way
 nicer than C++).  It's curiously old-fashioned compared to some of the
 languages above, but perhaps that's an advantage.

 Meantime, in the background, Javascript is becoming more and more important,
 even though people don't really like to use it (the Good Parts
 http://shop.oreilly.com/product/9780596517748.do book is excellent BTW).  It
 is becoming more important on the client (see elsewhere) and is also starting
 to play a role on the server when writing asynchronous servers (again, see
 elsewhere).  This has triggered various attempts to make it more friendly:
 Coffeescript http://coffeescript.org/ is a pre-processor that makes it look
 more like Python; Clojurescript https://github.com/clojure/clojurescript
 allows programs written in Clojure to be "compiled" to Javascript; Closure
 http://code.google.com/closure/ (completely unrelated to Clojure) is a
 compiler from Javascript to Javascript (!).

 Increasingly, languages are supporting distributed and parallel computing.
 Erlang http://www.erlang.org/ was an early success in this space. Clojure, as
 mentioned above, is also pushing new ideas (actors and software transactional
 memory).  It's also possible that we need a completely new paradigm -
 non-deterministic computing
 http://my-inner-voice.blogspot.com/2012/01/many-core-processors-everything-you.html.

3D Printing

 Is as mindblowingly awesome as jet packs, IMHO.


Andrew