Andrew Cooke | Contents | Latest | RSS | Previous | Next

C[omp]ute

Welcome to my blog, which was once a mailing list of the same name and is still generated by mail. Please reply via the "comment" links.

Always interested in offers/projects/new ideas. Eclectic experience in fields like: numerical computing; Python web; Java enterprise; functional languages; GPGPU; SQL databases; etc. Based in Santiago, Chile; telecommute worldwide. CV; email.

Personal Projects

Choochoo Training Diary

Last 100 entries

Surprise Paradox; [Books] Good Author List; [Computing] Efficient queries with grouping in Postgres; [Computing] Automatic Wake (Linux); [Computing] AWS CDK Aspects in Go; [Bike] Adidas Gravel Shoes; [Computing, Horror] Biological Chips; [Books] Weird Lit Recs; [Covid] Extended SIR Models; [Art] York-based Printmaker; [Physics] Quantum Transitions are not Instantaneous; [Computing] AI and Drum Machines; [Computing] Probabilities, Stopping Times, Martingales; bpftrace Intro Article; [Computing] Starlab Systems - Linux Laptops; [Computing] Extended Berkeley Packet Filter; [Green] Mainspring Linear Generator; Better Approach; Rummikub Solver; Chilean Poetry; Felicitations - Empowerment Grant; [Bike] Fixing Spyre Brakes (That Need Constant Adjustment); [Computing, Music] Raspberry Pi Media (Audio) Streamer; [Computing] Amazing Hack To Embed DSL In Python; [Bike] Ruta Del Condor (El Alfalfal); [Bike] Estimating Power On Climbs; [Computing] Applying Azure B2C Authentication To Function Apps; [Bike] Gearing On The Back Of An Envelope; [Computing] Okular and Postscript in OpenSuse; There's a fix!; [Computing] Fail2Ban on OpenSuse Leap 15.3 (NFTables); [Cycling, Computing] Power Calculation and Brakes; [Hardware, Computing] Amazing Pockit Computer; Bullying; How I Am - 3 Years Post Accident, 8+ Years With MS; [USA Politics] In America's Uncivil War Republicans Are The Aggressors; [Programming] Selenium and Python; Better Walking Data; [Bike] How Fast Before Walking More Efficient Than Cycling?; [COVID] Coronavirus And Cycling; [Programming] Docker on OpenSuse; Cadence v Speed; [Bike] Gearing For Real Cyclists; [Programming] React plotting - visx; [Programming] React Leaflet; AliExpress Independent Sellers; Applebaum - Twilight of Democracy; [Politics] Back + US Elections; [Programming,Exercise] Simple Timer Script; [News] 2019: The year revolt went global; [Politics] The world's most-surveilled cities; [Bike] Hope Freehub; [Restaurant] Mama Chau's (Chinese, Providencia); [Politics] Brexit Podcast; [Diary] Pneumonia; [Politics] Britain's Reichstag Fire moment; install cairo; [Programming] GCC Sanitizer Flags; [GPU, Programming] Per-Thread Program Counters; My Bike Accident - Looking Back One Year; [Python] Geographic heights are incredibly easy!; [Cooking] Cookie Recipe; Efficient, Simple, Directed Maximisation of Noisy Function; And for argparse; Bash Completion in Python; [Computing] Configuring Github Jekyll Locally; [Maths, Link] The Napkin Project; You can Masquerade in Firewalld; [Bike] Servicing Budget (Spring) Forks; [Crypto] CIA Internet Comms Failure; [Python] Cute Rate Limiting API; [Causality] Judea Pearl Lecture; [Security, Computing] Chinese Hardware Hack Of Supermicro Boards; SQLAlchemy Joined Table Inheritance and Delete Cascade; [Translation] The Club; [Computing] Super Potato Bruh; [Computing] Extending Jupyter; Further HRM Details; [Computing, Bike] Activities in ch2; [Books, Link] Modern Japanese Lit; What ended up there; [Link, Book] Logic Book; Update - Garmin Express / Connect; Garmin Forerunner 35 v 230; [Link, Politics, Internet] Government Trolls; [Link, Politics] Why identity politics benefits the right more than the left; SSH Forwarding; A Specification For Repeating Events; A Fight for the Soul of Science; [Science, Book, Link] Lost In Math; OpenSuse Leap 15 Network Fixes; Update; [Book] Galileo's Middle Finger; [Bike] Chinese Carbon Rims; [Bike] Servicing Shimano XT Front Hub HB-M8010; [Bike] Aliexpress Cycling Tops; [Computing] Change to ssh handling of multiple identities?; [Bike] Endura Hummvee Lite II; [Computing] Marble Based Logic; [Link, Politics] Sanity Check For Nuclear Launch; [Link, Science] Entropy and Life

© 2006-2017 Andrew Cooke (site) / post authors (content).

Python Metaprogramming

From: "andrew cooke" <andrew@...>

Date: Wed, 16 Apr 2008 22:23:08 -0400 (CLT)

Below I'll give an example of metaprogramming in Python.  The
motivation is database access, but the approach is very general.


Motivation
----------

I was thinking of writing a Python program that displays information
from a database, letting the user edit values.  The data have quite a
lot of structure, so I was going to map certain groups of values to
objects.  I could use an existing ORM package, but my experience with
them hasn't been that positive (in particular, I find SQL to be very
useful and want to use it more than is normally possible with standard
ORM).  So instead I started looking for a way to intercept changes
made to the objects so that I could automatically modify the database.


Initial Thoughts
----------------

What I needed then, was a way to intercept changes to attributes (I
didn't want to force the use of "setters" since that is not natural
Python).  It turns out that you can do this using Descriptors, which
are described at http://users.rcn.com/python/download/Descriptor.htm

However, the problem with a simple Descriptor approach was that I
ended up repeating myself: attribute "x" needed both a descriptor and
a method to do the database work.  It seemed to me that things would
be simpler if I could somehow generate the Descriptor (ie the
attribute) automatically, using code that recognised the method.

In other words, if a class has the method "set_x" I wanted that to
trigger the generation of an attribute which, using Descriptors, would
call "set_x" when the attribute changed.

I started by looking at factories, but finally it dawned on me that I
what I really wanted was "metaprogramming" and that led me to
http://www.python.org/download/releases/2.2.3/descrintro/ which, near
the end, has a very nice description of a almost exactly this.  So the
following is pretty much a copy of that.


Implementation
--------------

Below I'll paste the code.  Hopefully the formmating won't go too far
wrong...


class _AutoWriteDescriptor(object):

    '''Descriptor for AutoWrite.

       Implements an attribute that allows an initial setting then
       calls the setter.
    '''

    def __init__(self, name, ignore_identical=True, prefix='_set_'):
        self.name = name
        self.ignore_identical = ignore_identical
        self.prefix = prefix

    def __get__(self, obj, objtype=None):
        self.check_dictionary(obj)
        return obj._auto_write_dict[self.name]

    def __set__(self, obj, new_value):
        self.check_dictionary(obj)
        if self.name in obj._auto_write_dict:
            if (self.ignore_identical is False or
                new_value is not obj._auto_write_dict[self.name]):
                attr = getattr(obj.__class__,
                               "%s%s" % (self.prefix, self.name), None)
                obj._auto_write_dict[self.name] = attr(obj, new_value)
        else:
            obj._auto_write_dict[self.name] = new_value

    def __delete__(self, obj):
        raise AttributeError("cannot delete AutoWrite attribute")

    def check_dictionary(self, obj):
        if getattr(obj, "_auto_write_dict", None) is None:
            raise AttributeError(
                '''AutoWrite dictionary missing.
                   Class %s probably does not call AutoWrite.__init__'''
                % obj.__class__)

So this (above) is a simple Descriptor.  The only thing to understand
about Descriptors is that they are attributes.  So the methods above
live on the attribute, not on the main object.  When Python accesses
an attribute it checks and, if it's a Descriptor, it invokes the
appropriate method (get if the attribute is being read, set if its
value is being changed, delete if the attribute is being deleted).

The advantage of doing things this way (which is what Python's "new
classes" is all about really) is twofold.  First, you end up uniting
methods and functions (this is emphasised a lot in the Python docs,
but isn't really that interesting, as far as I can see).  Second, you
have an efficient way to change how certain attributes behave.  This
is much better than changing the main object's __getattr__ and
_setattr__ (or whatever thay are called), because the extra code is
only invoked for the "special" attributes - there no speed penalty
otherwise.

I've not said what this Descriptor actually does, but it should be
clear from the discussion earlier.  If the value is being read, the
actual value is pulled from a dictionary that is stored on the main
object.

If the value is being set for the first time (ie when the object is
populated from the database) the value is stored.  Subsequent updates
(ie when the object is modified via the user interface) call the
setter method.  The setter method is "_set_..." (the prefix variable).


class _AutoWriteMeta(type):

    '''Metaclass for AutoWrite.

       Constructs the attributes for AutoWrite using _AutoWriteDescriptor.
    '''

    def __init__(cls, name, bases, dict):
        super(_AutoWriteMeta, cls).__init__(name, bases, dict)
        prefix = dict.get('_auto_write_prefix', '_set_')
        ignore_identical = dict.get('_auto_write_ignore_identical', True)
        for attr in dict.keys():
            if attr.startswith(prefix):
                name = attr[len(prefix):]
                setattr(cls, name,
                        _AutoWriteDescriptor(name, ignore_identical, prefix))

This (above) is the metaclass - effectively is a class factory and
it's called when the class description is parsed by Python.  Once you
know that (and assuming you know that everything in Python is
basically a dictionary) you can predict the rest: the dictionary of
methods is read and for each one that starts with "_set" a Descriptor
is generated.


class AutoWrite(object):

    '''Superclass for active attributes.

       Subclasses should define _set_'name' for active attributes.  The
       named attributes are then automatically generated with the
       following behaviour:
       - Initial setting stores the value (typically the initial value is
         stored during creation, reading from a database)
       - Setting a new value that is not equal to the old value will call
         the set method.  That method should either return a value (which is
         what will be stored) or throw GlobalStateChanged, to indicate that
         the entire system needs to re-read values.
       The _set_'name' method should have the signature (self, new_value).

       The ignore_identical and prefix options can be set via the class
       dictionary.  For example, to use method names like 'on_change_x'
       which are called always (even if the value being set is identical):

           class MyClass(AutoWrite):
               _auto_write_prefix = 'on_change_'
               _auto_write_ignore_identical = False
    '''

    __metaclass__ = _AutoWriteMeta

    def __init__(self):
        self._auto_write_dict = {}

The Descriptor and metaclass are all that you need, really.  This base
class is just a nice way of putting everything together.  It ensures
that the right metaclass is used and that the dictionary used to store
the values is created.  So to use everything above you just subclass
this.


Tests
-----

The following tests show this in action:

class Simple(AutoWrite):

    def __init__(self):
        AutoWrite.__init__(self)
        self.last_setter_called = None

    def _set_x(self, new_value):
        self.last_setter_called = new_value
        return new_value

    def _set_inc(self, new_value):
        self.last_setter_called = new_value
        return new_value + 1


class BadInheritance(AutoWrite):

    def __init__(self):
        pass

    def _set_x(self, new_value):
        pass


class Options(AutoWrite):

    _auto_write_prefix = 'set_'
    _auto_write_ignore_identical = False

    def __init__(self):
        AutoWrite.__init__(self)
        self.last_setter_called = None

    def set_x(self, new_value):
        self.last_setter_called = new_value
        return new_value


class AutoWriteTest(unittest.TestCase):

    def test_direct(self):
        simple = Simple()
        simple.x = 1
        self.assertEqual(simple.x, 1)
        self.assertEqual(simple.last_setter_called, None)
        simple.x = 2
        self.assertEqual(simple.x, 2)
        self.assertEqual(simple.last_setter_called, 2)
        try:
            del simple.x
            self.fail("no exception when deleting attribute")
        except AttributeError:
            pass
        self.assertEqual(simple.x, 2)

    def test_inc(self):
        simple = Simple()
        simple.inc = 1
        self.assertEqual(simple.inc, 1)
        self.assertEqual(simple.last_setter_called, None)
        simple.inc = 2
        self.assertEqual(simple.inc, 3)
        self.assertEqual(simple.last_setter_called, 2)
        try:
            del simple.inc
            self.fail("no exception when deleting attribute")
        except AttributeError:
            pass
        self.assertEqual(simple.inc, 3)

    def test_bad_inheritance(self):
        bad = BadInheritance()
        try:
            bad.x = 1
            self.fail("expected failure due to no __init__ call")
        except AttributeError, e:
            self.assertNotEqual(
                e.message.find("AutoWrite dictionary missing"), -1)
            self.assertNotEqual(e.message.find("BadInheritance"), -1)

    def test_options(self):
        options = Options()
        options.x = 1
        self.assertEqual(options.x, 1)
        self.assertEqual(options.last_setter_called, None)
        options.x = 1
        self.assertEqual(options.x, 1)
        self.assertEqual(options.last_setter_called, 1)

The test_direct method above shows what is happening.  Setting "x" to
1 the first time works like any normal attribute.  The value can be
read back as expected.  Setting it again (to 2) triggers the callback
which, in this case, just sets the attribute "last_setter_called".


Hope that made sense,
Andrew

Useful Responses to Python Metaprogramming

From: "andrew cooke" <andrew@...>

Date: Thu, 17 Apr 2008 09:00:30 -0400 (CLT)

There are some useful comments here -
http://groups.google.com/group/comp.lang.python/browse_thread/thread/e4144d9c8fafe29a

I will update my own code, but not the post here (this clunky web site /
mail archive isn't that easy to update....)

Andrew

Another, Simpler Python Meta-Programming Example

From: "andrew cooke" <andrew@...>

Date: Tue, 5 Aug 2008 21:24:38 -0400 (CLT)

Here I wanted each class to display self._text in a different way:

class _LexicalMeta(type):
    '''
    Constructs __str__ using _template
    '''

    def __init__(cls, name, bases, dict):
        super(_LexicalMeta, cls).__init__(name, bases, dict)
        template = dict['_template']
        setattr(cls, '__str__', lambda self: template % self._text)


class _BaseLexical(object):
    '''
    Base class for lexical objects.
    '''

    __metaclass__ = _LexicalMeta
    _template = None

    def __init__(self, text):
        self._text = text


class Word(_BaseLexical):
    _template = '"%s"'

class Float(_BaseLexical):
    _template = 'float:%s'

class Integer(_BaseLexical):
    _template = 'integer:%s'

class Symbol(_BaseLexical):
    _template = '%s'


So, for example,
  str(Work('abc')) == '"abc"'
  str(Integer('123')) == 'integer:123'

Andrew

Comment on this post