C[omp]ute

Welcome to my blog, which was once a mailing list of the same name and is still generated by mail. Please reply via the "comment" links.

Always interested in offers/projects/new ideas. Eclectic experience in fields like: numerical computing; Python web; Java enterprise; functional languages; GPGPU; SQL databases; etc. Based in Santiago, Chile; telecommute worldwide. CV; email.

© 2006-2015 Andrew Cooke (site) / post authors (content).

[Python] Geographic heights are incredibly easy!

From: andrew cooke <andrew@...>

Date: Sun, 13 Jan 2019 20:47:20 -0300

Wanted to add heights to bike rides in Choochoo, given the GPS
coordinates.  At first I considered stealing the data from Strava, but
their T&C don't like it and anyway I couldn't find it in the API.
Then I considered Google's Elevation Service, but it's \$5 for 1,000
values.  Then I considered the free OpenStreetMap equivalent, but that
seemed to be broken.  Then I realised it's pretty damn easy to do by
hand!

Turns out that the Space Shuttle scanned the entire Earth (except the
Poles) at a resolution of one arcsecond (about 30m on the equator) and
the data have been publicly released.

The project is called SRTM, and the 30m data are "v3" or SRTM3.  More
info at https://wiki.openstreetmap.org/wiki/SRTM and

http://dwtkns.com/srtm30m/ and there's an excellent (although buggy)
blog post on how to parse the data at
code at https://github.com/aatishnn/srtm-python

Because the coords are in the file name there's no need for any kind
of RTree or similar index - you just take your coords and infer what
stick it in an array, and return the array value!

My own code to do this is at
https://github.com/andrewcooke/choochoo/blob/master/ch2/sortem/__init__.py
and includes bilinear interpolation (you could cut+paste that code
except for the constructor which is specific to Choochoo - just replace
the messing around w Constant with a literal directory string).

The tests are at
https://github.com/andrewcooke/choochoo/blob/master/tests/test_sortem.py
and from the contours there, which are plotted across 4 tiles, it's
pretty clear that the interpolation is seamless.

So easy!  I thought this would take a week of work and I did it all
this afternoon....

Andrew

Previous Entries

From: andrew cooke <andrew@...>

Date: Tue, 25 Dec 2018 17:30:46 -0300

125g butter
1 egg
60g sugar (or more?)
45g cocoa
130g flour (w raising powder)
120g chocolate

Mix butter, egg and sugar.  Sieve in and mix cocoa and flour.  Add a
little water if necessary - want a thick, sticky mix, as solid as
possible, but not powder.

Break the chocolate into pieces and add.

Cool in fridge.  Pre-heat oven to 180C.

Place blobs of dough on lightly greased baking paper on tray.  Cook
for 15m.  Should flatten but not spread much.

Not very sweet, except for the chocolate.  Very cocoay and good
texture.  Good w ice-cream?

Variations: more sugar?  vanilla?

Andrew

Efficient, Simple, Directed Maximisation of Noisy Function

From: andrew cooke <andrew@...>

Date: Fri, 14 Dec 2018 18:04:10 -0300

Maybe this is already known - I guess it must be - but I just dreamt
it up myself.  I need to find the max (could be min, obvs) of a noise
function in 1D.

Since it's noisy I can't assume it's smooth, hope it has a global
minimum, and bisect.  I need something more robust.

At the same time, I don't want to be doing repeated evaluations of the
function where they're not needed, so any iterative deepening of a
search should re-use old values when possible.

So here's the solution:

* Evaluate the function at 5 equally spaced points across the range
(including the two extremes).

* Throw away two end points.

* Expand the remaining 3 points back to 5 by inserting new points at
the mid-points of the existing points.

* Repeat until you're happy.

The trick is to know which end points to discard.  It could be one
either end, or two from one end.

What I do is calculate the average of the x values at the points,
weighted by the function values there.  Then I compare this to the
unweighted average.  If it's higher, then the max is towards the high
end, so discard a low point.  Or vice-versa.  And repeat for a second
point.

Here's an example.  The lines are (x, f(x)) pairs:

[(0.0, 0), (0.25, 5), (0.5, 10), (0.75, 1), (1.0, 1)]
[(0.25, 5), (0.375, 12), (0.5, 10), (0.625, 2), (0.75, 1)]
[(0.25, 5), (0.3125, 11), (0.375, 12), (0.4375, 10), (0.5, 10)]
[(0.3125, 11), (0.34375, 12), (0.375, 12), (0.40625, 10), (0.4375, 10)]
[(0.3125, 11), (0.328125, 11), (0.34375, 12), (0.359375, 13), (0.375, 12)]

On the first step, both ends were discarded.  On the second, two from
the "high" side, etc.  The maximum is 13 at 0.359375, roughly.

And here's the code:

def expand_max(lo, hi, n, f):
data = [(x, f(x)) for x in (lo + i * (hi - lo) / 4 for i in range(5))]
for _ in range(n):
print(data)
while len(data) > 3:
w = sum(x*fx for (i, (x, fx)) in enumerate(data)) / sum(fx for (x, fx) in data)
m = sum(x for (x, fx) in data) / len(data)
if w > m:
del data[0]
else:
del data[-1]
x = (data[0][0] + data[1][0]) / 2
data.insert(1, (x, f(x)))
x = (data[-2][0] + data[-1][0]) / 2
data.insert(-1, (x, f(x)))
x_max, fx_max = None, None
for (x, fx) in data:
if x_max is None or fx > fx_max:
x_max, fx_max = x, fx
return x_max, fx_max

Andrew

Bash Completion in Python

From: andrew cooke <andrew@...>

Date: Fri, 7 Dec 2018 08:02:11 -0300

https://www.reddit.com/r/Python/comments/a3vg7q/i_made_a_tutorial_for_writing_bash_completion/

https://github.com/CarvellScott/completion_utils

Andrew

[Computing] Configuring Github Jekyll Locally

From: andrew cooke <andrew@...>

Date: Sun, 25 Nov 2018 15:02:59 -0300

If you have docs that already exist, and that github displays on
Github Pages, then you may want to run jekyll locally - just to save on
round-tripping time or when github (as today) starts thoring errors
for no reason.

The instructions at
https://help.github.com/articles/setting-up-your-github-pages-site-locally-with-jekyll/
are pretty confusing if you already have things working on github.

What you need to do is:

* Install requirements as described (may need to run as root)

* In your local repo, at the root of the jekyll pages (for me, the
docs directory), create the Gemfile described in step 2 and then
install jekyll.

* Run jekyll in that same directory (step 4)

In other words, SKIP steps 1 and 3 and do everything in the docs
directory.

At least, that seemed to work for me.

Andrew

From: andrew cooke <andrew@...>

Date: Tue, 20 Nov 2018 08:10:21 -0300

Introductory higher math.

http://web.evanchen.cc/napkin.html

Andrew

[Bike] Servicing Budget (Spring) Forks

From: andrew cooke <andrew@...>

Date: Tue, 6 Nov 2018 11:49:06 -0300

Lower priced mountain biks often come with forks that contain a coil
spring (rather than an "air spring").  These tend to not get a lot of
love on the intenet.  That's partly justified - an air fork is much
more adjustable and has better damping.  But the spring forks do have
some advantages too.  Most of all, they're very low maintenance - most
people don't touch them once they've bought them.

Despite that, a little care can help keep your fork working well and
extend its life.  I just serviced mine and I thought I'd make a few
notes here ot help others.

First, I am not 100% sure what my fork is.  I removed the stickers
years ago.  I think it may be a Suntour XCM (26").  Whatever, the
general principles below should apply to pretty much and budget fork.

There are two things you can do.  The simplest (and most important) is
to remove the lowers.  More complicated is to open up the "insides"
too.

Removing Lowers

First, remove the two bolts holding the front brake (I'm assuming
disc) so that it's no longer connected to the fork.  Also remove the
clip or zip tie or tape that holds the hose to the fork.  Then you can
ignore the brake (until the end, when we need to put it back on).

(Don't press the brake lever with the caliper loose - you can push the
hydraulic pistons out).

Turn the bike upside down, so it's sitting on the bars (or put it in a
stand if you have one I guess) and remove the wheel.  Have a look at
the bottoms of the forks.  On mine each has a 10mm nut.  Remove the
nuts and see if the you can slide the lowers off.  Don't use violence
- just pull them up with your hands while keeping the bike on the

Possibly they won't come off.  This is because the rods inside the
fork tend to stick in the ends of the lowers.  We can loosen them by
pushing the threads that the nuts were fastened on back "into" the
lowers.

To do this, take a piece of wood, lay it on top of the threaded rod,
and strike with a hammer.  You may need a few attempts (change the
position of the wood since the rods makes a hole!) before the threaded
rods moves noticeably.  Once both sides are loose, you should be able
to pull the lowers off, no problem.

With the lowers off, clean the exposed stanchions and the inside of
the lowers (eg with a stick and cloth - if the cloth gets stuck
inside, blow it out from the other end).

The stanchions are steel, so can rust.  This is why it's worth
servicing them - we want them to stay smooth and clean.  So rub some
grease on them (I use random car grease - nothing fancy).

To reassemble, first put some more grease on the insides of the seals.
Then slide the lowers back on the stanchions.  Push them down and the
rods will re-appear out of the ends.  Replace and tighten the nuts
(fairly tight - you don't want these coming undone...).

Finally, replace the wheel and re-attach the brakes (once the caliper
is in place, apply the brakes and then tighten the bolts - that will
help tighten the bolts with the calipers in the right position to
avoid rubbing).

Looking Inside

If you want to look "inside" the fork, first remove the lowers (as
above).

Next, use an appropriate tool from Suntour to unscrew the two caps on
the "top" of the fork (one on each side).  This is easier to do with
the bike lying on its side (or upright in a stand).

I'd really enourage buying the right tool.  Mine is just a plastic
wrench (looks like a big bottle opener or measuring spoon).  You can
get the caps off without it, but getting the spring side back on is
tricky (because the spring is compressed) and without the tool you're
more likely to cross-thread the cap.

If you have a lockout you may need to move the lockout knob before
removing the cap.  Mine just levers off (carefully insert a flat
screwdriver underneath and lift).

It's interesting to see what's inside, but there's not much to do
except clean and grease.  I removed an elastomer damper from "inside"
my spring so that the forks had a little more travel, but I'm not sure
I noticed much difference.  On the other side, my fork has a damper,
but it's a sealed unit with nothing to adjust.

(BTW I couldn't remove the "foot" on the spring side because the
bottom-out damper held it in place.)

Assembly is the reverse process.  Be careful replacing the cap on the
spring side as it's easy to cross-thread.  My caps are plastic so I
didn't tighten them crazy-tight.

I hope that helps.  With a little care you can keep these babies
rust-free and they'll last forever.

Andrew

[Crypto] CIA Internet Comms Failure

From: andrew cooke <andrew@...>

Date: Sat, 3 Nov 2018 17:37:18 -0300

It's difficult to tell exactly what happened, but my guess from
reading between the lines is that the CIA had a bunch of fake sites
that were used as endpoints for communication.  Somehow the Iranians
detected a pattern in the way these sites were generated (eg matching
comments in the HTML source) that let them identify the sites and, via
traffic monitoring, work out who was using them.

https://in.news.yahoo.com/cias-communications-suffered-catastrophic-compromise-started-iran-090018710.html

Andrew

[Python] Cute Rate Limiting API

From: andrew cooke <andrew@...>

Date: Thu, 25 Oct 2018 21:13:23 -0300

It's just a decorator - https://pypi.org/project/ratelimit/

Andrew

[Causality] Judea Pearl Lecture

From: andrew cooke <andrew@...>

Date: Fri, 19 Oct 2018 12:00:01 -0300

Got confused by his book years ago, but this lecture is very good.

Andrew

[Security, Computing] Chinese Hardware Hack Of Supermicro Boards

From: andrew cooke <andrew@...>

Date: Thu, 4 Oct 2018 14:36:32 -0300

https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies

Damn.  Even I have a SuperMicro board...

Andrew

SQLAlchemy Joined Table Inheritance and Delete Cascade

From: andrew cooke <andrew@...>

Date: Fri, 28 Sep 2018 09:10:21 -0300

A self-contained test that should be self-explanatory
https://github.com/andrewcooke/choochoo/blob/db-stats/tests/test_inheritance.py

Trying to make Choochoo robust to deleted data.  It's complicated.

Andrew