- Lana works again (i.e. it doesn’t cheerfully move the bytecode around in memory while you’re trying to run it)
- I’ve implemented the Hough transform and the inverse Hough transform
- The following ImageLana code runs:
go = procedure() a=imgload("testvertline.png") b=a.hough() c=b.dehough(0) c.normalise() a.view(0) b.view(1) c.view(2) end go() wait()
And I get the following images out:
Here’s a lovely little presentation in which Ken Perlin talks about his noise function – Perlin noise – which he thought up while he was working on TRON (the original,) and which everyone in graphics uses when they want some vaguely naturalistic looking noise. Clouds, wall textures, height maps – it’s all Perlin.
I keep everything in Subversion. Lecture notes, bits of writing, everything. Including bits of miscellaneous code, the prototypes of which sometimes blossom (or metastasize, depending on your point of view) into full-sized projects.
To this end, I have a “miscellaneous code” repo with subdirectories for each new thingy. Most of these are very small, but sometimes they get large or I might want them to become public, in which case I have to turn part of this repo into a whole new repo.
To do this, I need to:
- dump the repository
- filter the repository to include only the bit I want
- change paths
- populate a new repo with the new data
svnadmin dump svn/misc_code | svndumpfilter include \ --drop-empty-revs \ --renumber-revs /barry >barry.dump
Now you can create the new repo and load the data into it:
svn create svn/newrepo svn load svn/newrepo < barry.dump
Also, pretty much none of this will work with binary data because of the hell that is md5 checksums being screwed up. But for text-only repos, it should be fine.
Well, that bug’s fixed. Simple matter of changing the order in which reference counts are incremented and decremented. Valgrind reports no problems from the unit tests under both memcheck and exp-ptrcheck. Now for more documentation (which is about halfway through its initial release.)
Well, the documentation is coming along. That is, it’s been started (as Doxygen pages) and I’m working through it. But now I have a horrible garbage collection problem. The code
create().foo = 1
crashes horribly. What happens is that an object gets created, then gets deleted because we’re never going to use it again, then we try to assign a value to it. Dammit. Need to rejig the ordering.
I’ve reached what feels like a fairly significant milestone in Lana today – the following transcript shows dictionaries and iterators working, with all the objects and temporaries being garbage collected successfully. There is, of course, a bunch of rather more extensive unit tests to write now, but it’s starting to feel like a proper language.
Script started on Sun 24 Jul 2011 11:43:44 BST white@cranberry:~/misccode/script/lib/lana$ cat foo f=function() a=dict() a=0 a="wobble" a["hello"]=56 print("ITERATOR START") for i in values(a) print(i) endfor print("") for i in keys(a) print(str(i) + " " + str(a[i])) endfor end f() white@cranberry:~/misccode/script/lib/lana$ valgrind ./testmain foo ==4441== Memcheck, a memory error detector ==4441== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al. ==4441== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h for copyright info ==4441== Command: ./testmain foo ==4441== ITERATOR START 0 wobble 56 0 0 1 wobble hello 56 ==4441== ==4441== HEAP SUMMARY: ==4441== in use at exit: 0 bytes in 0 blocks ==4441== total heap usage: 160 allocs, 160 frees, 60,172 bytes allocated ==4441== ==4441== All heap blocks were freed -- no leaks are possible ==4441== ==4441== For counts of detected and suppressed errors, rerun with: -v ==4441== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 18 from 7) white@cranberry:~/misccode/script/lib/lana$ Script done on Sun 24 Jul 2011 11:43:58 BST
Having an actual day job (Wii programming) has meant that work on Lana has had to take a back seat. Combining this with the fact that I’m now doing the tricky stuff, development has slowed down somewhat. But here’s what’s new:
Notwithstanding what I said in an earlier post, Lana now has garbage collection! This works for strings, objects, dictionaries, everything. I’ve used the same techniques Python uses:
- Reference counting for all managed data – whenever a managed entity (such as a string or object) is referred to in a value, a new entity is created, or a reference to an entity is copied into a new value, its reference count is increased; when the new reference goes away the reference count is decreased; and when the count is zero the entity is destroyed. This is made rather more complex by there being two distinct allocation models – SimpleMalloc, used for strings and the like and malloc()/free() are used; and Complex/SimpleNew, where the entity is some subclass of GarbageCollected – and in the Complex case can contain references to other entities.
- This idea of referenced entities also containing references means that plain reference counting isn’t enough – entities which refer to each other in a loop will never be deleted. Therefore we also have a cycle detector which uses a list of all the reference counted entities which can refer to other entities (so called Complex entities.) It can detect entities in a cycle which are not referred to from outside the cycle, and delete them. This has to be run periodically, and we currently do it with the Lana
These are containers which can be keyed on, and can contain, any type of value:
a=dict() a="hello" a["foo"]=bar b=create() a["object"]=b b.dict = a # created a cycle!
These proved somewhat hairy – possibly the hairiest thing in Lana. That’s because of the way variables work. When a variable is encountered in the code, what actually happens is that the virtual machine stacks a Ref – a reference to the variable. If a subsequent instruction wants to read the variable, it just calls the internal
popval() method, which will pop the Ref from the stack and dereference it, returning the variable’s contents to the instruction code. If we want to set the value – and only the code for the
OP_SET instruction does this – we pop the reference without dereferencing it and write to the variable it refers to.
Exactly the same technique is used for object properties – these are represented by PropRef values, which contain a pointer to the object and the ID of the property.
Naturally I wanted to do the same thing with dictionaries – have a type of value (a “DictRef”) which refers to a dictionary entry. To do this, I need to have the value contain both the key and a reference to the dictionary. However, if the key can be any Lana value, I need to be able to create a value type which contains both a dictionary reference and any value type. Nastily recursive, that definition. So I cheated, and use a special “packed value” in which the type field holds the hash key type and the allocation type flag (SimpleMalloc, etc.) is set to a special DictRefAlloc type, half the value field holds the key itself (which is the “primary part” of the key value,) and the other half holds the dictionary pointer. Nasty, and it means that values with both secondary and primary parts – PropRefs, DictRefs themselves, and some of the advanced types I may create later like delegates and closures – can’t serve as keys. It’s a small price.
- Iterators – these are in the C++ code but there’s no Lana
forstatement. Also, I’ve just realised that I need to iterate over both keys and values, which opens a small can of worms.
- Array lists – again, the basic object exists in the C++ code but isn’t linked into Lana’s syntax.
- User iterables – i.e. objects the user creates in C++ (or Lana I suppose) could have the iterable interface, allowing
forto work with them.
- Optimisation of common instruction sequences such as
- Serialisation – this is a big one, and comes close to the “point” of the language.
- Delegates in the C# sense – values which contain both a method reference and the object to call it on, as a single value: very useful for event-driven code!
- Closures – I have some notes on how to do this, by using a value to hold both the function pointer and the environment in which it was created (or a subset thereof.)
- Creating a basic applet
- Registering the applet by writing a server file
- Testing the applet
- The Meter class – making the code OO
- Adding a timer
- Adding the wattage
- Files to download
Last time we looked at the basics of Python, and how Linux allows access to system information via file-like objects inside the
/dev filesystem. We then looked at writing a Python function which got the battery information and calculated the power drain in watts.
Now, we’re going to actually build a panel applet to display this value at regular intervals, and write the special file Gnome needs to be able to run the applet and insert it in a panel.
Let’s start our code off with a whole bunch of imports, for the various Python libraries we’ll need; and the
get() method we wrote last time. We’ll put all this into a file called
#!/usr/bin/env python import gnomeapplet import sys import pygtk pygtk.require('2.0') import gtk import gtk.glade import gnome.ui import os.path import re import gobject def get(): f = open('/proc/acpi/battery/BAT0/state') for x in f.readlines(): if x.find('present rate')>=0: d,data=x.split(':') ma,d = data.strip().split() ma = float(ma)/1000.0 if x.find('present voltage')>=0: d,data=x.split(':') mv,d = data.strip().split() mv = float(mv)/1000.0 f.close() return ma*mv
A basic applet
Gnome uses the Bonobo object model to connect components together. This lets us have many different programs, displaying data in their own way inside components belonging to other programs. And this is how panel applets work: your applet is a Bonobo component, displaying its data as part of the Gnome Panel app.
To write a Bonobo applet, you need to provide a function which Bonobo will call. We’ll call this the “factory function.” It takes a Bonobo container widget and the activation IID (a unique string for every bonobo object type, we can ignore it here.) We pass this function to a special
gnomeapplet method called
bonobo_factory(). This will set up the container widget for us and then call our function.
Inside our factory function we should create our GTK widgets, and add them to the applet, and tell the applet (and its widgets) to show itself.
Our code will therefore go something like this for a very simple applet:
def factory(applet,iid): widget = gtk.Label("test!") applet.add(widget) applet.show_all() return gtk.TRUE # indicate success! gnomeapplet.bonobo_factory("OAFIID:GNOME_Wattometer_Factory", gnomeapplet.Applet.__gtype__, "wattometer", "0", factory)
This is pretty much the simplest panel applet there is, displaying a single label with the word “test!” in it. Everything in here should be straightforward, apart from all those arguments to
bonobo_factory(). These are:
- The bonobo-activation iid of the factory (more on this later.)
- The type of the object, which will be an applet – bonobo factories are actually used all over Gnome for all sorts of things, but here we want an applet. We get the type using Python’s reflection mechanism.
- A description.
- A version.
- The factory function we’re providing.
Of course, if you run this, it won’t do anything – we haven’t told Gnome about it.
Registering the applet
We now need to tell Gnome about the applet. We do this by providing a server file and putting it into
/usr/lib/bonobo/servers/, where it will be read by the bonobo activation server daemon. Here’s our file, which we should call
<oaf_info> <oaf_server iid="OAFIID:GNOME_Wattometer_Factory" type="exe" location="/home/white/misccode/wattometer/wattometer.py"> <oaf_attribute name="repo_ids" type="stringv"> <item value="IDL:Bonobo/GenericFactory:1.0"/> <item value="IDL:Bonobo/Unknown:1.0"/> </oaf_attribute> <oaf_attribute name="name" type="string" value="Wattometer"/> <oaf_attribute name="description" type="string" value="Wattage applet"/> </oaf_server> <oaf_server iid="OAFIID:GNOME_Wattometer" type="factory" location="OAFIID:GNOME_Wattometer_Factory"> <oaf_attribute name="repo_ids" type="stringv"> <item value="IDL:GNOME/Vertigo/PanelAppletShell:1.0"/> <item value="IDL:Bonobo/Control:1.0"/> <item value="IDL:Bonobo/Unknown:1.0"/> </oaf_attribute> <oaf_attribute name="name" type="string" value="Wattometer"/> <oaf_attribute name="description" type="string" value="Wattage applet"/> <oaf_attribute name="panel:category" type="string" value="Utility"/> <oaf_attribute name="panel:icon" type="string" value="iconimage.png"/> </oaf_server> </oaf_info>
Complicated, yes? However, it’s pretty much a standard format for applets. You can see it registers two servers: the first is the executable (type “exe”), which is our Python program. The second is the applet factory itself, which is told about the exe so it can use our program to fill itself with our widgets.
The only things you need to worry about are:
- Making sure the IID’s are unique. The IID of the factory is the IID which we should use in our program’s call to
- The name and description in the factory block
- Making sure the factory’s location attribute points to the exe
- Making sure the exe’s location attribute points to the location of the python program
You might also want to change category and image, too.
Once you’ve set this up correctly, and copied it into the servers directory, the Gnome Panel’s “Add To Panel…” dialog will contain the new applet. Naturally, you’ll have to be an administrator to be able to put the server file into its new home!
Testing the applet
If you just run this program from the command line it won’t do much: it’s not being called from inside the activation server, so there’s no applet framework for it to plug its widgets into. We’ll modify our code so that we can run it:
def factory(applet,iid): widget = gtk.Label("test!") applet.add(widget) applet.show_all() return gtk.TRUE # indicate success! if __name__ == '__main__': # testing for execution if len(sys.argv) > 1 and sys.argv == '-d': # debugging mainWindow = gtk.Window() mainWindow.set_title('Applet window') mainWindow.connect('destroy', gtk.main_quit) applet = gnomeapplet.Applet() factory(applet, None) applet.reparent(mainWindow) mainWindow.show_all() gtk.main() sys.exit() else: gnomeapplet.bonobo_factory("OAFIID:GNOME_Wattometer_Factory", gnomeapplet.Applet.__gtype__, "wattometer", "0", factory)
Let’s look at those changes.
First, we use the “if name=main” trick to make sure we’re actually being run from the command line. If the code is just being included in another program, we won’t do anything. It’s not entirely necessary here, but it’s good practice. Then, we look to see if we’ve been called with a “-d” argument. If so, instead of going the normal route of plugging into an existing widget (the applet,) we
- create a window
- create an applet widget
- call our factory, creating our widgets inside the applet
- make the applet part of the window
- show everything
- run the GTK main loop until we exit
So now, if we run the program with
python wattometer.py -d
we’ll see a tiny window pop up with our applet inside it, just like a normal application. There are no title bar buttons, so you’ll have to close it by killing the process. Ctrl-C from the command line should do it.
To keep everything tidy, we’re going to put all the user interface code into a class called
Meter. All our factory function will do is create a new
Meter object. Add the following code, and change the factory method as shown:
class Meter: # this is the constructor def __init__(self,applet,iid): self.applet = applet self.ebox = gtk.EventBox() self.label = gtk.Label("test") self.ebox.add(self.label) applet.add(self.ebox) applet.connect("destroy",self.cleanup) applet.show_all() def cleanup(self,e): pass # pythonese for "do nothing" def factory(applet, iid): Meter(applet,iid) return True
If you’re new to Python, note the following oddities of Python’s object syntax:
- Methods definitions always have
selfas their first argument, even though this isn’t passed in in the normal way
- Methods and instance variables must be preceded by
self.when accessed from a method
- The constructor is called
- Like all variables in Python, we don’t need to declare instance variables – just setting them to values inside the constructor will do that.
Two functional changes have also been made here: the label is created inside an EventBox, so we can catch clicks and the like; and the applet now calls a cleanup method when it’s closed – which does nothing, but it’s always a good idea to have it there in case you need to add something to it.
Everything’s now nicely contained, and we can start to build the Meter functionality.
Adding a timer
It’s not much use just displaying the wattage once; we need to show it every few seconds. To do this, we’ll use a timeout timer, which will call a given method at regular intervals.
We start a timer by adding a method called
settimeout() to our class:
def settimeout(self,delay): if self.source>=0: gobject.source_remove(self.source) self.source=gobject.timeout_add(delay*1000, # delay in milliseconds self.timeout_callback, self)
What this method does is destroy a timer if one already exists, and then create a new one. That way, we can call the method several times if we want to change the interval.
We’ll also need to add some code to our constructor to initialise the source value and start the timer. We’ll also initialise a counter for testing purposes:
def __init__(self,applet,iid): ... self.counter = 0 self.source = -1 self.settimeout(2) # 2 seconds
We’ll use the counter in our callback method:
def timeout_callback(self,event): self.counter = self.counter+1 self.label.set_label("count %d" % self.counter) return True
If you run this you’ll find that the label says “test!” for the first five seconds. This is because the correct data isn’t put there until the timeout runs. This is no good for our purposes, so we’ll put the code to change the label into a separate method which we’ll run from both the constructor and the timer tick:
def __init__(self,applet,iid): ... self.counter = 0 self.source = -1 self.settimeout(2) # 2 seconds self.setlabel() def setlabel(self): self.label.set_label("count %d" % self.counter) def timeout_callback(self,event): self.counter = self.counter+1 self.label.set_label("count %d" % self.counter) return True
Run this with
-d, or even as an applet if you’ve installed the server file, and you’ll see a slowly increasing count. Nearly there!
Adding the wattage
Finally, we’ll want to actually call our
get() function to output the ever-changing wattage. Get rid of the counter, and replace the new
setlabel() method. One little quirk – if the system reports we’re using a very small about of battery power it doesn’t mean that we’ve suddenly gone into Hyper-Efficient Battery Usage Mode, it means that we’re plugged into the mains. Although it could be argued that this is a Hyper-Efficient Battery Usage Mode… anyway, we’ll check for that case:
def setlabel(self): v = get() if v<0.001: s = "(charging)" else: s = "%0.1fW" % v self.label.set_label(s) return True
Note how we’re formatting the wattage: the format string
%0.1f means we should use one decimal place of precision.
And we’re done! You now have a working wattage applet!
You can download the files developed in these articles as a zip file from here. They are:
GNOME_Wattometer.server, which should be edited to change the location of
wattometer.pyand put into
wattometer2.py, an improved version which uses a drop-down menu to allow the user to change the sampling rate
I called it Lana, some wag called it Jimscript. Anyway, as those who know me will know by now, my projects tend to expand until I need to embed a scripting language in them. And my current project is no exception. Previously, however, I’ve always just bolted in some kind of Forthlike, and hang the readability. This time, I wanted to do it properly – either embed Lua or Python, or do a proper language.
What I wanted this time is a language that would allow me to “decorate” my projects’ data structures with tiny fragments of arbitrary code, and for the code to be fully integrated with my data structures. Lua and Python’s integration – particularly with C++ – is still pretty revolting, so once again I had an excuse to do it myself. And earlier today, Lana ran (after a whole bunch of other unit tests) the following program:
# create an object a = create() # set some values in it a.val1=1 a.val2=2 # set a function in it (i.e a method) a.setvals = function(a,b) this.val1 = a this.val2 = b end # and another one which does a calculation a.somecalc = function() returns return this.val1+this.val2 end # assert that the calculation gets the correct value assertInt(3,a.somecalc()) # make a clone of the object b = clone(a) b.setvals(3,5) # assert that the calculation now gives a different result assertInt(8,b.somecalc()) # create a new function as a first-class value # but first copy the old function somewhere safe! oldfunc = a.somecalc newfunc = function() returns return this.val1-this.val2 end # and set this in 'a' - it should also change in 'b' a.somecalc=newfunc assertInt(-1,a.somecalc()) assertInt(-2,b.somecalc()) # now change 'b' back to the old calc; 'a' should not change b.somecalc=oldfunc assertInt(-1,a.somecalc()) assertInt(8,b.somecalc()) # all done delete(a) delete(b)
Quick list of features:
- Fairly powerful language – repeat/until, while/endwhile, break and continue, if/elseif/else/endif, and (I’m afraid) goto.
- Reasonably rapid – code is compiled into bytecode before execution.
- Fully interactive – each line of input is compiled to bytecode and run immediately, unless you’re compiling a function expression.
- Functions are first class objects, so you can create functions inside functions, assign them to variables and properties and so on. No closures yet, because that requires garbage collection – but I can do that with simple refcounting for this one case, so maybe non-local variables and closures will appear.
- No garbage collection – I could add it, but it would be missing the point – the language needs to be fast. I’ve lived without GC in C++ for a good few years, I can cope without it in this.
- Strings, however, are tidied up properly.
- Because functions are first class objects, functions are just values and can be stored anywhere else values are stored – inside Lana objects, inside your own objects (which can also be Lana objects) etc.
- Source code is not stored, but can be regenerated at any point from the bytecode, including comments. A lot of work has gone into making this work correctly, and it’s a big focus of the unit tests.
- Object properties (members and methods) are stored in fast hashes, based on the Python hash table algorithm.
- It’s easy to register new native C++ functions, and native C++ methods for Lana objects.
- Writing a power consumption applet
- Using Bonobo (sorry)
- Very quick intro to Python
- Reading the battery info from a pseudofile interface
- Writing a function to calculate the power
So that I can see the power consumption of my shiny new netbook second by second, I’m going to write a Gnome panel applet which will display it constantly.
Now, this may not work in Unity, the new desktop environment for Ubuntu, but I’m not using it (yet) and a lot of people aren’t happy with it, so I’m ignoring it.
First, a confession. This applet uses Bonobo, which is deprecated. It’s also, it seems, currently the only way to write Python applets using standard repository packages, until the new GObject Introspection API is stable. When that happens, I’ll write a post porting this code.
Another alternative might be a rewrite in C, using this version as a prototype – and that’s something I’m seriously considering.
I’m going to use Python – it’s a good language for doing this sort of task, with lots of helpful text processing stuff and not too many barriers to getting something up quickly. For me, Python sits neatly between a scripting language and a software engineer’s language. If you don’t know Python, hopefully you’ll at least be able to follow along, but I’ll get you started by quickly introducing the oddest thing about the language.
Python doesn’t use curly brackets like Java and C to delineate blocks of code. Instead, it uses a colon on the line before, and an increase in indentation. When the block is finished, the indent decreases again. For example:
def foo(a,b): if a>b: print "A is greater than B" print "so I'll call bar()" bar(a,b) else: print "Other way round, so I'll call baz()" baz(a,b) def anotherfunction(): ...
You see that there’s no keyword or symbol saying we’re at the end of an “if” statement, or even at the end of a function definition – it’s all in the indenting.
First we need to write the code to get the power drain in watts, and just print it on the console. We’ll write this as a function called
getpower() which returns a floating point value.
To do this, we’re going to need to look at the battery state, which is in a special file called
This isn’t actually a file – it’s a pseudofile. It looks like a file, but reading it runs some special code which outputs some text describing the state of the battery. This sort of thing happens a lot in UNIX-style operating systems – the pattern is called “Everything is a File.” If you look around, you’ll find pseudofiles which represent devices (in
/dev), files which represent processes (in the top level of
/proc) and many files which represent system state (such as our battery file.)
Note: Your machine may use a different name for the battery – please check. If I were writing this properly, I’d probably make the code find the battery file itself by scanning the directory, and I’d have to make it deal with multiple batteries.
The contents of this fake file look something like this:
present: yes capacity state: ok charging state: discharging present rate: 773 mA remaining capacity: 3145 mAh present voltage: 12067 mV
We’re going to need to
- extract the “present rate” and “present voltage” values
- convert them from milliamps and millivolts into amps and volts
- and multiply them together to give the current power drain in watts
Luckily this is very easy in Python!
We start by opening the battery state file for reading:
def getpower(): f = open('/proc/acpi/battery/BAT0/state')
If you’re not familiar with Python,
def is used to define a new function, which in this case takes no parameter. Blocks of code are marked by a colon, a new line, and a change of indentation – this is the weirdest thing about Python to a (say) Java or C programmer.
Now we scan the file line by line, using the
foreach construct, which repeats a block of code for each element of a list, and the file method
readlines() which returns a list of lines in a file:
for x in f.readlines():
First we see if the current line has “present rate” in it. If it has, we split the line into two strings – the part before the colon, and the part after. Here we make use of the handy ability of Python to return lists of things, making it look like we’re returning two values:
if(x.find("present rate")>=0): dummy,data = x.split(':')
We throw away the string before the colon, and just use
data. This string looks something like this: ” 654 mA”
We need to remove the leading space, and split the “654 mA” by the space in the middle, giving us just the “654″ part (or whatever the amperage actually is.) We can do this by chaining a couple of methods together:
ma,dummy = data.strip().split()
strip() returns a string without leading or trailing spaces, and the
split() will split that string into two using the whitespace, leaving “654″ in
ma and the “mA” in
dummy. Finally, we need to turn this string into a floating point number:
amps = float(ma) / 1000.0
We now have the current in amps, as a floating point value.
We do exactly the same thing to get the volts:
if x.find("present voltage")>=0: dummy,data = x.split(':') mv,dummy = data.strip().split() volts = float(mv) / 1000.0
We scan the whole file, doing these two checks on every line. Finally, we get to the end of the loop. We can now close the file, and return our power drain – calculated by multiplying the current and the voltage:
f.close() return amps*volts
And that’s that. If we put this altogether in a test program, we can print out the current:
#!/bin/python def getpower(): f = open('/proc/acpi/battery/BAT0/state') for x in f.readlines(): if(x.find("present rate")>=0): dummy,data = x.split(':') ma,dummy = data.strip().split() amps = float(ma) / 1000.0 if x.find("present voltage")>=0: dummy,data = x.split(':') mv,dummy = data.strip().split() volts = float(mv) / 1000.0 f.close() return amps*volts print getpower()
That function is the core of our applet, which we can now build around it.
Next, we’ll start looking at building the applet itself.