Having an actual day job (Wii programming) has meant that work on Lana has had to take a back seat. Combining this with the fact that I’m now doing the tricky stuff, development has slowed down somewhat. But here’s what’s new:

Garbage collection

Notwithstanding what I said in an earlier post, Lana now has garbage collection! This works for strings, objects, dictionaries, everything. I’ve used the same techniques Python uses:

• Reference counting for all managed data – whenever a managed entity (such as a string or object) is referred to in a value, a new entity is created, or a reference to an entity is copied into a new value, its reference count is increased; when the new reference goes away the reference count is decreased; and when the count is zero the entity is destroyed. This is made rather more complex by there being two distinct allocation models – SimpleMalloc, used for strings and the like and malloc()/free() are used; and Complex/SimpleNew, where the entity is some subclass of GarbageCollected – and in the Complex case can contain references to other entities.
• This idea of referenced entities also containing references means that plain reference counting isn’t enough – entities which refer to each other in a loop will never be deleted. Therefore we also have a cycle detector which uses a list of all the reference counted entities which can refer to other entities (so called Complex entities.) It can detect entities in a cycle which are not referred to from outside the cycle, and delete them. This has to be run periodically, and we currently do it with the Lana gc() function.

Dictionaries

These are containers which can be keyed on, and can contain, any type of value:

These proved somewhat hairy – possibly the hairiest thing in Lana. That’s because of the way variables work. When a variable is encountered in the code, what actually happens is that the virtual machine stacks a Ref – a reference to the variable. If a subsequent instruction wants to read the variable, it just calls the internal popval() method, which will pop the Ref from the stack and dereference it, returning the variable’s contents to the instruction code. If we want to set the value – and only the code for the OP_SET instruction does this – we pop the reference without dereferencing it and write to the variable it refers to.

Exactly the same technique is used for object properties – these are represented by PropRef values, which contain a pointer to the object and the ID of the property.

Naturally I wanted to do the same thing with dictionaries – have a type of value (a “DictRef”) which refers to a dictionary entry. To do this, I need to have the value contain both the key and a reference to the dictionary. However, if the key can be any Lana value, I need to be able to create a value type which contains both a dictionary reference and any value type. Nastily recursive, that definition. So I cheated, and use a special “packed value” in which the type field holds the hash key type and the allocation type flag (SimpleMalloc, etc.) is set to a special DictRefAlloc type, half the value field holds the key itself (which is the “primary part” of the key value,) and the other half holds the dictionary pointer. Nasty, and it means that values with both secondary and primary parts – PropRefs, DictRefs themselves, and some of the advanced types I may create later like delegates and closures – can’t serve as keys. It’s a small price.

Next up

• Iterators – these are in the C++ code but there’s no Lana for statement. Also, I’ve just realised that I need to iterate over both keys and values, which opens a small can of worms.
• Array lists – again, the basic object exists in the C++ code but isn’t linked into Lana’s syntax.
• User iterables – i.e. objects the user creates in C++ (or Lana I suppose) could have the iterable interface, allowing for to work with them.
• Constructors.
• Optimisation of common instruction sequences such as OP_THIS OP_PROPREF.
• Serialisation – this is a big one, and comes close to the “point” of the language.
• Delegates in the C# sense – values which contain both a method reference and the object to call it on, as a single value: very useful for event-driven code!
• Closures – I have some notes on how to do this, by using a value to hold both the function pointer and the environment in which it was created (or a subset thereof.)

I called it Lana, some wag called it Jimscript. Anyway, as those who know me will know by now, my projects tend to expand until I need to embed a scripting language in them. And my current project is no exception. Previously, however, I’ve always just bolted in some kind of Forthlike, and hang the readability. This time, I wanted to do it properly – either embed Lua or Python, or do a proper language.

What I wanted this time is a language that would allow me to “decorate” my projects’ data structures with tiny fragments of arbitrary code, and for the code to be fully integrated with my data structures. Lua and Python’s integration – particularly with C++ – is still pretty revolting, so once again I had an excuse to do it myself. And earlier today, Lana ran (after a whole bunch of other unit tests) the following program:

Quick list of features:

• Fairly powerful language – repeat/until, while/endwhile, break and continue, if/elseif/else/endif, and (I’m afraid) goto.
• Reasonably rapid – code is compiled into bytecode before execution.
• Fully interactive – each line of input is compiled to bytecode and run immediately, unless you’re compiling a function expression.
• Functions are first class objects, so you can create functions inside functions, assign them to variables and properties and so on. No closures yet, because that requires garbage collection – but I can do that with simple refcounting for this one case, so maybe non-local variables and closures will appear.
• No garbage collection – I could add it, but it would be missing the point – the language needs to be fast. I’ve lived without GC in C++ for a good few years, I can cope without it in this.
• Strings, however, are tidied up properly.
• Because functions are first class objects, functions are just values and can be stored anywhere else values are stored – inside Lana objects, inside your own objects (which can also be Lana objects) etc.
• Source code is not stored, but can be regenerated at any point from the bytecode, including comments. A lot of work has gone into making this work correctly, and it’s a big focus of the unit tests.
• Object properties (members and methods) are stored in fast hashes, based on the Python hash table algorithm.
• It’s easy to register new native C++ functions, and native C++ methods for Lana objects.