Yeah, it's that time again!

As usual lots and lots of improvements in this new release including:

• complete nullable type support
  
• special notation for nullable types (T?) and enumerables (T*)
  
• improved booish behavior with nicer colors (it should work inside emacs now :))
• 'else' block for 'for' and 'while loops
• generics, generics, generics
• and generics!

With many many kudos to Avishay Lavie, Cédric Vivier and Marcus Griep!

Cédric deserves special mention for working on all fronts and really pushing for the release. Superb work!

What? - http://boo.codehaus.org/

Download - http://boo.codehaus.org/Download

Official irc channel - irc://irc.codehaus.org/boo

Have fun!

Almost four years ago the first feature request was entered into the boo issue tracker.

There should be a way to extend the parser so it could recognize custom measurement unit literals such as 1kg and 2cm.

Since then boo has improved a lot but with no solution for BOO-1 in the horizon.

I think I'm getting close to solve it in a interesting way using PEGs implemented as a graph of expression objects.

PEGs are very likable. Conceptually simple and composable.

Take the PEG that recognizes integer expressions involving the + and * operators:

grammar <- spaces addition eof
addition <- term ("+" spaces term)*
term <- factor ("*" spaces factor)*
factor <- [0-9]+ spaces
spaces <- (' ' / '\t')*
eof <- !.

where:

() means grouping.

* means zero or more matches.

+ means one or more matches.

/ is the prioritized choice operator. If the first expression succeeds, the whole expression succeeds. If the first expression fails, it backtracks and evaluates the second expression.

! is the not predicate operator which succeeds if its operand fails. It never consumes any input.

. matches any input.

The grammar can be translated to boo very simply using the peg macro from Boo.Pegs:

import Boo.Pegs

peg:
    grammar = spaces, addition, eof
    addition = term, --("+", spaces, term)
    term = factor, --("*", spaces, factor)
    factor = ++[0-9], spaces
    spaces = --(' ' / '\t')
    eof = not any()
    
assert grammar.Match(PegContext("  6*6 + 6 "))

I had to be a little creative in mapping the PEG operators to valid boo expressions because as it must be clear by now boo doesn't allow the introduction of completely new syntax and that's what the fuss is all about here.

I actually like the way it looks.

Implementing something more useful such as expression evaluation on top of that requires a few semantic actions operating a stack:

import Boo.Pegs
import System.Collections.Generic

stack = Stack[of int]()
push = stack.Push
pop = stack.Pop

peg:
    grammar = spaces, addition, eof
    addition = term, --("+", spaces, term, { push(pop() + pop()) })
    term = factor, --("*", spaces, factor, { push(pop() * pop()) })
    factor = ++[0-9], { push(int.Parse($text)) }, spaces
    spaces = --(' ' / '\t')
    eof = not any()
    
assert grammar.Match(PegContext("  6*6 + 6 "))
assert 42 == pop()

Semantic actions are just closures that get executed as matching succeeds. $text returns the text matched so far by the current rule.

Beautiful.

The underlying implementation based on a graph of expression objects really shines when one considers what it takes to extend the grammar above with support for hexadecimal literals:

peg:
    // rebind
    factor.Expression = hex_number / factor.Expression
    hex_number = "0x", ++hex_digit, { push(int.Parse($text[2:], NumberStyles.HexNumber)) }, spaces
    hex_digit = [0-9, a-f, A-F]
assert grammar.Match(PegContext("  0xa*2 + 11*0x02"))
assert 42 == pop()

It's not yet clear how this extensibility mechanism will be exposed at the boo language level but the simplicity at the peg level is encouraging.

One last feature worth pointing out is the ability to match based on a previously matched rule. For instance, the closing tag of a xml element must match the name in the starting tag:

import Boo.Pegs

peg:
    element = '<', tag, '>', content, '', @tag, '>'
    tag = ++(a-z)
    content = --(element / text)
    text = not "<", any()
    
assert element.Match(PegContext("<foo><bar>Hello</bar></foo>"))

I've found the idea for the last match operator @ first described in this article. Great idea.

I've been also greatly inspired by conversations I've had with Massi who's exploring similar territory and Jb during the last Mono Meeting in Barcelona Madrid and with Cedric over a beer in Paris. I think Massi will be pleased to know that I haven't given any thoughts to performance leaving all the fun to him.

Extensible parsing. Soon in a boo compiler close to you.

Lots of improvements in this release including a simpler way for writing macros, support for nested functions, a better interactive interpreter, error messages that include suggestions for misspelled names, exception filters, exception fault handlers and for loop IDisposeable.Dispose integration.

With many many kudos to Avishay Lavie, Cédric Vivier, Daniel Grunwald and Marcus Griep!

What? - http://boo.codehaus.org/
Download - http://boo.codehaus.org/Download
Official irc channel - irc://irc.codehaus.org/boo

Have fun!

Gaiaware has just announced the Gaia Programming Contest. A contest "... about creating an Ajax Application that will serve as a meeting place for people dedicated to solving environmental issues...".

Very good but there's more: "... no Close Source dependencies can be used which means that the end product must be compilable on Mono ...". Great!

Of course people using boo and db4o have a huge headstart. So what are you waiting for?

As I went through the conference memories I recovered this one conversation I had with Jim Purbrick after the boo presentation.

His idea would be to have boo as the language for building languages in Second Life.

Niiiiice.

What a great experience.

A chance to interact live with a dear friend. Free Software, Hacking, Women, Futurama, McDonalds, love spreading, Militant Atheism, Monty Python, Douglas Adams and the French Way.

Had lots of interesting exchange of ideas with Massi, ranging from "extensible parsing through composeable PEGs with  optimal performance" to Carlos Castañeda, Jesus Christ, meta-physics, religion and Pink Floyd. The Cryptonomicon really got me.

Got to put a face on Joachim and see how really cool Unity is.

Jeroen IKVM Frijters is a funny guy!

On Thursday I got to talk about db4o which led me to meet a few db4o users hanging around the conference.

Pedro Santos had an interesting question, how to monitor and control the usage of computational resources in a managed client/server application? In other words, how a sysadmin can make sure a specific client won't DOS the application?

For .NET servers running on Windows there are performance counters, what about Mono servers running on Linux?

I've also got to spread the gospel about boo for which I got a hugely positive response.

Thomas "Gaia" Hansen seemed to really get it and so we had lots of interesting discussions on how to take over the world boo style.

Jackson wants to hack on better nullable type support for boo!

Mark wants an extensible language where NullReferenceExceptions are impossible.

Miguel reassured me once again mcs won't be rewritten on top of the boo compiler infrastructure :)

The presentation material is here.

Looking forward to the next one.

So it's official now, I'll be speaking at the Mono Summit 2007.

It will be great to see you there!

Many thanks to the great folks at db4o for sponsoring my trip.

Oren talks about a simple but interesting macro to aid with mocking. I decided to see if and how the latest meta programming facilities I've been working on are actually useful. Here's the complete application, what do you think?

    namespace Adapter

    import Boo.Lang.Compiler
    import Boo.Lang.Compiler.Ast
    import Boo.Lang.Compiler.Ast.Visitors
    import Boo.Lang.Compiler.TypeSystem
    import Boo.Lang.Compiler.MetaProgramming

    class AdapterMacro(AbstractAstMacro):
        def Expand(macro as MacroStatement):
            if macro.Arguments.Count != 1 or not macro.Arguments[0] isa ReferenceExpression:
                raise "adapter must be called with a single argument"

            entity = NameResolutionService.Resolve(macro.Arguments[0].ToString())
            raise "adapter only accept types" unless entity.EntityType == EntityType.Type
            BuildType(macro, entity)

        def GetModule(node as Node) as Boo.Lang.Compiler.Ast.Module:
            return node.GetAncestor(NodeType.Module)

        def BuildType(macro as MacroStatement, type as IType):
            adapterInterface = [|
                interface $("I" + type.Name):
                    pass
            |]

            adapter = [|
                class $(type.Name + "Adapter")($adapterInterface):

                    theTarget as $(type.FullName)

                    def constructor(target as $(type.FullName)):
                        theTarget = target
            |]

            GetModule(macro).Members.Add(adapter)
            GetModule(macro).Members.Add(adapterInterface)

            for member in type.GetMembers():
                AddMethod(adapter, adapterInterface,  member) if member isa IMethod

            BooPrinterVisitor(System.Console.Out).Visit(adapterInterface)
            BooPrinterVisitor(System.Console.Out).Visit(adapter)

        def AddMethod(adapter as ClassDefinition,
                adapterInterface as InterfaceDefinition,
                method as IMethod):

            if not method.IsPublic: return
            if method.IsStatic: return
            if method.ReturnType.IsByRef: return
            if method.ReturnType.IsArray: return

            interfaceMethod = [|
                def $(method.Name)() as $(method.ReturnType.FullName):
                    pass
            |]

            forwarder = interfaceMethod.CloneNode()

            forwardInvocation = [| theTarget.$(method.Name)() |]
            for param in method.GetParameters():
                if param.IsByRef or param.Type.IsArray:
                    return

                forwarder.Parameters.Add(
                    ParameterDeclaration(
                        Name: param.Name,
                        Type: SimpleTypeReference(param.Type.FullName)))

                interfaceMethod.Parameters.Add(
                    ParameterDeclaration(
                        Name: param.Name,
                        Type: SimpleTypeReference(param.Type.FullName)))

                forwardInvocation.Arguments.Add(ReferenceExpression(param.Name))


            adapterInterface.Members.Add(interfaceMethod)
            adapter.Members.Add(forwarder)

            if method.ReturnType == TypeSystemServices.VoidType:
                forwarder.Body.Add(forwardInvocation)
            else:
                forwarder.Body.Add([| return $forwardInvocation |])


    code = [|
        import Adapter

        adapter int

        print Int32Adapter(42) isa IInt32
    |]

    try:
        module = compile(code, typeof(AdapterMacro).Assembly)
        module.EntryPoint.Invoke(null, (null,))
    except x as CompilationErrorsException:
        print x.Errors.ToString(true)

Today history was made. The first eclipse plugin written in boo and compiled down to java bytecodes by boojay has come to life. The plugin is a direct translation of the the plugin described here.
And the boo code:

    """
    Hello World eclipse plugin.

    A direct translation of
    http://www.eclipse.org/articles/Article-Your%20First%20Plug-in/YourFirstPlugin.html
    to boo.
    """
    namespace HelloWorldPlugin

    import org.eclipse.ui
    import org.eclipse.jface.action
    import org.eclipse.jface.dialogs
    import org.eclipse.jface.viewers

    class HelloWorldAction(IWorkbenchWindowActionDelegate):

        activeWindow as IWorkbenchWindow

        def run(proxyAction as IAction):
            shell = activeWindow.getShell()
            MessageDialog.openInformation(shell, "Hello from boojay!", "Hello World!")

        def init(window as IWorkbenchWindow):
            activeWindow = window

        def dispose():
            pass

        def selectionChanged(proxyAction as IAction, selection as ISelection):
            pass

If you think it makes sense to have a JVM backend for the boo programming language, join us.

This release includes bug fixes, performance improvements and better meta-programming capabilities [1].

Special thanks to Marcus Griep, Nick Fortune and Matt McElheny!

What? - http://boo.codehaus.org/
Download - http://boo.codehaus.org/Download
Official irc channel - irc://irc.codehaus.org/boo


Complete change log here.
Have fun!
[1] see the 'match' and 'data' macros in the boo-extensions project for examples

Consider the following simple application for a moment:

import org.eclipse.swt
import org.eclipse.swt.widgets

display = Display()
shell = Shell(display)
shell.setText("Hello!")
shell.setSize(200, 200)
shell.open()

while not shell.isDisposed():
    if not display.readAndDispatch():
        display.sleep()
        
display.dispose()

A boo application using the SWT java GUI library. Thanks to IKVM that's not only possible but very simple as well.

So what's the news? Well, Friday morning I was chatting with Klaus and he said to me "if you get boo to emit java bytecodes I'll do all my stuff in boo". How's that for a challenge? :)

Thanks again to IKVM, ObjectWeb ASM and the extensible boo pipeline architecture after a weekend of relaxed hacking boojay was born.

UPDATE: Just in case it's not clear, the generated class files DO NOT require IKVM in any way and can be executed in any compliant JVM.

I've finally took some time off this weekend to implement a simple object pattern matching facility as part of the newly created boo-extensions project.

Here's some code using the new 'match' and 'data' macros to implement an expression evaluator:

import Boo.PatternMatching
import Boo.Adt

def eval(e as Expression) as int:
   match e:
      case Const(value):
        return value
      case Add(left, right):
        return eval(left) + eval(right)

def simplify(e as Expression) as Expression:
    match e:
        case Add(left: Const(value: 0), right):
            return simplify(right)
        case Add(left, right: Const(value: 0)):
            return simplify(left)
        case Add(left, right):
            return Add(simplify(left), simplify(right))
        case _:
            return _

data Expression = Const(value as int) | Add(left as Expression, right as Expression)

e = Add(Add(Const(19), Const(0)), Add(Const(0), Const(23)))

print simplify(e)
print eval(e)
print eval(simplify(e))

'match' coupled with our recently acquired quasiquoting capabilities makes writing macros for boo a pleasant endeavour actually.

Dont believe me? Take a look at the data macro again.

This release includes bug fixes, improves on generic support and introduces a few metaprogramming facilities (still on early stage).

Many thanks to the growing boo community!

What? - http://boo.codehaus.org/
Download - http://boo.codehaus.org/Download
Official irc channel - irc://irc.codehaus.org/boo

Full change log here.

Have fun!

Boo meta methods are methods that take code trees as input and return code trees as output. In addition they must be marked with the MetaAttribute for the compiler to recognize them as so.

The compiler invokes meta methods during the type resolution phase and replaces the code tree at the point of invocation with the code tree returned by the meta method.

Multiple overloads can be specified in which case the types of the code tree arguments will be used for the purpose of overload resolution.

An example, assert

Assert can be invoked with one or two arguments:

    assert x is null // use the code as the assertion message
    assert x is not null, "x shouldn't be null" // custom exception or string

One way of implementing assert would be:

    [meta] def assert(condition as Expression):
        return [|
            if not $condition:
                raise AssertionFailedException($(condition.ToCodeString()))
        |]
    
    [meta] def assert(condition as Expression, exception as Expression):
        return [|
            if not $condition:
                raise $exception
        |]

$ is generally called the "splice" operator and it means "evaluate me at compilation time".

Alternatively, assert could declare a variable parameter list:

    [meta] def assert(*arguments as (Expression)):
        condition = arguments[0]
        if len(arguments) > 1:
            // assuming 2 arguments
            exception = arguments[1]
        else:
            exception = [| AssertionFailedException($(condition.ToCodeString())) |]
        return [|
            if not $condition:
                raise $exception
        |] 

An interesting aspect of boo's splicing semantics is exemplified by the subexpression:

    [| AssertionFailedException($(condition.ToCodeString())) |]

Another example, using

'using' provides for deterministic disposal of resources. The argument should implement the IDisposable interface to have its Dispose method called
at the end of a provided code block.

For instance, the following code:

    using socket=OpenConnection():
        socket.Send("ping")

    socket=OpenConnection()
    try:
        socket.Send("ping")
    ensure:
        if socket isa IDisposable: (socket as IDisposable).Dispose()

    [meta] def using(e as Expression, block as BlockExpression):
        temp = uniqueName()
        return [|
            $temp = $e
            try:
                $(block.Body)
            ensure:
                if $temp isa IDisposable: ($temp as IDisposable).Dispose()
        |]

More about quasi-quotation

Due to the reuse of syntactic elements in different contexts the parser needs to follow some conventions in order to infer the meaning of a quasi-quote expression.

Take the quasi-quote [| a as string |]. What does it mean? If we were to interpret it as an expression it would mean a try cast expression. If we were to interpret it as a type member it would mean a field definition.

For the inline form the convention is to try to interpret it as either an expression or an expression pair or an import declaration or a namespace declaration.

The block form is first probed for a type member definition then for a single statement, then for a block of statements.and then for a module.

Example:

    e = [| a as string |]

    assert e isa TryCastExpression

    

    f = [|

        a as string

    |]

    assert f isa Field

    

    m = [|

        namespace Spam

        

        print "Spam! "*3

    |]

    assert m isa Module

It should be possible to specify the exact context of quasi-quotation by using some special syntax but the specific details are not clear at this point.

Soon to reach a svn repository near you.

Oh, yeah, and many thanks to the people behind Template Haskell!

My good friend JB has just let me know it was his idea.

Sorry, JB :)

Georges showed me today how he was using a boo DSL to generate HTML:

   1:html() do:
   2:    body() do:
   3:        text("Hello, world!")

The idea is to use nested closures to represent the hierarchy of tags:

   1:        
   2:callable Block()
   3:
   4:def blockTag(tagName as string, block as Block):
   5:    print "<${tagName}>"
   6:    block()
   7:    print "</${tagName}>"
   8:    
   9:def html(block as Block):
  10:    blockTag("html", block)
  11:    
  12:def body(block as Block):
  13:    blockTag("body", block)
  14:
  15:def text(s as string):
  16:    print s

It works pretty well except that all those do keywords and parenthesis kind of get in the way of clarity for that particular use case.

What we really would like to write is something like:

   1:html:
   2:    body:
   3:        text "Hello, world!"

Which is clearer and more to the point.

Boo actually allows one to do such a thing by extending the language with macros.

Boo macros are objects that are invoked by the compiler during the compilation process to expand or transform the AST in a hopefully useful way.

Whenever the Boo parser finds code in the form: name expression_list (block)? it creates a MacroStatement node that will be later handled to a macro object for expansion.

Let's define our html, body and text macros for our DSL:

   1:import Boo.Lang.Compiler
   2:import Boo.Lang.Compiler.Ast
   3:
   4:class AbstractTagMacro(AbstractAstMacro):
   5:"""
   6:Maps a macro to a DSL method invocation.
   7:"""
   8:    override def Expand(node as MacroStatement):
   9:        invocation = MethodInvocationExpression(Target: ReferenceExpression(TagName))
  10:        invocation.Arguments.Add(CallableBlockExpression(Body: node.Block)) 
  11:        return ExpressionStatement(invocation)
  12:        
  13:    abstract TagName as string:
  14:        get:
  15:            pass
  16:
  17:class HtmlMacro(AbstractTagMacro):
  18:    override TagName:
  19:        get: return "html"
  20:        
  21:class BodyMacro(AbstractTagMacro):
  22:    override TagName:
  23:        get: return "body"
  24:        
  25:class TextMacro(AbstractAstMacro):
  26:    override def Expand(node as MacroStatement):
  27:        invocation = MethodInvocationExpression(Target: ReferenceExpression("text"))
  28:        invocation.Arguments = node.Arguments
  29:        return ExpressionStatement(invocation)

Now we can write code as we wanted to in the first place but there are a few issues with macros:

• writing macros demand a in depth knowledge of the compiler object model (oh, you didn't notice it?)
• macros cannot be used in the same assembly defining them (the compiler needs to instantiate them after all, right?)

Enter BOO-835. It's now possible to use regular method definitions to create DSLs:

   1:callable Block()
   2:
   3:def blockTag(tagName as string, block as Block):
   4:    print "<${tagName}>"
   5:    block()
   6:    print "</${tagName}>"
   7:    
   8:def html(block as Block):
   9:    blockTag "html", block
  10:    
  11:def body(block as Block):
  12:    blockTag "body", block
  13:    
  14:def text(s as string):
  15:    print s
  16:    
  17:html:
  18:    body:
  19:        text "Hello, world!"

The code landed in the repository just a few minutes ago so feel free to try it while it's hot.

Updated: jira issue code was wrong.

Around 4 years ago (!) there was this discussion about how to support some dynamic language features on top of mono. One of the topics was optimizing dynamic dispatching and apparently my suggestion was redirected to nul.

4 years later here I am finally implementing the idea in order to take boo's dynamic dispatching performance to the next level.

Before the optimization:

$ build/booi performance/duckoperators.boo
int*int: 1.101584
list*int: 29.0217312
dynamicDispatch: 51.9484224
staticDispatch 1.4921456

Each line reports how long it takes to execute the described operation with dynamic dispatching 5_000_000 times (except the last line which executes the same operation as the line before it but with static dispatching).

The first line tells us that it takes 1.10 seconds for boo to multiply two integer objects using dynamic dispatching.
The second line says boo takes 29.02 seconds to multiply a List instance by an integer using dynamic dispatching (dynamic dispatch over static methods).
The third line which is the most interesting one for our purposes here says that boo takes roughly 52 seconds to dynamically dispatch 5_000_000 instance method calls.

We can see a huge overhead over static dispatching.

After the optimization:

$ build/booi performance/duckoperators.boo
int*int: 1.101584
list*int: 27.755072
dynamicDispatch: 4.055832
staticDispatch 1.4821312

Niiiiiice.

So this first stab got it from 52 seconds down to 4 seconds. Not bad at all. A few changes and we'll have the same benefits for dynamic dispatching over static methods.

I hope this will have a huge impact on environments that rely heavily on dynamic dispatching such as Brail.

Unfortunately though this optimization is only available when building for the .NET 2.0 profile.

Soon in a source code repository near you.

I've just arrived in Seattle for the Microsoft DLR Compiler Lab. JB will be here soon and I heard Miguel is joining us as well. Fun!

Looking forward to getting my first Boo Silverlight application running.

One of the questions I'm here for is should we migrate Boo's duck typing support to be based on the DLR? On the pros side we get better integration with all the DLR languages and perhaps better performance when executing dynamic code. On the cons side it's an additional dependency (and not a very mature one for that matter).

Thoughts?

I started reviewing the overload resolution code in the boo compiler this week. Very old and, let's say, very interesting code. It was based on a fuzzy scoring system I have no idea how I came up with. Well, these things happen.

The new code is based on the concepts discussed here. Thanks Avish and Daniel for that!

A few test cases (mostly varargs related) had to be reviewed to comply with the improved behavior so expect a few compilation errors when updating to the new code and please let us know of any strange behavior.

Life looks good so let's consider a very simple data structure. A linked list:

   1:import System.Collections
   2:
   3:class LinkedList(IEnumerable):
   4:    
   5:    _head as Node
   6:    
   7:    def constructor(*values):
   8:        for value in values:
   9:            Add(