Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > [ANN] RubyMacros 0.1.0 Released

Reply
Thread Tools

[ANN] RubyMacros 0.1.0 Released

 
 
Caleb Clausen
Guest
Posts: n/a
 
      10-24-2008
RubyMacros is a lisp-like macro pre-processor for Ruby. More than just a
purely textual substitution scheme, RubyMacros can manipulate and morph
Ruby parse trees (in the form of RedParse Nodes) at parse time in just about
any way you see fit.

Macros are programmed in ruby itself. And since parse trees are represented
in RedParse format, they're easier to use (programatically) and more object-
oriented than other available ruby parsetree formats. (RedParse Node format
is actually designed to be straightforward to use and to represent the
structure of ruby source code very closely.)

== Benefits:
* Powerful and easy metaprogramming
* Create better DSLs
* Manipulate syntax trees to suit yourself
* Access local variables and other caller context unavailable to methods
* Macros as inline methods: should be slightly faster than equivalent methods

== Drawbacks:
Although in theory already as powerful as lisp macros, the current
implementation has a number of problems which added together make it merely
a proof of concept or toy at this point:
* pre-processing is very, very slow (because of RedParse)
* macro calls must be inside some sort of method;
* straight out macro calls at the top level won't work
* macros can't have blocks or receivers
* some ruby syntax is unsupported in files using macros
* files using macros must be loaded via Macro.require;
* Kernel#require will not recognize macros
* RedParse Node tree format will be changing slightly
* macros cannot be scoped
* no variable (or other) hygiene

== Requirements:
RubyMacros requires RedParse.

== Install:
gem install rubymacros

== Examples:
macro simple(a,b)
^a+^b)
end
def simple_user
p simple(1,2) #prints 3
end

#loop as a macro, should be a bit faster than the #loop method
macro loop(body)
while true
^body
end
)
end

#for more examples, see the examples/ directory

== New Syntax:
I have invented 3 new syntactical constructions in order to allow reasonably
easy to use macros. Macros themselves look just like methods except that
'macro' instead of 'def' is used to start the macro definition off. A form
literal is an expression surrounded by '' and ')'. The form escape operator
is '^'. '^' is a unary operator of fairly high precedence.

== Forms and Form Escapes:
Forms are an essential adjunct to macros. Forms represent quoted source
code, which has been parsed but not evaled yet. When a form literal is
executed, it returns a RedParse::Node representing the parse tree for the
enclosed source code. Within a form literal, a ^, used as a unary operator,
will escape the expression it controls, so that instead of being part of the
form's data, it is executed at the same time as the form literal, and the
result of an escaped expression (which should be a Node) is interpolated
into the form at that point. The whole effect is much like that of string
interpolations (#{}) inside string literals.

== How Macros Work
Typically, macros return a single form literal, which contains form escape
expressions within it which make use of the macro's parameters. However,
macro bodies may contain anything at all; more complicated macros will
likely not contain any forms. (Likewise, form literals may be used outside
macros, but the utility of doing so may be minimal.)

At parse time (well, really at method definition time, but in effect it's
much the same thing) method bodies are scanned for callsites which have the
names of known macros. When such a call is found, it is expanded as follows.
The parsetrees for the arguments to the callsite are passed as arguments to
the macro. The macro is expected to return a parsetree, which replaces the
macro callsite in the parsetree which contained it.

== License:
LGPL

 
Reply With Quote
 
 
 
 
Pit Capitain
Guest
Posts: n/a
 
      10-25-2008
2008/10/24 Caleb Clausen <(E-Mail Removed)>:
> RubyMacros is a lisp-like macro pre-processor for Ruby.


Sounds interesting. When will you release some code to look at?

Regards,
Pit

 
Reply With Quote
 
 
 
 
toomln
Guest
Posts: n/a
 
      10-25-2008
> When will you release some code to look at?

I was close to asking the very same question. But then I saw "gem
install", so it's available as gem.

Anyway, how does this approach compare to:

http://blog.drewolson.org/2008/06/ru...xperiment.html
http://weblog.raganwald.com/2008/06/...e-in-ruby.html

May I humbly ask, how do I run the examples? I tried calling running
the example/*_wrap.rb scripts with ruby18 and ruby19 but only got
Syntax errors:

ruby 1.8.7 (2008-08-11 patchlevel 72) [i386-cygwin]
ruby 1.9.0 (2008-08-26 revision 18849) [i386-cygwin]
 
Reply With Quote
 
Caleb Clausen
Guest
Posts: n/a
 
      10-25-2008
On 10/25/08, toomln <(E-Mail Removed)> wrote:
>> When will you release some code to look at?

>
> I was close to asking the very same question. But then I saw "gem
> install", so it's available as gem.


Sorry, I didn't include any urls since I didn't have the rubyforge
project yet. Maybe I should have posted one anyway... but now the
project has been approved, and the code is up there, so try these:

http://rubymacros.rubyforge.org/README.txt
http://rubyforge.org/projects/rubymacros

>
> Anyway, how does this approach compare to:
>
> http://blog.drewolson.org/2008/06/ru...xperiment.html


I have to confess, I haven't looked into this project very much. This
one is using strings to represent sexps... you can do a lot with that,
but eventually you'll run into a glass ceiling of things you want to
find in those strings that just can't be parsed readily with regexps.
(My 'with' macro, (in the examples/ dir) is an example of the kind of
thing that would be impossible.) The macros themselves require a
distressingly large number of calls to eval.

> http://weblog.raganwald.com/2008/06/...e-in-ruby.html


This is the best attempt I've seen to make something macro-like using
ParseTree's syntax tree format. I'm amazed at Reg's ability to
persuade ruby to do feats of gymnastics that I wouldn't have thought
possible... but the result here is still slightly to very clunky,
depending on which abstraction you're using.

Here's the shorter of his two of his attempts to write an andand macro
that operates like the && operator:

called_by_name(ur_and) { |x,y|
if temp = x
y
else
temp
end
}

Not so bad in itself, but all called_by_name's must be wrapped in a
'with', which must get tiresome.

The longer one is too horrible to contemplate. The equivalent in
RubyMacros is something like this:

macro andand(a,b)
if temp = ^a
^b
else
temp
end
)
#or maybe just ^a && ^b )
end

If you want to write a sexp processor, rewrite can help you do that,
and it should be possible to do things even macros can't that way. But
it's very hard to use.

This called_by_name looks considerably easier to use, but I think Reg
says somewhere that it's only useful for a subset of what macros do...

> May I humbly ask, how do I run the examples? I tried calling running
> the example/*_wrap.rb scripts with ruby18 and ruby19 but only got
> Syntax errors:


Aaaaaag! Don't tell me I published broken code.... I did! Ok, I just
fixed it, but gems hasn't found the new version yet, hopefully that
won't take long, but til then, here's a couple of direct links to the
fixed version:
http://rubyforge.org/frs/download.ph...cros-0.1.1.tgz
http://rubyforge.org/frs/download.ph...cros-0.1.1.gem

(This is all because of a misguided attempt to keep rdoc from
crashing... I give up on rdoc for the time being.)

You were correct to try invoking the *_wrap scripts. "ruby -rubygems
example/simple_wrap.rb" should do the trick.

 
Reply With Quote
 
Reg Braithwaite
Guest
Posts: n/a
 
      10-26-2008
Caleb Clausen wrote:

> Here's the shorter of his two of his attempts to write an andand macro
> that operates like the && operator:
>
> called_by_name(ur_and) { |x,y|
> if temp = x
> y
> else
> temp
> end
> }
>
> Not so bad in itself, but all called_by_name's must be wrapped in a
> 'with', which must get tiresome.


The use of "with" is a deliberate design choice. Rather than making
macros global and "automagical," you state what you are using and where
you are using it. This is somewhat akin to writing 'require
such-and-such' in each ruby source file.

Of course, some people like magic, and if you look at Rails, the
initializers and environment.rb file allow you to sprinkle magic
throughout your project implicitly. My feeling when I designed rewrite
was that that if I started with explicit "with," it would easy to build
implicit into a project or framework later.

> The longer one is too horrible to contemplate.


> Speak for yourself


> The equivalent in RubyMacros is something like this:
>
> macro andand(a,b)
> if temp = ^a
> ^b
> else
> temp
> end
> )
> #or maybe just ^a && ^b )
> end


^a && ^b ) is a little too metacircular for my taste, but I put it
to you that rewrite allows you to define your own syntactic replacement
using && if you want to. Now to get more specific. called_by_name is
actually not a way of doing macros, it's a way of writing functions with
call by name semantics.

Rewrite actually provides a facility for code rewriting, which is one
level *above* simple unhygienic macros. A traditional unhygienic macro
is a way of saying "when you see something that looks like a method
call, replace it with the following code, performing substitutions here
and here and here." Rewrite supports this as well as a number of other
arbitrary rewriting rules.

For example, you can say "when you see foo.select { ...blah... }.map {
...blah-blah }, replace it with a single call to .each that performs the
selction and mapping with out iterating over the collection twice.

Now, called_by_name is actually a macro written using rewrite. So it's a
meta-macro. I would say that gievn your example, the macro is better
because it does not "compile" into a function call, whereas anything
built with called_by_name will be rewritten as a function call. If what
you want is the fastest, tightest code, use a ruby macro or use rewrite
to directly rewrite the function call as an if statement.

If you want to compare rewrite and ruby macros more directly, there's a
little thing I wrote called Unhygienic. It does code rewriting "by
example." Now, I use the term "andand" to refer to
http://andand.rubyforge.org/, so here is how to write part of that gem
using Rewriting by example:

Unhygienic.from(:receiver, :message, [arameters]) {
receiver.andand.message(parameters)
}.to {
lambda { |andand_temp|
andand_temp.message(parameters) if andand_temp
}.call(receiver)
}

By the way, I use lambdas a lot in my rewrites to try to alleviate the
pain of Ruby's scoping rules. If you like to live a little more
dangerously (and the example above does), this rule can be made shorter:

Unhygienic.from(:receiver, :message, [arameters]) {
receiver.andand.message(parameters)
}.to {
temp.message(parameters) if (temp = receiver)
}

Both examples are longer than the macro or called_by_name definitions.
The "from" says "here is a snippet of code where receiver, message, and
parameters are placehilders for an expression, and expression, and a
list of expressions. The "to" says "when you find that, replace it with
this, plugging in the placeholders."

The idea here is that you can use any arbitrary ruby expression, not
just something that looks like a function call. In this case, you are
making something that looks like a method call expand into something
else entirely.

My motivation with Rewrite was very specific: I was trying to show that
we have alternatives to wide scale opening of core classes to implement
DSLs and syntactic abstractions. This goal necessarily encompassed
providing an alternative to existing idioms like #andand or
Symbol#to_proc or #try. The goal of Ruby macros seems to be a little
different, and thus the two gems work in different ways.

Overall, I wish Ruby macros every success and hope that people get
excited about syntactic abstractions.
--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
ara.t.howard
Guest
Posts: n/a
 
      10-26-2008

On Oct 24, 2008, at 2:23 PM, Caleb Clausen wrote:

> == Benefits:
> * Powerful and easy metaprogramming
> * Create better DSLs
> * Manipulate syntax trees to suit yourself
> * Access local variables and other caller context unavailable to
> methods
> * Macros as inline methods: should be slightly faster than
> equivalent methods



only manipulating the syntax tree seems like a real advantage from
here, generating a dsl is already as nearly painless as it could be in
ruby, and methods do in fact have access to local variables and other
caller context

cfp:~ > cat a.rb
def context &block
eval 'a += 40', block
block.call
end

a = 2
context{ p a }
p a



cfp:~ > ruby a.rb
42
42


can you show us something that cannot be done using ruby currently,
which macros make possible?


cheers.


a @ http://codeforpeople.com/
--
we can deny everything, except that we have the possibility of being
better. simply reflect on that.
h.h. the 14th dalai lama




 
Reply With Quote
 
Caleb Clausen
Guest
Posts: n/a
 
      11-02-2008
On 10/26/08, ara.t.howard <(E-Mail Removed)> wrote:
>
> On Oct 24, 2008, at 2:23 PM, Caleb Clausen wrote:
>
>> == Benefits:
>> * Powerful and easy metaprogramming
>> * Create better DSLs
>> * Manipulate syntax trees to suit yourself
>> * Access local variables and other caller context unavailable to
>> methods
>> * Macros as inline methods: should be slightly faster than
>> equivalent methods

>
>
> only manipulating the syntax tree seems like a real advantage from
> here, generating a dsl is already as nearly painless as it could be in
> ruby, and methods do in fact have access to local variables and other
> caller context


Sorry for the late reply... I thought this thread had died.

You seem to have left out inline methods, for whatever they're worth.

Yes, DSLs in ruby are currently very easy to create and use; I
anticipate that macros will make DSLs slightly harder to write, but
easier to use. Ruby DSLs tend to be very natural looking to users, but
there are often small compromises to usability that won't make much
sense to domain users, such as the need to begin some words with a
colon and the need to use 'do' at certain places in the language for
no apparent reason. If you use macros to define your DSL, it should be
possible to rid oneself of those features. The situation is already
pretty good, but macros will make it slightly better. I say "will",
because, to be honest, at the moment most of the sugary convenience
features needed for nicer DSLs are not present.

As an example of a DSL (perhaps the wrong word in this case...) that
could be written in macros, there is iterate for common lisp:
http://common-lisp.net/project/iterate/doc/index.html
Iterate is a looping mini-language with special syntax for many common
looping tasks. It looks kind of like what list comprehensions do for
you in python, but more powerful. Now I imagine that something like
this could be written entirely with methods.... but it would be too
slow. (I admit, tho, that I don't understand iterate -- or lisp in
general -- very well. If someone out there wants to correct my
misapprehensions, please feel free.)

> cfp:~ > cat a.rb
> def context &block
> eval 'a += 40', block
> block.call
> end


This is a slick way of getting to your caller's lvars. I would have
passed in a Binding myself, but this way is probably a little cleaner.

But, the caller must pass a block (or binding) in order to make this
work. Sometimes, that's not a problem. Sometimes it is. For instance
(I've run into this) if you want to create an api that works exactly
like Regexp#match, you'll find that it can't be done. #match sets its
caller's $~ (a local variable); methods can't do that. In the past,
I've passed in an optional Binding to handle this case, but
practically speaking, it was a little too clumsy. Using a block
instead is a better idea, but you're still changing the interface used
by your custom #match. The whole point is to re-use your user's
existing knowledge about #match.... if he has to remember, "oh yeah,
and if you use $~ or other matching variables, you have to pass an
extra block to #match", then that's not an effective re-use of
existing knowledge; it might as well be a new interface.

> can you show us something that cannot be done using ruby currently,
> which macros make possible?


A recently requested new feature on ruby-core was __DIR__, which acts
like __FILE__, but returns the directory containing the current source
file. As a macro, that is:

macro __DIR__
File.dirname __FILE__ )
end

Now maybe (now that I've seen your block-as-binding trick) you can
actually write this as a method, something like,

def __DIR__(&ctx)
File.dirname(eval("__FILE__",ctx))
end

I have no ruby ATM, and can't check if that works or not, sorry. But
if it does, it will have to be called like __DIR__{}, instead of
__DIR__. I'd find that a little jarring.

Another recent request was a 'with' keyword, which operates like
instance_eval, but only changes the default receiver for code in the
block passed in, and not self as seen by instance variables. I have an
implementation of this as well (in the example directory of
RubyMacros), but for various reasons I'm unsatisfied with it right
now, so I'd rather not post it.

I'm not claiming that either of these macros is actually a good idea;
I'm just trying to illustrate the possible.

It's likely that quite a few of the features for ruby that get
requested could actually be implemented by macros. It's probably
appropriate that most of these requests are rejected; we don't really
need a lot of global changes to the language. However, if users can
write their own macros to scratch some of these itches, that's a
better solution. They get the feature they want in just the program
that needs it, and the rest of us get a stable, predictable language
without a lot of weird new features in all other ruby programs.

 
Reply With Quote
 
Trans
Guest
Posts: n/a
 
      11-03-2008
On Nov 2, 5:40=A0pm, "Caleb Clausen" <(E-Mail Removed)> wrote:

> It's likely that quite a few of the features for ruby that get
> requested could actually be implemented by macros. It's probably
> appropriate that most of these requests are rejected; we don't really
> need a lot of global changes to the language. However, if users can
> write their own macros to scratch some of these itches, that's a
> better solution. They get the feature they want in just the program
> that needs it, and the rest of us get a stable, predictable language
> without a lot of weird new features in all other ruby programs.


Or we could just write our programs in "Macro-Ruby" and to put a death
nail in so called predictable language.

_._ _._
|||| ||||
||||_ ___ _||||
| || .-'___`-. || |
\ / .' .'_ _'. '. \ /
/~~| | (| b d |) | |~~\
/' | | | ' | | | `\
, /__.-: ,| | `-' | |, :-.__\ ,
|'-------( \-''""/.| /\___/\ |.\""''-/ )------'|
| \_.-'\ / '-._____.-' \ /'-._/ |
|.---------\ /'._| _ .---. =3D=3D=3D |_.'\ /--------.|
' \ / | |\_\ _ \=3Dv=3D/ _ | | \ / '
`. | | \_\_\ ~~~ (_) | | .'
`'"'|`'--.__.^.__.--'`|'"'`
\ /
`,..---'"'---..,'
:--..___..--: TO DSL...
\ /
|`. .'| AND BEYOND!
| :___: |
| | | |
| | | |
|.-.| |.-.|
|`-'| |`-'|
| | | |
/ | | \
|_____| |_____|
':---:-'-:---:'
/ | | \
jgs /.---.| |.---.\
`.____; :____.'


T.

 
Reply With Quote
 
ara.t.howard
Guest
Posts: n/a
 
      11-03-2008

On Nov 2, 2008, at 3:40 PM, Caleb Clausen wrote:
>>

>
> Sorry for the late reply... I thought this thread had died.
>
> You seem to have left out inline methods, for whatever they're worth.


yeah - if they are really a lot faster it's worth considering but, for
now, i'll assume they're not...

>
>
> Yes, DSLs in ruby are currently very easy to create and use; I
> anticipate that macros will make DSLs slightly harder to write, but
> easier to use. Ruby DSLs tend to be very natural looking to users, but
> there are often small compromises to usability that won't make much
> sense to domain users, such as the need to begin some words with a
> colon and the need to use 'do' at certain places in the language for
> no apparent reason. If you use macros to define your DSL, it should be
> possible to rid oneself of those features. The situation is already
> pretty good, but macros will make it slightly better. I say "will",
> because, to be honest, at the moment most of the sugary convenience
> features needed for nicer DSLs are not present.
>
>


that's an interesting point. not sure about it though - if people are
writing dsls that are not ruby support debugging becomes quite
difficult. still, i get it.


>
>> cfp:~ > cat a.rb
>> def context &block
>> eval 'a += 40', block
>> block.call
>> end

>
> This is a slick way of getting to your caller's lvars. I would have
> passed in a Binding myself, but this way is probably a little cleaner.
>
> But, the caller must pass a block (or binding) in order to make this
> work. Sometimes, that's not a problem. Sometimes it is. For instance
> (I've run into this) if you want to create an api that works exactly
> like Regexp#match, you'll find that it can't be done. #match sets its
> caller's $~ (a local variable); methods can't do that. In the past,
> I've passed in an optional Binding to handle this case, but
> practically speaking, it was a little too clumsy. Using a block
> instead is a better idea, but you're still changing the interface used
> by your custom #match. The whole point is to re-use your user's
> existing knowledge about #match.... if he has to remember, "oh yeah,
> and if you use $~ or other matching variables, you have to pass an
> extra block to #match", then that's not an effective re-use of
> existing knowledge; it might as well be a new interface.
>


i wouldn't strictly agree with your analysis, but i do agree that it's
very hard to do so. the key is having a context or marker which
allows the method to be safely used on all objects, this is what tagz
does for html/xml generation to avoid this - basically methods called
on any 'self' have easy access to the caller. for instance

class Object
def LikeRegexp
LikeRegexpObject.new(self)
end
ed

LikeRegexp.match .....


is one workaround. i do clearly see the value of being inside an
object though. however, is the implication that all macros are global?


>> can you show us something that cannot be done using ruby currently,
>> which macros make possible?

>
> A recently requested new feature on ruby-core was __DIR__, which acts
> like __FILE__, but returns the directory containing the current source
> file. As a macro, that is:
>
> macro __DIR__
> File.dirname __FILE__ )
> end
>
> Now maybe (now that I've seen your block-as-binding trick) you can
> actually write this as a method, something like,
>
> def __DIR__(&ctx)
> File.dirname(eval("__FILE__",ctx))
> end
>
> I have no ruby ATM, and can't check if that works or not, sorry. But
> if it does, it will have to be called like __DIR__{}, instead of
> __DIR__. I'd find that a little jarring.
>


def __DIR__
filename = caller[0][/^(.*):/, 1]
File.expand_path(File.dirname(filename))
end


stolen wholesale from Ramaze (lot's of good stuff in there )


> Another recent request was a 'with' keyword, which operates like
> instance_eval, but only changes the default receiver for code in the
> block passed in, and not self as seen by instance variables. I have an
> implementation of this as well (in the example directory of
> RubyMacros), but for various reasons I'm unsatisfied with it right
> now, so I'd rather not post it.
>



def with &block
scope = Scope.new

instance_variables.each do |ivar|
scope.instance_variable_set ivar, instance_variable_get(ivar)
end

scope.instance_eval &block
end

hacky? yes. but it works well enough for ActionView...



> I'm not claiming that either of these macros is actually a good idea;
> I'm just trying to illustrate the possible.
>
> It's likely that quite a few of the features for ruby that get
> requested could actually be implemented by macros. It's probably
> appropriate that most of these requests are rejected; we don't really
> need a lot of global changes to the language. However, if users can
> write their own macros to scratch some of these itches, that's a
> better solution. They get the feature they want in just the program
> that needs it, and the rest of us get a stable, predictable language
> without a lot of weird new features in all other ruby programs.
>



well now that's something everyone can agree on! seriously, the
project looks super interesting - just trying to think of a real use
case.

hrrrrm. could we possibly use it to skin the

self.ivar = value

problem?


and, to repeat from above, are the global? i'm hoping macros can be
scoped to an object like instance methods....


cheers.


a @ http://codeforpeople.com/
--
we can deny everything, except that we have the possibility of being
better. simply reflect on that.
h.h. the 14th dalai lama




 
Reply With Quote
 
Pit Capitain
Guest
Posts: n/a
 
      11-03-2008
2008/11/3 ara.t.howard <(E-Mail Removed)>:
> hrrrrm. could we possibly use it to skin the
>
> self.ivar = value
>
> problem?


Of course:

irb(main):001:0> require "nolocal-spike"
=> true

irb(main):002:0> class C
irb(main):003:1> attr_accessor :a, :b, :c
irb(main):004:1> def initialize &blk
irb(main):005:2> instance_eval( &nolocal( &blk ) )
irb(main):006:2> end
irb(main):007:1> end
=> nil

irb(main):008:0> C.new do
irb(main):009:1* a = b = 5
irb(main):010:1> c = a + 2
irb(main):011:1> end
=> #<C:0x2fce8e0 @c=7, @b=5, @a=5>

Two years ago I was experimenting with modifying the AST using plain
Ruby. (On Windows, I couldn't use parsetree for example back then.)
The code above is one of the results. The same should be easy with the
actual libraries.

Regards,
Pit

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[ANN] rubymacros 0.1.5 Released Caleb Clausen Ruby 0 07-07-2009 04:15 PM
[ANN] RubyMacros 0.1.4 Released Caleb Clausen Ruby 0 05-22-2009 04:36 PM
[ANN] rubymacros 0.1.2 Released Caleb Clausen Ruby 0 05-01-2009 05:05 PM
Battlefield 2 New Patch Released (yet Again...) Silverstrand Front Page News 4 07-20-2005 04:33 PM
ANN: Anyplace Control 2.6 released Yura Goncharuk Wireless Networking 0 09-28-2004 10:13 AM



Advertisments