Some text processing questions

Hello all,

In an effort to read and process non-trivial command files for a
transactor in a testbench, I came upon a couple of questions regarding
text / string processing in VHDL:

1) It seems to be impossible to pass "line" variables into functions,
therefore making it difficult to implement such useful functions as
is_whitespace_only(line) and is_comment(line). For the meantime I made
them accept strings and read the line into a string. Isn't there a way
around this ? How do you prefer to work with "line" type vars ?

2) VHDL's strong typedness makes it difficult to work with strings in
a flexible way. For instance, the following "would be" code doesn't
really work:

var msg := string;
....
....
if (something) then
msg := "hello";
else
msg := "bye";
end if;

Because "msg" can't be declared as an unconstrained string. However,
to give it length would mean to constrain all strings assignable to it
to this length. If I say:

vsr msg := string(1 to 5);

Then I can assign "hello" to it (length 5) but not "bye". This is
quite painful. How can I solve this problem ?

In general, could you post useful code snippets for string / text
processing, and / or point to non-standard libraries you use ? I found
txt_util (http://www.stefanvhdl.com/vhdl/vhdl/txt_util.vhd) pretty
helpful for making some operations less painful.

TIA
Eli

Eli Bendersky

On 30 Mar 2007 07:44:24 -0700, "Eli Bendersky"
<> wrote:

>In an effort to read and process non-trivial command files for a
>transactor in a testbench,

Before I try to answer your question, it's worth noting that many
engineers have simply given up on this; instead they let the
VHDL compiler do the work for them. Your transactor provides
a bunch of useful procedures, and then you create a test case
not in plain text but in procedural VHDL, calling those procedures.
It's usually quite easy to write your set of procedures so
that a succession of calls to them looks almost like a script.
And of course you have the full power of the language for
doing conditionals, loops and what-have-you.

Anyways, after that digression:

> I came upon a couple of questions regarding
>text / string processing in VHDL:

No complete answers, I'm afraid, because text processing in VHDL
somewhat sucks; but there *are* good answers to at least some
of your questions.

>1) It seems to be impossible to pass "line" variables into functions,
>therefore making it difficult to implement such useful functions as
>is_whitespace_only(line) and is_comment(line). For the meantime I made
>them accept strings and read the line into a string. Isn't there a way
>around this ?

Given a line variable L, pass the string value (L.all) as a string
parameter to the function. You can also access individual characters
in L by writing subscripts or slices:

L(1) is the first character
L(L'length) is the last character
L(1 to 3) is the first 3 characters
L.all is equivalent to L(1 to L'length)

A LINE is merely a pointer to a string, making it possible to
create strings of arbitrary length. Not nicely, but possible.

Your string-testing functions can, of course, have unconstrained
input parameters. That is one thing that VHDL does really
beautifully.

>2) VHDL's strong typedness makes it difficult to work with strings in
>a flexible way. For instance, the following "would be" code doesn't
>really work:

In fact, it doesn't work at all, because you can't create a variable
of unconstrained type...

>var msg := string;
>...
>...
>if (something) then
> msg := "hello";
>else
> msg := "bye";
>end if;

Sure, but again line variables can come to your rescue:

procedure copy_string_to_line(L: inout line; S: in string) is
begin
deallocate(L); --- has no effect if L was already null
write(L, S);
end;
....
if (difficult) then
copy_string_to_line(L, "hard");
else
copy_string_to_line(L, "exceptionally easy");
end if;

And don't forget that you can create functions that return
unconstrained strings. So, having created an unknown-length
string in your line variable L, you could do...

impure function make_string_from_something(.....) return string is
variable L: line;
...
begin
--- mess around until you have the right stuff in L
return L.all;
end;

Impure function, because it probably needs to call some
procedures such as WRITE(...). Yuck.

However, this code fragment represents a memory leak because L
goes out of scope on function return, but its contents have not been
deallocated. (At least, that's my understanding. I don't think
the LRM says anything about how to deal with this.) This is,
to put it mildly, a pain; you want to deallocate L *after*
returning the value it references. Yuck again. If you can
set an upper limit on the length of L, there is a hack^wfix:

impure function ....
variable L: line;
variable N: natural;
variable result: string (1 to MAX);
begin
... --- manufacture variable-length result in L
N := L'length;
assert N <= MAX; --- paranoia
result(1 to N) := L.all;
--- storage referenced by L will not be recovered when
--- L goes out of scope, so we must deallocate it here:
deallocate(L);
return(result(1 to N));
--- storage for "result" is automatically deallocated
--- when it goes out of scope
end;

I'd LOVE to know a better answer to that one... anybody???
Maybe it's better to give up on functions, and pass things
to and from procedures instead. Makes your package's
API nastier, though.

I'll say it once more... c'mon, folks, where's the ability to
overload assignment in VHDL? That would make it SOOOO much
easier to write stuff like string processing packages.....
--
Jonathan Bromley, Consultant

DOULOS - Developing Design Know-how
VHDL * Verilog * SystemC * e * Perl * Tcl/Tk * Project Services

Doulos Ltd., 22 Market Place, Ringwood, BH24 1AW, UK

http://www.MYCOMPANY.com

The contents of this message may contain personal views which
are not the views of Doulos Ltd., unless specifically stated.

Jonathan Bromley

Eli Bendersky wrote:
> 1) It seems to be impossible to pass "line" variables into functions,
> therefore making it difficult to implement such useful functions as
> is_whitespace_only(line) and is_comment(line). For the meantime I made
> them accept strings and read the line into a string. Isn't there a way
> around this ? How do you prefer to work with "line" type vars ?
It needs to be an inout variable.

> 2) VHDL's strong typedness makes it difficult to work with strings in
> a flexible way. For instance, the following "would be" code doesn't
> really work:
>
> var msg := string;
> ...
> ...
> if (something) then
> msg := "hello";
> else
> msg := "bye";
> end if;
>
> Because "msg" can't be declared as an unconstrained string. However,
> to give it length would mean to constrain all strings assignable to it
> to this length. If I say:
>
> vsr msg := string(1 to 5);
>
> Then I can assign "hello" to it (length 5) but not "bye". This is
> quite painful. How can I solve this problem ?
Instead I use subprograms for transactions. A few other things
I consider hard in file based testbenches:
. Hard to make them reactive (respond to a signal transitioning).
Without this, a small disturbance in the design (add a clock
latency in anything you interact with) will require a change
in your file.
. Hard to have any useful language constructs without creating
a mini-language (that is interpreted rather than compiled).
. Clock based or small transaction based IO is slow.

My rule of thumb is to use textio for reading large transaction
values (like an network packet, a video image, ...). Use VHDL
for small transaction processing (like CpuRead, CpuWrite,
SendCharacter, ...).

Note others have solved these problems, and like the file based
approach - I prefer a different method.

Cheers,
Jim

Jim Lewis

Eli,
>> 1) It seems to be impossible to pass "line" variables into functions,
>> therefore making it difficult to implement such useful functions as
>> is_whitespace_only(line) and is_comment(line). For the meantime I made
>> them accept strings and read the line into a string. Isn't there a way
>> around this ? How do you prefer to work with "line" type vars ?

> It needs to be an inout variable.

I did not note you said function until I read Jonathan's post.

Have you tried variable with in for a function?
I suspect you are right, but that would seem to
be a language limitation that needs to go.
Otherwise, how would I write,
is_empty

I would hate to have to test: L'length = 0 all the time.
Not too readable.

OTOH, you can make your other subprograms work by changing
their functionality to:
remove_whitespace
remove_comment

This is what the read procedures do.

>
>> 2) VHDL's strong typedness makes it difficult to work with strings in
>> a flexible way. For instance, the following "would be" code doesn't
>> really work:
>>
>> var msg := string;
>> ...
>> ...
>> if (something) then
>> msg := "hello";
>> else
>> msg := "bye";
>> end if;
>>
>> Because "msg" can't be declared as an unconstrained string. However,
>> to give it length would mean to constrain all strings assignable to it
>> to this length. If I say:

Create a constant that defines your maximum length token.

variable message : string (1 to MAX_TOKEN_LEN) ;

then you can read character by character using text until it fails.
Alternately, put your variable length tokens first in the file and
use VHDL's built-in read[file, string] to read them from the file
(not a line, so don't do readline until after).

As I mentioned, I don't read files except for big data sets
which are values and not tokens, so what I describe above
for reading tokens will have the normal bugs and flaws.

Cheers,
Jim

Jim Lewis

On Fri, 30 Mar 2007 07:51:30 -0800, Jim Lewis
<> wrote:

>>> 1) It seems to be impossible to pass "line" variables into functions,
>
>Have you tried variable with in for a function?

That would make good sense ("const ref") but can't be done;
function parameters must be of class constant. Ouch.

>I suspect you are right, but that would seem to
>be a language limitation that needs to go.

Agreed.

>Otherwise, how would I write,
> is_empty

You can't Not even if the function is impure. This is
something you may like to look at for VHDL++; something
halfway between a function and a procedure - allowing
side-effects and maybe inout or output arguments, but
unable to consume simulation time. Right now the
function/procedure distinction conflates these two
issues, and it's kinda inconvenient.

>OTOH, you can make your other subprograms work by changing
>their functionality to:
> remove_whitespace
> remove_comment
>
>This is what the read procedures do.

You can do all that sort of stuff, but it remains a fact that
the idiom

target := func(args);

is exceedingly convenient and rather more readable than
the procedure-driven equivalent.

--
Jonathan Bromley, Consultant

DOULOS - Developing Design Know-how
VHDL * Verilog * SystemC * e * Perl * Tcl/Tk * Project Services

Doulos Ltd., 22 Market Place, Ringwood, BH24 1AW, UK

http://www.MYCOMPANY.com

The contents of this message may contain personal views which
are not the views of Doulos Ltd., unless specifically stated.

Jonathan Bromley

On Fri, 30 Mar 2007 16:32:36 +0100, Jonathan Bromley
<> wrote:

>Given a line variable L, pass the string value (L.all) as a string
>parameter to the function. You can also access individual characters
>in L by writing subscripts or slices:
[...]

What I *forgot* to mention - and was forcibly reminded of
when I tried some of this myself a few moments ago - is
that you cannot make *any* reference through a null
pointer, so passing L.all to a function - or even enquiring
about L'length - will crash the simulator if L is null. On
the other hand, it *is* OK to put an empty string into
a line variable:

write(L, string'(""));

and you will then find that L'length=0 as expected;
L'range is "1 to 0", a null range.

I never said it was going to be easy or nice
--
Jonathan Bromley, Consultant

DOULOS - Developing Design Know-how
VHDL * Verilog * SystemC * e * Perl * Tcl/Tk * Project Services

Doulos Ltd., 22 Market Place, Ringwood, BH24 1AW, UK

http://www.MYCOMPANY.com

The contents of this message may contain personal views which
are not the views of Doulos Ltd., unless specifically stated.

Jonathan Bromley

> Create a constant that defines your maximum length token.
>
> variable message : string (1 to MAX_TOKEN_LEN) ;
>
> then you can read character by character using text until it fails.
> Alternately, put your variable length tokens first in the file and
> use VHDL's built-in read[file, string] to read them from the file
> (not a line, so don't do readline until after).
>
> As I mentioned, I don't read files except for big data sets
> which are values and not tokens, so what I describe above
> for reading tokens will have the normal bugs and flaws.

I don't understand. How can this help me write code like:

variable name: string; -- incorrect declaration

....
....
if (something) then
name := "jim";
else
name := "john";
end if;

assert false report "Hello " & name & ", nice to meet you !";

Is there no way to do this in VHDL at all ? I can pad "jim" with a
space to be of length 4, but that will screw up with the printout.

Tx
Eli

Eli Bendersky

On Mar 30, 5:32 pm, Jonathan Bromley <jonathan.brom...@MYCOMPANY.com>
wrote:
> On 30 Mar 2007 07:44:24 -0700, "Eli Bendersky"
>
> <eli...@gmail.com> wrote:
> >In an effort to read and process non-trivial command files for a
> >transactor in a testbench,
>
> Before I try to answer your question, it's worth noting that many
> engineers have simply given up on this; instead they let the
> VHDL compiler do the work for them. Your transactor provides
> a bunch of useful procedures, and then you create a test case
> not in plain text but in procedural VHDL, calling those procedures.
> It's usually quite easy to write your set of procedures so
> that a succession of calls to them looks almost like a script.
> And of course you have the full power of the language for
> doing conditionals, loops and what-have-you.
>
> Anyways, after that digression:
>
[...]

Thanks for the detailed reply. It's a real shame what you're saying,
since I think that a text based command file is in many ways more
versatile than commands coded into VHDL. Moreover, when coupled with a
monitor that logs the outputs to another file, it allows complex
result analysis with an external tool (like Perl) that has only two
text files to work on. This would be more difficult with the
transactor commands coded into VHDL.

Eli

Eli Bendersky

Eli
>> Create a constant that defines your maximum length token.
>>
>> variable message : string (1 to MAX_TOKEN_LEN) ;
>>
>> then you can read character by character using text until it fails.
>> Alternately, put your variable length tokens first in the file and
>> use VHDL's built-in read[file, string] to read them from the file
>> (not a line, so don't do readline until after).
>>
>> As I mentioned, I don't read files except for big data sets
>> which are values and not tokens, so what I describe above
>> for reading tokens will have the normal bugs and flaws.
>
> I don't understand. How can this help me write code like:
>
> variable name: string; -- incorrect declaration
>
> ...
> ...
> if (something) then
> name := "jim";
> else
> name := "john";
> end if;
>
> assert false report "Hello " & name & ", nice to meet you !";
>
> Is there no way to do this in VHDL at all ? I can pad "jim" with a
> space to be of length 4, but that will screw up with the printout.
>

OOPs. I thought you wanted to read a token from a file
and was not paying attention when you switched the problem:

variable buf : line ;

> if (something) then
write(buf, string'("jim")) ;
> else
write(buf, string'("john"));
> end if;

assert false report "Hello " & buf.all & ", nice to meet you !";
deallocate(buf) ;

I should have also looked at Jonathan's post first, as he
showed something similar. Note that write will allows you
to build multiple things up into the string and that you
need to deallocate it to clear it.

Cheers,
Jim

Jim Lewis

I'm with Jonathan on this one... Do your stimulus and your analysis in
vhdl. text-io is for data input/output (images, etc.) only.

Andy

On Mar 30, 2:05 pm, "Eli Bendersky" <eli...@gmail.com> wrote:
> On Mar 30, 5:32 pm, Jonathan Bromley <jonathan.brom...@MYCOMPANY.com>
> wrote:
>
> > On 30 Mar 2007 07:44:24 -0700, "Eli Bendersky"
>
> > <eli...@gmail.com> wrote:
> > >In an effort to read and process non-trivial command files for a
> > >transactor in a testbench,
>
> > Before I try to answer your question, it's worth noting that many
> > engineers have simply given up on this; instead they let the
> > VHDL compiler do the work for them. Your transactor provides
> > a bunch of useful procedures, and then you create a test case
> > not in plain text but in procedural VHDL, calling those procedures.
> > It's usually quite easy to write your set of procedures so
> > that a succession of calls to them looks almost like a script.
> > And of course you have the full power of the language for
> > doing conditionals, loops and what-have-you.
>
> > Anyways, after that digression:
>
> [...]
>
> Thanks for the detailed reply. It's a real shame what you're saying,
> since I think that a text based command file is in many ways more
> versatile than commands coded into VHDL. Moreover, when coupled with a
> monitor that logs the outputs to another file, it allows complex
> result analysis with an external tool (like Perl) that has only two
> text files to work on. This would be more difficult with the
> transactor commands coded into VHDL.
>
> Eli

Andy