C. C Language Interface

The Q interpreter provides an interface to the C programming language, which allows you to extend the interpreter with your own collections of "built-ins" in C modules, in order to access functions in C libraries and to take advantage of the higher processing speed of C functions for time-critical applications. We also refer to such C modules interfacing to the Q interpreter as external modules. On most systems supporting "shared" libraries (a.k.a. "dll"s), external modules are loaded dynamically at runtime, otherwise they have to be linked statically with the interpreter's main program to create a new interpreter executable. The Q interpreter has its own external module support for MS Windows; on UNIX systems, it uses the libtool package to implement this functionality. See GNU Libtool: (libtool)Top section `Shared library support for GNU' in GNU Libtool.

Conversely, Q scripts can also be executed from C, which allows you to use Q as an embedded macro or programming language or a term rewriting engine in your C/C++ applications.

In the following we first discuss Q's C module interface, libq, in some detail. In the final section of this appendix we then give a brief description of the libqint interface which is used to embed Q in C/C++ applications.

NOTE: As of version 6.0, Q now also has support for SWIG, the "Simplified Wrapper and Interface Generator", see http://www.swig.org. SWIG considerably simplifies the process of interfacing to existing C and C++ code. This feature still needs to be documented; for the time being, please refer to the README-SWIG file in the distribution for more information.

C.1 Compiling a Module

To provide for a platform-independent way of compiling and linking external modules, the Q programming system includes two tiny additional utilities which take care of the necessary and sometimes messy compilation details: qcc and qld.

Qcc is the module compiler. After you have prepared an external module (as explained in Writing a Module), you run qcc to compile your module to a library, a binary file which can be loaded by or linked into the interpreter.

The qcc program will compile each given source file and then invoke the linker on all object files to produce the output library. The synopsis of the qcc program is as follows:

qcc [options] [source-or-object-file …] [-- cc-options … [--link ld-options…]]

As indicated, qcc can process both C source and object files, and extra options can be passed on to compiler and linker after the -- option. All options understood by the qcc utility are listed below:

--dry-run, -n

Only print compilation commands, do not actually execute them.

--ext

Print the default output file extension and exit. This is usually `.la' on UNIX systems which denotes a libtool library, or `.dll' on Windows.

--help, -h

Print a short help message and exit.

--keep, -k

Do not delete intermediate files (object files and such) produced during the compilation process.

--mingw

Select the mingw compiler driver. Mingw is the native Windows port of the well-known GNU C compiler, gcc. This option is the default for dll compilation on MS Windows.

--msc

Select the MS Visual C compiler driver. Use this if you want to compile your module with the Visual C compiler on Windows.

--output=file, -o file

Specify the name of the library output file.

--verbose, -v

Echo compilation commands as they are processed.

--version, -V

As with the other Q programming utilities, this option causes version number and copyright information to be displayed, after which the program terminates.

The Q module linker, qld, is implemented as a shell script (thus it is not supported on Windows) which simply invokes libtool with the specified -dlopen and -dlpreopen options to create a new interpreter executable. This is only necessary if your system does not support shared libraries. Qld is invoked as follows:

qld --help | --version | -o progname [-dlopen module …]

The --help and --version options (which may also be abbreviated as -h and -V, respectively) work as usual. All other options are simply passed on to the libtool program, thus you can actually use any option recognized in libtool's "link" mode. (See GNU Libtool: (libtool)Invoking libtool section `Invoking libtool' in GNU Libtool, for details.) The interpreter executable is written to the specified output file progname (there is no default for the output file name, so the `-o' option must always be specified).

For a closer description of how qld is used see Linking and Debugging a Module, below.

C.2 Writing a Module

Writing a real external module can be a complicated task, just like any bit of C programming which goes beyond mere exercising. We cannot cover this process in detail here, although we hopefully provide enough information to get you started. You should take a look at the external modules bundled with the Q distribution for more substantial examples.

The procedure for making a C function callable from the Q interpreter is fairly straightforward. You first need a Q script (called a stub) which declares the external C functions you would like to use in your Q program. Next you have to write a C module which implements these functions. Finally, you run qcc to translate the C module to a library file which can be loaded by or linked into the interpreter. We discuss each of these steps in turn.

As a running example, let us implement list reversal in C. Our C version of this function will be called creverse. We first create a Q script creverse.q, which declares the creverse function as extern:

// creverse stub
public extern creverse Xs;

This tells the interpreter that when the creverse function is applied to a single argument, it should invoke the corresponding C function, which is assumed to be found in a shared library named after the stub script.

To implement the creverse function, we create a C module cerverse.c. The Q programming system comes with an interface library called libq which provides the necessary operations to inspect Q expressions given as argument values, and to construct new expressions which can be returned as the result of the function invocation. The interface operations are declared in the libq.h header file which we include in our C module. The C module must provide the necessary code for being initialized by the interpreter, which is done by putting a MODULE "header" at the beginning of the source file. The MODULE macro takes one argument, the name of the module. The external functions are then declared with the FUNCTION macro, which takes four arguments: the name of the module, the name of the external function (as given in the stub script), the name of the argument count variable, and the name of the argument vector variable. The FUNCTION macro is followed by the C block giving the definition of the function. In our example, the creverse.c module contains the following C code:

#include <libq.h>

MODULE(creverse)

FUNCTION(creverse,creverse,argc,argv)
{
  /* to be sure, check number of arguments */
  if (argc == 1) {
    /* expr is the data type used by libq to represent Q expressions;
       x is set to the argument expression, y is initialized to the empty
       Q list (mknil value) */
    expr x = argv[0], y = mknil, hd, tl;
    /* iscons(x,...) checks that x is a [|] expression and returns its
       head element and tail list */
    while (y && iscons(x, &hd, &tl)) {
      /* use mkcons to prepend the head element to the list y constructed
         so far */
      expr z = mkcons(hd, y);
      y = z; x = tl;
    }
    if (!y)
      /* signal error condition */
      return __ERROR;
    else if (isnil(x))
      /* well-formed list argument, return the constructed reversal */
      return y;
    else {
      /* argument was not a well-formed list; throw away the constructed
         value and indicate failure */
      dispose(y);
      return __FAIL;
    }
  } else
    return __FAIL;
}

A description of the various macros and functions provided by the libq library can be found in the libq.h header file. Extern functions are treated very much like the built-in functions provided by the interpreter. The external definition may be thought of as an "external rule" being applied to a Q expression. The definition may either return a Q expression, in which case the rule applies, or __FAIL, in which case the rule fails, and the interpreter goes on to try other equations supplied by the programmer. As a third alternative, the external definition may also return __ERROR, which causes evaluation to be aborted with an error message. Also note that built-in functions always take priority. Thus the interpreter first checks for a built-in rule, then for an external definition, and finally considers the equations in the Q script. Therefore it is possible to override an equational function definition with an external function. For instance, we might rename the creverse function in the declaration in creverse.q to stdlib::reverse, and to reverse in creverse.c. This makes our definition override the definition of reverse in stdlib.q. The latter definition will then only be applied when the external definition fails. (This is what the clib module actually does to override the definition of reverse as well as other operations in stdlib.q, see Clib.)

Also note that in order to prevent name clashes between external functions and ordinary C function names, the external function names are stropped with a special prefix (this is taken care of by the FUNCTION macro). To call an external function declared with FUNCTION from within your C module, you must therefore use the FUNCALL macro, which is invoked with the module name, function name and the argc and argv parameters as arguments. For instance, you would call the creverse function defined above as follows:

FUNCALL(creverse,creverse,argc,argv)

Back to our example. Before we can actually use the above function in the interpreter, we have to run qcc to translate the C module to a library object. In our example the process is fairly simple:

$ qcc creverse.c

If all went well, we now have a libtool library named creverse.la which is accompanied by some other libtool-generated files. (If you are trying this on a Windows system, you will see a dll file named creverse.dll instead.) If your system supports shared libraries then you can now simply run the creverse.q script with the Q interpreter as usual:

$ q creverse.q

(If at startup the interpreter complains that it could not load the module then your system probably does not support shared libraries. In this case you will have to create a new interpreter executable which links the module into the interpreter, as explained in Linking and Debugging a Module.)

Let's try it: You probably want to compare the running time of our list reversal function against a Q version of the same operation, which can be defined as follows (include this in the creverse.q script):

public qreverse Xs;
qreverse Xs:List        = foldl push [] Xs;

The following results were obtained on an Intel PIII-800 PC running Linux:

==> var l = [1..50000]

==> var r = creverse l; stats
0.033 secs, 1 reduction, 50000 cells

==> def r = qreverse l; stats
0.471 secs, 100002 reductions, 50004 cells

Not very surprisingly, the C function is indeed much faster, which is due to the extra pattern matching, value extraction and function call overhead of the interpreter.

It is also possible to declare a Q type as extern, and realize the type in C. Such a type must always be abstract (i.e., it must not have any constructor symbols, cf. Types), and the only way to get a value of such a type is by means of corresponding extern functions. For this purpose, the libq library provides the mkobj operation, which takes as its argument a pointer to the corresponding C object. For instance, consider an extern type Bar and a corresponding construction function bar declared as:

public extern type Bar;
public extern bar I J; // I and J integer

We might implement this type as a C struct containing a pair of integers as follows:

#include <libq.h>

MODULE(ctype)

typedef struct { long i, j; } Bar;

FUNCTION(ctype,bar,argc,argv)
{
  long i, j;
  if (argc != 2 || !isint(argv[0], &i) || !isint(argv[1], &j))
    return __FAIL;
  else {
    Bar *v = malloc(sizeof(Bar));
    expr x;
    if (!v) return __ERROR;
    v->i = i; v->j = j;
    return mkobj(type(Bar), v);
  }
}

Note that objects of an external type are completely "opaque" as far as the interpreter is concerned (just like file values). In particular, they will be printed using the notation <<typeid>> in the interpreter:

==> bar 1 2
<<Bar>>

The only way to access the contents of an external object is by means of corresponding C functions. For instance, an extern function which converts a Bar object to a tuple may be implemented as follows. The declaration:

public extern bartuple B;

The corresponding definition in the C module:

FUNCTION(ctype,bartuple,argc,argv)
{
  Bar *v;
  if (argc == 1 && isobj(argv[0], type(Bar), (void**)&v))
    return mktuplel(2, mkint(v->i), mkint(v->j));
  else
    return __FAIL;
}

Compile and load the script, using the same procedure as above, and try the following:

==> bartuple (bar 1 2)
(1,2)

(It is worth noting here that the isint and mkint functions used above only allow you to access and create integer values fitting into a machine integer. Integers of arbitrary sizes, which are represented as GMP mpz_t values, can be dealt with using the ismpz and mkmpz functions, see the libq.h header files for details.)

As indicated in our example, external objects are usually allocated dynamically, and will be freed automatically by the interpreter when they are no longer needed. This default behaviour is appropriate in most cases. However, you can also explicitly define a destructor function for objects of an external type in your C module as follows:

DESTRUCTOR(ctype,Bar,v)
{
  /* we could perform any other necessary cleanup here */
  free(v);
}

If such a destructor function is present, it will be used instead of the interpreter's default action to call free() when it disposes a Q expression of the corresponding type.

Occasionally, a module will also have some global internal data structures which have to be initialized and/or finalized. For this purpose, you can declare parameterless functions using the INIT and FINI macros, which will be executed before the script's initialization code, and just before the interpreter exits, respectively. Both macros take the module name as their single argument:

INIT(name)
{
  /* any code to be executed at startup goes here */
}

FINI(name)
{
  /* any code to be executed at termination goes here */
}

Note that the actual order in which the initialization/finalization routines of different modules are executed is unspecified. Furthermore, initialization routines will be executed before any of the script's initialization code (def and undef). Thus these routines should only be used to perform initializations and cleanup of private data structures of the module. Other initializations can be performed using appropriate def statements in the stub script.

C.3 Linking and Debugging a Module

If your system does not support shared libraries, or provides no means to dynamically load a shared library at runtime, you must run the qld program to produce a new interpreter executable which links in the required module. For instance, we could link the creverse module from the preceding section into the interpreter as follows:

$ qld -o myq -dlopen creverse.la

(You can also use the -dlpreopen option instead of -dlopen to force the module to be linked at load time, even if your system supports runtime loading. See GNU Libtool: (libtool)Link mode section `Link mode' in GNU Libtool, for more information.)

You then run the script as usual, but using the custom interpreter executable built with qld instead of the standard `q' program:

$ ./myq creverse.q

Building such a "preloaded" interpreter is also required when you want to debug a module, in which case the module usually must be linked statically into the interpreter. You can do this as follows (assuming that your C compiler understands the -g debugging flag):

$ qcc creverse.c -- -g

$ qld -o myq -static -g -dlopen creverse.la

If you now let your debugger execute `myq creverse.q', you should be able to debug creverse.c at the source level.

C.4 Embedding Q in C/C++ Applications

To call the Q interpreter from a C/C++ application, your program must link to the Q interpreter interface library, libqint. This library provides a number of operations with which you can load a script into the interpreter and then evaluate Q expressions. A closer description of the operations can be found in the qint.h header file of the library.

Before you can evaluate anything, you have to invoke the qexecv() or qexecl() routine to load a script or byte code file into the interpreter. These two routines work pretty much like the run command in the interpreter. The qexecv() function takes the script parameters as a vector of `char*' values, while qexecl() allows you to pass the parameters directly as char* arguments; otherwise the two functions operate in exactly the same way. As with the interpreter's run command, only one script can be loaded at any one time; if a new script is loaded with qexecv() or qexecl(), it replaces an existing one.

Two additional routines are provided, qexecvx() and qexeclx(), which work exactly like qexecv() and qexecl(), respectively, but take the script (or byte code file) itself as a binary string in the first argument. This allows you to specify the script to be executed directly in your application, without relying on some external script file.

Once a script has been loaded, you can repeatedly evaluate expressions with the qeval() routine, which takes the expression to be evaluated as a string argument. The contents of the string must be in Q expression syntax. The result is returned as a malloc'd string which must be freed by the caller; it contains the unparsed evaluated expression in the same format as it would be printed by the interpreter.

Here is a complete C example which employs the qeval() routine to implement a simple read-eval-print loop. The name of the script to be loaded and its parameters are taken from the command line of the program.

/* poor man's Q interpreter */

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <qint.h>

#define BUFSIZE 1024

int main(int argc, char **argv)
{
  int status;
  char s[BUFSIZE], *t;

  /* First load the script with args as given on the command line. */

  if ((status = qexecv(argv[1], argc-1, argv+1))) {
    char msg[1000];
    sprintf(msg, qstrerror(status), argv[1]);
    fprintf(stderr, "Error starting interpreter: %d: %s\n", status, msg);
    exit(1);
  }

  /* The read/eval/print loop. */

  printf("%s loaded, ready to rumble!\n", argv[1]?argv[1]:"empty script");
  printf("\nin>  "); fflush(stdout);

  while (fgets(s, BUFSIZE, stdin)) {
    int l = strlen(s);

    if (l > 0 && s[l-1] == '\n')
      s[l-1] = 0;

    t = qeval(s, &status);

    if (status)
      printf("err> %d: %s\n", status, qstrerror(status));
    if (t) {
      printf("out> %s\n", t);
      free(t);
    }
    printf("\nin>  "); fflush(stdout);

  }

  printf("\n");
  exit(0);

}

On Linux and most other Unix-like systems you should be able to compile and then run this program as follows (assuming that the C source is in a file named myq.c):

$ gcc -o myq myq.c -lqint
$ ./myq /usr/share/q/examples/fac.q

(On some systems it might be necessary to add more linker options to resolve additional dependencies. E.g., under Cygwin you'll need something like `-lqint -liconv -lgmp -lq'.)

A sample session with the program is shown below.

/usr/share/q/examples/basics.q loaded, ready to rumble!

in>  1+
err> 17: Syntax error

in>  1+1
out> 2

in>  fact 12
out> 479001600

in>  foldl (*) 1 [1..12]
out> 479001600

Obviously, the qeval() function is of limited usefulness if your application needs to construct the input expression from a C data structure, and/or inspect the evaluated result, e.g., to translate the result back to another C structure. For such applications libqint provides the qevalx() function which receives a binary expression object (of type qexpr) as input and returns the evaluated expression as another qexpr object. Operations are provided to construct and inspect qexpr objects which are analogous to the corresponding libq routines. Here is a simple example which constructs and evaluates a qexpr object using qevalx() and then checks the result value. It also illustrates the use of qexeclx() to load an "inlined" script into the interpreter.

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <qint.h>

int main(int argc, char **argv)
{
  int status;
  char *t;
  qexpr x;

  /* Our little hello world script. */
  char *script = "hello = writes \"Hello, world!\\n\";";

  /* Load the script into the interpreter, also pass arguments (the first
     argument is always the (fake) script name which becomes ARGS!0). */
  qexeclx(script, strlen(script), 2, "Hello!", "Hello world example.");

  /* Just for fun, print the arguments passed to the script. (We don't do any
     error checking, since there's not much that can go wrong here.) */
  t = qeval("ARGS", &status);
  printf("ARGS ==> %s\n", t);
  free(t);

  /* Evaluate the `hello' function from the script and print the result. */
  t = qeval("hello", &status);
  printf("hello ==> %s\n", t);
  free(t);

  /* Here's how to employ qevalx() to evaluate binary expression objects
     instead of their string representations. We construct the function
     application `writes "Hello, world #2!\n"' and evaluate it. Note that the
     string value *must* be allocated dynamically since it is taken over by
     the interpreter (and freed automatically together with the rest of the
     input expression when qevalx() is finished with it). */

  x = qevalx(qmkapp(qmksym(qsym(writes)),
                    qmkstr(strdup("Hello, world #2!\n"))),
             &status);

  /* Check the result. */
  if (status)
    printf("evalx() returned error code %d: %s\n", status,
           qstrerror(status));
  else if (qisvoid(x))
    printf("writes was executed successfully\n");
  else {
    t = qprint(x, &status);
    printf("writes returned some unexpected value: %s\n", t);
    free(t);
  }

  /* Get rid of the result. */
  qdispose(x);

  exit(0);

}

The program produces the following output:

ARGS ==> ["Hello!","Hello world example."]
Hello, world!
hello ==> ()
Hello, world #2!
writes was executed successfully

Using the qdef() function, it is also possible to set variables in the interpreter. You can either change existing variables of the script or create new variables in the global scope. Here is a simple example:

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <qint.h>

int main(int argc, char **argv)
{
  int status;

  char *script = "def TEST = 99;";

  qexeclx(script, strlen(script), 0);

  /* change the value of an existing variable */
  printf("TEST ==> %s\n", qeval("TEST", &status));
  status = qdef("TEST", qevalx(qparse("TEST+2", &status), &status));
  printf("TEST ==> %s\n", qeval("TEST", &status));

  /* create a new variable in the global scope */
  printf("MYTEST ==> %s\n", qeval("MYTEST", &status));
  status = qdef("MYTEST", qevalx(qparse("TEST+2", &status), &status));
  printf("MYTEST ==> %s\n", qeval("MYTEST", &status));

  exit(0);

}

The program produces the following output:

TEST ==> 99
TEST ==> 101
MYTEST ==> MYTEST
MYTEST ==> 103

This document was generated by Albert Gräf on February, 23 2008 using texi2html 1.76.