"QNEWS" ======= * 7.11 23 February 2008 - As suggested by Rob Hubbard and John Cowan, special Unicode characters in strings can now be specified using an escape of the form '\&name;', where name is any of the XML entity names specified here: http://www.w3.org/TR/2007/WD-xml-entity-names-20071214/ - clib now provides a set of additional int/float vector operations, as suggested by John Cowan on the mailing list a while ago. These allow you to treat byte strings as mutable C vectors of signed/unsigned 8/16/32 bit integers or float/double values. Operations to convert between such C vectors and corresponding lists of integers/floating point values are provided as well. - The glob and regex functions were moved back from system into clib. Also, a bug in the await function function was fixed (now it properly accepts timeouts specified in seconds, not millisecs), and support for mutex and semaphore timeouts was added to the try function (this requires a system with pthread_mutex_timedlock/sem_timedwait). - Added support for highres timers (clock_gettime et al) on systems where the POSIX timer extension is available. The system module provides a number of new functions (nanotime, nanosleep, etc.) to deal with these. Moreover, the builtin time function will now also make use of clock_gettime/gettimeofday if they are available. - Worked around some quirks with the MS Access ODBC driver which caused it to reject NULL and empty string parameter values in SQL insert statements. (Bug reported by Jiri Spitz.) The odbc_examp.q script now also works with MS Access on Windows. Moreover, as suggested by Tim Haynes, there are a couple of new functions to retrieve additional metadata about a data source. - Jiri Spitz also contributed a much improved AVL tree implementation for the container types (bag.q, dict.q, hdict.q, set.q). The improved algorithm has its origin in the SWI-Prolog implementation of association lists by R.A.O'Keefe and Jan Wielemaker (see assoc.pl at http://www.swi-prolog.org). Jiri rewrote the algorithm in Q and also added the deletion operations which were missing in the Prolog version. Tests indicate that updating the new data structures is at least twice as fast for large trees, quite a notable improvement! Thanks to all who contributed code and participated in the discussions leading up to this release! ------------------------------------------------------------------------------ * 7.10 17 December 2007 Another bugfix release which fixes some bugs in the clib module, see the ChangeLog for details. ------------------------------------------------------------------------------ * 7.9 8 December 2007 A bugfix release. Please see the ChangeLog for details. ------------------------------------------------------------------------------ * 7.8 15 October 2007 This release fixes some critical and a lot of smaller bugs in the interpreter, and also adds some important new features. The most important user-visible changes are listed below. As usual, please check the ChangeLog for details. IMPORTANT BUGFIXES: - Left-hand side 'where' clauses now work as advertized. In Q 7.7, these would sometimes cause the bytecode compiler to generate wrong code for free variables on the right-hand side of an equation. (Reported by Eddie Rucker.) - Fixed a couple of rather obscure bugs in the bytecode compiler related to wrapping around at end-of-file, which caused segfaults under certain circumstances. (Reported by Eddie Rucker.) This also fixes a long-standing bug which caused the parser to "wrap over" with an unfinished construct at the end of an imported module; this now properly provokes an error message. - Shell escapes ('! command') work again. These apparently stopped working at some time in Q <= 7.7 due to changes in the command lexer. NEW FEATURES: - Qualified (a.k.a. selective) import clauses, as suggested by John Cowan. In addition to the old-style "unqualified" import clauses ('import module' or 'include module'), which import the entire namespace of the given module into the client module, Q now also supports "qualified" imports using the following syntax: from module import name1, name2, ...; from module include name1, name2, ...; This has the effect of declaring the given symbols in the current namespace *without* importing the module itself. Thus the imported symbol 'name' is to be accessed as just 'name' or 'client::name' (where 'client' is the importing module's name) instead of 'module::name'. (You can also use *both* qualified and unqualified imports in concert to make both 'client::name' and 'module::name' work. The difference between 'from module import' and 'from module include' is that the latter also reexports the symbols by making them 'public'. This works similar to unqualified includes, but note that with qualified includes the symbols really become part of the namespace of the client module. For convenience you can also write from module import; from module include; without listing any symbols to just import/include everything. (It goes without saying that this should be used with care.) Again, this is roughly equivalent to 'import module' or 'include module', respectively, but the imported symbols are redeclared in the namespace of the client, without importing or including the module itself. In order to better support the new clauses, the symbol declaration rules are slightly more strict now to ensure proper diagnostics if different symbols with the same print name are accidentally imported from different sources. (Please see the documentation or the ChangeLog for details on this.) - Slim prelude. Following a discussion about namespace pollution and startup times on the mailing list, the prelude was modified in order to reduce both. Specifically, stdtypes.q and its parts (array.q, bag.q, etc.), as well as getopt.q are *not* included in the prelude any more, and most of the POSIX system interface has been unbundled from clib.q and moved to the new system.q module which also needs to be imported explicitly. (But note that clib.q still offers all the really essential stuff, namely C replacement ops, additional string/gmp functions, a few high-level file I/O functions including printf/scanf and friends, byte strings, references, multithreading support and, last but not least, the 'exit' function. These are considered so ubiquitous in Q programming and/or fundamental enough that they are retained in the prelude.) As a result, interpreter startup with the slim prelude is now more than twice as fast, and the global (builtin+prelude) namespace is much leaner now (395 versus 1068 symbols). This hopefully makes the interpreter easier to use, especially on older or embedded hardware. But note that this also breaks backward compatibility, as you will now have to import getopt.q, system.q and stdtypes.q (or its parts) in scripts which need the system operations and the container data structures. You're kindly asked to check your scripts by running them through the compiler with the -w option to find out whether they might be affected. :) - Extended tuple and string operations. Enumerations and comprehensions now also work with tuples, using the same kind of syntactic sugar. Moreover, the tuple operations in stdlib.q have been moved to the new tuple.q module and a few additional list operations (cat, map, do and friends, reverse) have been extended to also work with tuples. Similarly, the remaining string functions from stdlib.q have been moved to the string.q module, and this module now also overloads *all* the common list operations from stdlib.q so that they work on strings as expected. (This was already suggested by Alexander Nickolsky a while ago.) - Mutable sequences. Based on one of John Cowan's proposals on the mailing list, I've implemented some convenience operations to work with tuples, lists, streams and other containers of references. These can be found in the new reftypes.q module (needs to be imported explicitly). Please note that this module is still experimental. Bug reports and comments are appreciated. - New 'whois' command. This is an improved version of the 'whos' command which also lists, for each given symbol, the fully qualified names under which the symbol is available in the global namespace. Moreover, the 'whos' command was modified to list available alternatives instead of just printing an error message for ambiguous unqualified symbols, and to also report the line numbers where each symbol is declared or first used. - Improved warning messages. At long last, the '-w' option now reports the proper line numbers in undeclared symbol warnings. The --pedantic option now also warns about non-prelude symbols from unqualified imports which are used without proper qualification. This is useful to report a common source of errors, namely the accidental "reuse" of an unqualified symbol due to a missing local symbol declaration. Moreover, the undocumented warning level 3 is now officially available using the '--paranoid' option. In addition to '--pedantic', it also warns about undeclared free variables and unqualified imports even from the prelude. Note that '--paranoid' produces an excessive amount of diagnostics even with perfectly sound scripts, so its use is not really recommended, but it may occasionally be useful to check your script for missing declarations or mistyped identifiers. Last but not least, the new QWARN environment variable can now be used to set a default warning level. - Both interpreter and bytecode compiler now accept '-' as the main script, causing the script to be read from standard input. This is great for testing purposes, and can also be combined with "here" documents to embed Q programs in ordinary shell scripts. DOCUMENTATION AND EXAMPLES: - Of course, the manual has been updated to describe all the new features, and I have also greatly expanded Appendix B.4, "Running Scripts from the Shell", which now describes in much more depth and with examples how Q scripts can be configured to run as standalone programs. Also see the section on expression references in the "Clib" chapter for a description of the new reftypes.q module. - Thanks are due to Eddie Rucker for contributing various examples to the wiki. The latest version of his comma-separated values example (csv.q) is now included in the distribution. PACKAGING: - On Linux systems, the "all in one" package now also includes the new ncurses and Qt/Q interfaces, as well as all the available multimedia examples, the Pd plugin and the faust2pd and faust2sc scripts, which were previously provided as separate packages. - At the time of this writing, the ncurses and Qt/Q modules as well as the Pd/Q plugin haven't been ported to Windows yet. They will hopefully be provided in the near future. Please check the Q website or the Q mailing list for corresponding announcements. - Also note that most of the addon modules had to be updated for the new Q release, so if you install Q from source, make sure that you reinstall the latest versions of your addon packages as well. ------------------------------------------------------------------------------ * 7.7 11 June 2007 This release marks another important milestone with quite a few notable changes, mostly in response to various discussions, suggestions and feature requests from the Q mailing list. Special thanks are due to John Cowan, Rob Hubbard, Marco Maggesi, Alexander Nickolsky and Eddie Rucker for testing and bug reports, and to Rob Hubbard for providing the latest versions of his rational and polynomial libraries. The present release is also intended as a preview for the forthcoming "stable" 8.0 release. The only major missing feature right now appears to be the support for 64 bit systems, but this will (hopefully) be fixed before the 8.0 release. The most important changes in this release, which also coincide with the most popular feature requests, are: - user-defined operator symbols consisting of multiple punctuation symbols, such as (--); - a new kind of left-hand qualifiers which can be shared among different equations (thus you can now have conditions and local variable definitions spanning multiple equations for the same left-hand side); - a full implementation of Wadler-style "views", which extends the "custom unparsing" mechanism already available in previous releases, so that you can now also use these custom representations in pattern matching. Besides this, the latest "all-in-one" package also comes with new and improved versions for many of the addon modules, and now also includes the latest version of Rob Hubbard's polynomial library "QiX". Below is a more detailed list of all important changes and additions in the core distribution. - Various bugfixes, see the ChangeLog for details. - Various cleanups in the core distribution. In particular, we removed the bundled glob, regex and readline libraries which were ancient versions and didn't support unicode; we now use the available system libraries instead. We also removed the bundled gqbuilder and gnocl interface, as well as mod_q and the qcwrap utility; these will be made available as separate source packages. As a result, the core package is much leaner now, it also builds much faster and 'make distcheck' finally works. - -w2 a.k.a. --pedantic doesn't print undeclared variable warnings any longer. These warnings were a major annoyance since they forced you to declare variables used inside lambdas, list comprehensions and similar constructs. (For the time being, there is a new -w3 option which still allows you to print those warnings if you can't live without them.) - There's a new --cstacksize option and corresponding 'cstacksize' command which allows you to set the maximum C stack size to be used by the internal eval() routine. This allows you to catch C stack overflows which could occur during the evaluation of non-tail-recursive special forms, as reported by Eddie Rucker. The 'cstacksize' value is given in KB; the default is 256. If 'cstacksize' is set to zero then these checks are disabled. - The Q bytecode compiler, qc, has a new option -d a.k.a. --debug which causes a listing of the generated bytecode to be printed on standard output. Both the QVM instructions generated for the right-hand sides of equations and the matching automaton for the left-hand sides is printed. Please note that this option is only available if qc is invoked directly. - Uncapitalized identifiers created dynamically at runtime are now treated as function symbols (unless they are explicitly declared as variables using a 'var' declaration, which is now also available on the command line). The previous behaviour was to treat *all* symbols created at runtime as variables, which, as pointed out by Alexander Nickolsky, was inconsistent with the script language and therefore confusing. - There's a new syntax for qualifiers ('if' and 'where' clauses) which allows qualifiers to be shared by different equations for the same left-hand side. The new kind of qualifiers is written on the left-hand side of an equation, like so: fact N if N>0: = N*fact (N-1); otherwise: = 1; The lhs qualifiers work exactly like the "old-style" qualifiers which are written on the right-hand side, at the end of an equation, but while the latter are always confined to a single equation, the former have a scope which extends up to the next equation with either another lhs qualifier or an explicit left-hand side. Conditions and local variable bindings are then shared between all equations in the scope of the qualifier. Of course, lhs and rhs qualifiers can be mixed freely. For instance: strip C S where N = #S-1 if not null S: = strip C $ sub S 0 (N-1) if S!N = C; = strip C $ sub S 1 N if S!0 = C; otherwise: = S; Here, the scope of the condition 'not null S' and the local definition 'N = #S-1' encompasses the first two equations. - It is now possible to declare operators which consist of multiple punctuation symbols, such as: public (--) Xs Ys @(-); Xs:List--Ys:List = foldl (flip (filter . neq)) Xs Ys; To make this work it was necessary to change some of the lexical rules of the Q language, and so this is not 100% backward-compatible. Nevertheless, most existing scripts should work with no or minimal changes. Please refer to the manual for details. - Wadler-style "views" are now supported, which allow you to match against "virtual" constructors, using the existing customizable unparsing mechanism. This is particularly useful to provide pattern-matching capabilities on abstract data types which have their real constructors hidden, such as the 'Rational' type (and now also 'Complex', see below) as well as the container data types from the standard library. See the new section on "views" in the manual for details. (Note that previous Q versions since 7.2 already had a kind of "custom unparsing" mechanism on which the views implementation builds and which it supersedes. In fact, views still work for defining custom pretty-printing rules, the only difference is that the "unparsing" function is now called 'view' instead of 'unparse'.) - 'Complex' is now an abstract data type, very much like 'Rational'. The virtual constructor is (+:); (:+) is provided as an alias for Haskell aficionados. This is also used as the canonical view for 'Complex'. E.g., the imaginary unit 'i' will now be printed as '0+:1'. - 'lambda' is now a virtual constructor for the builtin 'Function' type, and there's a builtin rule for 'view' which unparses 'Function' objects as 'lambda' terms. This lets you match against compiled lambdas without having to play dirty tricks with valq . str, and makes the runtime compilation of lambdas mostly transparent to the programmer. E.g., the following will now work as expected: ==> var fact = \N.if N>0 then N*fact(N-1) else 1; fact \X1 . if X1>0 then X1*fact (X1-1) else 1 ==> def \PAT.BODY = fact; PAT; BODY X1 if X1>0 then X1*fact (X1-1) else 1 - External types with an associated view will now use the view to compare objects of the type for syntactic equality, using the same "equal if they print out the same" rule which also applies to ordinary expressions. This applies, in particular, to the builtin 'Function' type (see above). Semantic equality (=) on 'Function' objects is now undefined (used to be syntactic equality, but that is now already provided by (==)). - The builtin special form (==), which checks two expressions for syntactic equality and is also used internally by the QVM to implement non-linear rules, is now directly accessible in Q scripts. We also added a corresponding (!=) operator, which is just the negation of (==), to the prelude. - We added a new '_FAIL_' builtin, suggested by Alexander Nickolsky, which makes an entire reduction fail (as opposed to 'fail' which only aborts the current rule). - Clib now provides a basic interface to the readline library. See clib.q or the manual for details. - The standard library function 'cat' now works with an arbitrary mixture of lists and streams and will always return a list. This is necessary to ensure consistent behaviour of list and stream comprehensions with mixed arguments. Note that this implies that 'cat' is now always strict and won't work with infinite streams any more, so you have to use 'streamcat' for lazy stream concatenation instead. - The pattern-matching conditional 'case' is now an instance of 'Lambda', to make rebindings in nested scopes work properly. The 'matchp' helper function was removed and replaced by direct invokations of the 'eq true.(\X.true)' idiom for the same reason. As a bonus, cond.q now defines two additional special forms 'condfun' and 'casefun' which allow you to construct anonymous conditional and pattern-matching functions which take their data in a second parameter. - Lots of updates and improvements in the manual. Updates for the new features in this release can be found, in particular, in the sections on lexical syntax in chapter 3, declarations in chapter 5, user-defined operators in chapter 6, equations and qualifier syntax in chapter 7, views in chapter 8, special constructors and streams in chapter 9, lambda abstractions in chapter 10, and complex numbers in chapter 11. Some material in these sections has also been rewritten to better explain some of the more arcane features of the Q language. ------------------------------------------------------------------------------ * 7.6 18 December 2006 Another maintenance release. Fixes issues with temporary files being created in the wrong directory on Windows, and bugs in custom unparsing and shebang handling. Also adds some OSX compatibility patches (thanks to Andrew Berg) and renewed support for OpenZaurus 3.5.4. ------------------------------------------------------------------------------ * 7.5 12 September 2006 Another bugfix release. Fixes various segfaults in libq and libqint. ------------------------------------------------------------------------------ * 7.4 30 August 2006 Another bugfix release. This one fixes a memory allocation bug and missing unresolved forward (-w) warnings in the compiler. ------------------------------------------------------------------------------ * 7.3 21 August 2006 This release fixes a critical bug related to the new memoization feature introduced in Q 7.1, which could cause random expressions to remain unevaluated. Some other bugs in the bytecode compiler and the interpreter were also fixed (see the ChangeLog for details). In particular, left-hand sides of equations may now contain const values in the head, provided that they are followed by extra arguments. ------------------------------------------------------------------------------ * 7.2 27 June 2006 This release sports a major revision of Q's "numeric tower". There is a new abstract type `Real' which is a subtype of `Num' and the new supertype of both `Int' and `Float'. Rob Hubbard's rational number type has been integrated into the standard library, and the `complex.q' module has been overhauled to turn complex numbers into an algebraic type. Moreover, as suggested by John Cowan, the floating point arithmetic has been fixed to properly support IEEE INFs and NaNs, and `typec.q' now includes some additional Scheme-like "semantic" type-checking predicates which allow you to classify numbers according to the abstract kind of value they represent (complex, real, rational, integer), rather than the concrete data type they belong to. Please note that, due to the necessary changes in the floating point arithmetic, division by (exact or inexact) zeros does not fail any more, but returns an inf, -inf or nan value, depending on the numerator. To support user-defined pretty-printing, there is an experimental new hook into the expression printer, which lets you define custom unparsings for any built-in, user-defined or external data type, by just adding definitions for the new (predefined) `unparse' function. Examples for this can be found in the prelude.q script. As usual, please refer to the ChangeLog and the manual for the details. ACKNOWLEDGEMENTS: Special thanks again to Rob Hubbard for making his rational number module available and for helping to integrate it into the standard library. Thanks are also due to John Cowan for his thorough testing and suggestions to improve the new number implementation. ------------------------------------------------------------------------------ * 7.1 11 June 2006 This release sports some critical bugfixes and optimizations in the interpreter as well as some important new features. Most changes are there to improve the handling of special forms. In particular, Q now has built-in support for memoization of special arguments, call-by-pattern evaluation of arguments during pattern matching, built-in lambda, and syntactic sugar for if-then-else, lambdas, stream objects and enumerations, as well as lists and stream comprehensions. All the remaining little warts of Q which I am aware of have been cured as well. Despite all the new stuff under the hood, most changes are rather unobtrusive so most existing scripts should work with little or no changes. (But see below for a list of known backward compatibility issues.) As usual, please check the ChangeLog file and the language manual for all the gory details. An overview of the most important user-visible changes follows. COMPILER: - Improved warning messages: Undeclared variable symbols are _not_ reported with -w anymore (these messages are often a nuisance if you use lambda a lot), and function symbols will only be reported as undeclared at the end of a source file, if they are neither imported nor declared _anywhere_ in the file (where appearances of a function symbol on the left-hand side of equations and variable definitions count as implicit declarations). If you want to play it *really* safe, there is a new warning level -w2 a.k.a. --pedantic which warns about *all* function and variable symbols which are used without a prior *explicit* declaration (i.e., when using that option you really have to declare each and every symbol explicitly, to make the compiler happy). LANGUAGE: - Syntactic sugar for streams and comprehensions: Streams can now be written in the same kind of notation as lists, but using curly braces instead of brackets, as in `{1,2,3}' or `{X|Xs}'. The new notation is translated to an appropriate application involving the stream constructors `cons_stream' and `nil_stream' (which are now predefined as builtins) by the parser, and is also used when an expression is printed. Moreover, the parser recognizes the notations `{1..10}' and `{1..}' for finite and infinite stream enumerations, and the notation `[X..]' is now also supported for finite enumeration types. There's also new syntactic sugar for list and stream comprehensions. E.g., you can now write `[2*X^2 : X in Xs, X mod 2 = 0]' as a shorthand for `listof (2*X^2) (X in Xs, X mod 2 = 0)'. Stream comprehensions (streamof) work the same, using curly braces instead of the brackets. - New "grouping" syntax. Semicolons inside lists, streams and tuples can be used to further partition the sequence into nested tuples. E.g., [A,B;C,D] is the same as [(A,B),(C,D)]. This makes it easier to write stuff like lists of key-value pairs, or the clauses of a conditional expression. - Syntactic sugar for lambdas. Lambdas can now be written in the customary form \X.Y, where X is the lambda pattern and Y the lambda body, and the notation \X1 ... Xn.Y is provided as a shorthand for nested lambdas. - Syntactic sugar for if-then-else. Instead of `ifelse X Y Z' you can now also write `if X then Y else Z'. Likewise, `when X Y' can also be written as `if X then Y'. - Memoization of special arguments: There is a new memoization operator `&' belonging to the group of quotation operators. When applied to a special argument, this operator ensures that the argument will only be evaluated once, at the time it is first accessed in a non-special context. - Call-by-pattern: As suggested by Yann Orlarey, special subterms of non-special args of function applications will now be evaluated automatically as needed during the pattern matching process. Note that only non-special arguments are treated in this manner. Special arguments still remain unevaluated during pattern matching, so that special forms can be used in a macro-like fashion as before. - Other language extensions: There's an alternative syntax `type X == Y' to declare type aliases, and var declarations can now include an initializer (`var X = foo'); the latter was suggested by Rob Hubbard. Moreover, the `var' keyword can now be used in front of a variable symbol on the right- hand side of an equation or variable definition to escape free variable symbols and declare them on the fly. BUILTIN AND STANDARD LIBRARY FUNCTIONS: - The `lambda' function is now provided as a builtin which "compiles" lambdas to a special internal representation. (The old implementation is still available, but has been moved to the examples folder.) As a result, `lambda' itself and the stuff which depends on it are now generally much faster. Non-linear patterns and patterns involving the anonymous variable are now also handled correctly. Note that, despite the new look and feel, lambda is still just an ordinary function, so Q's capabilities to manipulate lambdas on the symbolic level are unimpaired, and list and stream comprehensions are still implemented in the library (but are also much faster now, due to some general performance gains in the implementation of special forms and the speedups of the lambda function). - The `switch' and `match' conditionals in the standard library have been replaced with new special forms `cond' and `case'. The new conditionals are more Lisp-like, and they also take advantage of the new grouping feature. Example: sign X = cond (X>0, 1; X<0, -1; true, 0); - The `+' and `-' operators can now be used to perform simple arithmetics on enumeration types. If X is an enumeration type member then X+N and X-N yield the Nth successor and predecessor of X, respectively, and if X and Y are members of the same enumeration type then X-Y yields the integer N s.t. X=Y+N. Moreover, typec.q offers a new typechecking predicate for enumeration type members (isenum). - The builtin trunc, round, float, int and frac functions now work consistently on both integers and floating point values, and floor and ceil functions have been added to math.q, as suggested by Rob Hubbard. - The stream.q module provides a few new functions (repeat, repeatn, cycle; strict, lazy and friends), and `cat' and `tuple' now work with streams, too. Additional work has been done to improve the efficiency of some of the other stream operations and prevent unnecessary reevaluations of stream members. - The clib.q module now offers so-called "sentinels" as an additional type of lazy expression references. Sentinels are evaluated when the sentinel object is garbage-collected. This provides a means to implement ordinary Q data structures which perform automatic cleanup in the same fashion as some built-in and external data types. - The getopt.q module has been added to the standard library, which provides a quick-and-dirty replacement for the GNU getopt_long function. BACKWARD COMPATIBILITY ISSUES: For the sake of getting rid of most of (all?) the remaining warts in the language and the library, there have been a few minor changes which break backward compatibility in some places. They are listed below: - The deprecated library functions `apply' and `compose' have been removed. If you still have old code using these, you will have to replace them with the builtins ($) and (.). - The `switch' and `match' conditionals have been superseded by `cond' and `case', so you will have to rewrite code using those. Translating the old constructs to the new ones should be a piece of cake, though. - Some of the thread operations in clib.q have been renamed. In particular, `cond' has been renamed to `condition' (to resolve a name clash with the new `cond' conditional), and `sem' has been renamed to `semaphore'. The corresponding types are now named `Condition' and `Semaphore', respectively. - List and stream comprehensions are now defined in cond.q. (Actually this shouldn't be an issue unless you explicitly use qualified identifiers for the listof and streamof operations.) ACKNOWLEDGEMENTS: This release also contains a new version of Tim Haynes' cgi.q script, and the initial release of Rob Hubbard's rational.q module. Both can be found in the examples directory. Thanks, Rob and Tim, for your contributions, and thanks also to John Cowan for proofreading the manual, and the members of the Q mailing list for helping to sort out various design and implementation issues. ------------------------------------------------------------------------------ * 7.0 18 February 2006 The latest and greatest Q release comes with quite a few notable changes and improvements which is why we made this a major release. For the most part, this release should be backward-compatible with Q 6.2, so existing scripts should still work with minimal or no changes. Below you can find a brief summary of the major user-visible changes. As usual, additional information can be found in the ChangeLog and the manual. 1. Unicode support. Character strings are now internally represented using the UTF-8 encoding, and the interpreter handles the necessary translations between the internal representation and the system encoding automatically and transparently. Existing 7 bit ASCII applications should continue to work without any changes, but if unicode support is enabled (which is the default if your system has the necessary bits and pieces) then string literals can now also contain extended characters from the entire Unicode character set which will be treated as single Char objects. In addition, Unicode characters can now also be used in identifiers and operator symbols. NB: Please note that the unicode features have not yet been tested extensively in non-Western locales, and thus should be considered "beta". Bug reports and other inputs, especially from Q programmers using Eastern locales, will be greatly appreciated. :) 2. Internationalization and localization support. The clib module now provides the necessary operations to program applications which deal with different language environments, such as setlocale, strfmon, strcoll, as well as interfaces to GNU iconv and gettext. 3. C/C++ API: The version numbers of the libq and libqint libraries have been incremented to account for binary incompatibilities with Q 6.0 and earlier. Thus you will have to make sure that you recompile modules for the latest release. (Actually, this change has been overdue since Q 6.1, so we took the opportunity to make up for that with this major release.) 4. Gnocl/Q GUI builder. As a bonus, the Tcl/Tk interface now includes special support for Peter G. Baum's Gnocl (http://gnocl.sf.net), a Tcl extension for creating GNOME/GTK+ applications with Tcl. A full-featured GUI builder based on the new Q-Gnocl interface, which generates native Q code, is also available. Please see modules/tk/gqbuilder/README for details. 5. Packaging. Managing binary releases for an ever-growing collection of separate add-on modules has slowly turned every major release of the core package into a big nightmare. So, starting with this release, while the core package and the add-on modules are still distributed as separate source tarballs, there will be only one binary package for the entire Q programming system. Thus, on systems for which we provide binaries, you can now just install the "one big package" and be done with it. ACKNOWLEDGEMENTS: Thanks are due to John Cowan for his help and suggestions concerning all things unicode and other issues, as well as his tireless testing and bug reporting to make the Cygwin port work. ------------------------------------------------------------------------------ * 6.2 13 July 2005 This release finishes off the revision of the tuple syntax which was started with Q 6.1 (see below for details). The deprecated 1-tuple syntax (X) has been removed, so that the construct (X) now always denotes a parenthesized expression. Moreover, this release also adds support for user-defined operators. E.g., you can now define yourself an exclusive-or operator with the same precedence as the built-in `or' as follows: public (xor) X Y @ 3; X xor Y = (X or Y) and not (X and Y); Please see the ChangeLog and the manual for details. ------------------------------------------------------------------------------ * 6.1 10 July 2005 This release sports a few bug fixes (see the ChangeLog for details), new when and unless conditionals in the standard library, a new sign-on message in the interpreter and, most importantly, the following changes in the Q language: - Per request by Tim Haynes, there's a new right-associative infix application operator ($) which has the same meaning as in Haskell. See the language manual for details. - Following a suggestion by John Cowan, we're currently in the process of straightening out Q's 1-tuple syntax which is a bit confusing for most Q programmers, as it interferes with the syntax of parenthesized expressions. In this release, the 1-tuple notation (X), where X is a primary expression, is still accepted but deprecated, and it produces a warning message. The new recommended 1-tuple syntax is (X,), as in Python. (Actually, all lists and tuples can now have a trailing comma behind the last element, see the language manual for details.) Note that beginning with the next release, the new syntax will be _mandatory_, and the notation (X) will _always_ denote a parenthesized expression, so you should now start fixing your scripts accordingly. - The deprecated Prolog-style % syntax for comments is no longer valid. ------------------------------------------------------------------------------ * 6.0 23 October 2004 This release adds SWIG support to the Q programming system. This is an important milestone in the development of the Q programming system, as it makes interfacing to existing C and C++ libraries much easier. SWIG is the "Simplified Wrapper and Interface Generator" which allows you to create C/C++ wrapper modules for various target languages in an easy way. More information about SWIG is available at http://www.swig.org. Preliminary documentation for Q's SWIG support and some examples can be found in the modules/swig directory in the sources. Note that until the SWIG-Q module becomes part of the official SWIG distribution, to use SWIG with Q you'll need a SWIG version which has been patched up to add support for the Q language. For the time being, a suitable SWIG package can be found on the Q homepage. IMPORTANT: To implement the SWIG support, the libq ABI had to be changed, so you will have to recompile your modules (and reinstall the latest RPMs of add-on modules from the Q homepage) to make existing modules work with the new Q version. ------------------------------------------------------------------------------ * 5.5 2 October 2004 This is a bug fix release. Moreover, the Windows package now includes the latest GGI from CVS. ------------------------------------------------------------------------------ * 5.4 14 September 2004 This release adds a libxml2/libxslt interface, a new profiling command in the interpreter (try `help profile' for more information), and a few bug fixes and updates for third-party software (see the ChangeLog for details). ------------------------------------------------------------------------------ * 5.3 15 April 2004 Another bug fix release (see the ChangeLog for details). As a bonus, you also get a new built-in function composition operator (.) which works like stdlib::compose, but can be used as an infix operator and also adapts itself to special forms in its operands. (Note that stdlib::compose is now deprecated and will be removed from the library in the future.) A closer description of the (.) operator can be found in the manual (try `help composition' in the interpreter). ------------------------------------------------------------------------------ * 5.2 29 February 2004 This is a minor update with some bug fixes, optimizations, and two new looping constructs (dowhile, for) in the standard library. ------------------------------------------------------------------------------ * 5.1 20 February 2004 This release brings an overhaul of the virtual Q machine, which is now fully tail-recursive, a new `enum' builtin along with the usual syntactic sugar for enumerations, a new `qcwrap' utility for creating C wrappers from Q scripts, and a couple of bug fixes and improvements in the build system. - Tail call elimination: The interpreter now also performs tail call optimization for builtins and special arguments, so that tail-recursive definitions involving special forms like (and then), (or else) and `ifelse' are executed in constant stack space. - Enumerations: Compiler and interpreter now understand the `[X..Y]' syntax for specifying ranges of integers, floating point numbers and enumeration type members. See Section 8.2 of the Q language manual for details. - C wrappers: With the new `qcwrap' utility, which is bootstrapped from the src/qcwrap.q script, you can now turn a Q script into a C program which can be compiled to create a self-contained binary executable. See Appendix B.5 of the manual for details. - Build system: Most notably, running configure and make in a separate directory now works, provided you have a VPATH-capable make utility like GNU make. Moreover, `make test' can now be run before the interpreter is installed. - Bug fixes: Most notably, lambda now finally checks that the pattern is linear -- computers should be fast enough now to afford that. ;-) ------------------------------------------------------------------------------ * 5.0 31 January 2004 This release sports some changes in the C interface and the standard library warranting a new major release number. It also includes important new features and bug fixes in the debugger and additional facilities for web programming. We expect this to become the stable release of the core system for quite some time now (bug fixes notwithstanding). The user-visible changes are listed below. As usual, all the gory details can be found in the ChangeLog. - Libq has been overhauled. Some old stuff has been removed and a few useful additional expression operations have been added. IMPORTANT: Since the binary interface has changed you'll have to recompile your external modules. Also make sure that you reinstall the latest versions of other add-on modules like Q-Audio, which are distributed as separate packages. - Brand new C->Q interface: The libqint library now allows you to embed the Q interpreter in your C/C++ applications in a direct fashion. See the last section of Appendix C in the manual for further details. - Web programming: There's a new Q module for the Apache web server, see mod_q/README for more information. Moreover, the new `curl' module provides an interface to libcurl which allows you to transfer files with URL syntax using a variety of different Internet protocols; see README-Curl for details. - Standard library: Some final cleanup and polishing. Specifically, new curry/uncurry functions have been added (see stdlib.q), the unzip operation has been overhauled, and scan/scan1 have been replaced with a semantically complete set of scan functions (scanl/scanl1/scanr/scanr1), which is also compatible with Haskell. The stuff which used to be in comp.q has been moved to prelude.q, along with the overloaded succ and pred operations from stdlib.q. - Debugger improvements: The command syntax has undergone another face-lift, and there are new commands to set breakpoints on function symbols. Readline support was added as well, and some bugs were fixed (in particular, the `n' command now works correctly with recursive evals). See Appendix D in the manual for more information. - Compiler warnings (as suggested by Mathias Bruessel): There is a new compiler option (-w a.k.a. --warnings) to enable warnings about undeclared function and variable symbols. If this option is in effect, then the compiler complains about each undeclared symbol on the right-hand side of a definition or equation, so you will have to declare or define all symbols before they are used, to make the compiler happy. This may be annyoing at times, but helps a lot to avoid silly typos in larger scripts, which might otherwise go unnoticed and produce hard-to-find bugs at runtime. ------------------------------------------------------------------------------ * 4.6 6 January 2004 This release features a much improved GGI interface and a new magick module which uses ImageMagick to load, save and manipulate images in a variety of file formats. Of course, you can also transfer the pixel data from ImageMagick images to GGI visuals and vice versa. BIG NEWS: Q is now on SourceForge! As of December 2003, Q has become a SourceForge-hosted project, which can be found at the following URL: http://sourceforge.net/projects/q-lang/ There you can now obtain released source and binary packages (check the "Files" link on the project page), as well as the latest and greatest development sources in CVS ("CVS" link). Two mailing lists for discussing Q development and usage are also available ("Lists" link). ------------------------------------------------------------------------------ * 4.5 14 October 2003 This release sports a new ggi module (general graphics interface, see http://www.ggi-project.org), and some improvements in the debugger (see the ChangeLog for details). ------------------------------------------------------------------------------ * 4.4 16 September 2003 The new ODBC module allows you to connect to ODBC-compatible databases (MySQL, PostgreSQL, ...). There are also a few minor changes and bug fixes, see the ChangeLog for details. ------------------------------------------------------------------------------ * 4.3.2 15 August 2003 Additional changes for BSD compatibility, bug fixes. ------------------------------------------------------------------------------ * 4.3.1 28 July 2003 Fixes for latest autotools and FreeBSD compatibility, bug fixes. ------------------------------------------------------------------------------ * 4.3 13 June 2003 Free variables can now be declared `const', meaning that they are read-only, i.e., they can only be assigned a value once. A couple of bugs were fixed, too, see the ChangeLog for details. ------------------------------------------------------------------------------ * 4.2.3 6 May 2003 Added some missing stuff to clib, and a new gdbm module which provides an interface to the GNU dbm library. ------------------------------------------------------------------------------ * 4.2.2 3 May 2003 Some additional improvements in the standard library. In particular, the new Ref type in the clib module now allows you to create mutable data structures in a direct manner. ------------------------------------------------------------------------------ * 4.2.1 25 April 2003 Improvements in the clib module (automatic int/float coercion in the printf operations, new openpty and forkpty operations) and some minor fixes in the documentation. ------------------------------------------------------------------------------ * 4.2 26 March 2003 Good news for Macintosh aficionados: Q has finally been ported to OS X! Moreover, the interpreter now provides signal handling via the new `trap' builtin, and the clib module has had a major overhaul and now provides well over 200 functions. Most essential UNIX system operations have been implemented, including the POSIX termios and BSD socket interfaces. Furthermore, some bugs have been fixed, and some quirks with realtime thread scheduling under Linux and Windows have been resolved. NOTE: Take a look at modules/clib/givertcap if you want to enable realtime scheduling in a non-root application under Linux. ------------------------------------------------------------------------------ * 4.1.3 18 February 2003 Minor code cleanup and bug fixes; see ChangeLog for details. ------------------------------------------------------------------------------ * 4.1.2 24 November 2002 Some minor bug fixes in the interpreter and the clib and dxl modules. ------------------------------------------------------------------------------ * 4.1.1 19 November 2002 Bugfix in the builtin catch function. ------------------------------------------------------------------------------ * 4.1 11 November 2002 This is a minor update with some cleanup in the standard library, various improvements in the debugger and q-mode.el, updated documentation, and fixes for Tcl/Tk 8.4 compatibility. Moreover, a new builtin function `which' has been added which facilitates package configuration. Please refer to the ChangeLog for details. ------------------------------------------------------------------------------ * 4.0 3 November 2002 Ok, enough features have been added since release 3.1 now, so I decided it's time for a major release again. This release includes all the cumulative changes of the 3.x series, some more bug fixes (see the ChangeLog for details), and the following additional changes: - Equations may now include an arbitrary list of conditions and where clauses, in any order. See below for details. - The `=' operator must now be parenthesized on *both* the left- and right- hand sides of equations and definitions. This means that some definitions in existing scripts might have to be changed, but this is a small price for enabling the compiler to find obscure errors where a definition is mistaken for an equality check (which can happen much too easily, e.g., when forgetting the semicolon at the end of an equation, or the comma at the end of a definition). - The tk module now includes additional operations for setting and retrieving variable values in the Tcl/Tk interpreter. - The sample exception scripts (except.q, clib_except.q, etc.) have been moved back to the examples directory, which is where they really belong. The most notable change in this release is that an equation may now include an arbitrary list of conditions and where clauses, in any order. Thus the arbitrary restriction to "where-if-where" constructs has been removed. As before, the conditions and where clauses are processed from right to left, whereas the individual definitions in a where clause are considered from left to right. As soon as a condition or variable definition fails, the entire rule is aborted. This allows you to have definitions like, for example, foo X = bar Y if qux Y where Y = baz Z if quux Z where Z = ...; which interleaves conditions with the definitions of required variables, or foo X = bar Y where Y = baz Z where Z = qux U where U = ...; which incrementally builds a value through a series of definitions. (Note that the latter is actually equivalent to `foo X = bar Y where U = ..., Z = qux U, Y = baz Z;' but for many examples the "backward" order is just more natural.) The debugger was modified as well, to acommodate for the above change. It now stops on the current rule every time a condition or definition has been processed, and always displays the current qualifier while it is being processed. This makes it much easier to keep track of a rule with many different conditions and local definitions. ------------------------------------------------------------------------------ * 3.5.5 28 October 2002 This release fixes segfaults in the builtin hash function and the tk module. ------------------------------------------------------------------------------ * 3.5.4 10 October 2002 This release fixes a compiler bug which caused wrong bytecode to be generated when parsing multiple right-hand sides involving where clauses for the same left-hand side. It also works around a bug in the MSVC sscanf function which affected the clib sscanf function (Windows only). ------------------------------------------------------------------------------ * 3.5.3 8 October 2002 This release fixes some obscure segfaults in the debugger and the builtin tuple concatenation function. ------------------------------------------------------------------------------ * 3.5.2 6 October 2002 This is a maintenance release which fixes some compilation quirks on the latest Linux distros which come with Tcl/Tk 8.4 and gcc 3.2. Other changes: - The variable file is now named .q_vars by default, so it is hidden on Unix systems. - I also started working on the floating point precision of the `save' command. The precision used by `save' is now larger to reduce rounding errors when reconstructing these values with `load'. Moreover, I added experimental support for saving exact floating point values represented in hexadecimal textual format. Currently this only works on systems with a C99-compatible printf function which has the %a conversion flag. (Add -DHAVE_ISO99_PRINTF to the list of compilation flags when compiling to enable this feature.) ------------------------------------------------------------------------------ * 3.5.1 19 September 2002 New in this release: - Bug fixes in the standard library (cond.q, list.q, stream.q) and in the interpreter's garbage collector (--gc option). - Larger default stack size. - Clib: New `gets' and `fgets' operations to read a line with the trailing newline. The former `gets' function (which reads a whole file at once) has been renamed to `fget'. ------------------------------------------------------------------------------ * 3.5 September 2002 This release features some more bug fixes and cosmetic changes in the interpreter and Emacs Q mode; see the ChangeLog for details. The most important user-visible changes are the following: - Local definitions (`where') are now permitted both before and after a qualifying condition. A `where' clause before the condition will only be processed after the condition has been checked. - The standard library now also includes a hashed dictionary type which can be used for arbitrary (not necessarily ordered) key values. See hdict.q. ------------------------------------------------------------------------------ * 3.4.3 August 2002 More bug fixes: The builtin syntactic equality predicate is now implemented non-recursively, to prevent C stack overflows on large expressions. Special interpreter commands like `echo' and `cd' now accept string arguments enclosed in single quotes. ------------------------------------------------------------------------------ * 3.4.2 July 2002 This release fixes .qinitrc/.qexitrc execution, and parse stack overflows which occurred when sourcing long lists and tuples in the interpreter. Moreover, the import command now also works correctly if the main module is the empty script. ------------------------------------------------------------------------------ * 3.4.1 18 June 2002 Bug fixes in the tk module. The interpreter doesn't lock up on tk and tk_reads any more, and the Tcl/Tk interpreter is now thread-local. (NB: Starting with this release, I'm using 3-level version numbers when appropriate to make it easier to handle frequent bugfix-only releases.) ------------------------------------------------------------------------------ * 3.4 14 June 2002 Again a bug fix release. Adds some functions to the C interface which help to alleviate some weird segfaults in the Windows version. ------------------------------------------------------------------------------ * 3.3 June 2002 Bug fix release. Fixes an annoying bug in the compiler which caused wrong bytecode to be generated if a variable definition followed an equation with a where clause. ------------------------------------------------------------------------------ * 3.2 May 2002 Maintenance release. Most notably, the size of the expression data structure has been reduced considerably (24 bytes per expression cell vs. 36-40 bytes in earlier releases) without any performance losses. Also adds support for bounded semaphores (see clib.q). ------------------------------------------------------------------------------ * 3.1 April 2002 This is a major new release featuring a lot of smaller bug fixes and optimizations, as well as some important new features, most notably: improved modularization facilities, pattern-matching and rule-local variable definitions, and multithreading support. See the more detailed list of user-visible changes below. (Note that 3.0 was never officially released, hence what follows is the list of cumulative changes since release 2.3.) - MODULES: This release cures one of the most apparent shortcomings in earlier Q versions: the lack of decent support for "programming in the large". The new module system is both simple and effective. Different scripts are now considered as separate modules, each with their own namespace, and you must *always* explicitly include a module to gain access to the public symbols of that module (with the exception of the prelude operations which are still available without explicit inclusion). The new syntax for including modules is: import id1, id2, ...; include id1, id2, ...; Modules may either be specified using their name (which must be a valid identifier) or the full pathname in double quotes (the .q suffix is supplied automatically if necessary). The import and include directives differ in that the latter causes all public symbols of the included module to be reexported, whereas the former only makes the symbols available in the importing module. To resolve name clashes between modules, you can import a module under an alias, e.g.: import "my/prelude" as myprelude; To resolve name clashes between public symbols in different modules, you can use qualified identifiers of the form modname::id. You can also redeclare an imported symbol under a new name: private mylib::foo X Y as bar; If you redeclare an imported symbol as a public symbol, it will be reexported. IMPORTANT: The first declaration of an unqualified symbol in a given module will now always introduce a *new* symbol; this has to be considered when porting pre-3.0 scripts. Moreover, new symbols in a module are now declared "private" by default, so you have to explicitly declare symbols with scope "public" if they are to be used outside a module. The namespace available in the interpreter is now that of the main module, and the interpreter also allows you to dynamically import (and unimport) additional modules in the global scope. As before, it is possible to gain access to *all* public and private symbols of the program (also in modules not directly imported in the main module) using qualified identifiers. The downside of all this good new stuff is that it really breaks backward compatibility in some places. But most scripts should work after a minor touch-up, adding some public keywords and module qualifiers here and there. Also note that cyclic inclusion chains are not allowed any more, as they cannot be handled in a single-pass compilation. - PATTERN BINDINGS: Variable definitions can now involve a pattern on the left-hand-side which is to be matched against the supplied value. E.g., def (X,Y) = foo Z; evaluates foo Z, matches it against (X,Y) and binds the free variables X and Y accordingly. Non-linear patterns like (X,X) also work as expected. An error is reported if the match fails. - ANONYMOUS VARIABLE: Q now supports the use of _ as an anonymous variable: hd [X|_] = X; tl [_|Xs] = Xs; This is handy if you don't care about a component value on the left-hand side. The anonymous variable can only be used on the left-hand side of a rule or definition. Multiple instances of the anonymous variable are matched independently from each other, therefore foo _ _ = 0; will be matched for *any* combination of arguments. The interpreter now also employs the anonymous variable symbol for denoting the result of the last expression, instead of the $ symbol used in previous releases. Note that, for backward compatibility, the underscore normally still counts as a lowercase letter, and thus any other identifier starting with _ is considered as a *function* symbol; the anonymous variable is the only exception from this rule. - LOCAL VARIABLE DEFINITIONS: Local variables in a rule can now be introduced by means of "where" clauses: foo X = bar Y Z where Y = f X, Z = g Y; Where clauses always come behind the right-hand side *and* qualifier of a rule, and the defined variables are available in *both* the right-hand side and the qualifier: foo X = bar Y Z if check Z where Y = f X, Z = g Y; Pattern-matching definitions are permitted as well. Here, a failing match causes the entire rule to fail. For instance, foo Z = bar X Z where [X|_] = Z; works like foo [X|Xs] = bar X [X|Xs]; but avoids repeating the list value on the right. Variable bindings in where clauses are performed in the given order, and each definition may refer to all left-hand side variables of the rule, as well as all variables introduced in earlier definitions. Each use of a variable refers to its most recent definition. Note that, in difference to languages like Haskell, "where" clauses cannot be nested, and they can only be used to introduce local *variable* definitions, i.e., "local equations" are not supported. - EXCEPTION HANDLING: The builtin functionality has not changed, except that now hard error conditions generate an exception of the form syserr N, where N is the error number. The syserr symbol is actually a constructor symbol of the new builtin type SysException, which in turn is a subtype of the predefined (abstract) type Exception. It is recommended that you use this scheme for your exception generating rules as well, deriving suitable subtypes of the Exception type for your exceptions. This discipline helps to make exception handling much more transparent, and allows you to discriminate over different exception types without having to resort to matching individual exception patterns. The new except.q standard library module implements exceptions along this line for all builtin and standard library operations; the additional graphics_except module provides exceptions for the optional graphics module. Note that these modules are not loaded by default, thus you have to include them explicitly if you want exceptions to be generated. - MULTITHREADING SUPPORT: This release provides POSIX-based multithreading. So, for instance, you can now perform a lengthy calculation as a background task simply as follows: ==> def TASK = thread (sum (nums 1 1000000)) ==> // do some other work ... ==> result TASK // get the result 500000500000 ==> stats // stats lists secondary threads once they're finished 0 secs, 1 reduction, 0 cells thread #1: 8.42 secs, 2000003 reductions, 2000007 cells The multithreading operations are actually implemented in the clib module. All the good stuff from the POSIX threads library is there: thread creation, termination, cancellation, mutexes, conditions, semaphores, ... See the clib documentation for more details. Note that for all this to work, you must have the pthread library on your system, and the interpreter must have been built with thread support (which is the default when the library is available). Also note that the interpreter with thread support is a little slower and also needs more memory, compared to the no-thread version. Moreover, the current implementation effectively serializes bytecode interpretation on the reduction level and thus user-level threads cannot really take advantage of multi-processor machines. - STANDARD LIBRARY: The standard library has gone through a major overhaul, to accommodate for the new language features. For the most part, the library should be backward-compatible, but note the following changes: - clib is now "officially" part of the standard library and is thus loaded in the default prelude - examples for the external modules are now installed in corresponding subdirectories of the examples directory - for convenience, the `in' symbol used in list and stream comprehensions is now a predefined relational operator, hence parentheses around the pattern and list expression are not needed anymore - removed obsolete std module - removed obsolete `let' operation from lambda.q (use where clauses instead) - moved the stuff in the type module into stdlib - new multithreading primitives in the clib module (see above) - renamed the simple conditional `cond' in cond.q (which collides with the new condition constructor in clib.q) to `ifelse' - new except and graphics_except modules (see above) - new dxl_file function in the dxl module (see dxl_sample.q for an example) - COMPILER: Bytecode files now have the executable bit set, and the bytecode format includes an additional #! header line which allows bytecode files to be executed directly as shell scripts. Note that this will only work with UNIX shells supporting the "#!" feature. ------------------------------------------------------------------------------ * 2.3 September 2001 This is a maintenance release featuring several bugfixes for Solaris compatibility (no functional changes). See the ChangeLog for details. ------------------------------------------------------------------------------ * 2.2 August 2001 Well, as you know, Q 2001 was supposed to be the last and final word, but I didn't get it quite right at the first attempt. :( This release is supposed to cure the remaining bugs and warts of release 2.1, and also introduces some new features which have been left out until now due to lack of time. The most important user-visible changes are: - Reorganized the sources and integrated all available modules into the main package. While I was at it, I also automakified and libtoolized the sources, which should improve portability. In particular, the C interface should now work on any platform supported by libtool, and it is now possible to "preload" modules into the interpreter, which is necessary when debugging a module and to make modules work on systems lacking shared library support. To facilitate this, I have added a new utility, `qld' a.k.a. the "Q module linker", to the package. With all the changes in the module system, the libq library is incompatible with earlier releases, so you will have to recompile your own modules. Please note that there are some minor changes in the libq interface (see the ChangeLog for details), so you'll probably have to modify your module sources accordingly. Sorry for breaking backward compatibility once again, but these changes were absolutely necessary to make the new module system work. - Compiler: The meaning of the -w compiler option has changed (it now turns those annoying warnings *on*, not off); also, the warnings about overlapping rules have been improved. - Interpreter: Now includes readline 4.2. Various improvements are: useable --gc option, bug fixes in the run command, and improved rule printing in the debugger. See the ChangeLog and the manual for details. - Library: The lambda.q script has been overhauled again, and graphics.q is no longer part of std.q, so you have to include this script explicitly. The currently available modules have all been integrated into the main package and are now installed along with the standard library. As a bonus, there is a new interface module for IBM's OpenDX data visualization software (UNIX/X11-based systems only). Since OpenDX is freely available under an open source license, this should be interesting news for researchers developing scientific applications using a Q/Octave combo. New features of compiler and interpreter: - Improved tail recursion optimization: Toplevel sequences (||) in rule rhs's and qualifiers as well as variable definitions are now compiled to bytecode employing direct manipulation of the evaluation stack. This has the important consequence that basic imperative-style looping constructs like loop X = foo X || loop X if bar X; = () otherwise; are now executed in constant stack space, as one might reasonably expect. - Exception handling: The new `catch' and `throw' builtins implement the usual catch/throw exception handling paradigm. This also allows you to handle runtime errors like memory or stack overflow. Furthermore, the `fail' function allows you to cause a rule to fail while it is already being executed. - Rule priorities: You can now explicitly specify rule priorities with the @N directive. ------------------------------------------------------------------------------ * 2.1 Dec 2000 "Q 2001" ***10th Q ANNIVERSARY*** If you look further down in this file, you'll notice that it is Q's 10th anniversary (well, almost), so you can reasonably expect something special. ;-) So here goes ... I decided that it was about time to release a polished version of the Q programming systems, which brings the language and the standard library out of its notorious beta status. With this "Q 2001" release, the core system, including the language definition itself, has been dubbed "stable." This means that future releases will now enforce backward compatibility, and concentrate on additional scripts and modules as well as improvements and new developments in the interpreter (such as multi-threading, which is planned for release 3.0, but don't hold your breath yet ;-). Please note that the directory layout has changed in the present release, so make sure that you ***completely remove any existing Q installation*** before installing the new release. Q 2001 features a bunch of small changes and additions, too many to list them here, so please check the documentation for details. The most important user-visible changes are pointed out below. *** BUG FIXES *** Some more (obscure) bugs were fixed: - C interface (wrong reference counting; missing argument checks; the sym macro now creates new symbols if they do not exist). - Rare segfaults in the lexical analyzers and the interpreter's main loop reader. - Several compiler bugs, which allowed mismatching declarations of variable symbols, and equations with a non-symbol constant as the head of an application on the left-hand side of a rule. Moreover, free variable symbols on the left-hand side of an equation now produce an error message. - Invoking the interpreter from Qpad now also works under Windows 2000/NT. Furthermore, the builtin `run' command has been rewritten to avoid the overhead of an execvp() call, which has the nice side-effect that it works now in Qpad too. *** LANGUAGE *** A lot of smaller fixes here, some of which also affect the library, rendering some old stuff obsolete. - The new Char type is the subtype of String which denotes the single- character strings. - Builtin relational ops now also apply to "enumeration types" (types solely consisting of nullary constant symbols), and logical operators also work on integers (doing bitwise logical operations). - New syntax for 1-tuples, operator sections, and quote/force/splice operators. - Character escapes in strings can now also use the \0xhh and \0ooo syntax to denote character codes in hexadecimal and octal. - New builtin functions (fclose, pred, succ, time, sleep) and some minor changes in existing builtins (ord, isconst, isfun). In particular, the `random' function now returns 32 bit random _integers_ instead of floating point values, and uses a new algorithm ("Mersenne Twister" by Makoto Matsumoto and Takuji Nishimura, see random.c). *** INTERPRETER *** You will find some new command line options, as well as new and improved commands and an overhauled command syntax (more "shell-like" for the "special" commands). In particular, I'd like to point you to the new `stats' command, which is quite useful for profiling purposes. Furthermore, the `break' command now allows you to control the interpreter's behaviour in response to exceptions (Ctl-C, break function, invalid qualifier). Again, please refer to the language manual (Appendix "Using Q") for details. A lot of small improvements have been made in the interpreter code, which will mostly be invisible to the user. The installation procedure has been "modernized" to separate architecture-dependent and -independent files. You will now find the library scripts and some other useful items in /share/q, whereas the architecture-dependent stuff like external modules goes to /lib/q. *** C INTERFACE *** You'll also find some new features in the C interface (global module initialization and finalization, access to GMP mpz values). The new `qcc' utility should make it easier to translate external modules in a system-independent manner. I tested this with Linux and Windows, other systems may require some work. If you have any patches to make qcc (and the C interface/shared library support in general) work on your system please let me know. (Future versions should probably use libtool, in order to provide dlopen support for a greater variety of systems.) *** LIBRARY *** Some old stuff which is now provided as builtins has been removed, other parts have been polished, and a few new functions have been added. The special.q script is now named cond.q, and contains only the conditional expression constructs (the other stuff which used to be in special.q, namely defer/force/splice, has been superseded by the builtin quote/force/splice operators). Some experimental changes in the 2.0 release have been sorted out; in particular, cons, push, pop, top, hd and tl have reverted to their previous (<2.0) behaviour -- cons/push/pop/top still apply to both lists and tuples, though. Some other trivial little changes: `zip' now takes two arguments instead of one (to improve compatibility with that other well-known functional language ;-); the operations in sort.q now take an order predicate as their first argument (very useful); and, for cosmetic purposes, `isempty' and `ismember' have been renamed to `null' and `member' throughout, and the `genlist' and `genstream' operations are now called `while' and `iterate'. Last not least, substantial progress has been made with the lambda.q script which now provides pattern-matching lambda abstractions (GOOD STUFF!). Thus expressions like `lambda (foo a (b,c)) (bar a c)' can now be used to match the function argument against a pattern and bind variables accordingly. Since list and stream comprehensions are implemented in terms of lambda, you get pattern bindings there as well. The bad news: since `(x,y,z)' is now a pattern and *not* a variable "list," multi-argument lambdas must now be denoted using nested lambda expressions like `lambda x (lambda y ...)', so you probably have to rewrite existing scripts to make them work with the new lambda construct. To compensate for the porting hassle, you get some additional goodies, namely lambdas and comprehensions with special arguments, a new pattern-matching conditional, and a `let' special form which lets you bind local variables in an expression. *** MISC *** Q emacs mode (q-mode.el) now supports auto-indent and filling. As far as I'm concerned, this one is finished, except for bug fixes (no more elisp hacking, please ;-). If anyone wants to improve it, just go ahead and send me the results. Some useful external modules are finally available from the Q homepage. At the time of this writing, there are interfaces to Tcl/Tk (John Ousterhout's GUI toolkit), GNU Octave (J.W. Eaton's GPL'ed MATLAB clone), and "clib", my first attempt at a comprehensive system interface. Clib will also speed up -- by orders of magnitude -- the most important list and string processing operations from the standard library, so make sure you take a look at this one. Last not least, the docs have been updated, so the distribution now includes the 6th edition of "The Q Programming Language." -- Given the time I'll need to do the final touches and make all this stuff available officially, it'll probably be the end of December before "Q 2001" actually becomes available. So Merry Xmas and a Good New Year to everybody! ------------------------------------------------------------------------------ * 2.0 May 2000 This is a MAJOR release with some important new features as well as many bug fixes. The most relevant changes are: - improved tuple construct (modified syntax; C vector implementation) - explicit short-circut logical connectives (no --complete anymore) - separate concatenation operator ++ - integers of arbitrary sizes (using the GNU MP package; *GOOD STUFF!*) - much improved interpreter (beginnings of a real command language) - interface to the C language (via shared libraries) I must admit that the first three items necessarily break backward compatibility. But I think that these changes in the Q language were really necessary to eliminate some (final?) notorious misfeatures, and that they are worth the porting effort anyhow. A summary of all changes can be found below (please refer to man page and info file for more details). - COMMAND LINE INTERFACE: The main interface to the Q programming system now is the "q" program, which incorporates the interpreter (formerly separate "qi" program). You can still run compiler and interpreter separately, but the interpreter can now automagically invoke the compiler when a source script is given. Both compiler and interpreter can at last be invoked without giving a source script, meaning of course that an empty script will be used. By default, however, the compiler will always include the definitions in the prelude.q script, which enables you to set up an environment which always includes certain non-builtin functions and variables. You can disable this with the --no-prelude option (meaning that only the built-in stuff is available), and you can specify an alternative prelude name with the --prelude option. You can now also run a script directly from the interpreter, using the new "run" command. A number of new command line options is provided and some options have been renamed (see man page or info file). The environment strings QPS and QFS have been removed; these options can now be set using command line options as well as corresponding interpreter commands. It is now possible to specify commands and command files on the command line using the new -c and -s options of the interpreter. Finally, the filename parsing/construction code of the compiler and interpreter now properly handles volume prefixes and ~ (homedir) notation (this requires that the HOME environment variable is set). - INTERPRETER: Some nasty bugs in the Q machine (segfaults when the expression stack was resized during an evaluation; built-in constants being considered as "reducible") have been fixed. A memory leak in the garbage collector has been plugged as well. Error handling in the interpreter has been improved (it now prints the offending line and indicates the position of the error). The interpreter's automatic garbage collection, which previously was invoked any time an evaluation was finished, has been disabled, as it was a serious memory hog (to implement it, the interpreter had to allocate a free expression block for each one actually in use!). Note that this doesn't affect operation of the interpreter, it only prevents unused expression memory from being returned to the system pool until the interpreter is restarted. If you really liked it the way it was and you have plenty of memory to waste, you can still build the interpreter with the "automatic gc" option by configuring with --enable-gc. This also enables an (undocumented) --no-gc interpreter option for disabling this feature at runtime. The interpreter now supports GNU readline, which provides command line editing and history as well as symbol and filename completion when the interpreter is run interactively. Furthermore, a new built-in "$" variable can be used to refer to the result of the last expression evaluation, and the interpreter now understands a number of commands which allow you to inspect and adjust various parameters, edit and run scripts and command source files, read online info, load and save variables, etc. Multi-line input mode is not supported any more. Instead, the line end can now be escaped using the `\' character, as it is in most other script languages. Furthermore, you can now put multiple expressions/commands on a single line, like so: def X = 99; foo X; bar X Multiple defs/undefs can be put into a single command: def X = 99, Y = 2*X; undef A, B, C Both compiler and interpreter now support C-like hexadecimal and octal notations for integers, and the interpreter also lets you specify an integer output format, s.t. integers can also be printed in octal or hexadecimal if this is desired. Three different formats ("standard", "scientific" and "fixed", each with a given precision) are available for printing floating point numbers. The startup and termination procedure of the interpreter has gone through a major overhaul. The "environment file" stuff is gone. Instead you can now set up your environment in a flexible manner by putting arbitrary interpreter commands in the ~/.qinitrc startup and the ~/.qexitrc termination files which are sourced when the interpreter starts up resp. exits in interactive mode (see the sample qinitrc/qexitrc files in the lib directory). The debugger has also been polished a bit, and some bugs have been fixed. It supports some new and improved commands, and now also shows the qualifier part of an equation. You can now walk around on the reduction stack and apply the step-over command to an arbitrary stacked rule, which facilitates debugging quite a bit. Moreover, you can either print only the filenames or full pathnames of scripts. Built-in variables ($, INPUT, ARGS etc.) are now readonly and hence cannot be re- or undefined. Six new built-in functions are provided: list and tuple, which convert between lists and tuples; sub, which extracts subsequences from strings, lists and tuples; version and sysinfo, which return as strings the interpreter's version number and the configure-generated host system description, respectively; and atan2, a two-argument version of the arcus tangent function. The built-in exponentiation operator now also allows the first operand to be negative, but only if the second operand is an integer. Interpreter and compiler now implement integers of arbitrary sizes, using the GNU multiprecision (GMP) package. Thus things like `fac 100' and the "stream of powers" (see the `fac.q' and `powers.q' examples) will now finally work as expected. :-) Multiprecision floating point numbers with IEEE-style rounding will probably follow in a future release, as soon as they are fully supported by GMP. - Q MODE: The emacs Q mode has been rewritten almost completely to support the new interpreter features. It also provides for better synchronization with the inferior interpreter process (via gnuserv/gnuclient), and incorporates a lot of other small improvements. See the Q info file for details. The Qpad Windows GUI frontend to the interpreter has been updated to the new version as well; as usual, you can find it on the Q homepage. - LANGUAGE: The Q language now supports \ to escape line ends in a string, integers of arbitrary sizes, hexadecimal and octal notations for integers, and multiple def's/undef's in a single statement (as in the interpreter). The new `extern' keyword allows you to declare both functions and (abstract) types as "external," providing an interface to functions written in the C language (see "C INTERFACE" below). Cross-checking of symbol declarations is now more strict. In particular, the number of arguments must also match in a non-special declaration. It is now possible to specify which initial arguments of a special form should be evaluated. E.g., you can use a declaration like special foo ~X Y; to indicate that foo has two special args, the first of which should be evaluated. Thus it is no longer necessary to fiddle with `apply' and "auxiliary" function symbols to achieve the same effect. Tuples now work in the same fashion as lists, using the binary constructor (|). That is, (X1,...,Xn) = (X1|(X2|...(Xn|()))). This also implies that the 1-tuple (X|()) is now different from the singleton expression X. I know that this breaks backward compatibility in some cases (it also affects some parts of the standard library), and that the (X|()) syntax for singleton tuples is ugly. But these changes are necessary to avoid some of the quirks with the pre-2.0 tuple construct (in particular, you can now have nested tuples as the last element of a tuple). To compensate for the porting hassle, the interpreter now implements tuples as C vectors (whenever possible), thus providing constant-time indexing and a much more space-efficient representation of sequences than can be achieved with lists (at the cost of slower matching and construction with large tuples, however; nothing is perfect ;-). Furthermore, concatenation, subscripting and the size operator now all work on tuples as expected, and the new built-in "Tuple" type can be used to match tuples of arbitrary sizes. There also is a new built-in sub function for extracting subsequences of strings, lists and tuples. Concatenation is now has its own operator symbol (++). I felt that this was necessary, in order to allow overloading of the (+) operator, e.g., for componentwise addition of lists and tuples. (This is already used in the new complex.q module, and will also turn out useful when implementing vector and matrix operations which might be added to the library in the future.) The --complete compiler option is gone, and the logical connectives (and) and (or) now evaluate their arguments completely (and require that both arguments evaluate to truth values). The short-circuit connectives are now called (and then) and (or else). So finally it is now possible to use both the "standard" and the short-circuit connectives in a single script, and the standard connectives now work in a manner consistent with the arithmetic operators. As already mentioned, the last three items above unfortunately break backward compatibility, and hence existing code will have to be rewritten, as has been done for the standard library. - C INTERFACE: A C interface via dll's resp. shared libs is now provided both for Windows and Unix systems supporting SUN's dlopen interface (SUN, Linux, maybe others?). The interpreter can load C functions from shared libraries at runtime, and provides an interface which lets such functions access and create Q expressions. External data types are supported as well. Please refer to the info file for details. - LIBRARY: Unfortunately, some of the language changes discussed above have had quite some impact on the standard library; I took the opportunity to straighten a few other parts of the library as well. Some functions related to pairs are gone, and the quoted app/list/tuple constructors have been moved as private functions to special.q, which is the only place where they are actually needed. The value function has been moved to special.q as well, and the cons function has been removed from stdlib.q (use push instead). Lexicographic list comparison is now part of stdlib.q, and stdlib.q now also provides comparison of truth values; the compare.q script has been removed. The lambda.q script has been overhauled once more, to prevent the function argument of a saturated _B or _S combinator in a special argument to be evaluated more than once, and to take care of embedded list and tuple constructors; I hope it finally works reliably now. Some changes have been made to the list and stream operations in stdlib.q and stream.q: + there is a new tuplecat operation in stdlib.q which concatenates a list of tuples + for compatibility with well-known languages like Haskell and MATLAB, each and exists have been renamed to all and any + the complete set of list functions (including push, reverse, etc.) is now available for streams as well + push now takes the element to be prepended as its first argument, and it now also works with tuples; pop, top, hd and tl work with tuples as well; a prepend function has been added which prepends an element given as second argument to a list given as first argument + new isempty predicates for lists, tuples and streams + the genlist function in stdlib.q now takes the predicate as the first argument; the range function is now called nums and can generate lists of both integers and floating point numbers; moreover, a numsby function has been added which allows you to specify the step size; similarly, the stream.q ints function has been replaced by numstream/numstreamby + the streamadd function has been renamed to streamcat2 + a list/stream transposition function has been added (works only with rectangular lists/streams, i.e., if all component lists/streams have the same size) The other user-visible changes in the library are: + a general purpose conditional function "switch" has been added to special.q + the constructor symbols of the stdtypes family (array.q, heap.q, set.q, bag.q, dict.q) are now public (looks less bewildering when printing such objects) + the istuple function has been added to type.q, and the pseudo type predicates isapp/iscons/ispair have been removed from that module + the inverse hyperbolic functions have been added to math.q, and the fmod function has been removed; the constants e and pi have been removed as well, as they can now easily be defined in the qinitrc file + a complex.q module has been added, which implements complex numbers as pairs of integer and floating point numbers; this still needs some testing, but I hope you'll like it ;-) + the std.q script now also includes graphics.q + the FLUSHPAGE variable has been removed from graphics.q; moreover, the GRAPHICS variable is not assigned a value any more, so that you *always* have to define this variable before you can use any of the output operations + plain.q is gone, instead there is a new script named prelude.q, which is by default included whenever a script is compiled or run (see also "COMMAND LINE INTERFACE" above); as distributed, this script now includes std.q so that all standard library functions are normally available in the interpreter, without having to include them explicitly - DOCUMENTATION: 5th edition of "The Q Programming Language", also updated man page and README file ------------------------------------------------------------------------------ * 1.9 Jan 1999 This is mainly a maintenance release and should be 100% backward-compatible to Version 1.8 (except for the discontinued Symantec C++ support, see below). Bug fixes: - corrected a =/== typo in path searching code (qbase.c) [thanks to Winfried Szukalski ] - qualified identifiers are now recognized correctly in the interpreter (qmlex.l) - interpreter in interactive mode now flushes its output and the integrated debugger runs correctly even if stdin is not a tty (qm.c, qmfuns.c) - SIGTERM is now handled to shut down compiler and interpreter in a clean manner (q.c, qc.y, qi.c) Changes: - Additional built-in flush and fflush functions. - Additional "b" (binary flag) for the file mode in fopen and popen. This flag suppresses the LF/CR-LF conversion on MS-DOS/Windows system, which is essential if the file is to be used as a binary file. On UNIX systems, this flag is ignored. - Added device settings in graphics.q for Windows; uncomment these (and comment out the corresponding UNIX definitions) when running under Windows. - Cleaned things up a bit, revived support for DJGPP, and added support for MS Visual C++. The support for Symantec C++ has been dropped, since I am no longer able to maintain it. - Added support for synchronizing with the parent process under MS Visual C++. The system interface (sys.{h,c}) now contains some MSC-specific code to implement a safe simulation of SIGINT and SIGTERM on Windows95/98/NT, which is used by the Qpad application (see below). - Updated documentation (fourth edition of "The Q Programming Language"). BIG NEWS: A new GUI frontend to the Q compiler and interpreter is now available for Windows95/98/NT -- check out the qpad application on the Q homepage. (Actually, I cannot test whether this program runs under NT, so any feedback of NT users is appreciated.) ------------------------------------------------------------------------------ * 1.8 Apr 1997 Yes, Q is still alive. ;-) This version fixes some things (mostly cosmetic changes). - Changes to the distribution: The Makefile now also includes a `html' target for creating documentation in HTML format. (Requires that you have the texi2html script.) The `dist' target now requires that you have gzip to build a tar.gz file. The Emacs Q mode (`q-mode.el') has been improved considerably. It now supports syntactical fontification (using the font-lock mode) and commands for locating the source lines referred to in the interpreter window. The latter feature also depends on the improvements in the 1.8 debugger (see below). - Changes to the programming tools: The notation `#!