In a ltetter on ACM.org the ACM President Cherri Pancake wrote: “We believe that ACM can help support research, discovery and learning during this time of crisis by opening the ACM Digital Library to all.”

And support they do.

After searching for a few things I’ve been surprised to discover a wealth of LISP-related papers in the proceedings of “LFP: Conference on LISP and Functional Programming“.

This conference produced 287 publications in the span of 1980-1994. Out of them all, 276 publications are available for download.

It featured quite a number of prominent contributors, discussed plenty of topics that are still relevant today, and overall provides a very interesting window into the age of glory, when LISP machines were a thing and the future for AI looked promising. Unfortunately, it also captured artifacts of IT archeology: “Tachyon Common Lisp: an efficient and portable implementation of CLtL2” that was claimed to be:

an efficient and portable implementation of Common Lisp 2nd Edition. The design objective of Tachyon is to apply both advanced optimization technology developed for RISC processors and Lisp optimization techniques. The compiler generates very fast codes comparable to, and sometimes faster than the code generated by UNIX C compiler

nowdays is nowhere to be found. But “The Python compiler for CMU Common Lisp”
fotunately has survived the test of time and in some form or shape can be found in SBCL.

Some articles have abstracts as if written last year, not 1984:

As the need for high-speed computers increases, the need for multi-processors will be become more apparent. One of the major stumbling blocks to the development of useful multi-processors has been the lack of a good multi-processing language—one which is both powerful and understandable to programmers.

Overall, “LFP: Conference on LISP and Functional Programming” is a very interesting thing to read. Don’t miss this oportunity as access will be open only through June 30, 2020.

https://dl.acm.org/conference/lfp

A new way of blogging about Common Lisp by Yehonathan Sharvit (@viebel).

A canonical answer, as formulated by the Practical Common Lisp, section 14. Files and File I/O is essentially the same as the one provided by the Common Lisp Cookbook (Reading a File one Line at a Time):

(let ((in (open "/some/file/name.txt" :if-does-not-exist nil)))
  (when in
    (loop for line = (read-line in nil)
        while line do (format t "~a~%" line))
    (close in)))

And basically it does the job.

But what happens if you deal with a log that has captured random bytes from a crashing application? Lets simulate this scenario by reading from /dev/urandom. SBCL will give us a following result:

debugger invoked on a SB-INT:STREAM-DECODING-ERROR in thread
#:  :UTF-8 stream decoding error on
#:   the octet sequence #(199 231) cannot be decoded.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [ATTEMPT-RESYNC   ] Attempt to resync the stream at a character boundary
                         and continue.
  1: [FORCE-END-OF-FILE] Force an end of file.
  2: [INPUT-REPLACEMENT] Use string as replacement input, attempt to resync at
                         a character boundary and continue.
  3: [ABORT            ] Exit debugger, returning to top level.

(SB-IMPL::STREAM-DECODING-ERROR-AND-HANDLE # 2)

The same will be reported on other Lisp implementations. However, dealing with this problem is not really portable, and requires platform-specific switches and boilerplate code.

For example, on SBCL it is possible to specify a replacement character in the external-format specification:

(with-open-file (in "/dev/urandom"
                      :if-does-not-exist nil
                      :external-format '(:utf-8 :replacement "?"))
  ;; read lines
)

Other Lisps require a different and incompatible external format specification.

But there are actually other ways to read a file line-by line. cl-faster-input looks into some of them. Namely:

• A standard read-line.
• read-line-into-sequence suggested by Pascal Bourguignon in a cll discussion. Unlike the standard read-line this function reads lines into a pre-allocated buffer, reducing workload on the garbage collector.
• read-ascii-line that is the part of the COM.INFORMATIMAGO.COMMON-LISP.CESARUM library.
• ub-read-line-string from the ASCII-STRINGS package that is a part of the CL-STRING-MATCH library

Please check the src/benchmark-read-line.lisp in the sources repository.

Benchmarks show that the ub-read-line-string outperforms the standard read-line approach, does not require platform-specific switches, and allows trivial character substitution on the fly (like up/down casing the text, replacing control characters etc.)

Sample usage (from the sources):

(with-open-file (is +fname+ :direction :input :element-type 'ascii:ub-char)
    (loop with reader = (ascii:make-ub-line-reader :stream is)
       for line = (ascii:ub-read-line-string reader)
       while line
       count line))

On developer’s desktop it takes 1.71 seconds to complete the benchmark with the standard read-line, and 1.076 seconds with the ub-read-line-string benchmark. Memory consumption is on the same level as the standard read-line, though significantly higher than the read-line-into-sequence.

On Clozure CL 1.9 the read-ascii-line benchmark fails. The ub-read-line-string falls into an infinite loop.

On Embeddable CL 16.0 all functions work, but the ub-read-line-string takes almost 10 times more time to complete than any of the alternatives.

Conclusion: It might be reasonable to look at different approaches for reading files line-by-line if you plan to deal with large volumes of text data with a possibility of presence of malformed characters. Check the sources of cl-faster-input for different ideas, tweak and run the benchmarks as it suits your tasks.

P.S. this post has been written in September of 2015 but never published. As it appeared to be pretty complete I decided to post it now, in the January of 2018. Stay tuned…

These functions scan input according to a provided format string. The format string might contain conversion specifiers or conversion directives to extract integers, floating-point numbers, characters, strings, etc. from the input and store it in the arguments.

For example, a format string for parsing components of an IP address might look like: “%3d.%3d.%3d.%3d” — the scanf function will parse 4 integers (maximum 3 digits each) that are delimited with dots and return them to the caller.

There are basically two ways to implement scanf:

1. as an interpreter, that scans format string and executes commands as they are retrieved.
2. as a translator to an intermediate language that, in turn, is compiled into machine code.

The trivial-scanf package (that comes as the part of the CL-STRING-MATCH library) takes the first approach. The trivial-scanf implementation reads one character at a time, and depending on the read character performs the designated operation. Underneath, it uses PROC-PARSE library to deal with the input. Outline of the function’s main loop looks as follows:

(iter
 (while (< fmt-pos fmt-len))
 (for c = (char fmt fmt-pos))
 (case c
   (#\%
    ;; process conversion directive
    )
   ((#\Space #\Tab #\Return #\Newline #\Page)
    ;; process white space characters
    )
   (otherwise
    ;; process ordinary characters
    )))

Conversion directives might have optional flags and parameters that must be taken into account. Simple directives, like %d, are handled in a straightforward way: input matching to the designated data type (digits) are bound to a string that is then parsed using corresponding function (parse-integer in this case).

However, the standard scanf also specifies a directive to match a set of designated characters. For example, directive ‘%[a-z0-9-]’ would scan input and return a string composed of letters, digits, and a dash from the current position, until first mismatch. In case, if we dealt with an octet-string (a string where every character is guaranteed to be a single byte in size), it would be feasible to interpret this directive using a table to mark characters that belong to the set. The trivial-scanf takes another approach: characters set directive is converted into a list of closures that serve as predicates for the input string binding operation. In our example, the list of closures would contain predicates for: (range #\a…#\z), (range #\0…#\9) (character #\).

trivial-scanf will be accessible through Quicklisp after the next packages update. At the moment you can clone the repository and install it locally.

Some usage examples:

(ql:quickload :trivial-scanf)
(snf:scanf "%3d.%3d.%3d.%3d" "127.0.0.1") => (127 0 0 1)
(snf:scanf "%d %[A-C] %d" "1  ABBA  2") => (1 "ABBA" 2)

This the first (almost alpha) release of the code, so some bugs are expected. Feel free to comment or submit them.

trivial-scanf is the part of the CL-STRING-MATCH library.

There is still a long path to walk so any contributions are welcome. Please take a look at the list of open issues and feel free to either comment or join.

Since SVG is an application of XML and XML is closely related to both SGML and HTML/XHTML one could suggest that it might be possible to use standard full-featured HTML-generators. Indeed, it is, and here is how.

CL-WHO of Edi Weitz actually transforms a tree of S-exp into a sequence of commands that ouput the corresponding XML/SGML into the given stream. And this is exactly what is needed to produce SVG from Lisp!

Let’s define a stream where the SVG data will be serialized:

(defparameter *svg* *standard-output*)

Now let’s define a macro that will tune CL-WHO to do the job we want it to do:

(defmacro with-svg (&body body)
  `(progn
     (format *svg* "<?xml version=\"1.0\" standalone=\"yes\"?>~%~%")
     (who:with-html-output
	 (*svg* nil :prologue
		"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">"
		:indent T)
       ,@body)))

Here we print an XML declaration because CL-WHO does not and we ask it to use prologue that is required by SVG. Indentation is optional and is useful for debugging purposes.

The with-svg macro is designed to envelope chunks of SVG code. Let’s translate this example for this macro:

(with-open-file (*svg* "out.svg" :direction :output
		       :if-exists :supersede
		       :if-does-not-exist :create)
  (with-svg 
    (:svg :width "4in" :height "3in" :version "1.1"
	  :xmlns "http://www.w3.org/2000/svg" :|xmlns:xlink| "http://www.w3.org/1999/xlink"
	  (:desc "This graphic links to an external image")

	  (:image :x "200" :y "200" :width "100px" :height "100px"
		  :|xlink:href| "myimage.png"
		  (:title "My image")))))

And don’t forget that empty tags are defined by the who:*html-empty-tags* list.

One last note is that this very approach is suitable for generating arbitrary XML with a truly universal tool that is CL-WHO.

• ZS3 1.0.7 is released.
• CLAZY is cleaned up a bit.

Announcements

• Next Boston Lisp Meeting: 2009-03-30 1800 at MIT 34-401B – Carl Eastlund on Modular ACL2.

Discussions

• What open source implementation of Lisp do you prefer and why?.
• Sequential sharpsign-plusses?.
• John Backus and function level programming.
• SBCL Slime and emacsW32.
• help with function please.

Jobs

• A number of Lisp and Lisp-related jobs: 1 2 3 4.

• Adrienne released the solutions to Lisp 3rd edition by Horn and Winston, see: NAILS
• clnuplot-0.2.2 is available.
• lift and moptilities.
• Ready Lisp version 20090130 now available.

Announcements

• European Lisp Symposium 2009.
• International Lisp Conference 2009.

Demos

• Parenscript tricks and parallelism.

Discussions

• Why Language Extensibility Matters.
• Any Lisp tutorials?
• How to put tab into format.
• Fexprs more flexible, powerful, easier to learn? See also challenged by Common Lispers.
• Input file with commas.
• What gives you power in CL, that you don’t have in other languages.
• Learning Lisp the hard way.
• Writing a Windows Service in Common Lisp.
• Kent Pitman’s essay on why lisp doesn’t have copying of lists a comment by Xah Lee.
• How to tokenize a string?
• How to COMPILE-FILE without a file?

Jobs

• Two good jobs at ITA Software.
• One job at Accenture.
• One more job.

• Talks for the international Lisp conference.

Demos

• plot-widget on demo on Allegro CL.

Discussions

• A Symbolics Lisp machine is being sold.
• Lisp on the JVM – a list of Lisp implementations on the JVM.
• Processing XML-formatted documents in CL.
• Do two uses of backquote return the same result?
• The preference of protected-form and cleanup-form in unwind-protect.
• How to efficiently parse a CSV file into lists?
• How to access several slot values in a single dispatch?
• Simple XML Serialization for a hashtable.
• How to create function args from a lisp.
• A nice algorithms question from Kenneth Tilton.

• ITA Software will start distributing software for the Free Software World (Qitab – the CL.net project).
• ANN: Mudball updates.

Demos

• Gruff – and RDF graph browser from Franz Inc.

Discussions

• Lisp meetings coming in the next week(s).
• A Loop curiosity
• How to build ECL on Windows?
• Unicode and CL compatibility.
• What ARM Lisp compilers are there?
• What Lisp to use on Windows?
• How to check whether there is ASDF installed?
• “Lisp server pages” implementations.
• Performance differences between struct and CLOS.
• Which graduate program uses Lisp as research tool?

Jobs

• New LispJobs: 1 and 2.