Dan's Musings

Release Common Lisp on Your First Day

Published on

2023-05-07T09:41:14-0600

Introduction

This article presents a procedure for how to release a Common Lisp application starting from nothing within an hour. It also is intended to be a guide for how to get programmers who are new to Common Lisp set up with proper tooling around testing and packaging their code.

Since this guide may or may not be used in a corporate setting, it is assumed that the developer may or may not have control over which operating system their laptop uses. Therefore, this guide is written to be cross-platform as much as possible. This guide supports the Linux, Mac, and Windows operating systems.

This guide makes more opinionated recommendations on tooling than other guides extant in the literature. These opinions are intended to maximize the speed of set-up and chances of successful deployment in the face of cross-platform and other issues.

Audience

This guide assumes that the audience is comprised of programmers who know how to program in at least one other language and how to copy commands into a command prompt.

Procedures

Windows Users: Before You Begin

This section only applies to Windows users.

Much of the Common Lisp tooling works on Windows, but is Unix-native. As such, it will get very confused sometimes when working with directory path names which contain spaces. Please make sure the USERPROFILE directory (Typically this folder is C:\Users\<username>) doesn't have any spaces in it.

If it does, and the laptop is not a work laptop, the following steps may be used to mitigate the problem:

  1. Create a local user with a username that has no spaces
  2. Sign in with that user
  3. Log into your Microsoft Account from that user.

Thereafter, all Common Lisp development should be done using this user profile.

If it is a laptop owned by the user's employer, the employer or their IT department should be consulted about how to deal with this problem.

Install The Compiler, Package Manager, and Unit Test Tools

Follow these instructions on the developer laptop.

  1. Install Roswell (lisp installler), SBCL (the compiler), Qlot (the package manager), and Rove (the unit test tooling). For this part, open up a shell and copy the following commands into it.
    • Linux:
      1. Install Roswell using your favorite package manager. Failing that, download the binaries from latest release and put them on the PATH.
      2. Run ros install sbcl-bin.
      3. Run ros install sbcl.
      4. Run ros install qlot.
      5. Run ros install rove.
    • Mac:
      1. Run brew install roswell (get HomeBrew if you haven't yet). Failing that, download the binaries from latest release and put them on the PATH.
      2. Run ros install sbcl-bin.
      3. Run ros install sbcl.
      4. Run ros install qlot.
      5. Run ros install rove.

    • Windows:

      1. Run scoop install roswell from a CMD prompt (get Scoop here if you haven't yet). Failing that, download the binaries from the latest release and put them on the PATH.
      2. Install MSys2 by running scoop install msys2.
      3. Run ros install sbcl-bin.

      4. At the time of writing, the latest version of Roswell is 22.12.14.113. It doesn't have an important pull request for Windows yet, so we need to make the patch ourselves. Replace the contents of the file %USERPROFILE%\scoop\apps\roswell\current\lisp\install+msys2.lisp with the following: ``` (roswell:include "util-install-quicklisp") (defpackage :roswell.install.msys2+ (:use :cl :roswell.install :roswell.util :roswell.locations)) (in-package :roswell.install.msys2+) #-win32 (progn (warn "msys2 is only required on windows"))

        (defvar msys2-arch) (defvar msys2-bits)

        (defun msys2-get-version () '("20230127")) ;;sha1 "309f604a165179d50fbe4131cf87bd160769f974" ;;(ironclad:byte-array-to-hex-string (ironclad:digest-file :sha1 path))

        (defun msys2-setup (argv) (let ((uname-m (roswell.util:uname-m)) (msys2-bits (or (and (ros:opt "32") 32) (and (ros:opt "64") 64) (cond ((equal uname-m "x86-64") 64) ((equal uname-m "x86") 32)))) (msys2-arch (if (= 32 msys2-bits) "i686" "x86_64")) (path (merge-pathnames (format nil "archives/msys2-~A.tar.xz" (getf argv :version)) (homedir))) (msys (merge-pathnames (format nil "impls/~A/~A/msys2/~A/" (uname-m) (uname) (getf argv :version)) (homedir)))) (if (probe-file (merge-pathnames (format nil "mingw~A/bin/gcc.exe" msys2-bits) msys)) (format t "msys2 have been setup~%") (progn (format error-output "Download ~a~%" (file-namestring path)) (force-output error-output) (when (or (not (probe-file path)) (opt "download.force")) (download (format nil "~Amsys2/Base/~A/msys2-base-~A-~A.tar.xz" (msys2-uri) msys2-arch msys2-arch (getf argv :version)) path)) (format t " done.~%") (expand path (ensure-directories-exist (merge-pathnames (format nil "impls/~A/~A/msys2/" (uname-m) (uname)) (homedir)))) (format t "extraction done.~%") (ql-impl-util:rename-directory (merge-pathnames (format nil "impls/~A/~A/msys2/msys~A" (uname-m) (uname) msys2-bits*) (homedir)) (merge-pathnames (format nil "impls/~A/~A/msys2/~A" (uname-m) (uname) (getf argv :version)) (homedir))) (uiop/run-program:run-program `(,(uiop:native-namestring (merge-pathnames "usr/bin/bash" msys)) "-lc" " ") :output t :error-output t)

                 (dotimes (i 3)
           (uiop/run-program:run-program
            `(,(uiop:native-namestring (merge-pathnames "usr/bin/bash" msys))
              "-lc"
              ,(format nil "~@{~A~}"
                       "pacman --noconfirm "
                       "-Suy autoconf automake pkg-config "
                       "mingw-w64-" *msys2-arch* "-gcc "
                       "make zlib-devel"))
            :output t
            :error-output t))

         ;;(uiop/run-program:run-program
         ;; `(,(uiop:native-namestring (merge-pathnames "autorebase.bat" msys)))
         ;; :output t
         ;; :error-output t)
         (setf (config "msys2.version") (getf argv :version)))))
 (cons t argv))

        (defun msys2-help (argv) (format t "~%") (cons t argv))

        (defun msys2+ (type) (case type (:help '(msys2-help)) (:install `(,(decide-version 'msys2-get-version) msys2-setup)) (:list 'msys2-get-version))) ```

      5. Run the ros install sbcl command from a CMD prompt.

      6. Run the ros install qlot command from a CMD prompt.
      7. Run the ros install rove command from a CMD prompt.

      NOTE: Roswell install commands (those beginning with ros install) must be run from a CMD prompt, but all other commands, especially roswell-based commands such as qlot and rove, should be run from MSys2.

      All further commands in this guide should be run from an msys2 mingw64 shell when on Windows.

Set up the Project Files

These steps walk through setting up a new Common Lisp project. This guide assumes the user is using git. In the steps, we will create a new project called ip-finder. This example is intended to fully show how to get a project into a releasable state. The project ip-runner as shown below prints out the public IP address of the caller as shown by https://icanhazip.com. The code laid out in this section is available as a git repository.

  1. Make a directory called common-lisp. (Making this exact folder name at a specific location is important due to ASDF system searching rules). This directory will house all of our projects written in Common Lisp.

    • Mac and Linux: Put this in the HOME folder, as in ~/common-lisp.
    • Windows: Put this in your USERPROFILE folder (typically the C:\Users\<username>), as in C:\Users\<username>\common-lisp.

      NOTE: If the USERPROFILE directory has spaces, it will cause roswell to malfunction. See "Windows Users: Before You Begin" above.

  2. Pick a name for the new Common Lisp project. This name should use all lower-cased characters in kebab case. In this document, we will use the name ip-finder, and create a folder with that name.

  3. From within the project folder created in step #2, make two folders, one called src and another called tests.

  4. Add a LICENSE file, if the code will be released open source. In our example, we will use the MIT License for the contents of our LICENSE file. Be sure to substitute fields in angle brackets when copying the below text.

    Copyright <year> <author>

Permission is hereby granted, free of charge, to any person obtaining a
copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be included
in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

  5. Optionally, add a README.md file, which in our example might look like this:

    # IP Finder
This project is written in Common Lisp. It is exists primarily as a
demonstration for how to set up Common Lisp Projects.

# How To Build From Source
Run `ros build ip-finder.ros`. This command will create an executable
named `ip-finder` which can be run to obtain the public IP address of
the caller as observed by https://icanhazip.com/ .

  6. From within the project folder created in step #2, copy the following content into the following files for Git:

    • .gitattributes:

      * text=auto

# Linux, Mac
*.sh text eol=lf

# Windows
*.ps1 text eol=crlf
*.bat text eol=crlf

# All OSes
*.lisp text
*.asd text
*.md text
*.lock text
qlfile text
.gitignore text

# Images
*.png binary
*.jpg binary

# Lisp build (and dependency) files

*.abcl binary
*.fasl binary
*.dx32fsl binary
*.dx64fsl binary
*.lx32fsl binary
*.lx64fsl binary
*.x86f binary

*.exe binary
*.dll binary
*.so binary

    • .gitignore:

      # Lisp build (and dependency) files

*.abcl
*.fasl
*.dx32fsl
*.dx64fsl
*.lx32fsl
*.lx64fsl
*.x86f

*.exe
*.dll
*.so
# Editor save filees

*~
.#*
.*.sw[a-z]

# Qlot files
/.qlot/
/.bundle-libs/

  7. Add an ASD file. This file tells Common Lisp (in particular, ASDF) how to find our code. The file will be named <project-name>.asd and should reside in the root folder of the project. In our case, the file will be named ip-finder.asd. Be sure to change the contents to fit project needs.

    (defsystem "ip-finder"
           :version "0.1.0"
           :author "Daniel Jay Haskin"
           :license "MIT"
           :depends-on (
                        "dexador"
                        "cl-yaml"
                        )
           :components ((:module "src"
                                 :components
                                 ((:file "main"))))
           :description "IP address finder."
           :in-order-to ((test-op (test-op "ip-finder/tests"))))

(defsystem "ip-finder/tests"
           :version "0.1.0"
           :author "Daniel Jay Haskin"
           :license "MIT"
           :depends-on (
                        "ip-finder"
                        "rove"
                        )
           :components ((:module "tests"
                                 :components
                                 ((:file "main"))))
           :description "Test system for ip-finder"
           :perform (test-op (op c) (symbol-call :rove :run c)))

Notice in the file how there are two systems created. One is the main system and the other is the test system. In the test system, the main system is listed as a dependency. Be sure to do the same when customizing this file for other projects.

Further, notice in our example how there are two dependencies in the main system. One is Dexador, an HTTP client library in common use. Another is a library called cl-yaml which allows programs to read and write YAML documents.

  1. Add a file named qlfile in the root directory. This will list out our dependencies and tell Qlot where to find them. See the Qlot documentation for more information. Put the following content in it:
    ql dexador
git dexador https://github.com/eudoxia0/cl-yaml.git :branch master

In our example project, we list two dependencies, dexador and cl-yaml. The dexador line is prefixed with ql, indicating it should be downloaded from Quicklisp. By contrast, the cl-yaml line is prefixed with git, indicating that it will be downloaded directly from its git repository location. (In reality, cl-yaml can also be had from Quicklisp, and should generally be gotten from there. This is merely for demonstration, to show that git repositories can be dependended upon directly.)

  1. Add the main source file named main.lisp in the src directory. Give it the following contents:

    (in-package #:cl-user)
(defpackage
  #:ip-finder
  (:use #:cl)
  (:documentation
    "
    The ip-finder command gets the public IP address from the caller as
    observed by `icanhazip.com`.
    "
    )
    (:import-from #:dexador)
    (:import-from #:cl-yaml)
    (:export
      main))
(in-package #:ip-finder)

(defun main (&key
              ;; Dependency Injection for tests
              (out-stream t)
              (http-get #'dexador:get)
              (url "https://icanhazip.com"))
    "Get my IP address."
    (let ((ip (string-trim
               #(#\Return #\Newline #\Tab #\Space)
              (funcall http-get url))))
      (format out-stream "ip: ~A" (cl-yaml:emit-to-string ip))))

  2. Add the main test source file named main.lisp in the tests directory. Give it the following contents:

    (in-package #:cl-user)
(defpackage #:ip-finder/tests
  (:use #:cl
        #:rove)
  (:import-from
    #:ip-finder))

(in-package :ip-finder/tests)

(deftest
  main
  (testing "main"
    (ok (equal (ip-finder:main :out-stream nil :http-get (lambda (addr)
                                    (declare (ignore addr))
                                    "  127.0.0.1\n\n\t "))
              "ip: \"127.0.0.1\""))))

  3. Run ros init in the root of the project with <project-name>.ros as the argument, as in ros init ip-finder.ros.

  4. Edit the file ip-finder.ros to add the following line to the section labeled init forms:

    (asdf:load-system "ip-finder")

    Add another line in the main function to call our main function that we wrote in the file src/main.lisp as well, so that it looks like this:

    #!/bin/sh
#|-*- mode:lisp -*-|#
#|
exec ros -Q -- $0 "$@"
|#
(progn ;;init forms
  (ros:ensure-asdf)
  #+quicklisp(ql:quickload '() :silent t)
  (asdf:load-system "ip-finder") ;; <-- THIS WAS PUT IN BY US
  )

(defpackage :ros.script.ip-finder.3892512754
  (:use :cl))
(in-package :ros.script.ip-finder.3892512754)

(defun main (&rest argv)
  (declare (ignorable argv))
  (ip-finder:main)  ;; <-- THIS WAS ALSO PUT IN BY US
  )
;;; vim: set ft=lisp lisp:

Build and Test the Project

These steps walk through testing and building a new Common Lisp project. At the end, we will have created a binary executable file which may be installed on any machine sharing the same operating system and architecture as that of the machine upon which it was built.

  • Install the project dependencies.

    • Linux and Mac:
      1. Run qlot install.
      2. Run qlot bundle.
    • Windows:
      1. Run qlot install.
      2. Run qlot bundle.

      3. Windows needs access to the LibYAML DLL. To do this, run the following commands in a MinGW64 prompt:

        pacman -S mingw-w64-x86_64-libyaml # Do this after libyaml is installed cp /mingw64/bin/libyaml-0-2.dll <project-root>/libyaml.dll

  • To run a REPL with the written code loaded, run the following:

    ros run

Then, from inside the REPL, run

    (load ".bundle-libs/bundle.lisp")
    (asdf:load-system "ip-finder")

  • To run the unit tests, run the command rove tests/*.lisp from the root directory of the project.

  • To build a self-contained executable, run the command ros build ip-finder.ros on the command line. This will build a self-contained executable called ip-finder. It will but this executable in the current directory, which should be the project root.

  • To run the program after it is built, run the <project-name> binary, as in ./ip-finder. It should output something like ip: "<ip-address>".

Conclusion

This document has outlined, from start to finish, how to create a deployable artifact from scratch in Common Lisp. Notable is what the guide has left out. Setting up the editor tooling (whether emacs-based, vim-based, or something more non-standard) is left as an exercise to the reader. Deploying a web service is a topic left for another day. However, it does get the reader started and building something useful right from the start.