Shiny Things

Emacs Book Intro

Wed, 17 Apr 2024 00:00:00 -0400

Here’s something I wrote for the introduction to the Emacs book I started years ago and will never, ever finish writing (“Emacs Mastery: Attaining Coding Supremacy”):

Emacs is a master craftsman’s tool. No, wait — Emacs is an empty canvas, complete with brushes, paints (oil and acrylic), charcoal, scrapers, sponges, turpentine, and a nuclear-powered color wheel.

Since I’m a musician and not a woodworker or painter, here’s an analogy I can get behind: Emacs is a synthesizer. At first, you play the presets and have fun making music. You don’t need all those knobs, but you do wonder what they’re for. One day you’re unhappy with the preset bass sound you’re using — it needs a sharper attack, a longer decay, and a brighter sound. You tweak a few knobs. They don’t do what you want at first, but after a bit of reading the manual (ugh!) you start to understand what some of them do and you achieve that sound you were looking for.

After reading more of the manual you begin to understand how the synth creates sounds and what’s really going on when you turn those knobs. You start finding things that the synth can do that you never imagined it could. Not only that, getting the sounds you want is easier so you’re spending less time tweaking knobs and more time making music.

Meanwhile, you’ve come across terms like “subtractive synthesis”, “wavetables”, and “frequency modulation” so you start reading up on what those things mean. You’re learning how sound synthesis works and that there are different ways of creating sounds. This knowledge helps you create new sounds and textures, opening up a universe of sound explosions you use to create mega-hit after mega-hit making you so mega-rich you mega-retire and give all your money to charity. A combination music/philanthropy prize is named after you and statues to you are erected everywhere.

That’s Emacs.

I’ve been using Emacs since about 1982, and I still discover new things it can do or new ways to be more efficient. It may not be possible to know all there is to know about Emacs, but that’s not the point. The point is to have a tool that helps me create what I want and be more efficient so I can create more. And knobs. Lots of knobs.

Asynchronous Audits are Awesome

Wed, 03 May 2017 00:00:00 -0400

Audited is a Ruby gem that adds change logging to Rails models. It’s great for figuring out what happened to a model or a group of related models over time.

This post was originally published on The Dirty Birds, the Chloe + Isabel tech team blog.

If you have an application that generates a lot of audit records they can start to slow down your application a bit. Not only do the audit records have to get written to the database, but since they are versioned the save process needs to read the database first to get the current max version number before the new record can be written.

Making these saves asynchronous can improve your Rails app’s response times. Testing with our application showed a 14% speedup with audit-intensive code. That is of course an anecdote, not data.

We had originally forked the Audited gem and modified it to work with multiple asynchronous job processing libraries (we use Resque). After we opened a pull request and some great help from project members, a discussion took place in the PR about whether the code should be merged into the Audited gem’s code base.

After some thoughtful comments pro and con, we decided to close the PR. Instead, we’ve come up with a way to do this that is simpler and doesn’t require that the gem be changed at all.

Show Me the Code!

To force audits to be saved asynchronously we started by monkeypatching the audit model’s save method. It puts the data to be saved in the work queue to be written to the database later.

We use Resque to manage work queues and perform work asynchronously.

This code lives in config/initializers/audited.rb. (Note that this is not the final version.)

module Audited
  class Audit
    alias_method :orig_save, :save

    def save(*_args)
      Resque.enqueue(AuditingJob::AsyncSave, attributes)
      true
    end
  end
end

The asynchronous job does the actual save, using orig_save.

class AuditingJob::AsyncSave
  # ...boilerplate removed...

  def self.perform(audit_attributes)
    Audited::Audit.new(audit_attributes).orig_save
  end
end

This works, but if you run this you’ll notice at least two problems. First, the created_at is set during the save in the asynchronous job which means that it is not the time that the original audit record’s save method was called. Second, the fields related to the user who made the audit and the request itself (IP address and request UUID) are blank. That’s because the Audit gem’s save code gets that information from Rails’ current controller. The problem is that inside this asynchronous job there is no “current controller”.

The solution is to set those values in the monkeypatched save method. Setting the created_at attribute directly is easy. As for the other values, though the save method doesn’t directly have access to the current controller the Audited gem already has a way to get it. We can use that to get information about the current user and the request.

One final thing we need to do is muck with the datetime attributes before sending the to the async job. Datetimes are serialized as strings, but the string format of a Time object is not directly readable by the database. So we turn it into a format readable by the database. (We’re jumping ahead a bit — we will be using a SQL INSERT directly; see below.)

Here’s a modified version of the monkeypatched save method:

module Audited
  class Audit
    alias_method :orig_save, :save

    def save(*_args)
      self.created_at ||= Time.current
      ctl = Audited.store[:current_controller]
      if ctl
        self.user_id ||= ctl.current_user&.id
        self.user_type ||= ctl.current_user&.class&.name
        self.request_uuid ||= ctl.request&.uuid
        self.remote_address ||= ctl.request&.remote_ip
      end
      self.username ||= self.user&.email

      # Need to hack datetime values so they're encoded properly
      attrs = attributes
      attrs["created_at"] = attrs["created_at"].to_s(:db)

      Resque.enqueue(AuditingJob::AsyncSave, attrs)
      true
    end
  end
end

We’re using Ruby’s safe navigation operator (&.) which was introduced in Ruby 2.3.0. If you’re running an earlier version of Ruby you can use the Rails try method instead.

Now that we’ve set all of these values, we need to prevent Audited from overwriting them when it saves the record in our asynchronous job. To do that, we use a SQL INSERT statements instead of calling the Audited model’s save (via orig_save). That means we have to not only create the INSERT statement but also determine the version number for the record we are about to save.

Here’s the new version of the asynchronous job:

class AuditingJob::AsyncSave
  # ... boilerplate removed...

  def self.perform(audit_attributes)
    # Using raw SQL is not only simpler, but it also bypasses Audited code
    # that (re)sets some values we don't want touched.
    Audited::Audit.transaction do
      audit_attributes["version"] =
        get_version(audit_attributes["auditable_id"], audit_attributes["auditable_type"])
      conn = Audited::Audit.connection
      column_names = audit_attributes.keys
      safe_values = audit_attributes.values.map { |v| conn.quote(v) }
      conn.execute "INSERT INTO #{Audited::Audit.table_name} (#{column_names.join(", ")})" +
                   " VALUES (#{safe_values.join(", ")})"
    end
  end

  def self.get_version(auditable_id, auditable_type)
    return 1 unless auditable_id && auditable_type

    curr_version = Audited::Audit
                     .where(auditable_id: auditable_id, auditable_type: auditable_type)
                     .maximum(:version)
    (curr_version || 0) + 1
  end
end

That’s it.

Phoenix - Seeding Your Tests

Tue, 08 Sep 2015 00:00:00 -0400

If you need to seed your test database when running Phoenix tests, you can require the seed file in test/test_helper.exs:

ExUnit.start

Mix.Task.run "ecto.create", ["--quiet"]
Mix.Task.run "ecto.migrate", ["--quiet"]
# Add this line:
Code.require_file("priv/repo/seeds.exs")
Ecto.Adapters.SQL.begin_test_transaction(Expensive.Repo)

You can, of course, require a different file instead.

Emacs and Rails Testing

Fri, 17 Jul 2015 00:00:00 -0400

I’m an inveterate Emacs user. It’s been my Swiss Army chainsaw of choice for over 33 years. Part of the beauty of Emacs is that it’s extensible and configurable. The relatively new¹ package manager and ELPA package archive are a great start, but sometimes it’s better — or simply more fun — to roll your own.

This post was originally published on The Dirty Birds, the Chloe + Isabel tech team blog.

I’ve developed a few Emacs Lisp functions that let me run our Rails RSpec tests at varying levels of granularity and within various environments.

Granularity: what to test.

Run all tests in a directory and its subdirectories.
Run all tests in a single file, by default the one that’s in the current buffer.
Run a single test or single context within a spec file.

Environment: where to run the test.

Within an Emacs compilation buffer, which has features like jumping right to the source code referenced in any error message output can be used to jump directly to the relevant line of code.
Within a shell running in an external terminal program such as the Mac OS X Terminal program or iTerm 2. Running within a shell instead of using compilation mode is necessary when you need to interact with the tests, for example if you insert a Byebug breakpoint in the code. To send text to these external programs, we’ll use AppleScript.
Within a shell running inside Emacs. This is left as an exercise for the reader.

Many of the code samples here come from my Emacs init files, which are available on Github at jimm/elisp. The bulk of this code can be found in ruby-testing.el and my-rails.el.

The Command

The first thing we need is the command we want to run. Variables that we will fill in <LOOK_LIKE_THIS>.

cd <RAILS_ROOT> && \
    echo > log/test.log && \
    RAILS_ENV=test bundle exec rspec --seed=<SEED> <RPEC_PATH_AND_LINE>...

The first line takes us to the root of the Rails project.

The second line empties out the log file without deleting it. If you move or delete a log file that Rails has open it won’t recreate a new one, so instead you can zero out the existing file by doing this. The third line runs the test.

The command to run the tests needs to specify a random seed and one or more paths and optionally lines within files. The random seed is used by RSpec to determine in what order tests are run. By default, we’ll use the value $RANDOM which is a magic variable defined by most Unix shells that returns a different value every time it is read. If a test fails sometimes but not others, that could be because the order of the tests matters to the outcome. (If that is the case, it’s a sign that the test needs to be rewritten so that it has no external or intra-test data dependencies.) In that case you may want to make sure you use the same random seed while debugging so that the tests are run in the same order.

RSPEC_PATH_AND_LINE can be a directory ($RAILS_ROOT/spec/models), file ($RAILS_ROOT/spec/models/foo.rb), or file plus line number ($RAILS_ROOT/spec/models/foo.rb:42). If a line number is specified, then the test(s) containing that line is run. The line number doesn’t have to be the first line of the test; any line in the test will do. Additionally, if the line number is outside of any single test but within a context or describe block, then all tests in that block will be run.

The rspec command can actually take multiple dir/file name arguments, but we’re only going to worry about running one right now.

Generating the Command

Here’s the elisp² code that generates the command. The function my-rails--rspec-command takes a seed value and a file name (or directory name or file-plus-line-number) and returns a string containing the command we want. It calls find-rails-root to search for the Rails root directory at or above the specified file’s directory. The two dashes in the function’s name are a convention that tells the reader that this is a locally-used internal function not intended for export or use by other code.

(defun my-rails--rspec-command (seed fname)
  (let* ((rails-root (find-rails-root (file-name-directory fname)))
         (rspec-cmd (if (file-exists-p (concat rails-root "bin/rspec"))
                        "bin/rspec"
                      "rspec")))
    (concat "cd " rails-root " && "
            "echo > log/test.log && "
            "RAILS_ENV=test bundle exec " rspec-cmd " "
            (my-rails--seed-arg-string seed) " " fname)))

The function that finds the Rails root directory at or above a file’s directory is find-rails-root. It uses the predicate function rails-root-p to determine if a directory is a Rails root directory. By long-standing Lisp convention, the names of predicate functions (those that return true or false) end with -p, like Ruby’s convention of using ?.

(defun rails-root-p (path)
  "Returns true if path is a Rails root directory. Uses a heuristic that
involves looking for known Rails directories."
  (and path
       (file-directory-p path)
       (file-exists-p (concat path "Rakefile"))
       (file-directory-p (concat path "app"))
       (file-directory-p (concat path "config"))
       (file-directory-p (concat path "lib"))
       (file-directory-p (concat path "public"))))

(defun find-rails-root (path)
  "Returns Rails root dir at or above PATH. Returns nil if PATH is nil or no
Rails root dir is found. Uses `rails-root-p'."
  (locate-dominating-file path #'rails-root-p))

The seed option for the command is generated by this function:

(defun my-rails--seed-arg-string (seed)
  "Returns \"--seed=SEED\". If SEED is 1, returns \"--seed=$RANDOM\"."
  (concat "--seed="
          (if (equal seed 1) "$RANDOM" (int-to-string seed))))

One more addition lets us run the test(s) that are at point³:

(defun my-rails--rspec-at-point-command (seed fname &optional line-num)
  (concat (my-rails--rspec-command seed fname)
          ":" (int-to-string (line-number-at-pos))))

Actually Running the Tests

Now that we can build the command we need, it’s simple to write functions that run the command for the current line, current file, or an entire directory.

There are two “environments” (a compilation buffer within emacs or an external terminal program) and really only two granularities (whatever is at point or a file/directory). We can abstract those options a bit by passing around functions. One pair of functions takes a command string and executes it in the proper environment. For those we use #'compile and #'send-to-iterm. The former function is built in to Emacs and the latter is defined below.

We could theoretically create another pair that determine granularity by returning the proper final argument for the command string, but when I starting thinking about doing that it seemed to me that the code would become unnecessarily obtuse.

(defun run-spec-using (run-func seed fname)
  "Run RSpec test FNAME from the rails root directory above it.
If SEED is 1, $RANDOM will be used. Calls
`my-rails--rspec-command' to generated the command to run.
RUN-FUNC must be a function such as `compile' that takes a string
and executes it."
  (funcall run-func (my-rails--rspec-command seed fname)))

(defun run-spec (seed fname)
  "Run RSpec test FNAME from the Rails root directory above it.
If SEED is 1, $RANDOM will be used. FNAME may contain extra line
number info (e.g., 'foo.rb::42')."
  (interactive "p\nF") ; possibly nonexistent file name so we can append ":NNN"
  (run-spec-using #'compile seed fname))

(defun run-spec-in-iterm (seed fname)
  "Run RSpec test at point in iTerm. If SEED is 1, $RANDOM will
be used."
  (interactive "p\nF") ; possibly nonexistent file name so we can append ":NNN"
  (run-spec-using #'send-to-iterm seed fname))


(defun run-spec-at-point-using (func seed)
  "Run RSpec test FNAME from the rails root directory above it.
If SEED is 1, $RANDOM will be used. Calls
`my-rails--rspec-at-point-command' to generated the command to run.
RUN-FUNC must be a function such as `compile' that takes a string
and executes it."
  (funcall func (my-rails--rspec-at-point-command
                 seed
                 (buffer-file-name)
                 (line-number-at-pos))))

(defun run-spec-at-point (seed)
  "Run RSpec test at point from the Rails root directory. If SEED is 1,
$RANDOM will be used."
  (interactive "p")
  (run-spec-at-point-using #'compile seed))

(defun run-spec-at-point-in-iterm (seed)
  "Run RSpec test at point in iTerm. If SEED is 1, $RANDOM will
be used."
  (interactive "p")
  (run-spec-at-point-using #'send-to-iterm seed))

interactive is a special form⁴ that tells the function what types the args are and how to retrieve or prompt for them. Since the seed argument for all of these functions is numeric and since I usually want to use the default value, we tell the functions to use their numeric or prefix argument for the seed parameter.

The final missing piece is send-to-iterm:

(defun send-to-iterm (str)
  "Send STR to the front window/session in iTerm. STR may contain
multiple lines separated by `\n'."
  (interactive "siTerm input: ")
  (let ((lines (split-string
                (replace-regexp-in-string "\"" "\\\"" str t t)
                "\n")))
    (do-applescript (concat
                     "tell application \"iTerm\"\n"
                     "	tell the current terminal\n"
                     "    tell the current session\n"
                     (mapconcat (lambda (s) (concat "write text \"" s "\"\n")) lines "")
                     "    end tell\n"
                     "	end tell\n"
                     "end tell\n"
                     ))))

Writing this article caused me to review the code and improve it by refactoring it a bit. I even found a glaring error in the design: I assumed that the “current line” was to be found by asking the current buffer, even when calling a function with a completely different file.

Emptying Full Client-Side Browser Storage

Wed, 04 Jul 2012 00:00:00 -0400

This morning I started seeing the following error on all preview pages generated by an app running on my development machine:

QUOTA_EXCEEDED_ERR: DOM EXCEPTION 22
in http://localhost:8080/sites/1341373150.915000/public_html/layout.less

This error message isn’t caused by the application, it’s because the generated HTML uses client-side LESS to render its CSS. The LESS processing had filled up local storage, because each preview generates new storage keys due to the ever-changing timestamp part of the URL.

The solution: here’s how to manually clear your browser’s local storage of any data whose keys match a particular value:

// Clears local storage of keys that match a regular expression.
function clear_local_storage(key_regexp, keep_key=null) {
  var r = new RegExp(key_regexp);
  for (var key in localStorage) {
    if (key != keep_key && r.test(key))
      localStorage.removeItem(key);
  }
}

A sample of its use:

clear_local_storage("^http://localhost:8080/sites.*layout.less(:timestamp)?$");

Since I wrote the app that’s publishing the pages that generate this error, I’m considering making the app inject this JavaScript code into the pages to make sure that end users never see this error message.

Identity Recall

Sat, 04 Jun 2011 00:00:00 -0400

io.com is going away. It’s funny how upsetting this is to me. I’ve been jimm@io.com since 1994 or so—maybe a year or two earlier than that. You know what I’m worried about most? All those open source projects, emails, and other digital resources that point to jimm@io.com are going to be pointing nowhere in a month. It feels like my online identity is being stolen. Except it’s not being stolen, of course—merely recalled.

A Slashdot commenter also pointed out the potential security risk: “Think about it, among other things whoever owns that domain now will be able to intercept all mail to io.com accounts, and with the quickness and suddenness of the transfer not everyone’s who uses those addresses is going to be able to completely transition off them before the transfer happens”

io.com was the ISP run by Steve Jackson Games—the company raided by the Secret Service because they were writing a role playing game called “Hacker”. SJ Games fought the Secret Service and won. I think the EFF grew out of that case.

io.com was bought by prismnet.com years ago. PrismNet changed hands a few times. The last guy who sold it to the current owner (for $20) didn’t sell the io.com domain. He kept it but let them use it until July 1, 2011. My guess is he wants to sell it to I/O Digital for a skillion dollars.

From now on, I’m jim@jimmenard.com. Also jim.menard@gmail.com, or jimm@prismnet.com. Time to start wrapping my head around being jim@jimmenard.com instead of jimm@io.com. Bye, jimm@io.com.

Play Framework Configuration

Sun, 25 Jul 2010 00:00:00 -0400

I’ve been using the Play framework for a big project at work. It’s fantastic. I almost hesitate to say it, but it’s like Ruby on Rails for Java. Watch the video that’s on the home page; it’s a good quick introduction.

The Tech Ops team at work wants control over configuration of the app. This makes sense for a few reasons. First, they’re responsible for running it so they should be able to tweak things like memory settings. Second, they need to tell the app where the database lives, what the password is, etc. Tech Ops also wants the app to run as a WAR inside an app server. That’s fine: Play comes with a war command that bundles your app into either an exploded WAR or a WAR file.

The problem is, Play doesn’t know how to read any configuration file other than the one that’s in its conf directory. At least, that’s what I thought. It turns out there is an undocumented feature of Play configuration files: If any configuration key is named@include.FOO, then the value is used as a path and that file is read, too.

A few caveats.

The path is relative to the application directory within the WAR. So, for example, to find a file in Tomcat’s conf directory, the path will have to be something like ../../../../../conf/play.conf.
The file that gets included isn’t treated exactly like the normal configuration file. Magic values like ${play.path} and ${application.path} are not interpreted.
The current Play framework id (similar to Rails’ environment) is ignored. However, since the include code is run after everything else is loaded, this isn’t really a problem since the external configuration file you’re loading is presumably for the environment you’re running.

There is also a plugin architecture, and a hook that runs after the configuration file is read. This means that, even if @include.FOO didn’t exist, you can write a plugin to do anything you want to alter the app’s configuration.

Update: Core Play developer Guillaume Bort wrote on the email list, “This feature is not yet documented because it is not ready: as you observed it the placeholders and the framework id stuff are not handled properly.” Caveat emptor.

Up, Up, and Away

Sat, 12 Sep 2009 00:00:00 -0400

Let’s say you are in a subdirectory, and there’s a Makefile in a parent directory. How do you run make? You climb back up to the parent directory and type “make”, or use “make -C parent_dir”. In either case, you have to know what the parent dir is. Here are a pair of scripts that do that for you.

The findup script’s job is to find a file in a parent directory. It prints the directory name if found, else it exits with status 1.

#! /bin/sh
#
# usage: findup file [file...]
#
# Finds any of the files in the current directory or any directory above and
# prints the directory name. If no such directory is found (we hit the root
# directory), exit 1. Note: will not find files in the '/' directory.

start_dir=`pwd`

while [ `pwd` != '/' ] ; do
    for f in $* ; do
        if [ -f $f ] ; then
            echo `pwd`
            exit 0
        fi
    done
    # Keep swimming...keep swimming...swimming, swimming, swimming...
    cd ..
done

# No parent directory
exit 1

The makeup script uses findup.

#! /bin/sh
#
# usage: makeup [args...]
#
# Finds the first makefile in the current directory or any directory above and
# then runs make, passing on any args given to this script.
#
# Relies on the "findup" command, which is found in this directory.

mfdir=`findup makefile Makefile`
if [ -z "$mfdir" ] ; then
    echo no makefile found
    exit 1
fi

# We've found a makefile. Use the -C flag to tell make to run from the
# directory where we found the makefile. We do this so that error messages
# produced during the make process are relative to the current directory. I
# think.
make -C $mfdir $*

ChucK Hack Fork

Mon, 03 Aug 2009 00:00:00 -0400

I’ve created a GitHub fork of ChucK, the “strongly-timed, concurrent, and on-the-fly audio programming language.” In my fork I’ve started to add missing features like string methods, and hope to add many more such as raw file I/O and perhaps some form of MIDI file I/O.

So far, I’ve added string.ch which is a getter and setter for string characters (really one-character length substrings) and string.substr. This allowed me to write a function in ChucK that converts strings like “c#4” to MIDI note numbers.

It looks like ChucK’s code was last updated in 2006, so I don’t feel to bad about forking it.

Update: I just read in the chuck email list archive that the team is planning to start active development again, and they will be adding file I/O. If they get it done before I do, that would be very nice.

A Weak Man Begs Forgiveness

Mon, 22 Jun 2009 00:00:00 -0400

This attempt at humor was written while recovering from the Swine Flu.

Dearest Colleagues,

I write this electronically-delivered letter to you in the hopes that you will forgive me. A terrible fate has befallen my lovely daughter V_______, and I am afrid that I now share in her misfortune. To protect you all from similarly horrible destinies, I must eschew your company this day.

It all started innocently enough, with a persistent cough. My eldest daughter would delicately hack to “clear her throat”. The coughs became more frequent and urgent. She developed a fever which burned her pure, innocent forehead. As time passed, she begain to writhe in agony, delirious with dreams of God knows not what Heavenly retributions and Hellish tortures.

My dearest wife E____ conveyed her to the pediatrician’s office on Sunday. Her doctor (a woman, by God! Ah, well. I know not what muse guides Science, but she must hold in favor the fairer sex) diagnosed V_______ with Swine Flu. The woman practiced her Art with confidence, perscribing the most modern treatments available.

We repaired to our home, powders and poultices in hand. The cure had barely taken hold of my dear daughter when I myself began to cough. At first, I denied the obvious. What strong man would not? Having fought in The War, my thoughts were not for my own safety or bodily condition but for that of my child.

I tell you now: Mr. Poe never wrote of horrors so small. The tiny demons that ravage my body even now defy all description, and to burden you with reports of the agony they cause does not merit further thought.

The submarine-like conditions of the Juicyorange office being the breeding grounds for animalcules it is, it is best that I remain far, far away. This is why I have decided to remain in an entirely different State of the Union. My sorrow at not joining you is only mitigated by the knowledge that you will not be exposed to the fiends that wrack my body.

In the fervent hope that this missive is not rerouted to your Spam folder,

With Deepest Respect,
James J. Menard
Babbage Engine Servitor