ActivePython 2.5.2.2 and 2.4.5.14 released

Get it here: http://www.activestate.com/products/activepython/

After a hiatus while I worked hard on the Komodo 4.3 release with the Komodo crew here, I’ve finally had the chance to update ActivePython 2.5 and 2.4 to the latest.

Full details in the ActivePython release notes. These releases update the core to Python 2.5.2 and 2.4.5.

Enjoy.

nightly updates for Komodo Edit (and the Komodo AUS)

I and others here have been hard at work on Komodo 4.3 (due to go final this week) so it has been a while since I’ve posted. One thing I’ve wanted to post about for quite a while is the Komodo auto-update system. I alluded to it waaay back when working on adding auto-update support to Komodo 4.2 but haven’t written anything about it since.

Last Friday gives me good reason to post about it: We now have a “nightly” channel for Komodo Edit!

Komodo Auto-update channels

There are three Komodo auto-update “channels”:

  1. “release”: This is the typical (and default) channel for installations of a final release of Komodo (e.g. 4.2.0, 4.2.1, 4.3.0). On this channel, Komodo will only update itself to the latest final Komodo release.

  2. “beta”: This is the typical (and default) channel for Komodo alpha/beta builds. On this channel, Komodo will update itself to the final or pre-release (i.e. alphas and betas).

  3. “nightly”: This is a channel that I finally got working on the server-side on Friday. Since the announcement of OpenKomodo and open sourcing of Komodo Edit we’ve been doing “nightly” builds of Komodo Edit (built on most nights :). These are publicly available here: http://downloads.openkomodo.com/komodoedit/nightly/

As of last Friday, if you are on the nightly update channel Komodo Edit will update to the latest nightly build. These are quite a bit more burning-edge that the “beta” channel. Often the only criteria for putting up a nightly is that the build worked for all platforms. So, occasionally some features are broken — though I think we do pretty well.

This channel is quite new for us though, so there may be some growing pains in the first couple of weeks. Please let me know if you have any problems with it. I think it will be pretty cool to easily always be running the very latest Komodo Edit.

At this time we aren’t yet doing public nightlies of Komodo IDE.

Setting the update channel for your Komodo installation

Currently there isn’t a prefs panel in Komodo to tweak auto-update settings — such as the channel you are one. There should be. I hope to get one in sometime after 4.3.0.

To set your Komodo channel edit “channel-prefs.js” in your Komodo installation as appropriate. On Windows and Linux this file is here:

INSTALLDIR/lib/mozilla/defaults/pref/channel-prefs.js

and on Mac OS X here:

INSTALLDIR/Contents/MacOS/defaults/pref/channel-prefs.js

It is a very simple file that looks like this:

// Valid values are "release", "beta" and "nightly" (internal-only).
pref("app.update.channel", "beta");

Happily that “internal-only” is no longer correct for Komodo Edit.

Other Komodo AUS Stuff

Komodo’s auto-update system, on the client side (i.e. the app), pretty much just uses the excellent Mozilla auto-update system. On the server-side we have our own (very simple) Django-based update server. On the build-side, we have our own Python scripts (mozupdate.py et al) for building all relevant partial and complete update packages as part of full builds.

I hope to post more about our AUS server and about our build tools. I think I could fairly easily package up our tools to provide a possible answer to Mozilla Bug 415181 (Package the MAR generation tools for easy external usage).

There used to be an open bug (found it: Mozilla Bug 375752) to convert some of the Bash shell scripts for Mozilla update package building to Python scripts. I see (by way of Mozilla Bug 410806) that that has at least partially happened with make_incremental_updates.py.

markdown2.py

I’ve started a python-markdown2 project: this is another Python implementation of Markdown:

Markdown is a text-to-HTML conversion tool for web writers. Markdown allows you to write using an easy-to-read, easy-to-write plain text format, then convert it to structurally valid XHTML (or HTML).

No release package yet (setup.py, setuptools, easyinstall, ezsetup? — my head is spinning), but you can get the file directly from SVN: markdown2.py.

This is the first project for which I am really using Google Code project hosting. I’m really digging it. Nice bug tracker, nice enough wiki (all wikis should use a source control system as their backend), svn. Goodbye sourceforge.

open komodo is out

The wraps are finally off Open Komodo!

Check it out.

Quick build notes:

# Get the source
svn co http://svn.openkomodo.com/repos/openkomodo/trunk openkomodo
# Build Mozilla
cd openkomodo/mozilla
python build.py configure -k 1.0 --moz-src=cvs:1.8 --release \
    --no-strip --shared --tools
python build.py distclean all
cd ..
# Build Komodo
export PATH=`pwd`/util/black:$PATH   # Komodo's "bk" build tool
bk configure
bk build
# Run Komodo
bk run

If you have the build prerequisites setup, you should be able to cut ‘n paste the above. (Windows users, use the Windows-specific quick build steps in the README.txt.)

I have some (longer term) plans to reduce those build steps to:

./configure.py
mk

off to FSOSS

I’m off to FSSOS in a few hours. We’ll be opening up the Komodo sources next Wednesday, so Shane and I will be there to start the discussion about what Open Komodo and Snapdragon should focus on to best improve the tool story for open web development. (Currently the Open Komodo sources are available to a small group of mozillians. If you have some ideas and would like to take a peek, let me know and I’ll hook you up!)

I’m hoping to meet a few of the Mozilla folks that will be there: Benjamin to ask about breakpad (I want to get breakpad running for Open Komodo and Komodo builds), looking forward to Mike Beltzner’s talk on UE design at Mozilla, Eric Shephed to ask about how mozilla handles localization of their docs.

As well, if any of the mozilla folks involved in the AUS will be there, I’d love to talk with you to compare notes with Komodo’s auto-update system.

html5lib rocks (and a patch to preserve attribute order)

I’ve been playing with the Python html5lib package — having come across it reading Sam Ruby’s blog. What a fantastically useful library!

Originally my interest in it was with the discussion surrounding santization, and I expect to use it for that later, but today I’ve been playing with some general parse/filter/serialize code to support some preprocessing of HTML documentation for Open Komodo.

My code looks like this:

import sys
from html5lib import treebuilders, treewalkers
from html5lib.serializer.xhtmlserializer import XHTMLSerializer

def filter_play(path):
    p = html5lib.XHTMLParser(tree=treebuilders.getTreeBuilder("simpletree"))
    f = open(path)
    dom = p.parse(f)

    walker = treewalkers.getTreeWalker("simpletree")
    stream = walker(dom)
    #stream = MyPreprocessingFilter()

    s = XHTMLSerializer()
    outputter = s.serialize(stream)

    for item in outputter:
        sys.stdout.write(item)

filter_play(sys.argv[1])

One thing that bugged me a little with the output generated with this is that attributes on HTML elements get sorted, i.e. their order is not preserved. While totally cool for correctness, this reduces the utility of using diff or similar for comparing input with output. As well, I work on the Komodo IDE/editor and would like to consider using html5lib for an HTML reflow/beautifier feature at some point. Preserving attribute order for this will be important.

To that end, here is a small patch that adds the ability to preserve attribute order in serialized output. To use it:

  1. You need odict.py.
  2. You need to change the above code to:

    ...
    s = XHTMLSerializer(preserve_attr_order=True)
    ...
    

Obviously this isn’t something that would be ready to check-in to html5lib. Reasons why:

  • It only works for the “simpletree” treebuilder/treewalker. I’m not sure if it is feasible/practical to get it to work with some of the others (e.g. dom).
  • It unconditionally requires an external non-standard module (odict.py).
  • It should be optional on the parser because (a) using OrderedDict instead of dict would presumably have an undesired perf impact and (b) the attribute order normalization could be desirable for many users.

Maybe a better solution would be a custom “roundtriptree” tree type? Anyway, just throwing this up here to perhaps come back to later. I have to dig into the html5lib discussion list to see if this has come up before.

CakePHP view codeintel (autocomplete) in komodo

It is great to see more and more posts these days about adding functionality to Komodo. Only a few month back, daily blog posts about Komodo tended to be of the “Tried Komodo. Like it. Using it for blah.”. Now blog posts about Komodo are often “Tried Komodo. Like it. Using it for blah. Used the macro or extension system to add blah.”

For example, this post by Travis Cline shows a brilliant little hack to get Komodo’s PHP codeintel (autocomplete and calltips) to work with the implicit environment for CakePHP views (’.ctp’ files). It is a great example of the kinds of things you can do with Komodo macros.

Note to self: Look into providing this same functionality via a special CakeViewEnvironment class that does a similar thing and that is attached to any buffer for a CakePHP view. Every “buffer” in Komodo’s codeintel system (there is one buffer for each open file and each file used for autocomplete info) has an “env” attribute which is a instance of the “Environment” class. This class defines special runtime environment information — typically just Komodo preferences and environment variables. However, subclasses can provide other info. An example is the “KoJavaScriptMacroEnvironment” that is attached to a buffer for editing a Komodo JavaScript macro in the editor. This environment class hooks up the Komodo JS API catalog so that you get autocomplete for Komodo JS macro API.

I have to look into (1) documenting this (when we have the Open Komodo wiki setup, that’ll probably be the right place) and (2) ensuring that extensions can provide these environment classes and do interesting things with them. Currently it might require some custom work on the codeintel engine to hook this up. But codeintel Environment classes are the plan.

installing hg 0.9.4

Cleaning out some notes of mine, it seemed a shame to just throw out notes on installing hg. I’m happy to update this with sections for other platforms and other Linux distros if people send them my way.

Installation on Ubuntu (< Gutsy)

Note: only Ubuntu Gutsy has hg 0.9.4, and my Ubuntu isn’t yet Gutsy.

sudo apt-get install python2.4 python2.4-dev
wget http://www.selenic.com/mercurial/release/mercurial-0.9.4.tar.gz
tar xzf mercurial-0.9.4.tar.gz
cd mercurial-0.9.4
sudo make install
which hg    # should be /usr/local/bin/hg
hg debuginstall

Installation on Linux (with no package mgmt support):

Presumably this works just as well on other Unix-y platforms.

# You must have a Python >=2.4 installation and first on your PATH.
cd tmp
wget http://www.selenic.com/mercurial/release/mercurial-0.9.4.tar.gz
tar xzf mercurial-0.9.4.tar.gz
cd mercurial-0.9.4
python setup.py install
hg debuginstall

Installation on Mac OS X (using MacPorts):

sudo port selfupdate
sudo port search mercurial
# ensure this is version >= 0.9.4
sudo port install mercurial

Troubleshooting : install fails with “Another version of this port (mercurial @0.9.1_0) is already active.”

sudo port uninstall mercurial @0.9.1_0   # the old one
sudo port uninstall mercurial @0.9.4_0   # the broken new one
sudo port install mercurial

Installation on other platforms (Windows, Solaris, FreeBSD, …)

Look for an available binary package.

mercurial needs better end-of-line support

One real world issue with source control systems is the handling of end-of-line characters in text files. Currently Mercurial pretty much punts. The hg book says:

Note: The Windows version of Mercurial does not automatically convert line endings between Windows and Unix styles. If you want to share work with Unix users, you must do a little additional configuration work. XXX Flesh this out.

The hgrc man page suggests:

NOTE: the tempfile mechanism is recommended for Windows systems, where the standard shell I/O redirection operators often have strange effects. In particular, if you are doing line ending conversion on Windows using the popular dos2unix and unix2dos programs, you *must* use the tempfile mechanism, as using pipes will corrupt the contents of your files. Tempfile example:
    [encode]
    # convert files to unix line ending conventions on checkin
    **.txt = tempfile: dos2unix -n INFILE OUTFILE


    [decode]
    # convert files to windows line ending conventions when writing
    # them to the working dir
    **.txt = tempfile: unix2dos -n INFILE OUTFILE

However (1) unix2dos and dos2unix are generally not available on Windows machines and (2) if dos2unix isn’t available the “encoding” here will silently wipe out your file to empty content on checkin.

How is Mozilla handling this in their hg repository? Is it mandated that new files added on Windows use Unix line endings or is some kind of conversion for Windows attempted?

patch for “custom action” for mozilla updater

Here is a patch (against a slightly out of date updater.cpp on the Mozilla 1.8 branch) that would add support for a:

customaction "relative-path-to-executable"

action in the “update.manifest” for a partial or full update (.mar file) for the Mozilla update system.

I’m just chucking this up here quickly for lack of a better place to put it right now. Komodo uses the Mozilla update system and will possibly need this patch at some point.

Limitations:

  • It ignores the return value of the spawned executable.
  • It doesn’t support arguments to the executable.

an intro to Komodo extensions

Komodo uses the Mozilla extension mechanism — same .xpi files as Firefox to install an extension, same kind of bundle content in an extension. However, Komodo adds a number of “hooks” that can be used to customize Komodo with an extension (see the end of this post).

In Komodo 4.2 (currently in beta) we’ve been working at improving the extension story. Part of my work there has been to improve the tools for building them. To that end Komodo 4.2 now includes a sort of “SDK” with a few tools:

koext
A tool for building and generating stubs for Komodo extensions. A recently added a (very brief) intro to using koext to Komodo’s extension forum.
luddite
A tool for working with Komodo’s UDL (User-Defined Languages) system. The UDL system (new in Komodo 4.0) provides a way to define lexers for new languages. Lexers are used mainly for syntax coloring, but can also be used by Komodo Code Intelligence system for provide autocomplete and calltips. Eric wrote up a long intro to UDL a while back. UDL currently isn’t for the faint of heart, but it provides an execellent system for robust lexing of code languages — in particular it supports *multi-language* code (e.g. JavaScript in HTML, Ruby in RHTML, CSS in Django HTML).
codeintel
A tool to help writing a language support for Komodo’s Code Intelligence system. I’ll write more on this later.

These tools are all works in progress but they are used internally as part of normal Komodo development, so should be usable for Komodo extension authors.

Komodo’s koext tool briefly describes all the current Komodo extension “hooks”:

$ koext help hooks

  Many parts of Komodo's functionality can be extended with a
  Komodo extension. We call those "hooks" here. The following is
  a list of all extension hooks that Komodo currently supports.

  The "source tree files" sections below are conventions for
  placement of sources files. If you use these conventions, then
  `koext build' will automatically be able to build your extension
  properly.

  chrome
      Chrome is the collective term for XUL (content), JavaScript
      (content), CSS (skin), images (skin) and localized files
      (locale, typically DTDs) that can be used to extend the
      Komodo UI. This works in Komodo extensions in exactly the
      same way as any other Mozilla-base application (such as
      Firefox). See `koext help chrome' for some tips.

      source tree files:
          chrome.manifest
          content/            # XUL overlays, dialogs and JavaScript
          skin/               # CSS
          locale/             # localized files (typically DTDs)

  XPCOM components
      XPCOM components are placed here. These can be written in
      Python or JavaScript. (C++-based components are possible
      as well, but currently the Komodo SDK does not support
      building them.)

      source files:
          components/
              *.idl           # interface definitions
              *.py            # PyXPCOM components
              *.js            # JavaScript XPCOM components

  templates
      A file hierarchy under here maps into Komodo's "New File"
      dialog. For example, "templates/Common/Foo.pl" will result
      in a new Perl file template called "Foo" in the "Common"
      folder of the "New File" dialog.

      source files:
          templates/

  lexers
      Komodo User-Defined Languages (UDL) system provides a
      facility for writing regular expression, state-based lexers
      for new languages (including for multi-lang languages).
      ".lexres" files are built from ".udl" source files with
      the "luddite" tool (in this SDK). See `koext help udl' and
      Komodo's UDL documentation for more details.

      source files:
          udl/
              *-mainlex.udl   # a .lexres will be build for each of these
              *.udl           # support files to be included by
                              #   "*-mainlex.udl" files

  XML catalogs
      An extension can include an XML catalog (and associates
      schemas) defining namespace to schema mappings for XML
      autocomplete.

      source files:
          catalog.xml         # Note: This may move to xmlcatalogs/...

  API catalogs
      An extension can include API catalogs to provide autocomplete
      and calltips for 3rd party libraries. An API catalog is a CIX
      file (an XML dialect) that defines the API of a
      library/project/toolkit.

      source files:
          apicatalogs/        # .cix files here will be included
                              #   in the API catalog list in the
                              #   "Code Intelligence" prefs panel

  Python modules
      An extension can supply Python modules by placing then in
      the "pylib" directory of the extension. This "pylib" directory
      will be appended to Komodo's Python runtime sys.path.

      source files:
          pylib/

  codeintel
      An extension can provide the Code
      Intelligence logic (for autocomplete and calltips, for
      "Jump to Definition" and for the Code Browser in Komodo IDE)
      for new languages.

      source files:
          pylib/              # lang_*.py files here are picked up
                              #   by the codeintel system.

open komodo and the code

Yesterday we (ActiveState) announced Open Komodo, an open-source project seeded with much of the core of Komodo Edit and Komodo IDE with the goals of produce a platform/framework for and (codename Komodo Snapdragon) an IDE for client-side open web development.

That’s a mouthful. Shane and David have done a good job giving some wider perspective on what the Open Komodo project could mean (if all goes well). David went so far as to invent new language to make his points.

Some quick thoughts from a coder’s perspective:

  • The source will be available in a Mercurial repository in (quoting Shane paraphrasing Mike Shaver) “Two F**king Months!”. Early November — or earlier if we can.

  • Komodo is a Mozilla-based application with the added heavy use of PyXPCOM for much of the core logic. That means the app comes together like this:

    • Get a slightly tweaked mozilla build (C++, JavaScript, XUL).
    • Get a slightly tweaked Python build (C).
    • Add a bunch of core logic (Python). For example, the guts of Komodo’s Find/Replace system is written in Python — using Python’s unicode-aware regular expression engine.
    • Add Komodo chrome (XUL, JavaScript, CSS, DTDs).

    What this means is that to work on and add significant functionality to Komodo, all you tend to need to know is XUL, JavaScript and Python. From early on in Komodo’s development we’ve felt that this is one of Komodo’s aces in the hole: developing in the dynamic languages is so much faster. I remember David Ascher making the comment way back that if Subversion had been written in Python, it would have been ready years sooner. And now two of the primary DVCS, Mercurial and Bazaar, are written in Python.

  • Komodo uses the same extension mechanisms as Firefox. It is easy to build a .xpi to add functionality to Komodo. We really hope that a community of Komodo extension authors will develop.

  • Komodo builds and runs on Windows, Linux and Mac OS X. Given some work there is little reason the Open Komodo code base couldn’t be made to run well on Solaris, BSD, etc.

If any of this sounds interesting to you as an open-source tinkerer, then give Komodo Edit or Komodo IDE a try. The first app that will come out of the Open Komodo project (Komodo Snapdragon) will look and feel a lot like them.

In subsequent posts, and especially once the source code repository is up, I plan to blog here about Komodo’s internals.

mini-mick cometh

ewan and dad

Mini-mick (a.k.a. Ewan Mick) arrived about three weeks ago. I am so happy to have him and proud of my wife Alli.

In some ways, having a child is like a little boy’s dream come true: farts and poos are acceptable entertainment!

building MSI patch packages (.msp) with WiX

This post includes a complete and concrete example of building an MSI patch package (a .msp file to upgrade an existing .msi installation) with WiX.

Background

I’m responsible for building the ActivePython and Komodo installers at ActiveState. On Windows we build MSI packages for installation.

Currently I’m investigating auto-update support for Komodo 4.2. Because Komodo is based on Firefox/Mozilla we can benefit from the excellent Mozilla update system (I’ll write another post about our experience with it). However, integrating with an MSI-based installation isn’t something the Mozilla update system does out of the box: Firefox and Thunderbird don’t use MSI for their installers (they use NSIS), hence I suspect working with MSI was never a design consideration.

While working out how to best marry MSI and Moz update, I investigated producing MSI patch packages (.msp files) for Komodo updates. MSI is a complex and complicated technology (it would be nice if the latter, at least, wasn’t the case). Back in the day I used InstallShield for building our MSI packages, but now WiX is the best way to build .msi’s — by far. WiX helps a lot, but building appropriate MSI packages is still quite difficult. The following two pages helped me get to successfully building .msp’s. Hopefully this concrete example will help others too.

ActiveFoo 1.0

For this example we’ll build .msi installers for versions 1.0.0 and 1.0.1 of the the mythical “ActiveFoo” app (”activefoo-1.0.0.msi” and “activefoo-1.0.1.msi”). Then we’ll build a ‘.msp’ that will upgrade a 1.0.0 install to 1.0.1. We’ll have the following files:

1.0.0/
    activefoo.wxs       # This describes "activefoo-1.0.0.msi"
    config.wxi
    installimage/       # The ActiveFoo install image
        CHANGES.txt
        foo.exe
        README.txt
1.0.1/
    activefoo.wxs       # This describes "activefoo-1.0.1.msi"
    config.wxi
    installimage/       # The install image with changes for 1.0.1
        CHANGES.txt
        README.txt
    upgrade-1.0.0.wxs   # This describes the '.msp'.
make.py                 # 'python make.py' to build everything
README.txt

Here is a zip of the working files for this example, if you’d like to play along.

We have a simple install image with three files (foo.exe, README.txt and CHANGES.txt). The WiX code to build an installer for ActiveFoo 1.0.0 is 1.0.0/activefoo.wxs:

<?xml version="1.0" encoding="utf-8"?>

<?include config.wxi ?>

<Wix xmlns="http://schemas.microsoft.com/wix/2003/01/wi">
  <Product Name="$(var.ProductName)" Id="$(var.ProductCode)"
           Language="1033" Codepage="1252" Version="$(var.ProductVersion)"
           Manufacturer="Acme" UpgradeCode="$(var.UpgradeCode)">

    <Package Id="????????-????-????-????-????????????" Keywords="Installer"
      Description="$(var.ProductName)"
      Comments="blah blah" Manufacturer="Acme"
      InstallerVersion="200" Languages="1033" Compressed="yes"
      SummaryCodepage="1252" />

    <Media Id="1" Cabinet="media.cab" EmbedCab="yes" />

    <!-- Define some of the dir-structure. -->
    <Directory Id="TARGETDIR" Name="SourceDir">
      <Directory Id="ProgramFilesFolder" Name="PFILES">
        <Directory Id="INSTALLDIR" Name="$(var.InstallId)"
                   LongName="$(var.InstallName)" />
      </Directory>
    </Directory>

    <!-- Define the feature hierarchy (just one feature in this simple
         example). -->
    <Property Id="INSTALLLEVEL" Value="1000" />
    <Feature Id="core" Title="ActiveFoo" Description="The Foo core"
             Level="1">
      <ComponentRef Id="MainExe" />
      <ComponentRef Id="ReadMeFiles" />
    </Feature>

    <!-- Define all the components. -->
    <DirectoryRef Id="INSTALLDIR">
      <Component Id="MainExe" Guid="6ee6fda3-6f50-47bf-99b9-6031c720428e">
        <File Id="MainExe" Name="foo.exe" DiskId="1"
              src="installimage\foo.exe" Vital="yes" />
      </Component>
      <Component Id="ReadMeFiles" DiskId="1"
                 Guid="8f2255f3-3eaf-4c82-9688-3545cd9b2018">
        <File Id="README.txt" Name="README.txt"
              src="installimage\README.txt" />
        <File Id="CHANGES.txt" Name="CHANGES.txt"
              src="installimage\CHANGES.txt" />
      </Component>
    </DirectoryRef>

  </Product>
</Wix>

with some configuration variables included from 1.0.0/config.wxi:

<?xml version="1.0" encoding="utf-8"?>
<Include>
  <?define ProductCode = "cdc5e50f-b490-4a37-8ff6-22e3cb3d690e" ?>
  <?define UpgradeCode = "ed340ed8-aa91-4bf6-9dcf-d7f6f4d43737" ?>

  <?define ProductName = "ActiveFoo" ?>
  <?define InstallName = "ActiveFoo" ?>
  <?define InstallId = "AFoo10" ?>
  <?define ProductVersion = "1.0.0" ?>
  <?define ProductURL = "http://www.example.com/products/activefoo/" ?>
</Include>

(Note that this WiX project is simplistic. In a real world WiX project you’d likely have a UI element for a user UI, define Add/Remove Programs — ARP – properties, etc.)

Use the provided “make.py” script to build “activefoo-1.0.0.msi”:

C:\tmp\wix_and_msp> python make.py -v 100
INFO:make:build target '100'
DEBUG:make:running 'candle -nologo activefoo.wxs' in '1.0.0'
activefoo.wxs
DEBUG:make:running 'light -nologo -o ../activefoo-1.0.0.msi activefoo.wixobj' in '1.0.0'
INFO:make:'activefoo-1.0.0.msi' created

and install it. You should now have a “ActiveFoo” folder in your “Program Files”.

ActiveFoo 1.0.1

Version 1.0.1 has the following changes:

  1. The ProductVersion is incremented to 1.0.1. We aren’t change the ProductCode so this qualifies in MSI parlance as a “minor upgrade“, as opposed to a “small update” or a “major upgrade”).
  2. We’ve added a note to “CHANGES.txt” for the new release.
  3. We’ve removed the “foo.exe” file from the install image. This is so we can see how file removal can be accomplished with a “minor upgrade”. There is a lot of documentation out there than says that file removal can’t be done with an MSI minor upgrade. We’ll see that that isn’t true. I haven’t seen any justification for why minor upgrades shouldn’t remove files.

Normally, for these changes, the only updates to the WiX sources to build “activefoo-1.0.1.msi” would be to (a) update the “ProductVersion” string and (b) remove the File and Component elements for “foo.exe”. However, working from this comment in Minor and Major Upgrades Using IPWI:

If you need to remove any files or registry data during the upgrade, add
records to the RemoveFile or RemoveRegistry tables of the newer database.

I’ve found that to get WiX to put a RemoveFile entry for, in this case, “foo.exe”, I needed to add an explicit RemoveFile element:

      ...
      <Component Id="MainExe" Guid="6ee6fda3-6f50-47bf-99b9-6031c720428e">
        <!-- Note: This is how to explicitly remove files in an update. -->
        <RemoveFile Id="removefile1" On="install" Name="foo.exe"/>
      </Component>
      ...

The ProductVersion we updated in “1.0.1\config.wxi“:

C:\tmp\wix_and_msp>diff -u 1.0.0\config.wxi 1.0.1\config.wxi
--- 1.0.0\config.wxi    Mon May 28 17:33:01 2007
+++ 1.0.1\config.wxi    Mon May 28 17:33:03 2007
@@ -6,7 +6,7 @@
   <?define ProductName = "ActiveFoo" ?>
   <?define InstallName = "ActiveFoo" ?>
   <?define InstallId = "AFoo10" ?>
-  <?define ProductVersion = "1.0.0" ?>
+  <?define ProductVersion = "1.0.1" ?>
   <?define ProductURL = "http://www.example.com/products/activefoo/" ?>
 </Include>

Now we can build “activefoo-1.0.1.msi”:

C:\tmp\wix_and_msp> python make.py -v 101
INFO:make:build target '101'
DEBUG:make:running 'candle -nologo activefoo.wxs' in '1.0.1'
activefoo.wxs
DEBUG:make:running 'light -nologo -o ../activefoo-1.0.1.msi activefoo.wixobj' in '1.0.1'
INFO:make:'activefoo-1.0.1.msi' created

ActiveFoo 1.0.1 update

The basic process for building a ‘.msp’ is:

  1. Get an administrative install of the old version. I hadn’t known this before: An administrative install effective just extracts the file payload from an .msi into a given directory leaving a lighter .msi with just the MSI database tables. AFAIK this is the same thing as if you had built an “uncompressed MSI” — i.e. one in which <Package Compressed='no' .../>. make.py will put this in “1.0.1\build\before”.
  2. Get an administrative install of the new version. make.py will put this in “1.0.1\build\after”.
  3. Write a WiX file that describes the patch.
  4. Compile to a Patch Creation Properties (.pcp) file with WiX.
  5. Compile to a ‘.msp’ file with the “msimsp.exe” utility from the MSI SDK (Part of the Microsoft Platform SDK).

Here is a our WiX file describing the patch (1.0.1\upgrade-1.0.0.wxs) with comments inline:

<?xml version='1.0' encoding='windows-1252'?>

<Wix xmlns='http://schemas.microsoft.com/wix/2003/01/wi'>
  <!-- TODO: Update PatchCreation Id for each new patch.
             Can we just use WiX's '????????-????-????-????-????????????' ? -->
  <PatchCreation Id='e8ee6400-7877-47e4-9519-ce17e3f1d59b'
                 CleanWorkingFolder='yes'
                 WholeFilesOnly='no'
                 AllowMajorVersionMismatches='yes'
                 AllowProductCodeMismatches='no'>

    <PatchInformation Description="ActiveFoo 1.0.1 Patch"
                      Comments='blah blah'
                      Manufacturer='Acme'
                      Languages='1033'
                      Compressed='yes' />

    <!-- TODO: Play with other values of 'Classification'. Does msiexec's
         behaviour actually change for different values? -->
    <PatchMetadata Description="ActiveFoo 1.0.1 Patch"
                   DisplayName="ActiveFoo 1.0.1 Patch"
                   TargetProductName='ActiveFoo 1.0'
                   ManufacturerName='Acme'
                   MoreInfoURL='http://www.example.com/products/activefoo'
                   Classification='Update'
                   AllowRemoval='yes' />

    <!-- From <http://wix.sourceforge.net/manual-wix2/patch_building.htm>
         """
         The SequenceStart value is influenced by the number of files that
         the previous patch delivered, as well as the number of files that
         this patch will deliver. This tells PatchWiz.dll to start assigning
         File sequence numbers from this number. So if this patch ships 11
         files, and the next patch uses a SequenceStart of 1020, it will step
         on the 11th file’s assigned sequence number. In this case the next
         patch would use a SequenceStart of 1030, and 03 as the patch id to
         avoid conflicts with this patch. This scheme helps prevent this by
         coordinating the SequenceStart (file sequence numbers) with the
         patch sequence number. Also, note that the SequenceStart of the
         first patch must be greater than the number of files in the original
         installation. If the original installation contained more than 1000
         files(rare), then the SequenceStart for the first patch must be set
         to a higher value (e.g 2010.)
         """
    -->
    <!-- Name is max 8 chars. *How* unique does this have to be? -->
    <Family Name='Fam101' DiskId='2' MediaSrcProp='AFoo10_2_1_01'
            SequenceStart='1010'>
      <UpgradeImage Id='AFoo10Upgrade'
                    SourceFile='after\activefoo-1.0.1.msi'>
        <TargetImage Id='AFoo10Target' Order='1' IgnoreMissingFiles='no'
                     SourceFile='before\activefoo-1.0.0.msi' />
      </UpgradeImage>
    </Family>

    <TargetProductCode Id='cdc5e50f-b490-4a37-8ff6-22e3cb3d690e' />
  </PatchCreation>
</Wix>

Use make.py to build the patch:

C:\tmp\wix_and_msp> python make.py -v 101_upgrade
INFO:make:build target '101_upgrade'
DEBUG:make:running 'msiexec /a activefoo-1.0.0.msi TARGETDIR=C:\tmp\wix_and_msp\1.0.1\build\before'
DEBUG:make:running 'msiexec /a activefoo-1.0.1.msi TARGETDIR=C:\tmp\wix_and_msp\1.0.1\build\after'
        1 file(s) copied.
DEBUG:make:running 'candle -nologo upgrade.wxs' in 'C:\tmp\wix_and_msp\1.0.1\build'
upgrade.wxs
DEBUG:make:running 'light -nologo upgrade.wixobj' in 'C:\tmp\wix_and_msp\1.0.1\build'
DEBUG:make:running '"C:\Program Files\Microsoft Platform SDK\Samples\SysMgmt\Msi\Patching\MsiMsp.Exe" -s upgrade.pcp -p C:\tmp\wix_and_msp\activefoo-1.0.1-upgrade-1.0.0.msp -l upgrade.log' in 'C:\tmp\wix_and_msp\1.0.1\build'
INFO:make:'activefoo-1.0.1-upgrade-1.0.0.msp' created
INFO:make:To install the update, run:
  msiexec /p activefoo-1.0.1-upgrade-1.0.0.msp REINSTALL=ALL REINSTALLMODE=omus

You should now be able to install “activefoo-1.0.1-upgrade-1.0.0.msp” over an ActiveFoo 1.0.0 installation to upgrade to ActiveFoo 1.0.1. Note that some docs out there mention an MSI bug preventing installation of a ‘.msp’ by double-clicking on that. I’ve found that I am able to install by double-clicking on my WinXP box with Windows Installer V 3.01.4000.1823.

Notes/Limitations

  1. Having to explicitly put in RemoveFile elements to ensure upgrades remove them is a pain. It would be nice if WiX inferred that automatically. WiX v3 is slated to include “Patch creation support” and “ClickThrough”. Perhaps these will go a long way to making all of this easier.
  2. There are many variables to tweak here that I haven’t played with. I haven’t deployed any .msp’s built as describe here to users on any scale so I there may be gremlins lurking in this procedure.

I’d be happy to hear about others’ experiences working with WiX and MSI patches.

logging in to Gov. of Canada’s site an exercise in frustration

I love Canada. Couldn’t pick a better place to live. And I suppose if the following exercise is one of the worst annoyances in recent memory when dealing with my federal government, then that is a good thing.

How:

Canada Revenue unsupport browser 3: IE7/Win

pray:

Canada Revenue unsupported browser 2: Firefox 2/Win

tell:

Canada Revenue unsupported browser 1: Safari 2

am I meant to login to the Canada Revenue site to change my address? It is a good thing that you owe me money this year, or it would probably be easier for me to setup a website for you to login and get my tax information.

Computer programming is my day job and I happen to have access to a lot of machines — one of which is an old Windows 2000 machine that still has IE 6 on it. I was able to login to the epass system in this browser. However, I suspect that a lot of your users will not be able to login the moment they follow Microsoft’s Windows Update prompts to upgrade from IE6 to IE7.

mini-mick

ultrasound 1
Alli and I are happy to announce the upcoming birth of mini-mick (due mid-August sometime). Seeing the (6cm) foetus kick during our recent ultrasound visit was quite an experience. I am scared and excited.

Komodo Planet and Yahoo Pipes

I played a bit with Yahoo’s new Pipes web app. At ActiveState we have a number of blogs where many of us write about what interesting things we are working on. A number of those posts are about Komodo. As well, with the recent 4.0 release, Komodo has started popping up on blogs more and more — in particular, the very cool “Komodo Hacks” series from Brandon.

So, without further ado, I give you the Komodo Planet Pipe.

Let me know if there are other blogs I should add to this. Currently it just filters on posting titles matching “Komodo”. I tried to also include filtering on the posting content but Yahoo Pipes kept giving me (unhelpful) errors with that. I like the idea of Pipes, but overall I found the current experience to be too unresponsive to be pleasing.

Komodo babies

komodo babies

Long time, no post. I’ve been working on Komodo 4.0 — which we finally released yesterday after over a year of work! Top that off with the recent announcement by the British Zoo of the virgin birth of 5 baby Komodo dragons! Perfect.

Komodo 4.0 adds JavaScript debugging (amongst a bunch of other features) and

the reptiles are in good health and enjoying a diet of crickets and locusts.

Too fun.

Open letter to EQ3

Dear Sir,

I am writing to inform you that the customer service I received recently from EQ3 in Vancouver was totally unacceptable. Please forward this message to the appropriate person.

I ordered a Connick Pendant Lamp from your Yaletown Store on 10th June 2006 and was told that it would be delivered to the store in 2-3 weeks. When I called after 4 weeks to see if the lamp had arrived, I was then told that the 2-3 weeks was actually delivery time to Winnipeg and I would have to wait another week or so for the lamp to arrive in Vancouver.

Three months later in September I received a phone call from Kevin Smith [not his real name] indicating that the lamp had arrived in Vancouver. No apology for the delay was forthcoming. I was also informed that it could not be delivered to the Yaletown store and I would have to pick it up from the warehouse in a completely different area of Vancouver. As the warehouse was only open 9am to 5pm Monday to Friday, I took 2 hours off work and drove out of my way to collect the lamp on September 15th.

On arrival at the warehouse, I was notified by the warehouse employee that the truck that was meant to arrive the previous day had been delayed and that all customers should have been contacted. I was also notified that my lamp was destined to be delivered to the Yaletown store so I wasn’t expected to come and pick it up anyway.

Customer service up to this point had been substandard to say the least but degenerated to disgraceful when I called up Kevin Smith to inform him of the situation. Without any understanding of my circumstances he informed me that my drive to the warehouse was hardly out of my way and really, could eq3 be expected to call everybody and inform them of the delay? With this level of customer service, EQ3 cannot expect any of their current customers to shop in the store again and I have been left with a very poor opinion of the company.

I instructed Kevin Smith to refund my payment for the lamp and I am waiting to ensure this happens. If EQ3 wishes to be a successful franchise company I would recommend that it puts some effort into customer satisfaction.

Firefox and Thunderbird featurelets I want

I use Firefox and Thunderbird. (For the latter I recently switched from using mutt — the last luddite at work to finally switch to one of them new fangled GUI things for email.) Here are a couple fixes/featurelets that I want in them.

  • I set Firefox to open new URLs in a new tab. This is the behaviour I want most of the time. However, there is the odd time that I want to lock a particular browser window. I.e., I’d like subsequent decisions on where to open a new URL to ignore this window. For example, while editing this blog posting in my browser, I would like new URLs (ones I’m hunting for to link from this post) to open in another browser window. Ditto when I’m working on my calendar in the browser. Or in gmail. What if Firefox had a “pin” button (as provided in some Linux window managers)? That would be handy.
  • Say you previously sent this email:
        To: friend@example.com
        From: me@example.com
        Subject: Here are some pics from my weekend
    
    And, of course, you forgot to attach the pics. So you reply to your own email. In mutt and gmail when you reply to that email you get:
        To: friend@example.com
        From: me@example.com
        Subject: Re: Here are some pics from my weekend
    
    In Thunderbird, when you “Reply All” you get these headers:
        To: me@example.com
        Cc: friend@example.com
        Subject: Re: Here are some pics from my weekend
    
    That is just wrong. Even worse is when you just hit “Reply”:
        To: me@example.com
        Subject: Re: Here are some pics from my weekend
    
    I guess your friend just won’t see how your weekend went.
    [Update: I couldn’t find a bug for this so I added this one.]
    [Update 2: … and quickly closed: “fixed for 2.0″. Cool.]