jigdo - Jigsaw Download

Jigsaw Download, or short jigdo, is a tool designed to ease the distribution of very large files over the internet, for example CD or DVD images. Its aim is to make downloading the images as easy for users as a click on a direct download link in a browser, while avoiding all the problems that server administrators have with hosting such large files.

Article about jigdo on heise.de (c't-Magazin; German)

I have stopped development on this software long ago. But luckily, Steve McIntyre has taken over development - thanks Steve! Please head to einval.com for a forked jigdo version with new features.

Download

• This is all outdated, go to einval.com instead!
• jigdo-lite 0.7.3 for Linux, 994 kB (statically linked)
  Also available for Debian and Fedora Core
• jigdo-lite 0.7.2 for Windows, 1773 kB
  This version is capable of creating DVD-sized images on NTFS partitions (FAT32 only supports sizes up to 4 GB). In case you use WinZip, click on "Extract" and select "Use folder names", don't just drag the files to a directory.
  (Windows 98/ME users, try the older 0.7.1a version - 0.7.2 will not work for you due to a small problem.)
• jigdo-lite 0.7.1 for Solaris, 378 kB (statically linked binaries for Sparc)
• jigdo 0.7.3 source code, 630 kB
  jigdo is Free Software, distributable under the GNU GPL.
• External sources of jigdo binaries: RPMSeek, RPMFind, Mac OS X, FreeBSD
• Changelog - what is new in this version?

Help

• Documentation: jigdo FAQ (frequently asked questions)
• Documentation: The FAQ is part of the Debian jigdo mini-HOWTO. Look here if you have trouble using jigdo-lite.
• jigdo-user mailing list: General discussion about jigdo.
  Subscribe to jigdo-user, browse the archive or send mail directly to jigdo-user [at] lists [dot] berlios [dot] de.
  The mailing list is public - your mails will be archived, and it is not guaranteed that someone won't abuse the archive and send you spam.
• Documentation: The Debian CD FAQ is not directly related to jigdo, but worth checking if you have questions regarding the images you downloaded using jigdo.
• Debian-CD mailing list: Discussion about problems with the Debian CD images.
  Subscribe to debian-cd, browse the archive or send mail directly to debian-cd [at] lists [dot] debian [dot] org. This mailing list is also public.

Why Is It Needed?

jigdo was inspired by the problems that Debian Linux had with the distribution of its CD images. Debian is a 100% volunteer-driven organization which depends on donations to finance its infrastructure. For this reason, it also doesn't own any of the 300 servers around the world on which the distribution is mirrored.

For some time, the size of Debian has been a problem - Debian supports 11 different processor architectures, more than any other Linux distribution, and for every single one of these architectures, there are more than 10000 software packages - again more than any other distro. Today, a full Debian mirror needs roughly 100 GB of disc space!

Debian also offers the distribution in the form of CD images (and more recently DVD images). If these images were distributed as full .iso images on the mirrors, the size of a Debian mirror would double to 200 GB (or even triple to 300 GB if there were also DVD images) - nobody is prepared to mirror such a huge amount of data!

jigdo solves this problem by not requiring that the full image be stored on the servers. Instead, it can download the individual files from a normal Debian mirror and only assemble the image on the user's machine. Additionally, it can also do other useful things like helping with the mirror selection and upgrading images.

How Does It Work?

When someone has produced a large file they wish to distribute (for example, a CD image), they process this file with the jigdo-file tool. Among other things, jigdo-file creates a file with a .jigdo extension. To download the original file, a user only needs to tell the jigdo-lite download tool the URL of the jigdo file, it will extract all the necessary information and perform the download, fetching the data in many small pieces, possibly from several different servers.

So far, this is not exciting; there are already several other programs which allow you to split and reassemble large files, or which make their download easier. Also, at the moment only the shell script jigdo-lite can be used for downloading. Unfortunately, the jigdo GUI download manager shown below never reached a usable state.

The jigdo download tool (unfinished, not usable)

However, jigdo's approach is unique due to the way it identifies the small pieces that the large file consists of: For instance, if the file is a CD image, the individual files on the CD are contained somewhere within the image file. jigdo-file is capable of finding these files, so if copies of the individual files on the CD are stored separately on the server, jigdo-lite can be made to download these files one by one, and assemble the CD image on the fly on the user's computer. This scheme has several advantages:

• The large image does not need to be stored on the server, instead only the many small files contained in the image. This works with CD images, DVD images (both ISO9660 and UDF format), uncompressed zip files, tar archives... jigdo-file is format independent.
• In spite of the above, jigdo creates a bit-exact copy of the original image on the user's machine. (To achieve this, the directory data, boot block etc. of the image is stored in a special .template file which is distributed alongside the .jigdo file.)
• There is full control over where jigdo-lite will download the individual parts. It is possible to define mirrors so users can choose their nearest mirror.
• jigdo relies on standard HTTP/FTP, no support for special protocols like rsync, and no installation of special software is required on the mirrors.
• jigdo-lite supports resuming aborted downloads, or continuing the download with another mirror if the current one proves slow.
• It is possible to "upgrade" a CD image: If a new version of an image is released, jigdo-lite can download only the data that has changed, the remaining data is read from the outdated version.
• It is possible to release modified versions: If someone distributes a CD image and you have made a few small modifications, e.g. added some files, then you only need to upload the data for your modifications to your webspace. All the data from the original CD image is still fetched from the original site.

If you have read this far, you may be confused now, asking yourself what jigdo does and doesn't do. :-) It seems that the jigdo concept is a bit difficult to grasp at first. If you're still interested, have a look at the respective section of the HOWTO and at the Examples section of the jigdo-file manual.

Further jigdo Resources

• jigdo site for Debian CD images. Use jigdo-lite to download them.
• Debian jigdo mini-HOWTO
• JTE - Jigdo Template Extractor is a project by Steve McIntyre which modifies mkisofs to output .template files. This makes generation of templates for .iso images much faster than creating the image with mkisofs first and then feeding it to "jigdo-file make-template".
• snapshot.debian.net, which holds daily snapshots of the Debian archive, is a very useful resource if you are stuck with a broken Debian .jigdo file which makes jigdo-lite give you an error message "Aaargh - n files could not be downloaded. This should not happen!". To try to fetch the missing files from snapshot.debian.net, specify as "Debian mirror" an URL like http://snapshot.debian.net/archive/2009/09/02/debian/
  (Substitute the correct date - the date the broken .jigdo file was made.)

• Old and current releases: 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.6.0, 0.6.1, 0.6.2/bin, 0.6.3/bin, 0.6.4/bin, 0.6.5/bin, 0.6.6/bin, 0.6.7/bin/solaris, 0.6.8/bin/solaris/win, 0.6.9/bin/solaris/win, 0.7.0/bin/win, 0.7.1/bin/solaris/win, 0.7.2/bin/win, 0.7.3/bin

• Documentation for jigdo-file, the tool for low-level processing of .jigdo/.template files. There is also a more technical introduction to how jigdo works.
• Documentation for jigdo-lite, a shell script which uses jigdo-file to reassemble images. It is intended to fill the gap until the real jigdo GUI application is usable.
• Documentation for jigdo-mirror, a shell script which Debian mirror maintainers can use for efficient mirroring of Debian CD images offered as .jigdo files.
• jigdo project page at BerliOS. Thanks to this non-commercial SourceForge alternative for its services!