mkstatic: join binary and it's libraries together

You just built your program on your own notebook. And noticed, you don't have many of libraries on the other host. The host, where you want to check out your job. Nothing unusual. First approach to install 'em (lucky if you got root-access). Another approach is to recompile your program linking all the libraries statically; and it's good if you have static brothers for your dynlibs. So, it doesn't work everywhere. Especially, on production server.

But what if you just want to launch your program on another machine and you don't want to bother yourself with tons of libraries it depends on? To do this job quickly you may use mkstatic Perl script. It creates .static package which contains your binary and all it's libraries within.

No magic, again :-(

Surely, it doesn't perform anything you can't do manually (if you familiar with ld.so). But I think you won't do this job so accurate. So, what mkstatic does for you:

• it collects all dependencies of your binary_file (and remembers symlinks to libraries)
• it creates .tgz which is actually placed in Shell-file. This archive includes binary, libraries and bootstrapping code
• you may use binary.staic as usual binary file

I believe the latest point is the most critical. Because all the mess are hidden from you: all it works just like your original binary. With absolutely the same usage. And requires nothing from target host.

Known limitations

Keen reader may guess: "Hey, this will work for all binaries!". Yes, you can't [easily] copy your Chromium distribution this way to empty machine. Simply because it depends not only from libraries, but from many data (drivers for your Xorg, font configs). And mkstatic doesn't know anything about them. But you can use mkstatic, for instance, with Midnight Commander :-). Or any your executable which uses "data-free" libraries (libogg, libboost, libstdc++, etc).

Let's test it

Surely, it's better to test mkstatic in two machines: one which has all the bunch of libraries, and the fresh one. But I'll show you how I've tested this thing.

First of all, you have to build .static package. Use Midnight Commander' binary as example:

$ ./mkstatic -o /tmp/mc.static `which mc`
executable package is ready: /tmp/mc.static

I'm using xubuntu-12.04 (Precise Pangolin). As any Debian-like distribution it contains debootstrap utility. So, launch:

$ sudo debootstrap precise precise-chroot http://mirror.yandex.ru/ubuntu/
$ sudo cp /tmp/mc.static precise-chroot/tmp/
$ sudo chroot precise-chroot /bin/bash # now you're in test environment
# /tmp/mc.static

Wuala! Midnight Commander is working on your chroot environment, though you don't have libgpm.so within. You may say what Midnight commander is pretty simple. Surely! But you may use mkstatic with much more heavy binaries like mencoder which requires about 100 libraries. Or even Skype! All programs containing one executable binary file is mkstatic-friendly. Try it!

As usually, there is manual-page in package. See mkstatic --man for details.

P.S. If you just interested in approach self-extractable archive, you may see makself. It's widely used for binary installations on Unix world (Nvidia drivers, VirtualBox, etc).

Speedup file reading on linux

(Actually, this is pretty old post from my previos address.)

It's about how to speedup reading a pile of files from disk drives. Evident, such operation requires not so rarely - parsing couple of files, copying them over fast ethernet connection as like as moving them from one partition to another.

There are several things to speedup and I'm sure you know about IO-buffer size dependence, but one of main advantage achieves by "prereading" of data and doing this in right order, the things you usually can't control from userspace. Since the main obstacle while reading files is non-linear moving of disk drive heads, we should achieve as native physical order as possible.

This native order may be retrieved by using ioctl(FIBMAP) on opened file descriptor, but there are some limits: third argument of `ioctl' call presents pointer to integer - logical block being translated to physical on output, so obviously number of physical block able to be mapped is not very large. It may hit the limit on XFS and other huge FSes. There is also a big disadvantage of FIBMAP - it requires a superuser privilegies (don't know why). Instead of using old ioctl, new linux kernel provides an another one: FS_IOC_FIEMAP. This variant is much more flexible, universal and limit-safe. It also requires no superuser privilegies. This call provides you viewing of file as physical extents (even for filesystems allocating data by bitmaps, see flags). You can find much information in kernel documentation.

Here is the sample of how to retrieve first physical block by methods mentioned above:

#include <linux/fs.h>
#include <linux/fiemap.h>

uint64_t
get_physblock(const char *f)
{
    int fd = open(f, O_RDONLY);
    uint64_t block = ~0ULL;
    if (fd >= 0) {
#ifdef FS_IOC_FIEMAP
         union {
           struct fiemap fm;
           char buf[sizeof(struct fiemap) + sizeof(struct fiemap_extent) * 1];
         };
         memset(&fm, 0, sizeof fm);
         fm.fm_length = 1;       /* one byte mapping from logical offset=0 */
         fm.fm_extent_count = 1; /* buffer for one extent provided */
         if (ioctl(fd, FS_IOC_FIEMAP, &fm) != -1 && fm.fm_mapped_extents == 1)
           block = blk;
#else
         int blk = 0; /* first logical block */
         if (-1 != ioctl(fd, FIBMAP, &blk))
           block = blk;
#endif
        close(fd);
    }

    return block;
}

Test

Clearly right what relying on physical block ID, you may reorder files to read. In addition, there is a readahead(2) syscall which can be used to "preread" file data in VFS cache. It differs from the reading by read(2) since it has no "copy_to_userspace" overhead. Indeed, there is no much to talk about but give a test results. Testing principle is quite simple: read linux sources file by file. At first read them `as is', then apply readahead, and finally - preordering. FS cache between that cases may be purged by

$ echo 2 >/proc/sys/vm/drop_caches

Test results are following:

Method used	Time elapsed (sec)
as-is	33
readahead	26
reorder + readahead	14

It's not difficult to see what applying readahead, especially with preordering, gives much benefit. So, this method may be used for caching - there are several implementation engaged in popular Linux distributions, for example readahead package, used by default in Fedora and Ubuntu.

You can read details about fiemap on LWN page.

I hope this short note will convince you using this tricky calls when FS reading speed is valuable. Surely, this method worthy only for reading much of files, but not for couple of huge files since it's already self-ordered. For myself, I used this approach when developed library which creates dictionaries for classifying phrases - there was over 60000 of input files, and until then read this files consuming the most time.