handyfloss

This YouTube video shows Compiz Fusion running on my work computer. It has a fairly decent CPU (P4 3.00GHz), but no “useless” things like sound cards or (more relevant for this issue) graphics card. The only thing it has is an Intel 82865G graphics chip integrated in the motherboard. We are talking about an integrated chip (not dedicated graphics card) released in May 2003.

Judge the performance for yourself (take into account that the actual performance is higher, since the recording program to make the video also uses up some resources):

Intro

Any GNU/Linux user ends up hearing, sooner or later, the Ultimate Argument(tm) from a Windows fanboy:

If Linux is so good, and is given away for free, how can Windows still be so prominent?

Ironically, many Windows users answer the question themselves, when explaining why they can’t make the switch to Linux:

• There are no games for Linux
• Photoshop or AutoCAD are vital for me, and they only work under Windows
• I fear some pieces of hardware won’t work under Linux
• The web page of my bank only displays correctly under IE
• My friends/colleagues/business partners share documents in MS Office formats, and I need to be compatible

The concept than embraces all the preceding points, and answers the rhetorical question above, is vendor lock-in. I will try to explain the concept with a humble tale I have used twice so far in comments to entries in Enrique Dans’s blog.

Tales of bicycles and cars

Imagine a country with no bicycles.

One day a guy comes up with the idea of making them, and starts to produce, and sell, bicycles that we shall call of type A. Being an empty market, the A-type bicycles quickly triumph, and the maker makes a lot of money.

But some time later, a second guy devises a better bike design (type B), and decides to produce and sell it. The price and the quality are better, so when people buy a new bike or replace an old one, they tend to buy bikes of type B. Soon enough, the market is dominated by the new, better, bicycle.

Now imagine a country with no cars.

One day a guy comes up with the idea of making cars, and starts to produce and sell cars of type A. As cars need petrol to run, A-type gas stations develop in parallel to car sales. Building gas stations is expensive, but sales are guaranteed, as everyone has or will have A-type cars, and they need A-type petrol: their growth is synergistic.

But some time later, a second guy devises a better car design (type B) and decides to produce and sell it. The price and the quality are better, BUT drivers can not buy B-type cars, because there is no B-type gas station. The problem is that, since noone has a B-type car, making B-type gas stations is doomed to bankrupt. So, no B-type cars are sold, because there are no B-type gas stations, and B-type gas stations will not be made until B-type cars are popular!

The result is a vendor lock-in.

Consequences

When a market (such as the one in the tale above) is dominated by vendor lock-ins, the producers benefited by the lock-in have little, if any, incentive to make better products. Their sales are guaranteed by the lock-in situation, not by their superior product in a fair market.

On the other hand, other producers will have an extremely hard time for competing, as their products will be almost unusable for the buyers.

The moral is that the lock-in situation is bad for either the potential users of the product locked-out (the B-type car above) and the locked-in one (the A-type car). Even if a consumer would never choose the locked-out product, the lack of competition will adversely affect the evolution of the product they do choose. The lock-in is bad for everyone: all consumers and all producers but the locker ones.

And this relates to Windows vs. Linux in what way?

In a really straightforward way. Microsoft, cunning as they are, have tried their best to get as many lock-ins on the software market as they can. Ironically, instead of abhorring this practice, most Windows users happily continue not only using, but even defending the product. I shudder at the simpleton comment that “freedom is not using Linux, but using Windows and Linux whenever you feel like it”. Literally taken, it is a very wise argument. But unfortunately the reality is not so simple: using Windows helps enforce a lock-in that keeps Linux out (while in this case the contrary is not true). You can not use Windows/Linux 50/50, because Windows asks you for monogamy.

The many lock-ins that MS has forced down the throats of the users, while the latter still claim them to be benefits of Windows include (as mentioned at the beginning):

• Proprietary communication protocols that will not work with any other OS. This includes modifications on the IE web browser, so that web pages had to be done for it, and then be incompatible with other browsers. Or the MSN protocol, that is kept as closed as possible, to make free clones of the MSN client as little compatible as possible.
• Proprietary file formats that will not be possible to modify with tools other than the “official” MS ones: WMV for video, DOC, XLS, PPT for office documents.
• As much “Windows-only” software as possible, including games. Making games for platforms other than Windows ensues the wrath of MS, something that game makers can not take lightly, since their sales depend on the game actually running under Windows.
• As much “Windows-only” hardware as possible. The first idea that someone gets about an OS is that it is the piece of software that interacts with the hardware. If so, it is astonishing why it is not MS the one incorporating the drivers in the OS, instead of the hardware makers (hardware != software) providing them. We have all grown accustomed to buying printers, mice, external CD/DVD/HDs… with a CD with the “Drivers for Windows 98” or some such. Why? Windows can not make the drivers out of the blue, true. But the hardware makers can just make the necessary data public, so anyone will be able to make drivers. If the maker keeps these specifications secret, they will simply not sell anything.

One argument I tend to hear from Windows users is that in Windows you can do as much as you can with Linux, and that the technical advantages of Linux only show up if you are really an utter geek. This is one of (I hope) a series of entries in my blog, illustrating some cases where this doesn’t hold: I took advantage of tools provided by Linux in a way that anyone could have, not just geeks.

The moral of it all is that Windows encourages a lack of choice and flexibility that makes users tend not to be creative, and think the cage Windows keeps them in is actually a shelter from the storm, when it’s not. They think that what can’t be done with Windows, needs not be done. I think otherwise…

Today I will try to provide an example in which hard links can be useful. Under Windows XP hard links can be created, using the fsutil utility, but only for NTFS file systems, and only by the Administrator account (and only from the command line). If you want to learn more about links and specially Windows links, read this interesting sell-shocked.org article.

The problem

I download a lot of music from Jamendo, using the BitTorrent p2p protocol. After having downloaded a given album, I tend to leave the torrent open, so that people can continue uploading from my computer.

However, I also want to have my music collection tidy and ordered, so I immediately organize the newly-dowloaded songs moving them to a neat directory tree I have, will all my music.

So, there is a conflict between keeping the files in the bittorrent download/upload dir, and properly organizing them. I don’t want to have to wait until I decide to stop sharing a file to organize it, and I don’t want to risk deleting the files if I remove them from the bittorrent client before saving them elsewhere. I could get over all this by simply making a copy of the files… but then I would be filling twice as much disk space, and with GBs of shared files, this is not neat at all.

The solution

What I do is hardlink all the downloaded files to their final location. If I download all torrents to /scratch/ktorrent/, a downloaded album will look like that:

% ls /scratch/ktorrent/album1/
song1.ogg song2.ogg song3.ogg [...]

If I want to save the album under my artist1 directory, I do the following:

% mkdir /scratch/music/artist1/album1
% ln /scratch/ktorrent/album1/* /scratch/music/artist1/album1/

This way all the “song*.ogg” files will appear to be in both /scratch/music/artist1/album1/ and /scratch/ktorrent/album1/ at the same time.

Benefits:

1 – I can keep sharing the files in /scratch/ktorrent/album1/, while listening to and/or manipulating the /scratch/music/artist1/album1/ files as if I had 2 copies of each.

2 – The total size is not affected. The hard links do not “occupy” space (only a few bytes each).

3 – I can delete the files in the shared directory without any fear. Only the “copy” in /scratch/ktorrent/ disappears, while the other “copy” in /scratch/music/artist1/album1/ becomes the only copy (just as if it had always been a “normal” file, and the only one).

Recall that all files are hard links. Normally a given file is the only hard link to a given piece of data in the hard disk, but there can be more “links” pointing to that data. When we remove files, we only remove the “link” pointing to the data.

This is a short HowTo for making unicode work in Linux, specifically in the command line. Yet more specifically, in the konsole terminal. This is useful if you want to be able to use characters like ‘ñ’ or accents like in ‘á’ and ‘ö’.

1 – Modify your shell locale variables

You need locale settings that support UTF (for example en_US.UTF-8). For that, you can add the following lines to .tcshrc or whatever script run at login:

setenv mylang   en_US.UTF-8
setenv LANG     $mylang
setenv LC_CTYPE $mylang
setenv LANGUAGE $mylang
setenv LC_ALL   $mylang

The ‘$mylang’ thing is just because I’m lazy, and I might want to change them all in the future, and I don’t want to type too much.

2 – Modify your global locales

I don’t know if this is needed, but it doesn’t hurt. In Debian:

% dpkg-reconfigure locales

and follow the instructions, using en_US.UTF-8 or something similar as default.

3 – Modify the encoding of Konsole

In the menus:

Settings->Encoding->Unicode (utf8)

Make this permanent with:

Settings->Save as Default

Then choose xterm and not linux as keyboard setting:

Settings->Keyboard->Xterm (XFree 4.x.x)

You can make this permanent in the Session tab of:

Settings->Configure Konsole

namely inserting “xterm” in the box labeled “$TERM”.

If you follow these instructions, you will be able to introduce non-ASCII text in the terminal, and use non-ASCII filenames without problem.

I have been using Subversion for a while (after having it recommended by my colleague Thomas), and I must confess I’m a happy user. Subversion is a revision control system, designed to supersede, and replace, the (maybe) more popular CVS.

Subversion (svn) is good for much more than collaborative development, as a single person can keep track of versions of her own documents/scripts/whatever. Usually you only want the last version of whatever you work with. But whenever you find yourself saving a version somewhere else, to keep it like that even if further changes are made to the “current” version, svn is your friend. Whenever you wish you had saved an earlier version of the stuff you’re working with, you’re missing (know it or not) svn.

I came across a blog post [es] talking about Windows 7, the planned sucessor of the current Windows Vista. The same can be found elsewhere, e.g. in Ars Technica.

The article summarized some features that Windows users would like to see in W7. You can also see a picture with the whole W7 wishlist. What struck me was that, although the Redmond giant tried its best to copy every single innovation from free software, they still missed important points that users value enough to make a wishlist out of them.

Some points in the list are new and exciting. Some others are everyday things for us free software users, and it’s so amazing that Windows still does not include them:

1. Request for an integrated font manager
   One of the problems of proprietary software: the pieces each programs uses (including fonts) are property of the maker, so sharing is largely hindered. In Debian we have things like Defoma, and font management is quite lean in any distro, anyway.
2. Explorer toggle button to quickly show/hide hidden files or system files
   Files starting with a dot are hidden in Linux. All file managers I know of have the hability to show/hide them with a click or a shortcut (Ctrl-H in Thunar and Nautilus, no default but configurable shortcut for Konqueror).
3. Network/Internet bandwidth monitor
   Most, if not all, docks/taskbars in FLOSS desktops (Xfce, GNOME, KDE…) have a widget for that.
4. DirectX update on Windows Update/Microsoft Update
   I use Debian, and it manages the installed software with APT (other distros have other systems). With it, I run “aptitude update” and it searches the online repositories for the last version of all the packages that exist in them. When I do “aptitude safe-upgrade“, it automatically upgrades all the packages for which there are updates, and notifies me if some upgrade requieres to install a new package (without upgrading it until I agree to install that new package). And it’s been like this for years.

5. Infinite desktop, virtual desktop idea
   Although it probably refers to zooming interfaces, Linux has had the idea of virtual desktops for years.
6. Profile data: Move locations of all user folders and data to another location
   This is trivial in Linux since the dawn of its times.

7. Option to “Reopen Closed tabs” in IE
   Firefox has this option through add-ons like Tab Mix plus. Not only that, but many other things are possible, like: periodic reloading of some or all tabs, closing all tabs but the current one, duplicating tabs (along with all their history), freezing tabs (so they can not be accidentally closed or moved away from), change the name of the tab…
8. Auto clean of Temp folders
   Temporary file management in Linux is flawless. I never saw a tmp location full because the system forgot to clean it.
9. Provide Manual Duplex Printing in Windows Pring Dialog
   It is really lame to need to ask the maker of a big, monolithic, OS for stupid changes like that. The printing dialogs should be made by the desktop environment (a small part of the OS), or the application, and it should communicate with the printing server (another smaaaaall part of the OS). Details like that one should be fixed by updates in only one/some small packages related to the desktop environment.
10. IE should have a close button on each tab
    See point 7.
11. Disk Manager needs to have the ability to expand partitions
    Tools like GParted make partition management a breeze. In Windows, you need commercial third party tools for that. Tough luck.
12. Image (ISO, BIN) support in Windows
    What? In Windows you can not mount ISO images as if they were actual filesystems? In Linux, you sure can.
13. Family license
    It must suck to buy a copy of the OS and being able to use it only in one PC. With Linux and free software, you obviously don’t have this problem, and you don’t need to go crying to your dealer for a more mercyfull license
14. No dialog should take keyboard focus away from what you are doing
    With all serious desktop environments, you can configure this behaviour, as well as if focus follows mouse, or if you have to click on a window to make it active and so on.
15. Patch operating system without having to reboot
    With Linux, you only need to reboot if you install a new kernel (you can’t use a different kerner without rebooting). For everything else, you don’t need to.
16. Add folder size to data displayed by Windows Explorer
    Wow, it must suck being stuck with a single choice for a file manager (or any other task), and not being able to configure stupid things like that to your liking. Another con of Windows, I guess.
17. Live CD or DVD to boot from to recover from a crash or virus that would allow to transfer files
    But there is a tool for that task on Windows! It is called “Linux Live CD”, and many distros have it. I have read that it is pretty popular among some Windows users: when their system is utterly destroyed, a Linux Live CD can save the precious data in their disks.
18. Disallow removable (USB/Firewire) drives to default to next available drive letter when the letter is already used by other network drives
    I know the issue of wanting to have permanent names for given devices, no matter what. The solution is called udev.
19. Windows Mail should be minimizable to the system tray
    I use KMail and it is. Probably Thunderbird is, too. By the way… ever guess how similar to the former two Windows Mail is (by the looks in the Wikipedia article)?
20. Command Prompt should be improved
    Hehehe. I have no words.
21. Integrated Anti-Virus
    What is a virus? Please explain, I’m an ignorant Debian user!
22. More desktop themes should be offered in the default installation of the next version of Windows
    I thought Windows users wanted consistency and simplicity, and everything to look the way uncle Microsoft wanted. In Linux, we have soooo much to choose from. You doubt it? Take a look at KDE-look.org, or Xfce-look.org.
23. IE direct download – do not download to temp folder
    With any free browser (e.g. Firefox) you can choose the default dir for the downloads, and you can choose for each download where to put it (if you don’t want it in the default folder). Is it not like that in IE?

Maybe some slipped through, but I’m too tired to be more throughout.

Actually it is two applications I want to highlight: Filelight and Baobab. Both are disk usage analyzers, the former for KDE (see Figure 1), and the latter for GNOME (see Figure 2).

Figure 1: Filelight (click to enlarge)

Figure 2: Baobab (click to enlarge)

A disk usage analyzer is a tool to conveniently find out how much hard disk space different directories and files are taking up. It combines the effectiveness of the Unix du (if you never used it, stop here and do a man du in your command line immediately. If you do not know what that “command line” thingie is, whip yourself in the back repeatedly), with the convenience of a visual clue of how large directories are compared to one another.

From the two DUAs I mention, I largely prefer Filelight, for some reasons:

1 – When I want to open a terminal in a location chosen from the DUA window, with Baobab it’s two clicks away: “Open file manager here”, then “Open terminal here” in the file manager. With Filelight, it’s just one click: “open terminal here”. Plus Filelight has a handy locator bar at the top, showing the full path to the current location (useful to copy-and-paste with the mouse to an already open terminal).

2 – Filelight shows directories up to individual files. Baobab just dirs.

3 – With Filelight, navigation up and down (and back and forward) in the dir tree is a breeze (web browser-style). With Baobab, it’s a pain.

4 – The presentation is similar, but the one of Filelight is slightly nicer, with more info when the mouse is hovered over the graph.

Probably Baobab can be easily made to behave like Filelight. I just tried them both, and liked the latter better on first sight. I tried Baobab first, and I found some things lacking. When I tried Filelight, five minutes later, I just thought “These are the details Baobab was missing!”

The setup

A colleague wanted to edit a video (actually, three) for a presentation she intended to make in a laptop with Windows Vista and MS Office 2007. The video was a WMV, and the required edition included removing fragments, changing the speed of one of the fragments (and join it back with the others), and adding a soundtrack.

The problem

She could not, for the life of her, edit the damned thing on a Windows computer.

The solution

Why, Free Software, of course.

The motivation of this post

To help dispell two ideas: that “Windows is easy”, and that “With Linux, you waste your time finding out how to do things”.

The story

Part I – Linux

OK, so I proposed her to use some Free Software called Avidemux. Our first problem was that apparently Avidemux was unable to cut the video in pieces (it crashed at the attempt). After much perusing, and using the humble file command, I found out the reason: the WMV had no playing FPS set. Players, like MPlayer would reproduce it by guessing 25 or 30 frames per second, but editors need a precise value to count on. I readily fixed it by reencoding the video to 25fps with MEncoder:

% mencoder in.wmv -ovc lavc -nosound -fps 25 -lavcopts vcodec=wmv1 -o out.avi

Once a proper FPS given, I used Avidemux to split the file. However, I encountered a second problem: I couldn’t split the file anywhere. I could only cut it at points 10 seconds appart. I had to sweat a bit more to fix that, but I also learned something more in the way. Most (all?) compressed video formats use at least two kinds of frames: normal frames and keyframes. The latter are the frames where any player can seek to in the video. According to the man page of MEncoder:

keyint
maximum interval between keyframes in frames (default: 250 or one keyframe every ten seconds in a 25fps movie. […] Keyframes are also needed for seeking, as seeking is only possible to a keyframe – but keyframes need more space than other frames, so larger numbers here mean slightly smaller files but less precise seeking. 0 is equivalent to 1, which makes every frame a keyframe. […]

So here you are: the problem was the default value of some variable called keyint. To make the video seekable to any frame (so it could be cut at any point), I set keyint to 1:

% mencoder in.wmv -ovc lavc -nosound -fps 25 -lavcopts vcodec=wmv1:keyint=1 -o out.avi

Once the movie was split into parts with Avidemux, and the unwanted parts were removed, the next step consisted on playing one fragment faster. The problem here is that I don’t know how to make a variable FPS video, so we had to make it so all the video played at the same FPS, but a part was faster. How? Removing frames, of course. I used MPlayer to deconstruct the relevant fragment into individual frames (in PNG format):

% mplayer -vo png:z=2 fragment

The command above generates a whole lot of 0000xxxxx.png files, with frames ordered by the number in the filename. Next, I deleted every second frame. How? With a stupid GUI I don’t know, but from the command line it is trivial:

% rm -f 00*[13579].png

Now, I just re-constructed the video with half the frames, to get an effectively double-speed video, with same FPS as original:

% mencoder "mf://*.png" -mf fps=25 -o output.avi -ovc lavc -lavcopts vcodec=wmv1

If I am allowed to say it, the effect is really great. You wouldn’t tell the sped-up video from the original, except from the increased play speed.

Using Avidemux for joining the video fragments was a breeze, and it could even be done from the command line:

% avidemux fragment1.avi --append fragment2.avi --append fragment3.avi --save total.avi --quit

The last (Linux) part consisted on adding a soundtrack, which Avidemux can do, from a MP3, WAV, or another video. This was easy.

Part II – Windows Vista

OK, the last Linux step consisted on reencoding the video in some format that Vista could read. This was no immediate task, but after some tests, we made it. Windows Media Player could reproduce the movie with no problem.

Finally, we opened the wonderful Office 2007 in the shiny and new Vista laptop, and created a PowerPoint slide to insert the video (the rest of the presentation was already done). Everything seemed to work, but when we played the presentation, we discovered that either the video or the sound could be played (depending on how we had encoded the video in Linux), but not both simultaneously. WMP would play the videos just fine, but the embedded player in PowerPoint would not… go figure why. After at least 3 crashes of Office (yes, Office crashes), some bitching because we could not make any sense of the new Office interface (we are experienced pre-Vista and Linux users, and Windows is for idiots, right? We must be idioter than average) having to stand the fact that the semitransparent border of a window refused to disappear when we closed it (so we kept working with a blue-greenish stripe across a part of the desktop), and one Windows reboot (yes, Vista still hangs from time to time), we managed to insert and play the darned video. How? We just inserted two videos: one for which only the audio was playing, and another one for which only the image was showing. We then make these two objects to kick off at the same time, et voila!. Not the cleanest of solutions, but with Windows “everything just works”, right?

The moral

The moral of the first step (the FPS not being set) was that I had to play around for a while with my Linux tools, but the culprit was MS, and their lousy WMV. I have never produced a video with no FPS (and all other necessary metadata) set, because my FLOSS tools do it automatically. Secondly, I didn’t waste my time. Thanks to the usefulness of the FLOSS tools, I ended up learning something about movies, FPSs, and that they are required. I also learned about key frames, and seeking and cuting video streams.

On the other hand, for a much simpler job, we spent relatively (and maybe absolutely) longer with Windows, and we did lose our time with it. The problems we encountered with Linux were difficulties of the situation itself: the original WMV was flawed, the AVI we created had too high an inter-keyframe interval… and the FLOSS tools we used helped us fix them and learn in the process. In the case of Windows, the task was so simple, and all the problems we met were created by Vista. We didn’t learn anything from all of our struggle, because we only struggled against Windows (the GUI, the crashes, how to encode the video in Linux so that Vista could read it, why the darned Office would not play the audio or the video), not our problem (editing and embedding the file). All the time was devoted to learning how to overcome the limitations and errors of our tool, not to how to use our tool to perform some task, learning about the task itself in the process. Thus, it was wasted time.

I read in The Register that a zero-day vulnerability has been reported in Oracle 10g databases. I am by no means an expert in databases (“not an expert”, wow, what an understatement! I’m an ignorant), but I have my small war against people who regard proprietary DBs such as Oracle or IBM DB2 as far above free software alternatives such as MySQL or PostgreSQL. To put an example company with HUGE databases, Google uses MySQL. Actually, I just found in the previous link this post in an ex-Google employee’s blog, and I plan to show it to any half-wit parroting the motto that “big commercial solutions” are by default better than “hobbyist things” like free software (specially for DBs).

So, when I read the Register headline, I immediately thought of writing a post on how “bad” Oracle was. However, after actually reading the (short) article, I decided to change the main point of the post. Actually, what this case shows is how “bad” depending on proprietary software is. Quoting the Register article:

Oracle has reportedly created a fix but is not willing to break its quarterly patch release cycle to issue an update. The database giant’s next update is schedule for 15 January. In the absence of a patch no ready workaround is available, iDefense reports.

Holy crap! Oracle acknowledges that the bug is there, that it is dangerous, and that they do have a fix, but they friggin’ don’t want to release it!. Just because “it doesn’t fit” in their well-laid plans! No need to say that with free software this can not happen: there is no reason to hold on on bugfixes. And even if there was, anyone can write a patch, and release it, so there is no vendor locking the users to it, and deciding what to release and when.

I just found out that my regular backups at a couple of computers are filling up the corresponding disks (for the Spanish readers: ¡están petaos!), and I realized that it is because I was keeping a bunch of 200MB files uncompressed. Since the files are ASCII, full of numbers, most of which are actually zeros, they are perfect candidates for compressing them with tools like gzip or bzip2. Everybody knows that the latter is more efficient, but slower, so I made a small comparison:

Original file: 211MB
gzip: 4.5MB in 11 s (compress), 6.5 s (uncompress)
bzip2: 2.4MB in 1323 s (compress), 27 s (uncompress)

Yes, the compression with bzip2 is impressing: 88x compression, where gzip gets 47x (almost a 90% better compression). But the timing is poor: bzip2 is 120 times slower than gzip. For uncompression, bzip2 fares better: “only” 4 times slower than gzip. Where gzip can uncompress a file in about half the time it took to compress, bzip2 does the same almost 50 times faster (because compressing was soooo slow).

This case is anecdotal, but it nicely illustrates my experience in general.