handyfloss

I keep track of what pieces of music I listen to through last.fm since June 2006. This can be done automatically for you with Amarok, if you configure it to. As you can see in the following screenshot, I just listened to 30,000 tracks in this time!

(click to enlarge)

I listen to music continuously while I work, and I also count the music I listen at home too, so that’s why I have heard an average of 50-60 songs per day.

It is not uncommon to hear (mostly from Windows users berating Linux and its “useless console”) that one of the benefits of Windows is that everything can be done through a GUI. After all, clicking on icons and finding stuff in menus is more intuitive, and everything is easier that way. In contrast, with Linux you have to “type an awful lot of things, which is boring, slow, and difficult. And ugly”.

Now, don’t get me wrong, GUIs are great. I quite like them. What annoys me is the lack of command-line interface for some tasks. Both GUIs and CLIs have their place in computer use, and the wise should use each when appropriate. In this post I will try to illustrate a case where the automation allowed by using the CLI and some scripting is largely missed. The user (me) is forced to use an “intuitive” GUI, with the result that my patience takes a direct hit below the flotation line.

The first task I faced was to plot some orbitals of a molecule. The data for each orbital is saved in one file, and I am running a program that can read them and plot the given orbital (Molekel).

The following YouTube video, made by myself, shows the process of plotting 2 orbitals (I had to plot 17). Notice that, due to the program running so slow, the process takes around 1 minute per orbital!

Notice also that all the previous work has been done: choosing the colors of the background, atoms and orbitals, choosing the orientation, opening the atomic geometry… The comprehensive list of what to do for each orbital follows, with each line preceded by the point in time (seconds) when it happens:

1. 00.00 – Click on “Delete surface” to remove previous orbital
2. 07.80 – Click on “Load” to load a new orbital
3. 12.67 – Choose a file from the dialog window, and click on “Accept” to load it
4. 30.27 – Click on “Both signs”, because we want both positive and negative part of the orbital
5. 31.33 – Introduce a value for the isosurface (0.05) in the “cutoff” box
6. 33.13 – Click on “Create surface” to have Molekel render the isosurface
7. 37.33 – Isosurface appears
8. 37.93 – From a drop-down menu (called with right-click of the mouse), choose Snapshot -> RGB
9. 52.66 – “Save as” dialog box appears
10. 62.73 – Introduce filename for snapshot, and click “Accept”
11. 65.00 – We’re done, and can repeat the process for the next orbital

One can’t help but notice that 65 seconds are needed to make eight clicks and introduce a short text in two boxes! The issue is that human attention is necessary during the whole 65 seconds, because the time between actions is too short to do something else in between (although long enough to get on your nerves, like the full 15 seconds to have the “Save as” dialog appear).

Another obvious point is that from the two short texts introduced by the user, one (the value of the isosurface) is always the same, and only the other (the name of the file to save the snapshot as) varies. Also, only one click of the 8 we do is ever different (the choice of orbital file to read). It would be nice to have a robot do this task, the only data we would have to feed it being a list of orbitals (to read, and then to save a snapshot). But we can’t. We are stuck with this sluggish process!

In contrast, I will next show a case where some automation was made. The process is that of cropping the snapshots taken in the previous step (the Molekel thing). Sure, we could use GIMP, or some other GUI tool, but applying exactly the same process to a list of 17 images (and this is a short list, it could have been 1000) is the kind of thing that cringes for automation.

The following video shows the process:

Recall that it takes 4 minutes to process ALL the images. This may not sound like a huge improvement over the 18:25 that it (in principle) took the process above (17 x 65 sec). However, the time spent with Molekel scales linearly with the number of orbitals. 100 orbitals would need almost 2h. The automated cropping process would have taken more than 4 minutes, but only slightly more: maybe 5 or 6.

Also notice that the 4 minutes are full of decisions, and there is no repetitive, unnecessary task (except the fact of committing errors). Let’s take a look at the actions taken during the 4 minutes:

1. 00:11.00 – Open a Perl script I had half-done (another benefit of automation: you can reuse old stuff)
2. 00:17.87 – Shade window to take a look at the number and name of files to process
3. 00:21.33 – Change script accordingly
4. 00:47.93 – Save changes
5. 00:51.60 – Back to the CLI, and run the script
6. 00:55.00 – Ups, nothing happened!
7. 00:58.27 – Reopen the script, and look for the error
8. 01:05.53 – Found it. Fix it.
9. 01:07.33 – Save and execute
10. 01:06.87 – It works!
11. 01:13.00 – Finished running (0.36 sec per picture)
12. 01:21.73 – Open a cropped image in viewer
13. 01:22.93 – Realized the crop is wrong!
14. 01:30.73 – Alt-Tab to script file, to modify it
15. 01:55.67 – Save and execute again
16. 02:11.73 – Open the cropped images. The first one seems to be OK!
17. 02:29.80 – We reach one that is wrong
18. 02:34.20 – Back to the script, and fix it
19. 02:45.07 – Save, and back to CLI to re-run
20. 02:53.00 – Reopen in image viewer
21. 02:56.00 – Cropped part is not centered!
22. 03:02.00 – Back to the script, and fix it
23. 03:12.33 – Save and re-run
24. 03:19.73 – Reopen in image viewer
25. 03:27.87 – Yet another error: an image could have been cropped more, to hide an unwanted part
26. 03:32.40 – Back to the script
27. 03:40.80 – Rerun
28. 03:47.13 – Reopen images
29. 03:59.93 – See that all of them are correct. Stop and rest

Recall also that if I were to repeat both processes tomorrow, the image cropping would simply require to run the script again (0.36 seconds per image, and you can do something else in between, if you have 1000 images and don’t want to waste time waiting). The creation of the orbitals, on the other hand, would require to repeat the whole process again!! (65 seconds per orbital, plus you have to spend that time paying attention to the process. You can not run something and go away). And the whole problem with the creation of the orbitals is that there is no command-line way of doing it, to be able to automate it.

Remember my previous post about a problem with Flash in Firefox/Iceweasel? Now the second part.

After following my own instructions, I ended up with a Flash instalation that could play YouTube videos correctly, but some other Flash animations would not work. By chance, my computer at work would reproduce any Flash animation just fine, so… why would that be?

To find out the reason, I have compared what Flash-related packages I have installed in Homer (my computer at work) and Heracles (the one at home). The result is quite surprising:

Homer[~]: aptitude search flash
p   flashplayer-mozilla       - Macromedia Flash Player
p   flashrom                  - Universal flash programming utility
p   flashybrid                - automates use of a flash disk as the root filesystem
p   libflash-dev              - GPL Flash (SWF) Library - development files
p   libflash-mozplugin        - GPL Flash (SWF) Library - Mozilla-compatible plugin
p   libflash-swfplayer        - GPL Flash (SWF) Library - stand-alone player
p   libflash0c2               - GPL Flash (SWF) Library - shared library
p   libroxen-flash2           - Flash2 module for the Roxen Challenger web server
p   m16c-flash                - Flash programmer for Renesas M16C and R8C microcontrollers
p   vrflash                   - tool to flash kernels and romdisks to Agenda VR
Homer[~]: aptitude search swf
p   libflash-swfplayer        - GPL Flash (SWF) Library - stand-alone player
p   libswf-perl               - Ming (SWF) module for Perl
p   libswfdec-0.5-4           - SWF (Macromedia Flash) decoder library
p   libswfdec-0.5-4-dbg       - SWF (Macromedia Flash) decoder library
p   libswfdec-0.5-dev         - SWF (Macromedia Flash) decoder library
v   libswfdec-dev             -
p   pyvnc2swf                 - screen recording tool to SWF movie
v   swf-player                -
p   swfdec-mozilla            - Mozilla plugin for SWF files (Macromedia Flash)
p   swfmill                   - xml2swf and swf2xml processor

Yes, Flash works perfectly at Homer because it has no package installed with swf or flash in their name! And I don’t have any Gnash package installed, either. I removed all swf/flash-related packages on Heracles, and now Flash works perfectly in my home computer too.

I have released (how pretentious!) rip2ogg.py, the wonderful CD ripper everyone was waiting for. You can check its “home page” at isilanes.org.

Why did I do it? Well, one of the wonderful tools GNU/Linux provides to rip CDs is KAudioCreator, which is very neat. However, it has some shortcomings I wanted to overcome (again, how pretentious!):

1. It’s slow. It rips the CD to WAV and encodes the ripped WAVs to Ogg in parallel, while rip2ogg.py does both things sequentially. Yet rip2ogg.py is 40% faster! I have ripped a whole CD in 14 minutes with KAC, and in 10 minutes with r2o.
2. You can not have arbitrary character substitution, just one, and the interface for that is horrible. For example, with KAC it’s very simple to substitute every blank in the track name with an underscore. BUT I have found no way to provide KAC with two lists, so that it substitutes every character in the first list with the corresponding character(s) in the second list.
3. You can change the track title to get a “nice” filename for the Ogg, but the change is also reflected in the “track tile” tag. You can not tell KAC to substitute a “ñ” in the title for a “n” in the filename, but to keep the “ñ” in the “tag title” tag.
4. KAC is not able to rip all CDs. It sometimes chokes on DRM‘d CDs, and copes horribly with scratched surfaces. In contrast, the programs rip2ogg.py uses to rip have never failed for me. More than once command-line was my only way of ripping some rogue CDs. KAC simply couldn’t.

In the end, it all boils down to be able to control what the ripper is doing. To do so, I decided to make this simple script.

Obviously it is FLOSS (GPLv2), so use, modify and redistribute to your heart’s content!

This YouTube video shows Compiz Fusion running on my work computer. It has a fairly decent CPU (P4 3.00GHz), but no “useless” things like sound cards or (more relevant for this issue) graphics card. The only thing it has is an Intel 82865G graphics chip integrated in the motherboard. We are talking about an integrated chip (not dedicated graphics card) released in May 2003.

Judge the performance for yourself (take into account that the actual performance is higher, since the recording program to make the video also uses up some resources):

Yesterday morning, a new failure from Iberdrola turned the power supply of the whole campus off. So, here goes the updated list of blackouts I have been able to compile, with comments if any:

1. 2007-Dec-10 (I used the reboot of my computer to install kernel 2.6.22-3)
2. 2007-Oct-16
3. 2007-Aug-27 (at least three short power failures, 5-10 minutes apart)
4. 2007-May-19
5. 2006-Oct-21 (they warned beforehand)
6. 2006-Sep-14 (Orpheus fell, the DNSs fell, the DHCP servers fell)
7. 2006-Jul-04 (Orpheus didn’t fall)
8. 2006-Jun-16
9. 2006-Jun-13
10. 2006-Jun-08
11. 2006-Jun-04
12. 2006-May-26 (The card-based automated access to the Faculty broke down)
13. 2005-Dec-21
14. 2005-Dec-13

Summary: 14 blackouts in 728 days, or 52 dpb (days per blackout). 56 days since last blackout. Average dpb went up by 0.3.

First post in the series: here

One argument I tend to hear from Windows users is that in Windows you can do as much as you can with Linux, and that the technical advantages of Linux only show up if you are really an utter geek. This is one of (I hope) a series of entries in my blog, illustrating some cases where this doesn’t hold: I took advantage of tools provided by Linux in a way that anyone could have, not just geeks.

The moral of it all is that Windows encourages a lack of choice and flexibility that makes users tend not to be creative, and think the cage Windows keeps them in is actually a shelter from the storm, when it’s not. They think that what can’t be done with Windows, needs not be done. I think otherwise…

Today I will try to provide an example in which hard links can be useful. Under Windows XP hard links can be created, using the fsutil utility, but only for NTFS file systems, and only by the Administrator account (and only from the command line). If you want to learn more about links and specially Windows links, read this interesting sell-shocked.org article.

The problem

I download a lot of music from Jamendo, using the BitTorrent p2p protocol. After having downloaded a given album, I tend to leave the torrent open, so that people can continue uploading from my computer.

However, I also want to have my music collection tidy and ordered, so I immediately organize the newly-dowloaded songs moving them to a neat directory tree I have, will all my music.

So, there is a conflict between keeping the files in the bittorrent download/upload dir, and properly organizing them. I don’t want to have to wait until I decide to stop sharing a file to organize it, and I don’t want to risk deleting the files if I remove them from the bittorrent client before saving them elsewhere. I could get over all this by simply making a copy of the files… but then I would be filling twice as much disk space, and with GBs of shared files, this is not neat at all.

The solution

What I do is hardlink all the downloaded files to their final location. If I download all torrents to /scratch/ktorrent/, a downloaded album will look like that:

% ls /scratch/ktorrent/album1/
song1.ogg song2.ogg song3.ogg [...]

If I want to save the album under my artist1 directory, I do the following:

% mkdir /scratch/music/artist1/album1
% ln /scratch/ktorrent/album1/* /scratch/music/artist1/album1/

This way all the “song*.ogg” files will appear to be in both /scratch/music/artist1/album1/ and /scratch/ktorrent/album1/ at the same time.

Benefits:

1 – I can keep sharing the files in /scratch/ktorrent/album1/, while listening to and/or manipulating the /scratch/music/artist1/album1/ files as if I had 2 copies of each.

2 – The total size is not affected. The hard links do not “occupy” space (only a few bytes each).

3 – I can delete the files in the shared directory without any fear. Only the “copy” in /scratch/ktorrent/ disappears, while the other “copy” in /scratch/music/artist1/album1/ becomes the only copy (just as if it had always been a “normal” file, and the only one).

Recall that all files are hard links. Normally a given file is the only hard link to a given piece of data in the hard disk, but there can be more “links” pointing to that data. When we remove files, we only remove the “link” pointing to the data.

Following the “report” series started with my first summary of info about the music collection I listen to, I will update that info in this post.

The data (in parentheses the difference with respect to last report, 7 months ago).

Files

Total files        7036 (+1021)
  - Commercial     4465 (+301)
  - Jamendo        2533 (+713)
  - Other CC       31 (+0)
  - Other          7 (+7)
Total playtime     18d (+2d)
Disk usage         32GB (+5GB)
Artist count       817 (+99)
Album count        618 (+103)
MP3 count          0 (+0)
OGG count          7036 (+1021)


Last.fm

Playcount           26246 (+9502)

Most played artists Joaquín Sabina - 2252 plays (+810)
                    The Beatles - 983 plays
                    Silvio Rodríguez - 626 plays
                    David TMX - 599 plays
                    Siniestro Total - 536 plays

Most played songs   La del pirata cojo (J. Sabina) - 47 plays (+18)
                    Conductores suicidas (J. Sabina) - 46 plays
                    Pacto entre caballeros (J. Sabina) - 42 plays
                    Cuando aparezca el petróleo (E. Sánchez) - 42 plays
                    Y sin embargo (J. Sabina) - 40 plays

Amarok

Playcount         18186 (+5740)

Favorite artists  Ska-P - 93.73%
                  Juan Luis Guerra - 92.84%
                  Peiremans - 91.96%
                  Su ta Gar - 91.86%
                  La Caja Negra - 91.64%

Favorite songs    Tirado en la calle (E. Sánchez) - 97%
                  Fiesta pagana (Mägo de Oz) - 97%
                  Cuando aparezca el petróleo (E. Sánchez) - 97%
                  Las cuatro y diez (L.E. Aute and S. Rodríguez) - 97%
                  Wheels of fire (Manowar) - 97%

I just changed the theme from Contempt to Andreas09. I will miss the custom header, and I liked the previous style just a little bit better than this one. But for me the former was too narrow. From time to time I write really long posts, and they look horrible if the text is not wide. I also want to be able to add pictures with the size I deem appropriate, not with a tiny size to fit in the narrow blog column. So, I will be sacrificing looks for the sake of readability.

I just found out that my regular backups at a couple of computers are filling up the corresponding disks (for the Spanish readers: ¡están petaos!), and I realized that it is because I was keeping a bunch of 200MB files uncompressed. Since the files are ASCII, full of numbers, most of which are actually zeros, they are perfect candidates for compressing them with tools like gzip or bzip2. Everybody knows that the latter is more efficient, but slower, so I made a small comparison:

Original file: 211MB
gzip: 4.5MB in 11 s (compress), 6.5 s (uncompress)
bzip2: 2.4MB in 1323 s (compress), 27 s (uncompress)

Yes, the compression with bzip2 is impressing: 88x compression, where gzip gets 47x (almost a 90% better compression). But the timing is poor: bzip2 is 120 times slower than gzip. For uncompression, bzip2 fares better: “only” 4 times slower than gzip. Where gzip can uncompress a file in about half the time it took to compress, bzip2 does the same almost 50 times faster (because compressing was soooo slow).

This case is anecdotal, but it nicely illustrates my experience in general.