Unix HOWTOs and Tips Short unix command line administration tips and scripts

11Jul/140

Execute identical commands in all running OpenVZ containers on a hardware host

Suppose, that you have to execute some identical commands on all your running OpenVZ containers. For example you may want to check their disk usage, or ban an IP on all of them, or force a restart of their webserver for some reason. Typing the same commands over and over again is tiresome, and error prone. Suppose also, that you do not have an automated system like Ansible already (which by the way, I highly recommend) ...

Do not worry, just follow these steps:

1) Save the following script as allvpses_do, somewhere on your PATH, and make it executable:

#!/bin/bash

THECMD=$*

for i in `vzlist -o veid -H|  xargs`; do
   eval "\
   echo ;\
   echo ============================  VPS: $i ===================================;\
   vzctl exec $i $THECMD;\
   ";
done

2) Run a test command like this:

allvpses_do uptime

... or, you can try:

allvpses_do 'iptables --list -nv|grep DROP'

3) Enjoy your puppet master powers :-)

4Oct/120

Как да изпращаме HTML писма на кирилица, чрез PHP

Ако желаете да изпращате писма съдържащи кирилица, и форматиращи тагове, от PHP скрипт, можете да използвате следната функция:

function email_html($target, $sender, $subject, $htmlbody){
  $headers  = "MIME-Version: 1.0\r\n";
  $headers .= "From: {$sender}\r\n";
  $headers .= "Reply-To: {$sender}\r\n";
  $headers .= "Return-Path: {$sender}\r\n";
  $headers .= "Content-Type: text/html; charset=UTF-8; format=flowed\r\n";
  $headers .= "Content-Transfer-Encoding: base64 \r\n";
  $headers .= "X-Mailer: PHP/" . phpversion();

  $html = '<'.'!DOCTYPE'.' HTML>'.'<'.'head><'.'meta http-equiv="Content-Type" content="text/html; charset=utf-8">'.$htmlbody.'<'.'/body'.'><'.'/html>';
  $message = chunk_split(base64_encode($html));
  $subject = '=?UTF-8?B?'.base64_encode($subject).'?=';

  return @mail($target, $subject, $message, $headers);
}

Употребата и е проста (приемаме, че ще изпращаме писма с UTF-8 кодировка, съответно php скрипта ни или данните ни от базата също трябва да са в нея ... ):

$target = "targetmail@example.com";
$sender = "spammer@spammerhouse.com";
$subject = "Важно : това заглавие съдържа кирилица, но ще се покаже коректно в gmail.com и abv.bg";
$htmlbody = "Начало на тялото на съобщението....

Това е заглавен ред

А това е наклонен текст."; email_html($target, $sender, $subject, $htmlbody);
3Feb/120

PHP and curl: download multiple URLs using multiple http proxies, in parallel

Sometimes you may want to download some content from an http server that has a too restrictive banning policy (you may want to scrap/crawl/data mine the content for later processing).

If the above mentioned http server, has a policy to ban your IP, after too many requests in a short time, or a policy to allow only "Human" clients (modern browsers), and rejecting all others, you will quickly be stopped. In that case, to shortcut the ban, you may want to use public http proxies to anonymize your requests.

The problem with using proxies, is that most such public http proxies are either:
1) with restricted public bandwidth.
2) may sometimes return ads instead of the content that you want.
3) located in a foreign country to you, so ordinary the latency of the requests and the responses from them is too large, and that may cause timeouts, and failed downloads, which you have to process and try later the same URL with a different proxy server.

If you have to process many URLs, you can do it sequentially, but then the total processing time will be the sum of all the processing times, and as already said, each processing time can be quite large.

The solution of course is to download URLs in parallel. In that case the total processing time will be equal to the maximum of the processing times of the individual downloads, which may be many times smaller than the sum of them.

PHP and its CURL extension can help you with this task. Here is a project that does just that: Efficient PHP Parallel Downloader using multiple public http proxies

It capsulates the logic of retrying when a download was incomplete, and handles the parallel downloading of the URLs using an efficient CURL loop for processing of the proxy responses (using curl_multi_select, and not just polling). Using it, you can process hundreds of URLs downloading in parallel with a CPU usage < 1%.

Filed under: Uncategorized No Comments
9Nov/110

How to conveniently serve a folder of files over http

It is very simple actually (although not as fast as using a dedicated web server like nginx). Just use python:

cd YourFolder
python -m SimpleHTTPServer

That's it ! After running this line, the files in your current folder will be accessible over port 8000, so you can send your pears a link like this: http://YourIPHere:8000/ , and they will be able to read the shared folder.

You can change the port very easily too - just append it at the end of the last command, like this :

python -m SimpleHTTPServer 11001

That will run a webserver for the current folder over port 11001, instead of the default port 8000.

Stopping the python webserver process is simple too - just type Ctrl-C, like you would do for any other long running shell process.

Filed under: Uncategorized No Comments
5Nov/110

How to make tab switching snappy and fast again in Google Chrome

How to make switching of tabs in google chrome or chromium fast again after upgrading to version 14:

Switching of tabs is something that you need to do many times each day, if you are a power user and read many web pages. So slow tab switching  really affects the perception of Google Chrome as a snappy, fast and bugfree usefull browser.

So whithout further ado, here is the quick solution for the slow tab switching:

Move your bookmarks from the 'Bookmarks Bar' to a new folder under 'Other Bookmarks' and now the tab switching should be  snappy again in both Google Chrome and Chromium.

The 'Bookmarks Bar' is the default folder in which new bookmarks are added, and people like me, having the habit to bookmark every interesting page they visit for improving customization and later searchability, soon will collect a large number of bookmarks in this folder, thus slowing chrome more and more.

I can not remember the exact version where the slowdown started, but I think it was around Google Chrome version 14. Previous to that version, I had aproximately the same large amount of bookmarks, but chrome was snappy. After that, I thought that it was a problem with the development version of chrome, and I've used chromium for a while, because it remained fast (and it lags several versions behind on my machine). A couple of weeks ago, I've updated chromium to version '14.0.835.202 (Developer Build 103287 Linux) Ubuntu 11.04' and it started to lag, just like google chrome.

Moving the bookmarks to a different folder, as suggested here: http://code.google.com/p/chromium/issues/detail?id=87235

really helped, and google chrome is fast again :-) .

 

Filed under: Uncategorized No Comments
30Oct/110

Simple PHP cyr2lat command line transliteration filter from bulgarian to latin

Sometimes you need to easily convert some Cyrillic Bulgarian text to its latin equivalent (a process known as "romanization", see Romanization of Bulgarian ).

A possible use case scenario is making slugs for urls, containing bulgarian.

Since it is a common task, in the best Unix tradition, it is very usefull to have a simple command line filter, into which you can pipe the cyrillic text, and producing the romanized version in its output.

Here is a simple version of the command line filter cyr2lat, written in php, that does just that:

#!/usr/bin/env php
<?php
$cyr  = array('а','б','в','г','д','е','ж','з','и','й','к','л','м','н','о','п','р',
              'с','т','у','ф','х','ц','ч','ш','щ','ъ','ь','ю','я',
              'А','Б','В','Г','Д','Е','Ж','З','И','Й','К','Л','М','Н','О','П','Р',
              'С','Т','У','Ф','Х','Ц','Ч','Ш','Щ','Ъ','Ь', 'Ю','Я' );
$lat = array( 'a','b','v','g','d','e','zh','z','i','y','k','l','m','n','o','p','r',
              's','t','u','f' ,'h' ,'ts' ,'ch','sh' ,'sht' ,'a' ,'y' ,'yu','ya',
              'A','B','V','G','D','E','Zh','Z','I','Y','K','L','M','N','O','P','R',
              'S','T','U','F' ,'H' ,'Ts' ,'Ch','Sh' ,'Sht' ,'A' ,'Y' ,'Yu' ,'Ya' );

$in = fopen ("php://stdin","r");
while($line = fgets($in)){
    echo str_replace($cyr, $lat, $line);
}

Source of cyr2lat.php

To use it, just save it to a file named cyr2lat.php, then make this script executable by:

chmod 755 cyr2lat.php

... and possibly move it to a location in your path:

mv cyr2lat.php /usr/local/bin

or

mv cyr2lat.php ~/bin

After this, you can run for example:

echo "Това е текст на кирилица" | cyr2lat.php

and you will get:

Tova e tekst na kirilitsa

 

NB: This filter assumes that the input text is in the utf8 encoding. If you have an input text in the cp1251 encoding, just pipe it first through iconv, like this:

echo "Това е пак текст на кирилица, но този път с кодировка cp1251" |iconv -fcp1251 -tutf8 |cyr2lat.php

 

6Mar/110

Using jed with different encodings – utf8 and cp1251.

Maintaining websites through remote ssh connections, requires using console editors which should start fast, do not require huge amounts of memory and are easy to use.

The powerfull editors Vim and Emacs are not good for this task - although VIM is virtually guaranteed to be installed and it is fast, can be customized and so on, it is hardly easy to learn. Emacs may be easier, it has menus and tutorial, it is very customizable, but is not too fast to start.

In contrast, Jed is a very light weight editor which is ideal for quick changes, because it starts instantly, supports syntax highlighting for many common programming languages and has good UTF8 support. It has emacs like macro recorder, buffers, and auto indenting too. It also supports emacs keyboard shortcuts by default, so if you use emacs for longer programming sessions, you probably will like jed when you need to edit quickly some remote files.

Unfortunately bulgarian texts (and sites) are often written using 2 different common encodings, and so there is a catch - you need some tweaks for your programs (and jed in particular), to support both encodings equally well:

In order to edit utf8 files with jed, first, you need to have the correct bulgarian locale for utf8 generated.
You can do this using this command:

sudo localedef -i bg_BG -f UTF-8 bg_BG.UTF8

You can install jed and some jed goodies with the following instructions (Debian/Ubuntu):

sudo apt-get install jed jed-extra jedstate

For RedHat/Centos, try to google for a jed rpm, or try:

wget  ftp://ftp.pbone.net/mirror/centos.karan.org/el5/extras/testing/i386/RPMS/jed-0.99.18-5.el5.kb.i386.rpm
rpm -i jed-0.99.18-5.el5.kb.i386.rpm

Next, you add these lines to your .bashrc file:

alias jed.cp1251='LANG=C jed'
alias jed.utf8='LANG=bg_BG.UTF-8 jed'

Restart your shell, or do:

source ~/.bashrc

in your current shell, so that the changes can be applied.

After all this preparation, when you want to edit a file encoded using utf8, do it like this:

jed.utf8    FILENAME

For cp1251/windows1251, the command should be:

jed.cp1251   FILENAME

 

Note, that the selected encoding of your terminal application should match the encoding of your editor, or else you will see some funny characters instead of the proper ones. For this and other reasons, I recommend you to use the KDE's konsole application. It is stable, has good internationalization support, and it is pretty fast, even when you have a large scroll buffer.

You can also use PuTTY in case you are trapped in a windows only shop :-) .