Category Archives: Linux

Shorewall logcheck pattern

Posted on in Linux, Snippets

A logcheck ignore pattern to filter out shorewall REJECT and DROP entries (net is defined as eth0). /etc/logwatch/ignore.d.server/shorewall:

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ kernel:( \[ *[[:digit:]]+\.[[:digit:]]+\])? Shorewall:net2([a-z0-9]*):(REJECT|DROP)

Test if the regex works for your config:

sed -e 's/[[:space:]]*$//' /var/log/syslog | egrep '^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ kernel:( \[ *[[:digit:]]+\.[[:digit:]]+\])? Shorewall:net2([a-z0-9]*):(REJECT|DROP)'

Install Windows Server 2008 on KVM/VirtIO

Posted on in How-Tos, Linux, Server, Windows

An example how to install a Windows Server 2008 guest on KVM with VirtIO (on a LVM volume in the example). Besides the installation medium you need the VirtIO drivers for windows in order to be able to access the disk device. You can get signed binary drivers here. Then set up a Windows VM with a command like this:

virt-install --connect qemu:///system --arch=x86_64 -n win2k8 -r 1024 --vcpus=2 \
--disk pool=vmstore,size=50,bus=virtio,cache=none -c /path/to/win2k8.iso --vnc \
--noautoconsole --os-type windows --os-variant win2k8 --network network=subnet,model=e1000 \
--disk path=/path/to/virtio-win-1.1.16.iso,device=cdrom,perms=ro

When the guest is running, shut it down and edit the os section of the XML file to look like this (otherwise Windows setup won’t let you install on the disk):

virsh destroy win2k8
virsh edit win2k8
<os>
  ...
  <boot dev='cdrom'/>
  <boot dev='hd'/>
</os>
virsh start win2k8

Connect to the VNC console and start the installation process. When you reach the form to select a disk device you won’t see any devices available. Click on “Load drivers” at the bottom left and load the drivers from E:\viostor\wlh\{amd64|x86}. After the drivers are installed, you’ll see a disk device and can continue with the installation.

Sources

Fetchmail/Sieve on ISPMail setup – update

Posted on in How-Tos, Linux, Server

Quite some time ago I wrote a tutorial on how to integrate fetchmail and sieve into virtual mail with Postfix and Dovecot. As time passes and things change, here’s an update:

  1. I don’t use the sieve part anymore. Instead, I use the ManageSieve server provided by Dovecot. It integrates with Dovecot’s authentication system and you can use all kinds of clients supporting the managesieve protocol, e.g. Thunderbird or Roundcube.
  2. The ISPMail database structure changed since I wrote that tutorial which still relied on the DB structure for the Debian Etch structure. I updated the script to reflect both environments (take a look at the config file). However, it does not rely on any DB views anymore.
  3. The script (only the Fetchmail part) is now hosted on GitHub as I rarely use SVN anymore and the SVN may go offline in the near future. If anybody is interested in the Sieve part, just drop me a line.

Restore MySQL databases from raw *.frm files

Posted on in How-Tos, Linux, Server

I recently needed to restore data from a MySQL server where the host machine crashed and where I unfortunately didn’t have a proper dump backup – all I had was a backup of the MySQL data directory (/var/lib/mysql in case of Debian). After some googling I didn’t find a simple solution how to restore databases out of this backup. The solution which worked in the end was the following: I installed a fresh MySQL server in a virtual machine and replaced its data directory with the one from my backup (I had exactly the same MySQL versions on both machines). This allowed me to access the databases and create proper dumps which I could import in my real server later.

So, step one: in a virtual machine/spare server/local pc/whatever install a MySQL server and replace its data directory:

$ aptitude install mysql-server
$ /etc/init.d/mysql stop
$ mv /var/lib/mysql /var/lib/mysql.orig
$ cp -pr /tmp/backup/mysql /var/lib
$ chown -R mysql.mysql /var/lib/mysql

I also checked that file permissions match the normal permissions on Debian MySQL installations. Should be like this:

root@host:/var/lib/mysql# ls -al
drwx------  5 mysql mysql     4096 Mar  1 18:20 .
drwxr-xr-x 33 root  root      4096 Mar  1 18:20 ..
-rw-r--r--  1 root  root         0 Mar  1 18:07 debian-5.1.flag
-rw-rw----  1 mysql mysql 27262976 Mar  1 18:21 ibdata1
-rw-rw----  1 mysql mysql  5242880 Mar  1 18:21 ib_logfile0
-rw-rw----  1 mysql mysql  5242880 Mar  1 18:21 ib_logfile1
drwx------  2 mysql mysql     4096 Mar  1 18:20 database1
drwx------  2 mysql mysql     4096 Mar  1 18:21 database2
drwx------  2 mysql root      4096 Mar  1 18:08 mysql
-rw-------  1 root  root         6 Mar  1 18:08 mysql_upgrade_info

root@host:/var/lib/mysql# ls -al database1
drwx------ 2 mysql mysql 4096 Mar  1 18:20 .
drwx------ 5 mysql mysql 4096 Mar  1 18:20 ..
-rw-rw---- 1 mysql mysql   65 Mar  1 18:20 db.opt
-rw-rw---- 1 mysql mysql 8668 Mar  1 18:20 table1.frm
-rw-rw---- 1 mysql mysql  879 Mar  1 18:20 table2.frm
-rw-rw---- 1 mysql mysql 1520 Mar  1 18:20 table3.frm

Now you can try to start the server and look if your databases are readable:

$ /etc/init.d/mysql start
$ mysql -uroot -p -e "show databases;"
Enter password:
+--------------------+
| Database           |
+--------------------+
| information_schema |
| database1          |
| database1          |
| mysql              |
+--------------------+

Alright, if this works, simply dump your needed databases with mysqldump, transfer them to your server and import them normally.

$ mysqldump -uroot -p database1 > /tmp/database1.sql
$ scp /tmp/database1.sql user@server:/tmp

On the server:

$ mysql -uroot -p -e "create database database1;"
$ mysql -uroot -p database1 < /tmp/database1.sql

And don't forget to restore the temporary MySQL server to normal operation in case you need it later.

$ /etc/init.d/mysql stop
$ rm -rf /var/lib/mysql
$ mv /var/lib/mysql.orig /var/lib/mysql
$ /etc/init.d/mysql start

Use a LVM volume group with libvirt

Posted on in How-Tos, Linux, Server

A short howto how to use a LVM volume group with libvirt on Debian Squeeze (used for KVM VMs in my case). I assume your VG already exists and is dedicated for libvirt usage. In my case it’s /dev/vg1.

First of all, create the XML definition for the storage pool in /etc/libvirt/storage/vg1.xml. This is the minimal configuration needed, libvirt will extend it with things like UUID when you define it.

<pool type='logical'>
  <name>vg1</name>
  <target>
    <path>/dev/vg1</path>
  </target>
</pool>

Now you can tell libvirt about the new storage pool and let it start automatically.

$ virsh pool-define /etc/libvirt/storage/vg1.xml
$ virsh pool-start vg1
$ virsh pool-autostart vg1
$ virsh pool-info vg1

Creating virtual machines inside that storage pool is easy as pie:

$ virt-install -d --hvm --vnc --name=vm01 \
    --ram 512 --disk pool=vg1,size=10,bus=virtio,cache=none \
    --network network=default,model=virtio \
    --location=http://ftp.debian.org/debian/dists/squeeze/main/installer-amd64/ \
    --os-type=linux --os-variant=debiansqueeze

Cheers!

LaTeX build server with Git and Hudson on Ubuntu 10.04

Posted on in How-Tos, Linux, Server

LaTeX build server with Git and Hudson on Ubuntu 10.04

I’m currently working on a bigger paper for university using LaTeX. As it’s necessary to compile source files multiple times (especially when using BibTeX or TOCs), build runs can take quite some time. As an example, my current build script:

#!/bin/bash
BN=paper
pdflatex -interaction=nonstopmode $BN.tex
bibtex $BN
pdflatex -interaction=nonstopmode $BN.tex
bibtex $BN
pdflatex -interaction=nonstopmode $BN.tex
makeindex -s $BN.ist -t $BN.glg -o $BN.gls $BN.glo
pdflatex -interaction=nonstopmode $BN.tex
pdflatex -interaction=nonstopmode $BN.tex
rm -rf $BN.aux
rm -rf $BN.lof
rm -rf $BN.lot
rm -rf $BN.out
rm -rf $BN.toc
rm -rf $BN.bbl
rm -rf $BN.blg
rm -rf $BN.brf
rm -rf $BN.idx
rm -rf $BN.glo
rm -rf $BN.ist
rm -rf $BN.glg
rm -rf $BN.gls
rm -rf texput.log

This is OK on my workstation, but running a build on my notebook using a small 1.4 GHz single core processor can take up to a minute which is definitely too long. So I looked for solutions how to move the build process to a central server. As I was already using Git for source control on the project, I tried setting up a remote repository on the server which triggered a build using a post-receive script. This basically worked fine, but I wanted to go a step further. I had a look at CI servers and gave Hudson a try as it seems to have a lot of features while being quite easy to set up.

The result is the following: Hudson is polling the Git repository (can be remote or local, in my case it’s a self-hosted remote gitosis installation, but could be github too), starting a new build on changes and publishing the resulting PDF if successful. Hudson is accessible over https using an Apache2 server as frontend to a Tomcat installation.

Ready? Let’s go.

Continue reading →

Check broken links/rewrites with curl

Posted on in Development, How-Tos, Linux

I just did a change in the URL structure of a bigger site and had to check if all existing URLs are rewritten correctly to the new structure. First I extracted a list of all URLs from the CMS of the old version and wrote them to a text file, each line containing an URL. Then I changed the structure and used the following shellscript to check each URL with curl and output the result. Afterwards it’s easy to analyze the results and check for broken links.

#!/bin/sh
while read f
do
  echo $f
  echo ""
  curl -IL --silent $f
  echo "------------------"
  echo ""
done < $1

Usage:

$ ./checklinks.sh links.txt > stats.txt

/dev/pts errors on newly created Xen DomUs (Debian Lenny)

Posted on in Linux, Server

Today was the first time I had to create a new DomU after upgrading my Xen setup to Debian Lenny. When I booted the VM and logged in via xm console I got some strange errors when installing packages:

Can not write log, openpty() failed (/dev/pts not mounted?)

Additionally, after setting up SSH, I got the following error when logging in with SSH:

Server refused to allocate pty

Solution: install udev, reboot the VM and you’re good to go.