Digitalquill

In my last post I talked about how I setup ClamAV, Squid proxy and Dansguardian on a SheevaPlug. Overall this has been a great success. I have been testing it on my machines before I role it out onto the rest of the network.

I have encountered one problem. When I go to sites such as Google Keyword tool, or I run some of my more complicated scripts that take longer to run it seems to create problems. I am wondering if this is some sort of time out fault that I will be able to change in the setup of Squid or if it is a power issue with the SheevaPlug.

This is not an issue for most normal users, I have tried it on things like YouTube and BBC Iplayer and it works fine with those so I suspect it is something specific to the style of those pages that causes a problem.

I took delivery of my Sheeva Plug Sata Multi today. I bought it yesterday online and it came first thing this morning. Indeed I made a mistake on the order when I bought it with an EU plug so I emailed and then called NewIT who were very helpful and changed over the order without any fuss.

If you remember from my previous post I wanted this device to run a network content filter, Squid and Dansguardian and also to take control of the network I have at home. I therefore, went for the 8GB SD card with Ubuntu 9.04 pre-installed. I will take an image of this before I start to play with it although one can be downloaded from the NewIT website if I need to. I also went for the Sata version so that I can add hard disk space to the device in the future to make it into a NAS drive as well as a network content filter.

First Steps

So my first step was to simply plug the device into my home network, a very simple task of plugging in a network cable and plugging in the power.

I then logged onto my router which currently runs my DHCP and found that a device called Ubuntu had been given an IP address.
Opening up Putty (My preferred SSH client) and ssh’ing to the ip address, I was able to log into the device with the root login and the default password of ‘nosoup4u’

The first impression of the device while navigating around the system via SSH and running the ‘top –d1’ command seemed to be fairly quick, ok the device is not doing anything other than running base Ubuntu but still a good sign at this stage.

I also ran ‘free -m’ which showed 472mb of free memory and df –f to find the free disk space showed:

Filesystem            Size  Used Avail Use% Mounted on
tmpfs                 251M     0  251M   0% /lib/init/rw
varrun                251M   36K  251M   1% /var/run
varlock               251M     0  251M   0% /var/lock
udev                  251M  116K  251M   1% /dev
tmpfs                 251M     0  251M   0% /dev/shm
rootfs                7.4G  443M  6.6G   7% /

All looks in order and I was ready to start playing with the system to see what it could do, but first I wanted to take an image of the system as a backup.

Imaging the SD Disk

I was not sure how I was going to do this so I did a little bit of research for the best method, but I plumped for a simple TAR of the whole system using linux its self:

tar cvpzf backup.tgz --exclude=/backup.tgz /

remembering to exclude the backup file its self otherwise it would get into some sort of loop! The tar of the whole system took only 2 minutes and created a file that was 135 Mb. I then sent this across to my Western Digital MyBookWorld NAS drive which has a hacked Linux operating system on it

scp backup.tgz root@xxx.xxx.xxx.xxx:/shares/internal/backups/

Happy that I had a backup of the system I then went ahead with a system update before trying a setup of Dansguardian.

System update

I ran apt-get update and then apt-get upgrade and upgraded the whole system. It wanted to upgrade the following:

The following packages will be upgraded:
 root@ubuntu:/# apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
cron dhcp3-client dhcp3-common dpkg file gzip language-pack-en language-pack-en-base libcurl3-gnutls libgnutls26 libkrb53 libldap-2.4-2 libmagic1 libnewt0.52 libpam-modules libpam-runtime libpam0g
  libsasl2-2 libsasl2-modules libsqlite3-0 libssl0.9.8 libvolume-id1 libxcb1 lsb-base lsb-release ntpdate openssl perl perl-base perl-modules sudo tzdata udev wget whiptail
35 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 18.2MB of archives.
After this operation, 1565kB of additional disk space will be used.

This process was again fairly quick taking just 5 minutes most of which was taken downloading the 18mb as Karoo Internet is simply rubbish!
I restarted the device just to check everything was still ok

Shutdown –r now

It took less than a minute to start back up and was back with me.

I then changed the root password ‘passwd’ and I was ready to start my install of dansguardian.

Install of Dansguardian

The setup I wanted was one with Squid for the proxy, ClamAV for virus scanning and then Dansguardian for the content filtering. I followed a tutorial guide on how to forge
http://www.howtoforge.com/squid-proxy-server-on-ubuntu-9.04-server-with-dansguardian-clamav-and-wpad-proxy-auto-detection
but I would skip the auto-detection part as I did not need this and could always add it at a later date.

The first step is to install ClamAV

apt-get install clamav-daemon clamav-freshclam

I then needed to edit the ClamAV conf file so I ran

Vim  /etc/clamav/freshclam.conf

And found that vim was not installed so I installed it

Apt-get install vim

Once installed I could then edit the conf file above. FreshClam is the part of ClamAV that downloads the new virus definitions. The config file contains a line ‘checks 24’ which indicates that FreshClam will check for new definitions 24 times a day or once every hour. I decided that would be adequate o accepted the default values, So my freshclam.conf file looks like this:

# Automatically created by the clamav-freshclam postinst
# Comments will get lost when you reconfigure the clamav-freshclam package

DatabaseOwner clamav
UpdateLogFile /var/log/clamav/freshclam.log
LogVerbose false
LogSyslog false
LogFacility LOG_LOCAL6
LogFileMaxSize 0
LogTime no
Foreground false
Debug false
MaxAttempts 5
DatabaseDirectory /var/lib/clamav/
DNSDatabaseInfo current.cvd.clamav.net
AllowSupplementaryGroups false
PidFile /var/run/clamav/freshclam.pid
ConnectTimeout 30
ReceiveTimeout 30
ScriptedUpdates yes
CompressLocalDatabase no
NotifyClamd /etc/clamav/clamd.conf
# Check for new database 24 times a day
Checks 24
DatabaseMirror db.local.clamav.net
DatabaseMirror database.clamav.net

You then need to restart ClamAV

/etc/init.d/clamav-freshclam restart

It was then time to install Squid, again an easy process with apt

apt-get install squid

Then to make some modifications to the squid conf files:

Vim /etc/squid/squid.conf

Just adding ‘http_port 3128’  to the bottom of the file and then restart squid

/etc/init.d/squid reload

Finally it was the turn of Dansguardian, again a simple task using apt:

apt-get install dansguardian

Again some modifications to the dansguardian conf files

Vim /etc/dansguardian/dansguardian.conf

I simply added my SheevaPlug Server IP address to the ‘filterip =’ line
Uncommented the line:

contentscanner = '/etc/dansguardian/contentscanners/clamav.conf'

to enable the ClamAV virus Scanning and then commented out the line:

#UNCONFIGURED - Please remove this line after configuration

Then restarted dansguardian:

/etc/init.d/dansguardian restart

Testing the install

The system was then ready and I added the proxy settings to my web browser with the address of the SheevaPlug server and then a port of 8080

I then tried Google which was accessed no problem, and then google’d for some adult terms which were blocked.

I do not like the default ‘blocked site’ template that dansguardian uses but I am sure I can change that as it looks like a simple HTML page.
I also need to re-configure the router to stop it handing out ip addresses and limit access to it to the ipaddress of the Sheevaplug and then add DHCP to the sheevaplug but I will carry on with this at a later date.

The SheevaPlug review

I am very pleased with how well this has gone and the performance of the Sheevaplug. I will continue to use this content filtering on my own laptop to see how it goes and test the performance of the Sheevaplug. The first impressions and results are very good indeed.

EDIT:

I should have given a link to NewIT who sell these little SheevaPlug systems they can be found at http://www.newit.co.uk defiantly worth a try.

Eva is rapidly growing up (2 years and 4 months) and it will not be long before she is surfing the web. She already asks for websites to be brought up, she loves the night garden website and asks ‘press the night garden button’ every time I am sat with the laptop.

Although I like the idea of NetNanny and similar products, they are all focused around one PC and controlling that one PC or one profile on that PC. As an IT professional I have a number of computers all over the house, A Media Centre, a laptop for both Cheryl and myself and a PC in my office. With the new kitchen extension we will have another LCD TV along with a media centre PC. So the access to computers in the house is rather open.

I want to control the network we have at home rather better anyway. Currently I allow the router to do all the work, it handles the DHCP and the control of the network. I would like to have more control over that and as such a Linux server to control the network would be ideal.

This being so I would then be able to use Squid proxy to filter content on the network, I could then install Dansguardian on top of Squid to filter our Adult content and provide me with more control.

The only problem with this is that I do not like the idea of a server type PC, this both because I do not like the idea of having it running constantly as that could become costly, but also the size and noise of such a machine does not lend its self to the home environment.

I have been looking at low power PC options and have found the Sheeva plug system which is an ARM based mini PC which is the size of a standard power supply.There are a few options but the one that I was looking at was the Sheeva Plug Multi which takes an SD card for the OS. You can optionally buy an SD with an OS pre-installed Debian 6 or Ubuntu 9. Or you can download the CD Card Images from their site.

Sheva Plug

This is ideal as I want to have a system running Ubuntu as my tests with Dansguardian in VirtualBox have all been with Ubuntu and the install went very easily.

The system has a 1.2Ghz ARM processor with 512Mb of Flash Ram and 512 Mb of DDR2 Memory.It has a Gigabit Ethernet port, USB2 port and ESATA type II port. It does not have a VGA or other monitor port, the idea behind this is that you would access it via SSH which suits me. It also has the ability for another SD card to be plugged in or the USB and ESATA ports give the opportunity to add storage to the device effectively giving you a NAS solution.

The power consumption at about 19w is a little higher than some ‘Low power PC’ options, but at £89 I do not think that you can argue that, especially for the size of the device and being able to hide it away in a draw or behind a desk somewhere without it causing trouble with noise.

My only concern about this is that once I setup this system, I will change all the browsers settings of all the PCs on the network so that they run through the Squid proxy rather than directly to the router. I need to figure out how to stop the router accepting traffic from anything else other than the Squid proxy server so that you can not simply take out the proxy settings.

I have a fairly advanced Billion router (I forget the model) but I ams sure that somewhere on its configuration this will be possible.

I will be placing an order for this device and I will post her how my trials go with setting it up as both a home network server and as a content filter or net nanny.

I recently ran an apt-get update on my Debian Etch server as I wanted to install pSpell for php. I found that the apt repros were out of date. After and Internet search I found that they have been archived. This is worrying that the OS that I am running is now seemingly obsolete but this is a worry for another day.

I changed my repros in /etc/apt/sources.list to:

deb http://archive.debian.org/debian etch main contrib non-free
deb http://archive.debian.org/debian-volatile etch/volatile main contrib non-free
deb http://archive.debian.org/debian-security etch/updates main contrib non-free

and then ran an apt-get update again, this updated the repros, before I installed pspell i decided to run an apt-get update to ensure that everything else was up-to-date. It upgraded some minor stuff, but it did upgrade both Amavis-new and MySQL server.

Amavis-new was no problem other than having to manually restart SpamAssasin and ClamAV, however, the upgrade to MySQL changed the my.cnf (/etc/mysql/my.cnf) config file for mysql and seemingly turned on binary logging.

I was happy that the upgrade had gone ok and proceeded to install pSpell and work on the project I was working on when the server became sluggish and unresponsive.

After looking I found that the server was out of space, after and investigation I found that the /var/logs/mysql/ folder was huge with many log files of 100mb and over.

After looking at the my.cnf file I found that the binary logging lines were uncommented:

# WARNING: Using expire_logs_days without bin_log crashes the server! See README.Debian!
expire_logs_days       = 10
max_binlog_size         = 100M
binlog_do_db           = include_database_name
binlog_ignore_db       = include_database_name

So I commended out those lines and restarted mysql and after deleting 64Gb of Log files (created in a 2 hour period!) the server was back.

I will in future pay more attention to these upgrades. I have said this before but this is the very place that opensource software falls down. As a web host who runs several servers with many websites on each all earning money, I can not afford to have those servers go down.

If one can not rely on those updates to leave a perfectly working well configured server still working after the update one has to consider if this is the best OS to be using to host sites of any importance what  so ever.

When the open source community comes to the realization that the software that they create is relied upon by real people making real money and running real projects and it is not just some form of geeks project to see if we can make it work, and it pays more attention to its user base and ensuring continuity of the software then I am sure that open source of all forms will dominate the market, until such time one has to consider what is best for your business.

As you will have recently ready I am backing up all my media to Amazon S3. My photographs are stored on my Western Digital MyBook World which I have hacked to open up the Linux shell, and as such i use S3Sync on command line to back up my photographs. I have even written a bash script that does it as it backs up some other stuff as well.

The rest of the media is on an external drive on the Windows 7 media centre, I am looking for a decent app that will sit there and sync the videos from my digicam to Amazon S3. I have tried loads but they all seem to fall short, either they are buggy and do not work or they can not sync and just manage adhoc single transfers.

I need something that is automated that will sync once a day/week and any changes or new videos will be uploaded to Amazon.

If anyone can recommend any applications free or otherwise?

The title of this blog post will be a scandal in many people’s eyes; however, I will justify what I mean later in this post.

In the technical circles there are many that continually bash the big commercial software companies, Microsoft, Apple, Adobe and so on for producing software that contains bugs and security flaws and to some extent rightly so, in an ideal world software that is released should be perfect, however, this is almost impossible to achieve.

I am recently recovering one of my servers for another hack, and as you may have gathered from the subject of this post, this server was not a Windows server but a Debian Linux software. While trawling through the logs trying to find how and where they got in, I started to think, I have four dedicated servers, two Microsoft (windows 2000 and windows 2003) and two Linux (Debian and Ubuntu 8.x), and you know what, the Linux boxes have been hacked more times than I care to count and to date the windows boxes have never been hacked, they have been subjected to a denial of service attack (DOS) but never actually hacked.

This is not due to the use of the servers either; all my servers are used for similar things, indeed the windows boxes host more websites in total than the Linux ones.

You are going to come back and tell me that with the move to Ubuntu, Debian is no longer recommended or being actively developed, and that Ubuntu 8.x is old, and you would be completely right, however, Windows 2000 is 10 years old, much older than the version of Debian I am using and is in much the same situation. One can not be expected to move hundreds of websites simply to upgrade from a distro that the open source community has chosen to drop.

As a project manager on a large software development project (circa £3 million) I know very well the problems faced by software companies in producing fault free software. The amount of times that we have released a new version of our software only to find that a change to feature Z has actually broken something in feature A that we developed 3 years ago. It is the inherent problem with large software projects, to maintain the balance between testing and actually getting to the software released.

Risk management is key, we operate a system where the company critical aspects of the software have more rigorous testing than those within the bells and whistles functions that if broken would not jeopardies the running of the company.

I therefore recognise the difficulties faced by both software companies and the open source community; however, in our little software development team we have to answer to the company board as to why we have released buggy software, fortunately for us our Managing Director understands these difficulties.

Open source software has no commercial liability

Open source software has no commercial liability to keep it in check, by this I mean that the large software companies such as Microsoft have a commercial standing to maintain, share holders to answer to and balance sheets to tally. If they release bad software, all that is put into jeopardy, their commercial standing is damaged and as such they will do everything in their power to ensure that their software is as faultless as possible, and when faults are found that they patch them.

Open source software has no such commercial checks to keep it in line, and the very nature of community development results in a free for all in development with no management or corporate responsibility keeping it in check. This results in software that maybe fantastically feature rich and cleaver but just does not deliver the secure and stable environment now let alone in the future when the community chooses at a whim to drop that particular distro.

As both an web master and web host security is the most important aspect of my business. Down time for servers costs money, I am therefore considering my strategic position at this time.

It is something that I have often said, open source is not free, and perhaps the cost of the windows server license is actually worth it in comparison to the cost of the down time.

Simply because it is free just does not cut it anymore, if Linux and open source are to get to the standing that they want to, if they are to take a larger proportion of the desktop market, if they are to continue to establish themselves in the web hosting and server industries they need to become rather more commercially aware and commercially structures to deliver the stable secure systems that corporations and business require.

Having just posted a good write-up of Virtualbox I am having a strange problem with my Ubuntu Desktop. I decided to update it all, so ran the ‘apt-get update’ which was fine, but when I ran an ‘apt-get upgrade’ it decided it needed to download 150mb or updates. This was expected so I allowed to to go and get them, which was when the problems started.

The whole PC ground to a halt, task manager showed Virtualbox was using 100% of the CPU constantly.

I managed to get back onto the Virtualbox and cancel the upgrade which then released the CPU. However after repeating the upgrade the same thing happened.

Doing the same on the Virtual Debian works fine…

Very strange… Anyone else had any similar problems?

Following my recent re-install of my desktop machine I have followed the direction we are heading at work and setup Virtualbox with several environments each with a different OS.

The Desktop PC I am running is relatively old, Pentium 4 1.8Ghz with 3Gb of RAM.

I have the following running in Virtualbox

• Windows XP
• Windows Vista Ultimate
• Windows 7 Beta (more of that later)
• Ubuntu 8.10 Desktop
• Ubuntu 8.10 Server
• Debian 4
• Debian 4 with ISPConfig Installed (again more of that later)
• Ubuntu Studio.

I have been trying to get Mac OSX working, which apparently is possible, but as yet I have had no luck.

I want to get Windows 2003 Server up and running as this will then replicate my windows server.

The main idea behind this is so that I have have environments that I can test things in before implementing on my live servers. I have been playing with ISPConfig on a virtual environment to see if there is any mileage with utilizing it. It looks promising.

The main advantage of virtualisation is that firstly you are in an enclosed, non-live environment thus no real harm can be done, and secondly if a mistake is made you can roll back to a saved point in time. It also allows me to save a copy of a basic Ubuntu or Debian setup and quickly create environments from it, meaning that testing things like ISPConfig can be done easily.

In addition to testing software, I have environments setup that are exactly the same as my live web servers, each with a copy of any websites deployed on them, allowing me to check any changes work on the most critical sites before releasing them to the live server.

I would like to be able to ‘mirror’ at least my Linux server to a virtual environment, probably on a day-by-day basis using the rsync’d archive backup.

I have suffered several computer failures in the past month or so, including a big failure of my web server. This has caused allot of problems for me personally and my businesses.

I have therefore been looking at online backup services which seem to have been popping up all over the place recently.

My first port of call was the most well known one from Amazon S3 however their payment options do not suit my requirements, so while searching I found Memopal.

After signing up for their free trial, I was quickly impresses and signed up for their 150gb service which should allow me to backup most of my important stuff.

The service comes with a client which you install, the windows client has a GUI which is easy to use and setup, you just choose the folders you want to backup and it just does it in the background, it monitors those folders for changes or new items and backs them up as well.

It also keeps revisions of your files, although I have yet to test this.

They also have a linux client, which can be run from command line. This is ideal for backing things up from my NAS drive and directly from my web servers.

The system is fairly quick althought I have asked it to backup my 60gb digital photograph archive which says it will take over 5 days, whcih I gues is not bad, and this will only be an issue at the start as it catches up with the archive of files.

Firstly you should log in as root, and run `date`. This will show you the current time, date and timezone and then backup your current timezone setting

mv /etc/localtime /etc/localtime-old

Create a symbolic link the timezone you want from /usr/share/zoneinfo to /etc/localtime.

For example:

ln -sf /usr/share/zoneinfo/Europe/London /etc/localtime