Posts Categorized: shell

Tab Separated File Returns Wrong Number of Fields

Shell

So we had a CSV/TSV file with tabs separating each field but one of the fields was a description field that contained spaces. When parsing this TSV file some programs interpreted the spaces as field delimiters (TSV might be a bad idea!). Input file similar to:

The ‘Time Warner Cable’ field is all one… Read more »

Command Line Optimise JPEG image

This command significantly improved file size on disk although the quality value can be increased to adjust the image quality versus file size:

 

Batch Auto-Levels on Images

I had a thousand image files the kids had photographed when creating a stop motion video. An ‘auto-levels’ on many of the images was required to brighten them up so at the command line I used the following (with ImageMagick already installed of course):

ImageMagick is impressive – getting through each image in under… Read more »

Slurm & Environment Modules Bash Error

I had the following error message when trying to assign a job via Slurm (with Environment Modules installed). Since this has come up a few times for me I thought I would detail the solution:

This appears to be a problem related to the version of bash (pre shellshock?) so updating bash and/or the… Read more »

Selectively Checking Disk Usage Excluding Files and Folders

disk usage

Normally to check the disk usage of directories on a linux system I would run:

However, occasionally I want to exclude particular directories that take a long time to check and/or that I am not interested in anyway. The example below run from the root of the filesystem skips checks for /home /local /var… Read more »

Retrieving folder.jpg thumbnails: PHP Script

When using XBMC we usually use the ‘Files’ menu. The folder icons under ‘Files’ do not show a preview thumbnail unless there is an image in the folder named ‘folder.jpg’. I could manually search for folder images for each movie but decided that would be time-consuming so I put together a wee PHP script –… Read more »

Shell Script Example Squid Logs

I was trying to explain to a colleague a few days ago how a few shell commands can be really useful, when today I came across an example to try to illustrate. My problem was that I had 245 log files each about 70-80MB in size – roughly 4 million lines in each log file…. Read more »

Testing Proxy.pac Files

I decided to do a bit of checking on the proxy.pac file we were using. I grepped through our Apache access log files to pull out all of the IP addresses accessing the file over the year:

Armed with the list of IP addresses I next fired up pactester. Pactester Pactester is available from… Read more »

Coloring Bash Output

Colour bash shell script output

I wanted to add some colour to the output of my bash shell script and was able to do so with the following – very simple red and green:

Bash Shell Script to Automate FTP File Transfer

A little script to send a backup file to a remote FTP server:

Then run it every day from Cron.