Skip to main content

1.2 Shell Script Basics

Introduction

Shell scripts are text files containing a series of commands that are executed in order by the shell interpreter. They allow you to automate repetitive tasks, process data, and create workflows. It boils down to a series of commands that you would normally type in the terminal, but instead, you can save them in a file and run them all at once. This is especially useful for bioinformatics analyses where you often need to process large datasets with multiple steps.

Creating a Shell Script

Basic Structure

A shell script typically starts with a shebang line that tells the system which interpreter to use:

This is the contents of a basic shell script called script.sh:

#!/bin/bash

Making Scripts Executable

After creating a script, you may need to need to make it executable bby changing the permissions:

chmod +x script.sh

Then run it:

bash ./script.sh

Note the script is named with a .sh extension, but this is just a convention. You can name your script anything you like, as long as it has the appropriate permissions and the correct shebang line. It is sensible to use the .sh extension for clarity, as other languages have different extensions (e.g., .py for Python, .R for R). However, the extension does not affect how the script is executed.

Variables

Variables are like algebraic variables, but they can hold strings, numbers, or the output of commands. You can define variables to store data that can be used throughout your script:

#!/bin/bash

# Define variables
name="Lucy"
count=10

# Use variables
echo "Hello, $name"
echo "Count: $count"

# Use variables in commands
mkdir "$name"_data

The script would then be executed with:

bash ./script.sh

and you would get an output of: Hello, Lucy Count: 10 and a new directory called Lucy_data would be created in the current working directory.

Command Line Arguments

You can pass arguments to your script when you run it. These are accessed using special variables:

  • $1, $2, …: represent the first, second, etc. arguments
  • $@: represents all arguments as a list
  • $#: represents the number of arguments

Example script that takes arguments:

#!/bin/bash

# The first argument is the input to determine what genome reference to use, the second is a name for the project:
reference=$1
project_name=$2
echo "Using reference: $reference"
echo
echo "Project name: $project_name"

You would run this script with:

bash ./script.sh hg38 MyProject

And the output would be:

Using reference: hg38

Project name: MyProject

Basic Commands in Scripts

Echo

Print text to the terminal:

echo "This is a message"
echo "Current directory: $(pwd)"

Conditionals

If Statements

#!/bin/bash

file="data.txt"

if [ -f "$file" ]; then
    echo "File exists"
else
    echo "File does not exist"
fi

Common Test Conditions

  • -f file: file exists and is a regular file
  • -d directory: directory exists
  • -e path: path exists (file or directory)
  • -z string: string is empty
  • string1 = string2: strings are equal
  • num1 -eq num2: numbers are equal
  • num1 -lt num2: num1 is less than num2
  • num1 -gt num2: num1 is greater than num2

Loops

For Loops

Process multiple files:

#!/bin/bash

for file in *.fastq.gz; do
    echo "Processing $file"
    # Add processing commands here
done

Iterate over a range:

#!/bin/bash

for i in {1..10}; do
    echo "Number: $i"
done

And to iterate over a list of items:

#!/bin/bash

samples=("sample1" "sample2" "sample3")
for sample in "${samples[@]}"; do
    echo "Processing $sample"
done

While Loops

#!/bin/bash

count=0
while [ $count -lt 5 ]; do
    echo "Count: $count"
    count=$((count + 1))
done

Functions

Organize your code into reusable functions:

#!/bin/bash

# Define a function
greet() {
    local name=$1
    echo "Hello, $name!"
}

# Call the function
greet "Lucy"
greet "World"

Command Substitution

Capture the output of a command:

#!/bin/bash

# Using $()
current_date=$(date +%Y-%m-%d)
echo "Today is: $current_date"

# Count files
file_count=$(ls *.txt | wc -l)
echo "Number of text files: $file_count"

Practical Examples

Example 1: Process Multiple BAM Files

#!/bin/bash

# Process all BAM files in a directory
for bam in *.bam; do
    sample_name=$(basename $bam .bam)
    echo "Processing $sample_name"
    
    # Index the BAM file
    samtools index $bam
done

Example 2: Check File Existence and Size

#!/bin/bash

file=$1

if [ -z "$file" ]; then
    echo "Usage: $0 <filename>"
    exit 1
fi

if [ -f "$file" ]; then
    size=$(du -h "$file" | cut -f1)
    echo "File: $file"
    echo "Size: $size"
else
    echo "Error: File not found"
    exit 1
fi

Example 3: Quality Control Pipeline

#!/bin/bash

# QC pipeline for FASTQ files
input_dir="raw_data"
output_dir="qc_reports"

# Create output directory if it doesn't exist
mkdir -p $output_dir

# Process each FASTQ file
for fastq in ${input_dir}/*.fastq.gz; do
    sample=$(basename $fastq .fastq.gz)
    echo "Running FastQC on $sample"
    
    fastqc $fastq -o $output_dir
done

echo "QC complete. Reports in $output_dir"

Best Practices

  1. Use meaningful variable names: sample_name is better than x
  2. Add comments: Explain what your script does
  3. Check for errors: Use exit codes and conditionals
  4. Quote variables: Use "$variable" to handle spaces
  5. Use set -e: Exit script on any error
  6. Use set -u: Exit on undefined variables

Example of a robust script:

#!/bin/bash
set -e  # Exit on error
set -u  # Exit on undefined variable

# Script description
# This script processes FASTQ files

input_file=$1

if [ ! -f "$input_file" ]; then
    echo "Error: Input file not found"
    exit 1
fi

echo "Processing $input_file..."
# Add processing commands

Additional Resources