• Understand how the shell relates to the keyboard, the screen, the operating system, and users’ programs.
• Understand when and why command-line interfaces should be used instead of graphical interfaces.

• Human-computer interfaces (HCI)
  • Graphical User Interface (GUI)
  • Command-Line Interface (CLI)

• A program where users can type commands
  • simple commands (e.g. cd)
  • invoke complex programs (e.g. climate models)
• Many variants
  • bash (Bourne Again SHell)
  • zsh (Z SHell)

• GUIs are intuitive, but do not scale; the shell scales
• Many bioinformatics tools can only be used via CLI
  • (Systems like Galaxy try to wrap these tools)
• Automatable
  • package, repeat and reproduce analyses as scripts
  • combine tools into pipelines for large data volumes
  • high action-to-keystroke ratio
• Many bioinformatics analyses require remote server access, through the shell

• Start your terminal (if not already started)

$
$ ls

• Nelle Nemo: Marine Biologist to the stars
  • sampling gelatinous marine life in the Great Pacific Garbage Patch (North Pacific Gyre)
  • 1520 samples assayed for relative abundance of 300 proteins
  • analyse these results with goostats.sh (a program): 30s per sample

• Understand the similarities and differences between a file and a directory.
• Be able to translate an absolute path into a relative path and vice versa.
• Be able to construct absolute and relative paths that identify specific files and directories.
• Be able to use options and arguments to change the behaviour of a shell command.
• Be able to demonstrate the use of tab completion and explain its advantages.

• Part of the Operating System
  • organises data into
    • files (hold data)
    • directories/folders (hold files or directories)
• The shell has commands to
  • create
  • inspect
  • rename
  • delete

• Your account’s home
  • Location varies by operating system
    • Linux: /home/<USERNAME>
    • macOS: /Users/<USERNAME>
    • Windows: C:\Documents and Settings\<USERNAME> or C:\Users\<USERNAME>

cd
cd ~

• Nelle’s home: /Users/nelle

• Nelle’s home: /Users/nelle

ls
ls -F
ls --help
man ls
ls -j
ls -ltrh
ls -F Desktop
cd Desktop
pwd
cd ..
ls -Fa
cd
cd Desktop/shell-lesson-data/exercise-data
cd ~
cd /Users/<USERNAME>/Desktop/shell-lesson-data

1. cd .
2. cd /
3. cd /home/amanda
4. cd ../..
5. cd ~
6. cd home
7. cd ~/data/..
8. cd
9. cd ..

1. ../backup: No such file or directory
2. 2012-12-01 2013-01-08 2013-01-27
3. 2012-12-01/ 2013-01-08/ 2013-01-27/
4. original/ pnas_final/ pnas_sub/

ls -s
ls -S

• There is a parent directory: north-pacific-gyre
• Each sample is labelled:
  • unique ten-character ID (e.g. “NENE01729A”) for location, time, depth, etc.
  • files named, e.g. NENE01729A.txt
  • all 1520 files in same directory

• To be able to create a directory hierarchy that matches a given diagram.
• To be able to create files in that hierarchy using an editor or by copying and renaming existing files.
• To be able to delete, copy and move specified files and/or directories.

pwd
cd exercise-data/writing/
mkdir thesis
mkdir -p ../project/data ../project/results
ls -FR ../project

cd thesis
nano draft.txt

You may see instructions to hold the Control/Ctrl key written in several ways. E.g. to hold down Control and press X you might be instructed to press:

• Control-X
• Control+X
• Ctrl-X
• Ctrl+X
• ^X
• C-x

Files often have two parts, separated by a dot/period, e.g. myfile.txt.

• myfile is the filestem
• txt is the extension

cd ~/Desktop/shell-lesson-data/exercise-data/writing
mv thesis/draft.txt thesis/quotes.txt
mv -i thesis/quotes.txt .
ls thesis/quotes.txt

cp quotes.txt thesis/quotations.txt
ls quotes.txt thesis/quotations.txt
cp -r thesis thesis_backup
ls thesis thesis_backup

1. cp statstics.txt statistics.txt
2. mv statstics.txt statistics.txt
3. mv statstics.txt .
4. cp statstics.txt .

rm quotes.txt
ls quotes.txt

rm -i thesis_backup/quotations.txt

mkdir backup
cp creatures/minotaur.dat creatures/unicorn.dat backup/
ls *.pdb
ls p*.pdb
ls ?ethane.pdb
ls *ethane.pdb
ls ???ane.pdb

ethane.pdb methane.pdb

1. ls *t*ane.pdb
2. ls *t?ne.*
3. ls *t??ne.pdb
4. ls ethane.*

• Starting in shell-lesson-data/exercise-data/proteins

ls proteins
cd proteins
wc cubane.pdb
wc *.pdb
wc -l *.pdb

• By default the output from wc is written to the terminal, but we can redirect it with the > (right angled bracket) character

wc -l *.pdb > lengths.txt
ls lengths.txt
cat lengths.txt
less lengths.txt

sort ../numbers.txt
sort -n ../numbers.txt
sort -n lengths.txt
sort -n lengths.txt > sorted-lengths.txt
head -n 1 sorted-lengths.txt

• The pipe (|) symbol is like the redirect, except it passes the output to another command, rather than to a file.

sort -n lengths.txt | head -n 1
wc -l *.pdb | sort -n
wc -l *.pdb | sort -n | head -n 1

1. wc -l * > sort -n > head -n 3
2. wc -l * | sort -n | head -n 1-3
3. wc -l * | head -n 3 | sort -n
4. wc -l * | sort -n | head -n 3

• Nelle has created 17 files in the north-pacific-gyre directory
• She checks the output

cd north-pacific-gyre
wc -l *.txt
wc -l *.txt | sort -n | head -n 5

wc -l *.txt | sort -n | tail -n 5

ls *Z.txt

• To be able to write a loop that applies one or more commands separately to each file in a set of files.
• To be able to trace the values taken on by a loop variable during execution of the loop.
• Understand the difference between a variable’s name and its value.
• Understand why spaces and some punctuation characters shouldn’t be used in file names.
• To be able to see what commands have recently been executed.
• To be able to re-run recently executed commands without retyping them.

• Loops are a flow control mechanism in programming
• Loops allow us to repeat a command or set of commands for each item in a list
  • Apply to any number of files or items
• Reduce the amount of typing needed (reduces mistakes!)

head -n 5 basilisk.dat minotaur.dat unicorn.dat

for thing in list_of_things
do
    operation_using $thing    # Indentation not required, but readable
done

$ for filename in basilisk.dat minotaur.dat unicorn.dat
> do
>     head -n 2 $filename | tail -n 1
> done

$ for filename in *.dat
> do
>     echo $filename
>     head -n 100 $filename | tail -n 20
> done

cp *.dat original-*.dat

cp basilisk.dat minotaur.dat unicorn.dat original-*.dat

Instead we use a loop:

$ for filename in *.dat
> do
>     cp $filename original-$filename
> done

• To be able to write a shell script that runs a command or series of commands for a fixed set of files.
• To be able to run a shell script from the command line.
• To be able to write a shell script that operates on a set of files defined by the user on the command line.
• To be able to create pipelines that include shell scripts you, and others, have written.

cd proteins
nano middle.sh
head -n 15 octane.pdb | tail -n 5
bash middle.sh

nano middle.sh
head -n 15 "$1" | tail -n 5
bash middle.sh octane.pdb
bash middle.sh pentane.pdb

nano middle.sh
head -n "$2" "$1" | tail -n "$3"
bash middle.sh pentane.pdb 15 5
bash middle.sh pentane.pdb 20 5

# Select lines from the middle of a file.
# Usage: bash middle.sh filename end_line num_lines
head -n "$2" "$1" | tail -n "$3"

wc -l *.pdb | sort -n

# Sort files by their length.
# Usage: bash sorted.sh one_or_more_filenames
wc -l "$@" | sort -n

bash sorted.sh *.pdb ../creatures/*.dat

nano do-stats.sh

# Calculate stats for data files.
for datafile in "$@"
do
    echo $datafile
    bash goostats.sh $datafile stats-$datafile
done

bash do-stats.sh NENE*A.txt NENE*B.txt
bash do-stats.sh NENE*A.txt NENE*B.txt | wc -l

• To be able to use grep to select lines from text files that match simple patterns.
• To be able to use find to find files and directories whose names match simple patterns.
• To be able to use the output of one command as the command-line argument(s) to another command.
• Understand what is meant by ‘text’ and ‘binary’ files, and why many common tools don’t handle the latter well.

• grep is a very useful command-line tool that lets us find lines in files that match a pattern we want to look for

cd
cd Desktop/shell-lesson-data/exercise-data/writing
cat haiku.txt
grep not haiku.txt
grep The haiku.txt
grep -w The haiku.txt
grep -w "is not" haiku.txt
grep -n "it" haiku.txt
grep -n -w "the" haiku.txt
grep -nwi "the" haiku.txt
grep -nwv "the" haiku.txt
grep -r Yesterday .

• The find command finds files in the filesystem

cd shell-lesson-data/exercise-data
find .
find . -type d
find . -type f
find . -name *.txt
find . -name "*.txt"

• We can use the list of files produced by find as the input to other commands

wc -l $(find . -name "*.txt")
wc -l ./writing/LittleWomen.txt ./writing/haiku.txt ./numbers.txt
grep "searching" $(find . -name "*.txt")