Ben Chuanlong Du's Blog

And let it direct your passion with reason.

Count Number of Fields in Each Line

Sometimes, a structured text file might be malformatted. A simple way to verify it is to count the number of fields in each line.

Using awk

You can count the number of fields in each line using the following awk command. Unfortunately, awk does not take escaped characters into consideration …

Advanced Use of "head" and "tail"

It is suggested that you use Python instead of Shell to manipulate text files!!

Besides passing an unsigned integer as parameter to the option -n, you can also pass a signed integer to it. When a signed integer is passed to -n, it means from/to (inclusive/exclusive similar to …

Select Columns from Structured Text Files

Python pandas

My first choice is pandas in Python. However, below are some tools for quick and dirty solutions.

q

q -t -H 'select c1, c3 from file.txt'

cut

cut -d\t -f1,3 file.txt

awk

awk -F'\t' '{print $1 "\t" $3}' file.tsv 

Note: neither cut …

Tips for AWK

AWK Tutorial

  1. For small structured text files, it is suggested that you use the q command to manipulate it.

    For complicated logic, it is suggested that you use a scripting language (e.g., Python) instead. I personally discourage using of awk unless you have a large file (that q cannot …