How to sum up the values in a file

Once you know how to remove a column from a file, then a time will come where you'll want to sum up the values in a column. For this example, let's take a look at a file with numbers in it.

mike@shiner $ cat file-with-numbers
1
2
3
4
5
6

We can sum up the values in this file with the following command.

mike@shiner $ cat file-with-numbers | awk '{sum+=$1}END{print sum}'
21

We can see here that the sum of the numbers in the file total to 21.

How to find files larger than X

You can use the following command to find files that are larger than 50M.

mike@shiner $ find . -type f -size +50000k

The -type argument tells find that we want to look at files and the -size argument specifies files larger than 50,000k. We can use the following command to get a nicely formatted output with the name of the file and the size.

mike@shiner $ find . -type f -size +50000k -exec ls -lh {} \; | awk '{ print $9 ": " $5 }' 

Alternatively, we can do the following to get the same output.

mike@shiner $ find . -type f -size +50000k | xargs ls -lh | awk '{ print $9 ": " $5 }' 

xargs is a handy command that takes each line of input and passes it to the proceeding command, which in this case is ls -lh.

How to remove a column from a file

I often need to remove a column from a character delimited file so I can do some additional processing, such as counting up values. I'm going to use a file with the following contents as an example.

mike@shiner $ cat name_age_city.txt
Mike|35|New York
Jason|28|Dallas
Fox|32|Washington
Mike|28|Los Angeles
Ann|23|Denver
Dana|32|Louisville
Mike|32|Atlantic City
Ryan|30|Austin

We can use the cut command to extract the column with the age information.

mike@shiner $ cat name_age_city.txt | cut -d '|' -f 2
35
28
32
28
23
32
32
30

The -d option tells cut that the delimiter for the columns is the '|' character and the -f option specifies the second column. Let's say we want to know how many people have the same age. We can use the following command to enumerate these values for us.

mike@shiner $ cat name_age_city.txt | cut -d '|' -f 2 | sort | uniq -c
      1 23
      2 28
      1 30
      3 32
      1 35

If we need to extract multiple columns, cut gives us this functionality as well.

mike@shiner $ cat name_age_city.txt | cut -d '|' -f 1,3
Mike|New York
Jason|Dallas
Fox|Washington
Mike|Los Angeles
Ann|Denver
Dana|Louisville
Mike|Atlantic City
Ryan|Austin

Unfortunately, cut won't allow us to reorder the columns. We get the same results if we try "cut -d '|' -f 3,1". On a side note, we can use awk to give us functionality similar to cut. One advantage awk has over cut is the ability to reorder the columns.

mike@shiner $ cat name_age_city.txt | awk -F '|' '{print $3"|"$1}'
New York|Mike
Dallas|Jason
Washington|Fox
Los Angeles|Mike
Denver|Ann
Louisville|Dana
Atlantic City|Mike
Austin|Ryan

The awk syntax requires us to put the '|' character in the print output to get the same behavior as cut.