Un-obvious Linux tricks

[Mr. Bill]

I've been using Linux since 2005 -- and by 2005, Ubuntu Linux gave us a nice GUI, so I haven't needed to resort to the command line very often.  But sometimes ya gotta.  Linux is still occasionally a DIY project.

So this thread is for little useful tricks.  Nothing huge and complex, requiring years of experience to understand.  Just simple things, generally using the command line, that regular folk (e.g., NOT ARCHER ) might not be aware of.

Today's trick: comparing directories.  You may know that diff compares two files.  It wasn't until today that I discovered it can compare two directories, to tell you which files are present only in one or the other, or which are present in both but with file differences.

Code: [Select]

diff -qr directory1 directory2
Nifty!

[TexDaddy]

...that regular folk (e.g., NOT ARCHER ) might not be aware of...

I would never make the mistake of thinking Archer is regular folk. He is...exceptional. 

[Bradbn4]

dircmp (only on real Unix) can do the same thing.

now for some real fun, vim and improved version of vi dose have a nice feature - gvimdiff <file 1> <file 2>  will show not only the differences between both files, but highlight the specific character that is different.

When you say "vi" - do you say "v" "i", or is it like evil?

People who had classes say the "v" "i", those who learned it outside of classroom often say it as one word.  And folks who don't know Unix - are wondering why are we talking about "Village Inn"?

[Mountain State Prepper]

When you say "vi" - do you say "v" "i", or is it like evil?

I say "v" "i" seperately, never formally trained, just seemed right and what everyone else said.

I like the "diff" command, will have to remember that.   Here is one back to you...in a pinch and can't seem to find that big file taking up your space?...

"du -axk <filesystem> |sort -rn |more"   (or head)

I also hate that 'du' and 'df' can report different sizes....but that can get you going in circles...another thread perhaps one day .

[Archer]

dircmp (only on real Unix) can do the same thing.

now for some real fun, vim and improved version of vi dose have a nice feature - gvimdiff <file 1> <file 2>  will show not only the differences between both files, but highlight the specific character that is different.

When you say "vi" - do you say "v" "i", or is it like evil?

People who had classes say the "v" "i", those who learned it outside of classroom often say it as one word.  And folks who don't know Unix - are wondering why are we talking about "Village Inn"?

i use sdiff to compare two files:

~$ sdiff -w100 file1 file2
1                  1
2                  2
3                     |   33
4                  4
5                  5
                     >
6                  6
7                  7
8                  8
9                  9
10                     <

to only see one side:

~$ sdiff -lw100 file1 file2
1                     (
2                     (
3                     |   33
4                     (
5                     (
                     >
6                     (
7                     (
8                     (
9                     (
10   

[Archer]

I say "v" "i" seperately, never formally trained, just seemed right and what everyone else said.

I like the "diff" command, will have to remember that.   Here is one back to you...in a pinch and can't seem to find that big file taking up your space?...

"du -axk <filesystem> |sort -rn |more"   (or head)

I also hate that 'du' and 'df' can report different sizes....but that can get you going in circles...another thread perhaps one day .

df -h shows in human readable form:
$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/vzfs              80G   68G   13G  84% /
none                  768M  4.0K  768M   1% /dev

and du -h also shows human readable:
$ du -shx *
4.0K   tmp1
24K   tmp10
4.2M   tmp11
4.0K   tmp2
12K   tmp3
4.0K   tmp4
4.0K   tmp5
20K   tmp6
1.6M   tmp8
4.0K   tmp9

I like using "du -shx *" to show space usage in the current directory (see above). then i egrep out M and G to see the largest space users:
$ du -shx * | egrep 'M|G'
2.0M   aide
20M   Asst
100M   data
2.0M   forum
1.6M   homebrew
7.1M   images
9.3M   logs
2.0G   mail
825M   PDFs
42M   PDFs-books
1.6G   public_html
2.0M   smf_2-0-rc5_install.tar.gz
6.1M   tmp
2.1M   tsp

[Archer]

When you say "vi" - do you say "v" "i", or is it like evil?

People who had classes say the "v" "i", those who learned it outside of classroom often say it as one word.  And folks who don't know Unix - are wondering why are we talking about "Village Inn"?

it is pronounced: 'v' 'i'
as separate letters.

[Archer]

I would never make the mistake of thinking Archer is regular folk. He is...exceptional. 

i dont know everything, i've forgotten lots.... um, who am i?

[Archer]

another good command: tree
(if you dont have it, download it from your flavors site)

$ tree unix/
unix/
|-- hpux
|   `-- hpux-sites
|-- kab
|   |-- change-bigip
|   |-- check-mysql-slave-all
|   |-- cpu-info
|   |-- dus
|   |-- restart-webs
|   `-- ssh-test
|-- linux
|   |-- connect.c
|   |-- diskfree-2.1.0.tar.gz
|   |-- durep-0.9.tar.gz
|   `-- script-examples
|       `-- bash-variables
`-- sun

5 directories, 11 files

[Mountain State Prepper]

I like using "du -shx *" to show space usage in the current directory (see above). then i egrep out M and G to see the largest space users:
$ du -shx * | egrep 'M|G'

that is why i stick w/ "-k" and do the math in my head.  Using "-h" then piping it to "sort -rn" will but the 2G file toward the end of the output because it isn't a uniform size.  But grepping it out works too if you are just looking for large files without a sort from big>small.

[Mr. Bill]

it is pronounced: 'v' 'i'
as separate letters.

The way I pronounce it cannot be printed here.

[Archer]

The way I pronounce it cannot be printed here.

still better than emacs...

[Mountain State Prepper]

I use "view" a lot for read-only review.   Keeps you from making stupid mistakes when your attention is taken away.   Closing out by habit with ":wq!" is a killer if you forgot what "the -f" you are doing.

[Archer]

I use "view" a lot for read-only review.   Keeps you from making stupid mistakes when your attention is taken away.   Closing out by habit with ":wq!" is a killer if you forgot what "the -f" you are doing.

i've done that lots of times myself...... view is better for that as you say. i always save with ZZ w/o thinking about it.

[Bradbn4]

The way I pronounce it cannot be printed here.

The hard core original "vi" almost cost me a computer.  I got so disgusted by it's user unfriendliness I almost pushed the monitor off the work area. While vim is a fantastic improvement over vi, and the gvim version provides a worth while gui interface that beats the crap out of xemac's interface.  BTW: - vim sucks, but sucks less and still works well over very small 56k lines. This why I pronounce "vi" as in evil.   

My Unix trick of the moment.  If you don't have a spare computer to dedicate to Linux / Unix, get a copy of virtual box (vbox) and it is free.  https://www.virtualbox.org/   .   This software will let you build on your computer multiple Linux environments  without installing grub or other disk manager s/w.    On one computer I have 6 different types of Unix taking less than a few gigabytes of space.  So if you are testing for backwards compatibility of s/w between versions / styles of Linux it becomes a very affordable test platform.

The down side of Vbox?

It can / will run slower than native installed.  Mostly you will need to dedicate 1 - 2 gig of ram to the virtual instance for each OS you need to run at the same time.   Some older 64-bit dual core AMD processors had issues running virtualization of other 64-bit operating system.

useful utility programs & websites
   a2ps (ascii 2 post script).  I use this software to pretty print source code or other text files to almost any style.  https://www.gnu.org/software/a2ps/

  ntop - sort of like "top" but showing the network hogs in a system http://www.howtoforge.com/network_monitoring_with_ntop

FYI:  grab any of the good FAQ or one-line statements for sed, nawk, perl, etc.

[Skunkeye]

I like using "du -shx *" to show space usage in the current directory (see above). then i egrep out M and G to see the largest space users:

I use this guy all the time to find all files under the current directory and sort them by size (the largest files will end up at the bottom of the list):

  find . -type f -exec du -sk {} \; |sort -n

The "find" command is ridiculously useful, once you've mastered its somewhat tricky syntax and options.

[Twibble]

If you keep getting an error regarding your home IP, don't just delete it because it's irritating you.  If you do, when you can't get your printer to work and call in help, tell them that you did that before they spend an hour beating their head against the wall.

I might not have said that quite correctly.  I need to start using Linux again.  It's been a while.

[fritz_monroe]

The "find" command is ridiculously useful, once you've mastered its somewhat tricky syntax and options.

Find also works where things like ls won't.  For instance a directory with 4 million core files, ls will error out.  Find works great.

[Mountain State Prepper]

I use find to delete files of a certain age...

find /mydir -name "logfile123*.log" -mtime +7 -exec rm -f {} \;

[Archer]

I use find to delete files of a certain age...

find /mydir -name "logfile123*.log" -mtime +7 -exec rm -f {} \;

most finds have the -delete option:
find /mydir -name "logfile123*.log" -mtime +7 -delete

[Mountain State Prepper]

most finds have the -delete option:
find /mydir -name "logfile123*.log" -mtime +7 -delete

ah, very nice...

[Skunkeye]

One thing I usually do before using "find" to delete files, is to do the find command by itself without the "-delete" flag to see what files it's going to tag before I do the actual delete.  Linux is not so forgiving on recovering accidentally-deleted files...

[Archer]

One thing I usually do before using "find" to delete files, is to do the find command by itself without the "-delete" flag to see what files it's going to tag before I do the actual delete.  Linux is not so forgiving on recovering accidentally-deleted files...

i always do '-ls' first before the '-delete' to make sure i want to delete what it finds.....
*nix is not forgiving when deleting files...

[fritz_monroe]

i always do '-ls' first before the '-delete' to make sure i want to delete what it finds.....
*nix is not forgiving when deleting files...

Unlike the other operating systems, you are actually expected to know something about what you are doing.  That's one of the things that I really like about Linux, it doesn't try to save me from myself.

If I want to do a rm -rf * then it will allow me to do it.

[Archer]

If I want to do a rm -rf * then it will allow me to do it.

and then slowly die on you... whimpering and looking accusingly while doing so

[Mountain State Prepper]

There are a few commands I try not to wildcard.   I do it, but try really hard to better pattern match my chown's, chmod's, rm's etc.

About a year ago we had an admin (dba) try to chmod (777) a directory in a customers home directory.   Well she had the deadly mix of "-R" and "*" and ended up chmod'ing the entire /home mount.   Ooopsie Poopsie.   Tape restore orderd.   Thank god it was a cluster and we could fail over our databases....which we still had to comb over and reset permissions on.

[Archer]

There are a few commands I try not to wildcard.   I do it, but try really hard to better pattern match my chown's, chmod's, rm's etc.

About a year ago we had an admin (dba) try to chmod (777) a directory in a customers home directory.   Well she had the deadly mix of "-R" and "*" and ended up chmod'ing the entire /home mount.   Ooopsie Poopsie.   Tape restore orderd.   Thank god it was a cluster and we could fail over our databases....which we still had to comb over and reset permissions on.

i had a junior admin once go into /etc and want to delete old conf files (named file.conf.old.some-date). so he did:
rm *.*
oops...... next he got to learn how to restore the main DNS server...

[smurf hunter]

Pipes are the bomb (pun intended?)

I do this on test servers at work all the time to blast all files ending with ".log".  As fritz mentioned, also useful for purging big core files littered across your filesystem:

Code: [Select]

find . -name "*.log" -type f | xargs rm -f
Or search and replace a string in a file (let's replace "2012" with "2013":

Code: [Select]

cat file1.txt | sed -e 's/2012/2013/g' > file2.txt

Find number of open network sockets associated with a process:

Code: [Select]

lsof -p [b]<your PID here>[/b] | grep IPv6  | wc -l

[Skunkeye]

Code: [Select]

cat file1.txt | sed -e 's/2012/2013/g' > file2.txt


Many "sed" implementations have a "-i" or "--in-place" flag that lets you process the file in place, without doing all that redirection.  You can do: 

Code: [Select]

sed -i -e 's/2012/2013/g' file1.txt
to just do the search-and-replace in file1.txt.  GNU sed lets you specify an extension that will automatically backup the original file:

Code: [Select]

sed -i.bak -e 's/2012/2013/g' file1.txt
This will replace 2012 with 2013 in file1.txt and create a backup file "file1.txt.bak" before it modifies the original file.

Another handy command is "fuser" - give it a filename, and it will tell you any processes that currently are using that file.  With the "-m" flag, it will tell you any processes that have any files open on the filesystem you specify.  Handy if you need to unmount a disk and can't because some process has a file locked open.

[Archer]

Code: [Select]

find . -name "*.log" -type f | xargs rm -f

and you can prob use -delete instead of piping to xargs to delete the files.