The Survival Podcast Forum

Survivalism & Self Sufficiency Topics => Homesteading and Self Reliant Living => Do It Yourself - Projects, Ideas and How To => Topic started by: Mr. Bill on January 18, 2013, 07:05:35 PM

Title: Un-obvious Linux tricks
Post by: Mr. Bill on January 18, 2013, 07:05:35 PM
I've been using Linux since 2005 -- and by 2005, Ubuntu Linux gave us a nice GUI, so I haven't needed to resort to the command line very often.  But sometimes ya gotta.  Linux is still occasionally a DIY project.

So this thread is for little useful tricks.  Nothing huge and complex, requiring years of experience to understand.  Just simple things, generally using the command line, that regular folk (e.g., NOT ARCHER 8) ) might not be aware of.

Today's trick: comparing directories.  You may know that diff compares two files.  It wasn't until today that I discovered it can compare two directories, to tell you which files are present only in one or the other, or which are present in both but with file differences.

Code: [Select]
diff -qr directory1 directory2
Nifty!
Title: Re: Un-obvious Linux tricks
Post by: TexDaddy on January 18, 2013, 07:11:48 PM
...that regular folk (e.g., NOT ARCHER 8) ) might not be aware of...
I would never make the mistake of thinking Archer is regular folk. He is...exceptional.  ;)
Title: Re: Un-obvious Linux tricks
Post by: Bradbn4 on January 19, 2013, 08:50:26 AM
dircmp (only on real Unix) can do the same thing.

now for some real fun, vim and improved version of vi dose have a nice feature - gvimdiff <file 1> <file 2>  will show not only the differences between both files, but highlight the specific character that is different.

When you say "vi" - do you say "v" "i", or is it like evil?

People who had classes say the "v" "i", those who learned it outside of classroom often say it as one word.  And folks who don't know Unix - are wondering why are we talking about "Village Inn"?


Title: Re: Un-obvious Linux tricks
Post by: d3nni5 on January 19, 2013, 09:15:44 AM

When you say "vi" - do you say "v" "i", or is it like evil?


I say "v" "i" seperately, never formally trained, just seemed right and what everyone else said.

I like the "diff" command, will have to remember that.   Here is one back to you...in a pinch and can't seem to find that big file taking up your space?...

"du -axk <filesystem> |sort -rn |more"   (or head)

I also hate that 'du' and 'df' can report different sizes....but that can get you going in circles...another thread perhaps one day :).
Title: Re: Un-obvious Linux tricks
Post by: archer on January 19, 2013, 11:51:48 AM
dircmp (only on real Unix) can do the same thing.

now for some real fun, vim and improved version of vi dose have a nice feature - gvimdiff <file 1> <file 2>  will show not only the differences between both files, but highlight the specific character that is different.

When you say "vi" - do you say "v" "i", or is it like evil?

People who had classes say the "v" "i", those who learned it outside of classroom often say it as one word.  And folks who don't know Unix - are wondering why are we talking about "Village Inn"?



i use sdiff to compare two files:

~$ sdiff -w100 file1 file2
1                  1
2                  2
3                     |   33
4                  4
5                  5
                     >
6                  6
7                  7
8                  8
9                  9
10                     <

to only see one side:

~$ sdiff -lw100 file1 file2
1                     (
2                     (
3                     |   33
4                     (
5                     (
                     >
6                     (
7                     (
8                     (
9                     (
10   
Title: Re: Un-obvious Linux tricks
Post by: archer on January 19, 2013, 11:58:02 AM
I say "v" "i" seperately, never formally trained, just seemed right and what everyone else said.

I like the "diff" command, will have to remember that.   Here is one back to you...in a pinch and can't seem to find that big file taking up your space?...

"du -axk <filesystem> |sort -rn |more"   (or head)

I also hate that 'du' and 'df' can report different sizes....but that can get you going in circles...another thread perhaps one day :).

df -h shows in human readable form:
$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/vzfs              80G   68G   13G  84% /
none                  768M  4.0K  768M   1% /dev

and du -h also shows human readable:
$ du -shx *
4.0K   tmp1
24K   tmp10
4.2M   tmp11
4.0K   tmp2
12K   tmp3
4.0K   tmp4
4.0K   tmp5
20K   tmp6
1.6M   tmp8
4.0K   tmp9

I like using "du -shx *" to show space usage in the current directory (see above). then i egrep out M and G to see the largest space users:
$ du -shx * | egrep 'M|G'
2.0M   aide
20M   Asst
100M   data
2.0M   forum
1.6M   homebrew
7.1M   images
9.3M   logs
2.0G   mail
825M   PDFs
42M   PDFs-books
1.6G   public_html
2.0M   smf_2-0-rc5_install.tar.gz
6.1M   tmp
2.1M   tsp

Title: Re: Un-obvious Linux tricks
Post by: archer on January 19, 2013, 12:00:50 PM
When you say "vi" - do you say "v" "i", or is it like evil?

People who had classes say the "v" "i", those who learned it outside of classroom often say it as one word.  And folks who don't know Unix - are wondering why are we talking about "Village Inn"?

it is pronounced: 'v' 'i'
as separate letters.
Title: Re: Un-obvious Linux tricks
Post by: archer on January 19, 2013, 12:03:21 PM
I would never make the mistake of thinking Archer is regular folk. He is...exceptional.  ;)

i dont know everything, i've forgotten lots.... um, who am i?
Title: Re: Un-obvious Linux tricks
Post by: archer on January 19, 2013, 12:07:42 PM
another good command: tree
(if you dont have it, download it from your flavors site)

$ tree unix/
unix/
|-- hpux
|   `-- hpux-sites
|-- kab
|   |-- change-bigip
|   |-- check-mysql-slave-all
|   |-- cpu-info
|   |-- dus
|   |-- restart-webs
|   `-- ssh-test
|-- linux
|   |-- connect.c
|   |-- diskfree-2.1.0.tar.gz
|   |-- durep-0.9.tar.gz
|   `-- script-examples
|       `-- bash-variables
`-- sun

5 directories, 11 files
Title: Re: Un-obvious Linux tricks
Post by: d3nni5 on January 19, 2013, 12:20:55 PM

I like using "du -shx *" to show space usage in the current directory (see above). then i egrep out M and G to see the largest space users:
$ du -shx * | egrep 'M|G'

that is why i stick w/ "-k" and do the math in my head.  Using "-h" then piping it to "sort -rn" will but the 2G file toward the end of the output because it isn't a uniform size.  But grepping it out works too if you are just looking for large files without a sort from big>small.

Title: Re: Un-obvious Linux tricks
Post by: Mr. Bill on January 19, 2013, 12:49:49 PM
it is pronounced: 'v' 'i'
as separate letters.

The way I pronounce it cannot be printed here.

8)
Title: Re: Un-obvious Linux tricks
Post by: archer on January 19, 2013, 12:59:46 PM
The way I pronounce it cannot be printed here.

8)

still better than emacs...
Title: Re: Un-obvious Linux tricks
Post by: d3nni5 on January 19, 2013, 01:05:48 PM


I use "view" a lot for read-only review.   Keeps you from making stupid mistakes when your attention is taken away.   Closing out by habit with ":wq!" is a killer if you forgot what "the -f" you are doing. :)

Title: Re: Un-obvious Linux tricks
Post by: archer on January 19, 2013, 05:57:28 PM

I use "view" a lot for read-only review.   Keeps you from making stupid mistakes when your attention is taken away.   Closing out by habit with ":wq!" is a killer if you forgot what "the -f" you are doing. :)
i've done that lots of times myself...... view is better for that as you say. i always save with ZZ w/o thinking about it.
Title: Re: Un-obvious Linux tricks
Post by: Bradbn4 on January 20, 2013, 10:55:05 AM
The way I pronounce it cannot be printed here.

8)

The hard core original "vi" almost cost me a computer.  I got so disgusted by it's user unfriendliness I almost pushed the monitor off the work area. While vim is a fantastic improvement over vi, and the gvim version provides a worth while gui interface that beats the crap out of xemac's interface.  BTW: - vim sucks, but sucks less and still works well over very small 56k lines. This why I pronounce "vi" as in evil.   

My Unix trick of the moment.  If you don't have a spare computer to dedicate to Linux / Unix, get a copy of virtual box (vbox) and it is free.  https://www.virtualbox.org/   .   This software will let you build on your computer multiple Linux environments  without installing grub or other disk manager s/w.    On one computer I have 6 different types of Unix taking less than a few gigabytes of space.  So if you are testing for backwards compatibility of s/w between versions / styles of Linux it becomes a very affordable test platform.

The down side of Vbox?

It can / will run slower than native installed.  Mostly you will need to dedicate 1 - 2 gig of ram to the virtual instance for each OS you need to run at the same time.   Some older 64-bit dual core AMD processors had issues running virtualization of other 64-bit operating system.

useful utility programs & websites
   a2ps (ascii 2 post script).  I use this software to pretty print source code or other text files to almost any style.  https://www.gnu.org/software/a2ps/

  ntop - sort of like "top" but showing the network hogs in a system http://www.howtoforge.com/network_monitoring_with_ntop

FYI:  grab any of the good FAQ or one-line statements for sed, nawk, perl, etc.
Title: Re: Un-obvious Linux tricks
Post by: Skunkeye on January 22, 2013, 10:37:31 AM
I like using "du -shx *" to show space usage in the current directory (see above). then i egrep out M and G to see the largest space users:

I use this guy all the time to find all files under the current directory and sort them by size (the largest files will end up at the bottom of the list):

  find . -type f -exec du -sk {} \; |sort -n

The "find" command is ridiculously useful, once you've mastered its somewhat tricky syntax and options.
Title: Re: Un-obvious Linux tricks
Post by: Twibble on January 22, 2013, 10:42:25 AM
If you keep getting an error regarding your home IP, don't just delete it because it's irritating you.  If you do, when you can't get your printer to work and call in help, tell them that you did that before they spend an hour beating their head against the wall.

I might not have said that quite correctly.  I need to start using Linux again.  It's been a while.
Title: Re: Un-obvious Linux tricks
Post by: fritz_monroe on January 22, 2013, 11:04:40 AM
The "find" command is ridiculously useful, once you've mastered its somewhat tricky syntax and options.
Find also works where things like ls won't.  For instance a directory with 4 million core files, ls will error out.  Find works great.
Title: Re: Un-obvious Linux tricks
Post by: d3nni5 on January 22, 2013, 11:09:21 AM


I use find to delete files of a certain age...

find /mydir -name "logfile123*.log" -mtime +7 -exec rm -f {} \;

Title: Re: Un-obvious Linux tricks
Post by: archer on January 22, 2013, 01:31:03 PM

I use find to delete files of a certain age...

find /mydir -name "logfile123*.log" -mtime +7 -exec rm -f {} \;


most finds have the -delete option:
find /mydir -name "logfile123*.log" -mtime +7 -delete
Title: Re: Un-obvious Linux tricks
Post by: d3nni5 on January 22, 2013, 01:47:49 PM
most finds have the -delete option:
find /mydir -name "logfile123*.log" -mtime +7 -delete

ah, very nice...
Title: Re: Un-obvious Linux tricks
Post by: Skunkeye on January 22, 2013, 03:13:06 PM
One thing I usually do before using "find" to delete files, is to do the find command by itself without the "-delete" flag to see what files it's going to tag before I do the actual delete.  Linux is not so forgiving on recovering accidentally-deleted files...
Title: Re: Un-obvious Linux tricks
Post by: archer on January 22, 2013, 03:19:57 PM
One thing I usually do before using "find" to delete files, is to do the find command by itself without the "-delete" flag to see what files it's going to tag before I do the actual delete.  Linux is not so forgiving on recovering accidentally-deleted files...
i always do '-ls' first before the '-delete' to make sure i want to delete what it finds.....
*nix is not forgiving when deleting files...
Title: Re: Un-obvious Linux tricks
Post by: fritz_monroe on January 23, 2013, 04:49:28 AM
i always do '-ls' first before the '-delete' to make sure i want to delete what it finds.....
*nix is not forgiving when deleting files...
Unlike the other operating systems, you are actually expected to know something about what you are doing.  That's one of the things that I really like about Linux, it doesn't try to save me from myself.

If I want to do a rm -rf * then it will allow me to do it.
Title: Re: Un-obvious Linux tricks
Post by: archer on January 23, 2013, 01:07:31 PM
If I want to do a rm -rf * then it will allow me to do it.
and then slowly die on you... whimpering and looking accusingly while doing so
Title: Re: Un-obvious Linux tricks
Post by: d3nni5 on January 23, 2013, 01:38:58 PM


There are a few commands I try not to wildcard.   I do it, but try really hard to better pattern match my chown's, chmod's, rm's etc.

About a year ago we had an admin (dba) try to chmod (777) a directory in a customers home directory.   Well she had the deadly mix of "-R" and "*" and ended up chmod'ing the entire /home mount.   Ooopsie Poopsie.   Tape restore orderd.   Thank god it was a cluster and we could fail over our databases....which we still had to comb over and reset permissions on.

Title: Re: Un-obvious Linux tricks
Post by: archer on January 23, 2013, 01:45:57 PM

There are a few commands I try not to wildcard.   I do it, but try really hard to better pattern match my chown's, chmod's, rm's etc.

About a year ago we had an admin (dba) try to chmod (777) a directory in a customers home directory.   Well she had the deadly mix of "-R" and "*" and ended up chmod'ing the entire /home mount.   Ooopsie Poopsie.   Tape restore orderd.   Thank god it was a cluster and we could fail over our databases....which we still had to comb over and reset permissions on.

i had a junior admin once go into /etc and want to delete old conf files (named file.conf.old.some-date). so he did:
rm *.*
oops...... next he got to learn how to restore the main DNS server...
Title: Re: Un-obvious Linux tricks
Post by: Smurf Hunter on January 23, 2013, 02:02:19 PM
Pipes are the bomb (pun intended?)

I do this on test servers at work all the time to blast all files ending with ".log".  As fritz mentioned, also useful for purging big core files littered across your filesystem:

Code: [Select]
find . -name "*.log" -type f | xargs rm -f
Or search and replace a string in a file (let's replace "2012" with "2013":

Code: [Select]
cat file1.txt | sed -e 's/2012/2013/g' > file2.txt

Find number of open network sockets associated with a process:

Code: [Select]
lsof -p [b]<your PID here>[/b] | grep IPv6  | wc -l
Title: Re: Un-obvious Linux tricks
Post by: Skunkeye on January 23, 2013, 03:55:35 PM
Code: [Select]
cat file1.txt | sed -e 's/2012/2013/g' > file2.txt

Many "sed" implementations have a "-i" or "--in-place" flag that lets you process the file in place, without doing all that redirection.  You can do: 
Code: [Select]
sed -i -e 's/2012/2013/g' file1.txt
to just do the search-and-replace in file1.txt.  GNU sed lets you specify an extension that will automatically backup the original file:

Code: [Select]
sed -i.bak -e 's/2012/2013/g' file1.txt
This will replace 2012 with 2013 in file1.txt and create a backup file "file1.txt.bak" before it modifies the original file.

Another handy command is "fuser" - give it a filename, and it will tell you any processes that currently are using that file.  With the "-m" flag, it will tell you any processes that have any files open on the filesystem you specify.  Handy if you need to unmount a disk and can't because some process has a file locked open.
Title: Re: Un-obvious Linux tricks
Post by: archer on January 23, 2013, 04:12:55 PM
Code: [Select]
find . -name "*.log" -type f | xargs rm -f
and you can prob use -delete instead of piping to xargs to delete the files.
Title: Re: Un-obvious Linux tricks
Post by: Smurf Hunter on January 23, 2013, 04:14:42 PM
good point.  I guess it's a habit, since I could have something beside find at the left of the |

Code: [Select]
ls -1 *.foo | xargs rm
Title: Re: Un-obvious Linux tricks
Post by: archer on January 23, 2013, 05:49:24 PM
good point.  I guess it's a habit, since I could have something beside find at the left of the |

Code: [Select]
ls -1 *.foo | xargs rm
that is true, there are soo many different ways to do things in *nix
Title: Re: Un-obvious Linux tricks
Post by: Mr. Bill on March 07, 2013, 12:49:07 PM
HOW TO CHANGE AN ENVIRONMENT VARIABLE FROM THE COMMAND LINE:
[edited to add: This is for bash, ksh, and similar shells; not for csh.]

export NAME=value

Example: change the default text-pager used by man and other programs:

export PAGER=/bin/less

Okay, yes, I feel like a complete "Where's the ANY key?" newbie at the moment.  The last time I changed an environment variable from the command line was maybe 5 years ago.  I had no memory of how to do it, and it took me forever to find it in the gigantic and mind-numbing man bash documentation.

I had to do this on my web host because for some reason $PAGER was pointing at an incorrect location, and I couldn't even get man to work without adding -P /bin/less to the command line every single time.

Anyway, I scrawled out some notes to myself on a piece of paper, under the heading
Getting man to use less
which at least geve me a laugh when I saw what I'd written. ;D
Title: Re: Un-obvious Linux tricks
Post by: archer on March 07, 2013, 02:45:03 PM
give me a call next time if you get stuck and need help. my phone is always (groan) on.
Title: Re: Un-obvious Linux tricks
Post by: BillP38 on March 07, 2013, 04:35:38 PM
  I've used Linux on all my personal machines for the last few years and have found the following web page helpful. The site is makeuseof.com, I've never had a problem with the site or any of the files downloaded.
 
  http://www.makeuseof.com/tag/an-a-z-of-linux-40-essential-commands-you-should-know/
Title: Re: Un-obvious Linux tricks
Post by: d3nni5 on March 07, 2013, 05:19:09 PM
HOW TO CHANGE AN ENVIRONMENT VARIABLE FROM THE COMMAND LINE:

export NAME=value

Example: change the default text-pager used by man and other programs:

export PAGER=/bin/less

Okay, yes, I feel like a complete "Where's the ANY key?" newbie at the moment.  The last time I changed an environment variable from the command line was maybe 5 years ago.  I had no memory of how to do it, and it took me forever to find it in the gigantic and mind-numbing man bash documentation.

I had to do this on my web host because for some reason $PAGER was pointing at an incorrect location, and I couldn't even get man to work without adding -P /bin/less to the command line every single time.

Anyway, I scrawled out some notes to myself on a piece of paper, under the heading
Getting man to use less
which at least geve me a laugh when I saw what I'd written. ;D


Mr Bill,

You can also set these up in a .profile in your $HOME directory, if you are using bash or ksh or similar shell.  Those using tcsh or csh can look at the set and setenv    commands.

Example

$ cd $HOME
$ cat .profile
EDITOR=vi
PATH=$PATH:/my/extra/path/dirs
ORACLE_HOME=/my/oracle/home
ORACLE_BASE=/my/oracle/base
export EDITOR PATH ORACLE_HOME ORACLE_BASE
LD_LIBRARY_PATH=$ORACLE_HOME/lib
export LD_LIBRARY_PATH
Title: Re: Un-obvious Linux tricks
Post by: Bradbn4 on March 07, 2013, 05:20:16 PM
HOW TO CHANGE AN ENVIRONMENT VARIABLE FROM THE COMMAND LINE:

export NAME=value

Example: change the default text-pager used by man and other programs:

export PAGER=/bin/less

Okay, yes, I feel like a complete "Where's the ANY key?" newbie at the moment.  The last time I changed an environment variable from the command line was maybe 5 years ago.  I had no memory of how to do it, and it took me forever to find it in the gigantic and mind-numbing man bash documentation.

I had to do this on my web host because for some reason $PAGER was pointing at an incorrect location, and I couldn't even get man to work without adding -P /bin/less to the command line every single time.

Anyway, I scrawled out some notes to myself on a piece of paper, under the heading
Getting man to use less
which at least geve me a laugh when I saw what I'd written. ;D

Also note:  the method used to export from a terminal window changes based on your shell type.  I know that method works for ksh and I believe it also works for bash.  I also know it will fail for csh shell types.

often the ~/.profile ~/.cshrc file tweak your enthronement.  On other boxes the /etc/profile or /etc/.login  type files can be used to load system wide defaults.


man ksh
man bash
man csh

will give you the common interfaces to a shell

many man pages also support the key word search as in

man -k login

if the index needs to be rebuild due to s/w installation on some boxes a
catman -w
command might be necessary to access the man pages defined in the MANPATH statement.

set
and
env
command will dump out the environment you are running with.
if you have "short cuts defined" the
alias
command will help you figure out why typing "vi" runs "vim" (sometimes)
also a "which vi" can help you figure out where a program is located. (Mostly)

Title: Re: Un-obvious Linux tricks
Post by: fritz_monroe on March 07, 2013, 07:25:04 PM
For anyone looking to go a little geeky on a podcast, I really like the Going Linux Podcast (http://goinglinux.com/).  Great for anyone.

A bonus is that if you run into a problem in Linux and e-mail them, the hose if pretty quick to respond.
Title: Re: Un-obvious Linux tricks
Post by: Mr. Bill on March 07, 2013, 07:51:47 PM
give me a call next time if you get stuck and need help. my phone is always (groan) on.

Thanks.  I almost did that, but I was able to have a better temper tantrum doing it myself. ;D

...Those using tcsh or csh can look at the set and setenv    commands. ...

Yeah, that was my other problem, and thank you for making me realize what the issue was.

I was using csh.  I have a scrawled note to myself from a few months ago listing the set and setenv commands.  Yesterday I used chsh to change my shell from csh to bash (thanks for telling me how to do this, Archer!).  Today I tried using set and setenv and they didn't work.  Well, duh.
Title: Re: Un-obvious Linux tricks
Post by: Smurf Hunter on March 07, 2013, 10:26:14 PM
In the early days of my career we had to worry about the different *nixes:  SunOS, BSD, PC-UX, Linux

That forced us to learn common denominator skills, like plain old "vi" (not vim), and universal commands that worked in all shells.
Today, 99% of unix environments I encountered are linux, and the world is a far simpler place.
Title: Re: Un-obvious Linux tricks
Post by: Mr. Bill on April 07, 2013, 06:06:02 PM
I just had to haul this out of my ~/bin directory yesterday: it's a very simple script that tries to contact a site over and over, and beeps when it succeeds.

Code: [Select]
#!/bin/bash

# pingring
# by WSS
# version 2, 2013-04-07
# Pings a site until it responds, then plays a tone.
# Requires package "sox" for tone generation.
# Usage:  pingring [site]
#   where site = an IP address (111.22.333.44) or site name (abc.example.com)

# default site to ping, if not specified on command line [CHANGE THIS FOR YOUR SITUATION]:
DEFAULTSITE="192.168.1.1"
# time to wait for ping response before giving up (seconds):
WAITTIME=3
# sleep time before trying again (seconds):
SLEEPTIME=17

if [ "$1" == "" ]
then
  SITE=$DEFAULTSITE
else
  SITE=$1
fi

while ! ping -c 1 -w $WAITTIME $SITE ; do
  date
  sleep $SLEEPTIME
done

echo "PING RESPONSE RECEIVED"
date
# play a short synthesized tone
play -q -n -c1 synth sin %-24 sin %-9 sin %-5 sin %-2 fade q 0.2 1 0.5

exit 0

To use this:

What is this for?  Obviously you can use it to check if a website is alive or dead.  For those of us with erratic Internet service, it's a good tool to check whether your own Internet connection is up-and-running.  For this purpose I suggest playing around with tracepath (or traceroute or similar tool) to map out the nearby part of your Internet connection to the rest of the world.  Find some nearby branch point through which all your traffic appears to pass, and use that as your DEFAULTSITE.  Then, next time your Internet service goes down, you can just type pingring and go do something else until you hear the bleep.
Title: Re: Un-obvious Linux tricks
Post by: archer on April 07, 2013, 08:07:28 PM
neat little tool
Title: Re: Un-obvious Linux tricks
Post by: AverageDude on April 08, 2013, 05:34:18 AM

Why not just use the ping "-a" option or the wget audible options?
Title: Re: Un-obvious Linux tricks
Post by: Mr. Bill on April 08, 2013, 12:48:55 PM
Why not just use the ping "-a" option or the wget audible options?

Thre's always umpteen ways to do anything in Linux.  I never noticed ping -a, but I just tried it and I don't get anything audible on my machine, so that won't work for me.

I'm not sure that wget does what I want here.  I just want a response from Computer X at a particular IP address; I don't want to download a file, or even be dependent on whether Computer X has an http or ftp server running.  (Also I don't see any audible options in man wget, at least for the version I've got installed here.)

Also, part of my goal when I wrote Version 1 several years ago was just to get practice in writing a simple shell script.  I posted it here partly to help out people who have never done anything like that.  Even if you've never seen any computer program in your entire life, you can probably work out what it does just by reading it.  (Well, except for that play command at the end, but I just got that by stealing someone else's "bleep" and fiddling with the parameters until I liked the sound.)
Title: Re: Un-obvious Linux tricks
Post by: Smurf Hunter on April 08, 2013, 01:51:03 PM
when wget doesn't do it, I use curl.

http://curl.haxx.se/docs/manpage.html (http://curl.haxx.se/docs/manpage.html)
Title: Re: Un-obvious Linux tricks
Post by: ChEng on April 09, 2013, 05:54:32 PM
The hard core original "vi" almost cost me a computer.  I got so disgusted by it's user unfriendliness I almost pushed the monitor off the work area. While vim is a fantastic improvement over vi, and the gvim version provides a worth while gui interface that beats the crap out of xemac's interface.  BTW: - vim sucks, but sucks less and still works well over very small 56k lines. This why I pronounce "vi" as in evil.   

...

I never took the time to get familiar with vi, just never worked out.  Having gotten my start with CP/M, TRS-DOS and then MS-DOS, I got used to some simple text editors (never liked line-oriented editors, like ed.)  In that vain, I quickly picked up on nano (and pico, which is much like it) for simple text editing.  It reminds me of simpler stuff like MS-DOS' edit.  If your *nix doesn't have one or the other, they are available in most repositories.
Title: Re: Un-obvious Linux tricks
Post by: Smurf Hunter on April 09, 2013, 09:45:53 PM
I'm not a vi wizard, well perhaps compared to folks today I am :)

I kind of view mastery of the editor to be on par with driving a manual transmission.  You can certainly get by fine without knowing it, but as an old mentor used to say "it gets you closer to the machine".
Title: Re: Un-obvious Linux tricks
Post by: Bradbn4 on April 09, 2013, 10:17:52 PM
I never took the time to get familiar with vi, just never worked out.  Having gotten my start with CP/M, TRS-DOS and then MS-DOS, I got used to some simple text editors (never liked line-oriented editors, like ed.)  In that vain, I quickly picked up on nano (and pico, which is much like it) for simple text editing.  It reminds me of simpler stuff like MS-DOS' edit.  If your *nix doesn't have one or the other, they are available in most repositories.

I used programs like "edt", teco,  - I have built pico, pine, and others for working on my shell account when I signed up for the intrenet. 

I have also boot strapped my own compiler and tool set without root on various Unix boxes.

gnu.org is a fine site to obtain the source code so you can tweak / twiddle the program to do exactly just what you want.

Tip - use gvimdiff <file 1> < file 2> to show a side by side difference between two files and the editor will highlight which character(s) are different.

Tip #2 - if you need to print out some text files - locate and use a2ps (ASCII 2 postscript) and it can reformat source code any which way you want.  It can add line / page numbers, it can also pretty print out support many programming different languages. 

Tip #3 - if you want a good free editor for program development - look at netbeans

Title: Re: Un-obvious Linux tricks
Post by: archer on April 09, 2013, 10:31:42 PM
instead of gvimdiff, i use sdiff. it displays a side by side comparision and can show only differences if needed
Title: Re: Un-obvious Linux tricks
Post by: Bradbn4 on April 10, 2013, 07:01:52 PM
instead of gvimdiff, i use sdiff. it displays a side by side comparison and can show only differences if needed

true - often I use "sdiff -w 180  <file 1> < file 2>  the "sdiff" truncate the line length so you may not always see what really changed. 

sometimes I just use "diff" will provide you a bare minimum data - and "dircmp" is useful for looking at a whole lot of files.


if you are working in "large" programming env - consider creating a tag file - global might be one of the tools that can make a different types of tag files depending on your development environment.   

Note: A tag file is useful to cross reference program calls, data structures, etc

Today I used the basic script below to re-parse a configuration file to test directory structures.  I will often use this brute force method over awk, sed, naw, etc because it is easy to modify.  I will use the "cut command" to part out parts of the line.

#!/bin/ksh
file="/path/to/file.txt"
# while loop
while read line
do
        # display line or do something on $line
   echo "$line"
done <"$file"

to see a ksh script run in full "debug mode" I will often type
set -x
<program>
set +x

and if the output is large I will use

script /tmp/mylogfile

<commands here>
ctrl-d

then I issue dos2unix command like

cd /tmp
dos2unix mylogfile > mylogfile.txt
to rip off the end of line characters.

If you hare trying to communicate with another box using "ssh" and having issues.  I find that doing the "ssh -v myid@unixbox2" provides good level of debugging.

If you are tried blowing you budget on high end communications programs like exceed; look at putty.

Title: Re: Un-obvious Linux tricks
Post by: Smurf Hunter on April 10, 2013, 10:49:59 PM
Nice!

Though I always expect sed and awk to impress chicks.

Here's same Unix cat humor:

Code: [Select]
$ cat "door: paws too slippery"
  cat: cannot open door: paws too slippery
 
$ cat "food in tin cans"
  cat: cannot open food in tin cans
Title: Re: Un-obvious Linux tricks
Post by: d3nni5 on April 11, 2013, 04:37:00 AM
Nice!

Though I always expect sed and awk to impress chicks.

Here's same Unix cat humor:

Code: [Select]
$ cat "door: paws too slippery"
  cat: cannot open door: paws too slippery
 
$ cat "food in tin cans"
  cat: cannot open food in tin cans


An oldie but a goodie...

Code: [Select]

$  cd /pub
$  more beer
$  more beer
$  more beer
$  cd /home
$  sleep 28800


there is also a t-shirt out there that says "I read your email" I always found comical. :)
Title: Re: Un-obvious Linux tricks
Post by: archer on April 11, 2013, 12:40:50 PM
and another oldy by goody

Unix SEX :{ look; gawk; find; sed; talk; grep; touch; finger; find; flex; unzip; head; tail; mount; workbone; fsck; yes; gasp; fsck; more; yes; yes; eject; umount; makeclean; zip; split; done; exit:xargs!!;)} (source: someone’s signature in the Debian mailing lists).

Title: Re: Un-obvious Linux tricks
Post by: ChEng on April 11, 2013, 05:08:23 PM

An oldie but a goodie...

Code: [Select]

$  cd /pub
$  more beer
$  more beer
$  more beer
$  cd /home
$  sleep 28800


there is also a t-shirt out there that says "I read your email" I always found comical. :)

LOL, with me, though, that much beer would require more like:

Code: [Select]
sleep 86400

Can't handle it like I did when I was younger.  ;D
Title: Re: Un-obvious Linux tricks
Post by: Mr. Bill on August 10, 2018, 10:48:37 AM
Well, I think this qualifies as an un-obvious Linux trick.  But it's pretty nerdy, so consider yourselves warned.

Format a USB flash drive with UDF, and preserve Linux file ownership, permissions, and attributes (https://highbro.ws/node/31)

Quote
Scenario:
  • You're a Linux user, trying to prepare for disaster. So you want to copy all your files to a USB flash drive or other portable external drive.
  • After the disaster, you'll want to rebuild a Linux system. Which means you'll want all those files to have their original owner, group, permissions, and attributes. Which means you can't use FAT32/VFAT (the format of all commercial flash drives), because it's unable to store that data with the files.
  • But you also want your flash drive to be readable, and preferably writable, on Windows and Macintosh systems, since in your post-disaster world, you may not have access to a Linux system for a while. Which means you can't (easily) use a Linux format like EXT4, because this is unreadable on Windows/Mac unless you install additional software.
UDF (Universal Disk Format) provides a mostly-satisfactory solution. ...

...<SNIP many pages of technical how-to>...

So, is this worth the effort?

Maybe.

In our household, we have a specific use scenario. Both my wife and I use Linux on our main desktop computers. If one computer dies, we could share a computer for a while. We've already set up both our machines with two users, just in case. So a UDF backup flash drive would give us a quick recovery.

But we're also planning for other disasters, like our house burning down, or a sudden need to visit a sick relative. In this situation, we'd be able to carry all our files with us. And if we only have access to non-Linux computers, we'll still be able to retrieve a copy of our household inventory for the insurance company, or whatever. ...
Title: Re: Un-obvious Linux tricks
Post by: Bradbn4 on August 10, 2018, 08:12:45 PM
For Linux I started to use VM type software - easy to just save the whole environment. 
I use to run bare metal linux - but I figured it was easier to play with VM and all.  Back then I had removable drive kits so I just cloned the boot drive when I needed it.

But good option for backup -

 
Title: Re: Un-obvious Linux tricks
Post by: surfivor on August 11, 2018, 02:42:27 AM
 I think I missed this thread ..

Anyway, I have used vi and vim. vi is kind of primitive. I have also used emacs, but I currently use sublime:
https://www.sublimetext.com/

linux is an ideal development environment.

 Here is a type of command I have often used. Here I am searching source files for the string 'back-button', but I do not want it to search in the node_modules directory as those are modules brought in by webpack and are not my code. I want to search all files named either *.js, *.jsx, *.slim, *.coffee, or *.erb. Some files are compressed .js files with no new lines. These are generated files. I don't want to search them and they mess up the search by printing massive output when matched.
This part says ignore all lines that are more than 299 characters and takes care of that. That means a line that big without a newline character
Quote
awk 'length<300'

full command:
Quote
find . \( -type d -name 'node_modules' -prune  -o -name "*.js" -o -name "*.jsx" -o -name "*.slim" -o -name "*.erb" -o -name "*.coffee" \) -exec grep 'back-button' {} \; -print | awk 'length<300'


find large use of space:

Quote
du -h -d 5 / 2>/dev/null | sort -hr | head -n 50

du -h / 2>/dev/null | grep -P '^[0-9\.]+G'

kill various processes based on their name:

Quote

ps -ef | grep massruby | egrep 'node|webpack' | awk '{ print $2 }' | xargs kill -9


ps -ef | egrep 'rails server|react_on_rails|redis-server|status_job:runner|sidekiq' | awk '{ print $2 }' | xargs kill -9


========

other stuff from notes, what does this do ? I forget, I guess my notes here were lacking as I copied down stuff I tried but forgot to put in my notes what they do.

Quote
find . \( -name "*.rb" -or -name "*.slim" -or -name "*.js" -or -name "*.jsx" -or -name "*.scss" -or -name "*.erb" -or -name "*.css" \) -o  -path './db' -prune -o -path './shared/uploads' -prune -o -path './client/node_modules' -prune -o -path './.git' -prune -o -path './log' -prune -o -path './tmp' -prune

find . \( -name "*.rb" -or -name "*.slim" -or -name "*.js" -or -name "*.jsx" -or -name "*.scss" -or -name "*.erb" -or -name "*.css" \) -o -not \( -path './db' -prune \) -not \( -path './shared' -prune \)  -not \( -path './client/node_modules' -prune \) -not \( -path './shared/uploads' -prune \) -not \( -path './.git' -prune \) -not \( -path './log' -prune \) -not \( -path './tmp' -prune \) -not \( -path './.Trash-1000' -prune \) -not \( -path './public' -prune \) -not \( -path './lib' -prune \)


 I also use git a great deal, and have been on linux mint but was on ubuntu previously. Another good one is egrep:
 
 find either red_apples or yellow_pairs
Quote
egrep 'red_apples|yellow_pairs' *.js
Title: Re: Un-obvious Linux tricks
Post by: surfivor on August 11, 2018, 03:04:28 AM

Cygwin is a cool way to get a Linux shell on Microsoft Windows
http://www.cygwin.com