Files #

Find files and run commands #

-exec command {} \;: run the command for each matched file:

find . -iname '*.pdf' -exec pdfgrep -Hni "storage" {} \;

-exec command {} +: append all found files to the command (the command is not run for each matched file):

find . -iname '*.pdf' -exec pdfgrep -Hni "storage" {} +  2> /dev/null

Search string in files #

ripgrep is known to be the fastest for code searching:

rg -g "*.h" -g "*.c" string # find the string in C files

Coreutils #

See coreutils gotchas for more.

dd #

Bypass the Linux VFS as much as possible to avoid systems stalls, especially with slow devices.
When dd is done, no further sync needed.

dd bs=2M if=disk.img of=/dev/sda status=progress iflag=direct oflag=direct

other methods of copying image:

cp myfile.iso /dev/sdb
cat /dev/cdrom > myfile.iso
gzip -9 < /dev/sdb > /tmp/myimage.gz

Get extension of a file #

echo xxxx.wmv | rev | cut -f1 -d'.' | rev

Truncate a file #

# colon is equal to true: http://stackoverflow.com/questions/3224878/what-is-the-purpose-of-the-colon-gnu-bash-builtin
: > /something
true > something

see http://www.vidarholen.net/contents/blog/?p=479

Subshell #

# */ all directories within a directory
# cd to each dir and run make
for dir in */; do ( cd "$dir" && make ); done

GNU parallel #

Repeatedly do something #

Run a command without input argument for N times.

seq 10 | parallel -n0 echo hi

Log to file #

Run from input and dump to corresponding output file:

parallel customScript -c 33 -I -file {} -a -v 55 '>' {.}.output ::: *.input

Input from file #

parallel ssh {} :::: host.list

Multiple input sources #

Generates all combinations of the input sources:

parallel echo ::: A B C ::: D E F

Output (the order may be different):

  A D
  A E
  A F
  B D
  B E
  B F
  C D
  C E
  C F

show progress: #

parallel --progress
parallel --bar

lsof #

List all open files

• Processes using a file? (fuser equivalent): lsof /path/to/file
• Open files within a directory: lsof +D /path
• By program name: lsof -c apache
• By users: lsof -u user1,user2
• AND’ing selection conditions: lsof -u www-data -c apache
• By pid:lsof -p 1
• All network activity by a user: lsof -a -u name1 -i
• Except certain pids: lsof -p ^1
• TCP and UDP connections: lsof -i tcp
• By port: lsof -i :25

SSH #

SSH Config #

Maintain persistent connections to frequently-visiting hosts:

Host bsd* linux* oasis*
    HostName %h.csie.ntu.edu.tw
    ControlMaster auto
    ControlPath /tmp/.%r@%h-%p
    ControlPersist 600
    IdentityFile /home/yunchih/.secret/ssh/yunchih@csie_rsa

Bitbucket & Github SSH Config

Host github.com
    HostName github.com
    User git
    PreferredAuthentications publickey
    IdentityFile ~/.secret/ssh/id_ecdsa_github

Host bitbucket.org
    IdentityFile ~/.secret/ssh/id_ecdsa_bitbucket

Proxy via a third-party host into an internal host

Host dest
    ProxyCommand ssh -q proxy_host -W [destination IP]:[destination port] 

SSH Reverse Proxy #

Connection to port 8888 of remote host will be redirected to localhost of current host. Most usefully run as a systemd service or bash inifinite loop to maintain a connection into a NAT or firewall-protected host. (remote holds public unrestricted IP, local host is protected)

ssh -R 8888:[localhost]:22 [remote] -N

SSH Forward Proxy #

Connection to port 8888 of localhost will be redirected to [another host]:[another port] from [remote]. This enables local machine to access a service available only from [remote].

ssh -L 8888:[another host]:[another port] [remote] -N