xargs command in linux ; xargs vs exec

 Linux and Unix xargs command tutorial with examples

https://shapeshed.com/unix-xargs/

 

xargs
 - command line for building an execution pipeline from standard input.

-  reads items from standard input as separated by blanks and executes a command once for each argument. 

echo 'one two three' | xargs mkdir
ls
one two three

xargs vs exec 

The find command supports the -exec option that allows arbitrary commands to be performed on found files. The following are equivalent.

find ./foo -type f -name "*.txt" -exec rm {} \; 

find ./foo -type f -name "*.txt" | xargs rm

So which one is faster? Let’s compare a folder with 1000 files in it.

time find . -type f -name "*.txt" -exec rm {} \; 0.35s user 0.11s system 99% cpu 0.467 total

 time find ./foo -type f -name "*.txt" | xargs rm 0.00s user 0.01s system 75% cpu 0.016 total

Clearly using xargs is far more efficient. In fact several benchmarks suggest using xargs over exec {} is six times more efficient.

grep : Using grep to search for a string that has a dot in it

https://stackoverflow.com/questions/10346816/using-grep-to-search-for-a-string-that-has-a-dot-in-it

 in grep,  . means "any character" in a regex.

-F  : which would interpret the pattern as a fixed string
-w :  Checking for full words, not for sub-strings using grep -w


grep uses regexes; . means "any character" in a regex.
 If you want a literal string, use grep -F, fgrep, or escape the . to \.
Don't forget to wrap your string in double quotes. Or else you should use \\.

How to copy files from one machine to another using ssh [duplicate]

https://unix.stackexchange.com/questions/106480/how-to-copy-files-from-one-machine-to-another-using-ssh

Syntax:

scp <source> <destination>

To copy a file from B to A while logged into B:

scp /path/to/file username@a:/path/to/destination

To copy a file from B to A while logged into A:

scp username@b:/path/to/file /path/to/destination

Installing CMake

https://cmake.org/install/
https://cmake.org/download/
https://gist.github.com/1duo/38af1abd68a2c7fe5087532ab968574e#file-centos-install-cmake-from-source-md

Compile from source and install

tar zxvf cmake-3.*
cd cmake-3.*
./bootstrap --prefix=/usr/local
make
make install 

Validate installation

cmake --version

cmake version *.*.*
CMake suite maintained and supported by Kitware (kitware.com/cmake). 

Install, uninstall RPM on linux (CentOS, RHEL, Fedora)

https://phoenixnap.com/kb/how-to-install-rpm-file-centos-linux

Install RPM on Linux

sudo rpm –i sample_file.rpm

sudo yum localinstall sample_file.rpm 

Remove RPM Package

sudo rpm –e sample_file.rpm

Using grep -v on multiple arguments

https://www.thegeekstuff.com/2011/10/grep-or-and-not-operators

Using grep -v on multiple arguments

grep -v -e foo -e bar file

grep -Ev 'word1|word2'

Index a string in linux

https://unix.stackexchange.com/questions/303960/index-a-string-in-bash

Substring Extraction

${string:position}

Extracts substring from $string at $position.

If the $string parameter is "*" or "@", then this extracts the positional parameters, starting at $position.

${string:position:length}

Extracts $length characters of substring from $string at $position.

stringZ=abcABC123ABCabc
#       0123456789.....
#       0-based indexing.

echo ${stringZ:0}                       # abcABC123ABCabc
echo ${stringZ:1}                       # bcABC123ABCabc
echo ${stringZ:7}                       # 23ABCabc 

echo ${stringZ:7:3}                     # 23A
                                        # Three characters of substring.


# Is it possible to index from the right end of the string?

echo ${stringZ:-4}                      # abcABC123ABCabc
# Defaults to full string, as in ${parameter:-default}.
# However . . . 

echo ${stringZ:(-4)}                    # Cabc
echo ${stringZ: -4}                     # Cabc
# Now, it works.
# Parentheses or added space "escape" the position parameter.

Linux comm command

http://www.unixcl.com/2009/08/linux-comm-command-brief-tutorial.html


From COMM(1) man page, the options available are:

-1 suppress lines unique to FILE1
-2 suppress lines unique to FILE2
-3 suppress lines that appear in both files

comm - compare two sorted files line by line

comm <(sort a.txt) <(sort b.txt)

Sed Command in Linux

Replacing all the occurrence of the pattern in a line : The substitute flag /g (global replacement) specifies the sed command to replace all the occurrences of the string in the line.

$sed 's/apple/lemon/g' test.txt

https://www.tecmint.com/linux-sed-command-tips-tricks/


https://www.geeksforgeeks.org/sed-command-in-linux-unix-with-examples/

https://likegeeks.com/sed-linux/

Redirect stderr and stdout in Bash

https://stackoverflow.com/questions/637827/redirect-stderr-and-stdout-in-bash


You can redirect stderr to stdout and the stdout into a file:

some_command >file.log 2>&1 

To append file
echo "foo" 1>> bar.txt 2>&1 

Mount disks with HFS+ volumes ( used in Mac OS) in CentOS

http://opensysblog.directorioc.net/2015/07/centos-mount-disks-with-hfs-volumes.html
https://www.centos.org/forums/viewtopic.php?t=67360


https://askubuntu.com/questions/332315/how-to-read-and-write-hfs-journaled-external-hdd-in-ubuntu-without-access-to-os

$ sudo rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

$ sudo rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
$ sudo yum install kmod-hfsplus

$ sudo yum install kmod-hfs 
$ sudo yum install hfsplus-tools

after install, you might need reboot

By default, the partition will be mounted in read-only mode
use force option to activate read-write mode

$ sudo mount -t hfsplus -o force -o rw /dev/sdc2 /media/disk1/

Add a New Disk Larger Than 2TB to An Existing Linux

see more details in
https://www.tecmint.com/add-disk-larger-than-2tb-to-an-existing-linux/


# list current partitions

[lee@hwang29 ~]$ sudo fdisk -l




# move to disk
[lee@hwang29 ~]$ sudo fdisk /dev/sda


 
d : delete a partition  # deletes the partition. It will delete the data on the disk
w : write table to disk and exit

# write partition with parted
[lee@hwang29 ~]$ sudo parted /dev/sda

(parted) mklabel gpt   # set partition table format to GPT (GUID Partition Table)
(parted) mkpart primary 0GB 3000GB # create primary partition, assaing disk capacity
(parted) quit


# Creating an ext4 File System
[lee@hwang29 ~]$ sudo mkfs.ext4 /dev/sda

# mount disk
[lee@hwang29 ~]$ sudo mount /dev/sda /data_1

# add entry to /etc/fstab for permanent mounting
[lee@hwang29 ~]$ vim /etc/fstab
/dev/sda    /data_1    ext4    defaults    0    0


tip for naming the disk
- /dev/disk/by-id/scsi-SATA_ModelName_partion1 instead of  /dev/disk/sda
- if you are using multiple disk/partition and one of them is failed, it is easy to identify which disk and partition goes wrong
- if you change, add, move disk, it might cause problems because other disk already use name sda, sdb

How to read a file into a variable in shell?

https://stackoverflow.com/questions/7427262/how-to-read-a-file-into-a-variable-in-shell


1. make file name list

ls | grep .gz > file_name_list.txt


2.  read file into shell variable & do 'for'

value=$(<file_name_list.txt)
for name in $value
do
/program_path/program -i /file_path/${name}
done

mount iso file in linux

https://askubuntu.com/questions/164227/how-to-mount-an-iso-file

1. Create a directory to serve as the mount location:
   

   sudo mkdir /mnt/iso
   

2. Mount the ISO in the target directory:
   

   sudo mount -o loop path/to/iso/file/YOUR_ISO_FILE.ISO /mnt/iso
   

3. Unmount the ISO:
   

   sudo umount /mnt/iso
   

Qt: XKEYBOARD extension not present on the X server.

check details in
https://github.com/spyder-ide/spyder/issues/3713


From replies

athlonshi commented on 4 Jul 2017

For those using Exceed X Server, you may also need to change Xconfig setting. Go to Xconfig --> Settings --> Protocol... --> Extensions --> check XKEYBOARD and save the settings.

Oreki47 commented on 6 Oct 2017

I'm using TightVNC with Ubuntu 14.04.
It works on my machine with:
export XKB_DEFAULT_RULES=base
export QT_XKB_CONFIG_ROOT=/usr/share/X11/xkb
The set rules has to come first.

How to do MD5 checksum

http://linux.byexamples.com/archives/198/md5-checksum-how-to/

md5 is used to verify data integrity, the correctness of files you downloaded.

[admin@localhost]$ md5sum your_file

check multiple file

1) create md5sum.txt file

1st column : md5 string     2nd column : location of the file.

[admin@localhost]$ more md5sum.txt
9e9ccb13b31defc2b5207646d9ba07b5 Ibd11-1_1.fastq.gz
df040df4307bc24f54a69c9774d5a72f Ibd11-1_2.fastq.gz
a3fa0b9a3d9b2d843319b7ea3374318b Ibd11-2_1.fastq.gz
be29dcc8407a09ab1f0316727c1651bf Ibd11-2_2.fastq.gz
8e6f2cbaaf124be4b96b0e03e4ffa680 Ibd11-3_1.fastq.gz
8586451a651c37d4b6b9d72998f581e6 Ibd11-3_2.fastq.gz

[admin@localhost]$ md5sum -c md5sum.txt
Ibd11-1_1.fastq.gz: OK
Ibd11-1_2.fastq.gz: OK
Ibd11-2_1.fastq.gz: OK
Ibd11-2_2.fastq.gz: OK
Ibd11-3_1.fastq.gz: OK
Ibd11-3_2.fastq.gz: OK 

nohup - running program as background in linux

nohup ./workflow.sh &     # running as background

output is saved to  nohup.out


for terminate nohup

kill -9 PID number   # you can check PID number with 'top' command

TOP command examples on Linux to monitor processes

for more information, visit
http://www.binarytides.com/linux-top-command/

$ top

top - 18:50:35 up  9:05,  5 users,  load average: 0.68, 0.52, 0.39
Tasks: 254 total,   1 running, 252 sleeping,   0 stopped,   1 zombie
%Cpu(s):  2.3 us,  0.5 sy,  0.0 ni, 97.1 id,  0.2 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   8165300 total,  6567896 used,  1597404 free,   219232 buffers
KiB Swap:  1998844 total,        0 used,  1998844 free.  2445372 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND             
17952 enlight+  20   0 1062096 363340  88068 S   4.8  4.4   0:49.33 chrome              
14294 enlight+  20   0  954752 203548  61404 S   2.1  2.5   2:00.91 chrome              
 1364 root      20   0  519048 105704  65348 S   0.6  1.3  17:31.27 Xorg                
19211 enlight+  20   0  576608  47216  39136 S   0.6  0.6   0:01.01 konsole             
   13 root      rt   0       0      0      0 S   0.3  0.0   0:00.10 watchdog/1          
   25 root      20   0       0      0      0 S   0.3  0.0   0:03.49 rcuos/2             
 1724 enlight+  20   0  430144  36456  32608 S   0.3  0.4   0:03.60 akonadi_contact     
 1869 enlight+  20   0  534708  52700  38132 S   0.3  0.6   0:53.94 yakuake             
14040 enlight+  20   0  858176 133944  61152 S   0.3  1.6   0:09.89 chrome

USER - The system user account running the process.
%CPU - CPU usage by the process.
%MEM - Memory usage by the process
COMMAND - The command (executable file) of the process

Display full command path and arguments of process - 'c'

Press 'c' to display the full command path along with the commandline arguments in the COMMAND column.

%CPU %MEM     TIME+ COMMAND                                                    
  0.0  0.0   0:00.00 /usr/bin/dbus-launch --exit-with-session /usr/bin/im-laun+ 
  0.0  0.1   0:01.52 /usr/bin/dbus-daemon --fork --print-pid 5 --print-address+ 
  0.0  0.3   0:00.41 /usr/bin/kwalletd --pam-login 17 20                        
  0.0  0.0   0:00.00 /usr/lib/x86_64-linux-gnu/libexec/kf5/start_kdeinit --kde+ 
  0.0  0.3   0:01.55 klauncher [kdeinit5] --fd=9                                
  0.0  0.2   0:00.13 /usr/lib/telepathy/mission-control-5                       
  0.0  0.1   0:00.00 /usr/lib/dconf/dconf-service                               
  0.0  0.4   0:01.41 /usr/lib/x86_64-linux-gnu/libexec/kdeconnectd              
  0.0  0.2   0:01.09 /usr/lib/x86_64-linux-gnu/libexec/kf5/kscreen_backend_lau+

Display all CPU cores - '1'

Pressing '1' will display the load information about individual cpu cores. Here is how it looks -

top - 10:45:47 up  1:42,  5 users,  load average: 0.81, 1.14, 0.94
Tasks: 260 total,   2 running, 257 sleeping,   0 stopped,   1 zombie
%Cpu0  :  3.6 us,  3.6 sy,  0.0 ni, 92.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  3.1 us,  3.6 sy,  0.0 ni, 93.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  7.6 us,  1.8 sy,  0.0 ni, 90.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  9.6 us,  2.6 sy,  0.0 ni, 87.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   8165300 total,  7118864 used,  1046436 free,   204224 buffers
KiB Swap:  1998844 total,        0 used,  1998844 free.  3410364 cached Mem

Batch mode

Top also supports batch mode output, where it would keep printing information sequentially instead of a single screen. This is useful when you need to log the top output for later analysis of some kind.

Here is a simple example that shows the Cpu usage at intervals of 1 second.

$ top -d 1.0 -b | grep Cpu

Linux find command

for more information, visit
http://www.binarytides.com/linux-find-command-examples/


basic syntax
$ find location comparison-criteria search-term

searches for files by their name

$ find ./test -name "abc.txt"
./test/abc.txt

wildcards

$ find ./test -name "*.php"
./test/subdir/how.php
./test/cool.php

all sub directories are searched recursively. So this is a very powerful way to find all files of a given extension.

Trying to search the "/" directory which is the root, would search the entire file system including mounted devices and network storage devices. So be careful. Of course you can press Ctrl + c anytime to stop the command.

add swap space

if you run softwares that require a large amount of memory, you need to add more swap space

You have three options: create a new swap partition, create a new swap file, or extend swap on an existing LVM2 logical volume. It is recommended that you extend an existing logical volume.

https://www.centos.org/docs/5/html/Deployment_Guide-en-US/s1-swap-adding.html