Linux: Solution for 'Not Able to Access Remote Server'

If you're playing with the Linux servers, one of the bothering issue is that you're not able to access other server from the current server

Or, MySQL replication or other service will not work even if you're able to ping from one server to the other or vice versa

Or, you may not be able to access the Tomcat service through your browser

Or, any other such connectivity issue, here is the solution that I used:

Just reset all the Firewall(Iptables) rules and you're done!


First get into SUDO and check the applied rules. It should be looking like below:

[root@myhost ]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:opsession-prxy

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)

target     prot opt source               destination



In my case, the highlighted line is the culprit! It just says to REJECT all incoming connections !

Either you can modify the rules by removing this line, or just like me, remove all the rules !!

Here is how to remove/flush  the firewall rules:



[root@myhost ]# iptables --flush


Yes, that's it!

Linux: awk - Group By Count File Data

While its pretty easy to do 'Group By' at database level, 'awk' enables us to do same at file level.

Consider a file as below:

# cat test.csv
anita 111
rama 555
david 555
raj 111
shaik 222
naren 222

Here, I want to know how many names in the 1st column are having same values in the second column. 

That's nothing but a simple Group By in MySQL or any other database. 

So, load to a table and execute the query.

But, if the file having crores of data, MySQL query would be a costly operation.

In this time, 'awk' command will come for your help.

Below is the command:

# awk  '{arr[$2]++;}END{for(i in arr)print i, arr[i] ;}' test.csv

555 2

111 2

222 2

Here, awk is fetching the second column into an array and counting the appearance.

If, the file is comma separated(CSV), use the below command:

# awk  -F, '{arr[$2]++;}END{for(i in arr)print i, arr[i] ;}' test.csv

555 2

111 2

222 2

Linux: Concatenate In Linux

Here is the commands to concatenate in Linux.

Consider a file as below:

[root@myserver 6048]# cat test.txt
12
0
123445456
234565443
3883
33

My requirement is to generate update/delete statement for each of this value. Think of a file with millions of records.

My update statement would be:

update mytable set col1='Yes' where col2=' value in the file';

Here's the solution:

[root@myserver 6048]# cat test.txt | sed "s/^/update mytable set col1='Yes' where col2='/g;s/$/';/g"
update mytable set col1='Yes' where col2='12';
update mytable set col1='Yes' where col2='0';
update mytable set col1='Yes' where col2='123445456';
update mytable set col1='Yes' where col2='234565443';
update mytable set col1='Yes' where col2='3883';
update mytable set col1='Yes' where col2='33';
update mytable set col1='Yes' where col2='';
update mytable set col1='Yes' where col2='987676';


You may redirect the output to some other file, as below:

[root@myserver 6048]# cat test.txt | sed "s/^/update mytable set col1='Yes' where col2='/g;s/$/';/g" > updates_test.sql


You may also use the below query for the purpose:


[root@myserver 6048]# sed  "s/^/update mytable set col1='Yes' where col2='/g;s/$/';/g"  test.txt > updates_test.sql


Ping me for any doubts.. +Kiran Yadagere  | facebook.com/kiranyadagere

Linux: Adding Numbers in a File

Here's a Linux command to add numbers inside a file:


[root@myserver misc]# cat num.test
1
2
3
4
5


This is a sample file with only 5 lines. Think of a file with millions of lines. Either you can do with MS - Excel, which may hang(!!) or use Linux command - 'awk'- which is powerful and easier:

[root@myserver misc]# awk '{s+=$1} END {print s}' num.test
15

As simple as that..!!

Linux: Sorting a File

Here's simple 'sort' command that will remove the duplicate entry from any file and sort in ascending order:

Consider a file with few numbers:

[root@myhost tmp]# cat testSort.txt
23
4
56
001
34
3
 

To sort it:

[root@myhost tmp]# sort -u testSort.txt > sortd_testSort.txt
 
wherein, -u --> Unique


Its sorted:

[root@myhost tmp]# cat sortd_testSort.txt
001
23
3
34
4
56
Oops! Though it worked, it didn't work correctly. Because the file contains numeric data.

So use the below:

[root@myhost tmp]# sort -u -n testSort.txt > numeric_sortd_testSort.txt wherein, -n --> for numeric data

 
And, now its correct:

[root@myhost tmp]# cat numeric_sortd_testSort.txt
001
3
4
23
34
56
However, this works only for the small files. Smaller than space available with '/tmp' partition.

If you want to sort huge files, use the below:

[root@myhost lib]# sort -u -n -T. testSort.txt > numeric_sortd_testSort_new.txt wherein, -T. --> T indicates the location used as temporary space. By default its '/tmp'.

'.' indicates current directory

Linux: Crontab - Brief

I always get confuse whenever I want to set a new cron job. The confuse is with regard to the options too be set!
For those who new to 'cron', its nothing but, an event scheduler in Linux. That means, you can schedule any script to run at any time you wanted to. Its just the system/server should be up and running!

cron job is specific to every user in Linux/Unix. So, one can't see other's cron unless the necessary privileges or sudo root access given.

Whatever, here is the options in cron:


To check cron jobs:

[root@localhost kiran]# crontab -l
no crontab for root

To set cron jobs:

[root@localhost kiran]# crontab -e

After adding, here is how it looks:

[root@localhost kiran]# crontab -l

##Script to test

00 */2 1-31 * 0,2,3   sh /home/kiran/test.sh >> /dev/null

Every Cron job should be given with 5 options:

• minute -> 0-59
• hour -> 0-23
• day of month -> 1-31
• month -> 1-12 
• day of week -> 0-7 (0 is Sunday )

In the above example:

00 -- 0th Minute

*/2 -- Every 2 hours

1-31 -- Every day (1 to 31)

* -- Every Month

0,2,3 -- Sunday,Tuesday,Wednesday

Linux: awk - Grouping Data in a file

Consider the below file:

# cat test.csv
aa 1 qwer
ab 2 tyui
aa 3 poiu
ab 2 mnb
bb 1 njio
ba 2 njtwe

test.csv  is a tab separated file with 3 columns. 

Here, I want to segregate the whole lines with matching 1st and 2nd columns into separate files. 

Like below:

# cat file_bb_1.csv

bb 1 njio

# cat file_ba_2.csv

ba 2 njtwe

# cat file_ab_2.csv

ab 2 tyui

ab 2 mnb

# cat file_aa_3.csv

aa 3 poiu

# cat file_aa_1.csv

aa 1 qwer

Though you can do this manually, think of a file with more than million lines.

Here, 'awk', being an powerful data manipulation tool,comes to our help. 

Below is the command we can use:

#cat test.csv | awk '{a=$1;b=$2; print $0 >> "file_" a "_" b ".csv"}'

[You can give any name instead of 'file_']

rsync: Remote File Transfer

While we usually deal with huge data, we come across situations where we need to copy such data from one server/host to a remote host. 

One of the useful command is 'rsync', which is faster than the traditional scp command!

Below is the syntax with example :

[root@myhost file_path]# rsync -avrz /file_path/* user@12.12.12.12:/destination_path/ >> /log_path/rsync.log 2>> /log_path/rsync.err

where as, 

-a -> Its same as --archive, which means to say preserver everything as it is. No changes between source and destination

-v -> Its same as --verbose, which displays information about the current execution

-r -> Its same as --recursive, copy directory and sub-directories, recursively

-z -> Its same as --compress, which compress data while its being transferred