while loop with ssh issue

somehow while loop with ssh only read the first line from the input file & exit without reading the next lines.
eg:

$ cat server.txt
server1
server2
server3
$ while read server
> do
> echo $server
> ssh $server uptime
> done < server.txt
server1
13:15:08 up 49 days, 1:07, 5 users, load average: 0.00, 0.00, 0.00

the uptime command will only be run on the server1 but not on the server2 or server3. to resolve this, use the -n option with ssh, which prevents ssh from reading from standard input:
> ssh -n $server uptime

alternatively, you can use for loop. should be no issue ;)

$ for server in `cat server.txt`
> do
> echo $server
> ssh $server uptime
> done

for more info - http://sial.org/howto/shell/while-ssh/

chage: can't lock shadow password file

i got this error "chage: can't lock shadow password file" when executing below command:

# chage -E 2008-04-28 username

google said that this error because of these 2 files exist /etc/passwd.lock or /etc/shadow.lock. they are created when a user is made with useradd command. they contain the PID of the command creating them (useradd). once the PID is gone (useradd exits) the locks no longer apply and should probably have been removed by the process that created them.

in the other word, if you remove these files, you should be able to use back the chage command above. but in my case it's still failed!

turn out that the server just rebooted & one of the start up script is hanging. the script is to start an application & need to su to one user & turn out that the user's password has expired! (got this when try to su to the user - "You are required to change your password immediately (password aged)" & hang there!)

the hanging script!

root 31001 31000 0 May02 ? 00:00:00 /bin/sh /etc/rc3.d/S99apps start
root 31002 31001 0 May02 ? 00:00:00 su - user -c /apps/startapps

killed those processes & settled! :)

p/s: changing a users passwd or locking/unlocking an account does not change, create, or remove these files.

sent file as an attachment from unix

for my reference & knowledge sharing on how to sent text file as an attachment from unix.
most of the ways i found on net, they suggest to do like this:

$ uuencode filename.txt attachment_name.txt | mailx -s "file attach" user@domain.com

it works fine but you'll not get the same format as your original file. in other word the text file format totally screw up!

so you should using this instead:

$ unix2dos -437 filename.txt | uuencode attachment_name.txt | mailx -s "file attach" user@domain.com

go figure out yourself what -437 is for :P

random number generator

i just happen to know that we can easily generate random number using $RANDOM

$ echo $RANDOM
12517
$ echo $RANDOM
18362
$ echo $RANDOM
26219
$ echo $RANDOM
26056
$ echo $RANDOM
30305
$ echo $RANDOM
9606

but if you want to get smaller random number eg. less than 100, you can do this:

$ number=$((RANDOM%100))
$ echo $number
75

20 random numbers less than 100:

$ count=1
$ while [ $count -le 20 ] ; do number=$((RANDOM%100)); echo $number; let "count += 1"; done
93
14
19
96
61
86
63
48
45
26
11
0
89
54
11
84
17
10
99
12

random number less than 10:

$ number=$((RANDOM%10))
$ echo $number
4

5 random numbers less than 10:

$ count=1
$ while [ $count -le 5 ] ; do number=$((RANDOM%10)); echo $number; let "count += 1"; done
3
4
9
6
7

and so on...

keychain rocks!

do you ever feel tired to key in your password or passphrase every time you want to ssh into your servers OR you need to run automate jobs (cron job) to ssh into specific servers but unfortunately unable to do it since you still need to key in your password or passphrase manually. well worry no more, here come the solution: use keychain!

first of all you'll need to setup the authentication using public key instead. then download the keychain from http://dev.gentoo.org/~agriffis/keychain/ & install the keychain into your Linux workstation.

ashterix@desktop:/home/ashterix $ ls -l bin/keychain
-rwx------ 1 ashterix u_026776 53740 Feb 27 08:25 bin/keychain

put this into your .profile or .bash_profile

keychain ~/.ssh/id_rsa
. ~/.keychain/`hostname`-sh

and you're done! the first time your login you'll ask to enter your passphrase of course.

KeyChain 2.6.8; http://www.gentoo.org/proj/en/keychain/
Copyright 2002-2004 Gentoo Foundation; Distributed under the GPL

* Initializing /home/ashterix/.keychain/desktop-sh file...
* Initializing /home/ashterix/.keychain/desktop-csh file...
* Initializing /home/ashterix/.keychain/desktop-fish file...
* Starting ssh-agent
* Adding 1 ssh key(s)...
Enter passphrase for /home/ashterix/.ssh/id_rsa:
Identity added: /home/ashterix/.ssh/id_rsa (/home/ashterix/.ssh/id_rsa)

ashterix@desktop:/home/ashterix $

if you log off & login again, the ssh-agent already there & you no need to key in your passphrase again.

KeyChain 2.6.8; http://www.gentoo.org/proj/en/keychain/
Copyright 2002-2004 Gentoo Foundation; Distributed under the GPL

* Found existing ssh-agent (19507)
* Known ssh key: /home/ashterix/.ssh/id_rsa

ashterix@desktop:/home/ashterix $

then from here you can ssh to any servers without key in the passphrase.
the same if you want to run automate ssh cron jobs, the passphrase automatically loaded, there's no need to key in the passphrase manually. cool eh :)

ashterix@desktop:/home/ashterix $ ssh server

ashterix@server :/home/ashterix $

for more info read here:
http://www-128.ibm.com/developerworks/library/l-keyc2/
http://www.gentoo.org/proj/en/keychain/index.xml
http://www.gentoo.org/doc/en/keychain-guide.xml

FAQ during interview for UNIX/LINUX system engineer/admin

I'm compiling a never_ending_list of questions that frequently asked during interview session for UNIX/LINUX system engineer/admin post. No, I'm not looking for new job right now or preparing for an interview, just want to make a list and share my experience. hopefully it could help others for their interview. even though you do all these tasks almost everyday, sometimes all the sudden your brain went blank during the session... dang!!! so be prepared!
you are most welcome to add yours in the comment!

here are the questions i've been asked, those that i can remember.
in no specific order:

1. Explain suid, sgid & sticky bit
2. What is sudo?
3. When you create a file in unix, by default what permissions will be applied to it? How do you change this setup?
4. Journaling filesystem? What is it?
5. What is /proc?
6. Explain steps to add new hard disks untill to mirror them
7. NFS export options/features/security
8. Autofs - Is it possible to mount the directory without cd into it?
9. What is stale file handle error in NFS?
10. What is the difference between UDP and TCP?
11. How do you list open files?
12. How to check number of files limit by kernel?
13. What is the difference between hard and soft mount?
14. Explain boot process
15. What is PAM?
16. What performance monitoring tools do you use?
17. Explain load average
18. What is ssh? How to setup trust between accounts/machines?
19. NIS administration/commands
20. How to limit certain users accessing certain machines?
21. How you troubleshoot slow performance?
22. Jumpstart & kickstart servers & process
23. Package management for linux & solaris - how to use them?
24. Solaris mirroring
25. What is zombie process?
26. Scripting skills, to automate certain jobs, cron
27. How do you perform changes on a few machines?
28. Xwindows options/features/security
.
.

looking for the answers? No... you have to find the answers yourself :P

zero the log files

what the different between these 2 commands?

# cat /dev/null > /var/log/messages

# > /var/log/messages

both commands zero the /var/log/messages file
but using > will not spawn a new process.
so you'll need to restart syslogd. (only on solaris but not on linux)

search engine for UNIX system administrators

today while browsing i found this site. quite useful :)

coolcommands.com is a search engine for UNIX system administrators. Our database provides access to useful commands for all flavours of UNIX including:

Sun Solaris IBM AIX HP/UX
Linux SCO UNIX OpenBSD
SGI Irix Compaq Tru64 FreeBSD

coolcommands.com is NOT

coolcommands.com is not an online database of man pages but rather a query tool to find commands or one-line scripts which provide a function in a UNIX environment.

coolcommands.com is cool because

We not only provide a searchable database of commands and one-line scripts, we also allow you to search by category making it easier to find that command you are looking for. Each coolcommand provides a summary, description and example to make your life as a sysadmin easier.

give it a try!

ssh authorized_keys

just now, one user complained that he can't use his public key as an authentication to remote server.
he already generate the public key & did all the necessary steps but still the remote server asked for password after key in the public key.

i tried used my id and it worked just fine, so it could be something wrong with his id or his public key. try to regenerate his public key but still the same thing.
as you see from below verbose message of ssh, it still asked for password after key in the public key:

user1@local_server $ ssh -v remote_server
OpenSSH_3.8.1p1, OpenSSL 0.9.7a Feb 19 2003
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to remote_server [ip_address] port 22.
debug1: Connection established.
debug1: identity file /glb/home/user1/.ssh/identity type -1
debug1: identity file /glb/home/user1/.ssh/id_rsa type 1
debug1: identity file /glb/home/user1/.ssh/id_dsa type -1
debug1: Remote protocol version 1.99, remote software version OpenSSH_3.8.1p1
debug1: match: OpenSSH_3.8.1p1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_3.8.1p1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-cbc hmac-md5 none
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192)>
Enter passphrase for key '/glb/home/user1/.ssh/id_dsa':
debug1: read PEM private key done: type RSA
debug1: Authentications that can continue: publickey,gssapi-with-mic,password,hostbased
debug1: Next authentication method: password
user1@remote_server's password:

a friend of mine (trew) found the solution... thanks :)
it is actually because his home directory is group writable (775). for security reason, authorized_keys file cannot be writable by anyone else, this also apply to the parent directory of authorized_keys file, in this case - his home directory.
for more info refer to http://dsd.lbl.gov/~ksb/Scratch/ssh_user_setup.html at the authorized_keys section.

Happy SysAdmin Day!

Happy SysAdmin Day for me & all sys admin around the globe!
hopefully your users & servers will let you rest today :P


Today, Friday, July 28th, 2006, is the 7th annual System Administrator Appreciation Day. On this special international day, give your System Administrator something that shows that you truly appreciate their hard work and dedication.
http://www.sysadminday.com/

random password generator

This is a very simple tool to generate a random password and almost available on all unixes.

$ cat /dev/urandom | strings -n 8 | head -5
?8L803^+}
JFn95:`_
RF+Q~8ff
]r'kZhVG
>2MJ,!]j

Just pick one as your password :)

smart way to mount NFS file system

sometimes NFS mount point can hang when you do a listing (ls -al) due to certain reasons such as server down, network problems etcetera.

to avoid this is it advisable to make the mount point down one level of root.

instead of:

# mount NFS_server:/data /data

in this case, if the mount point hang,

# ls -al /

would take a long time until the NFS mount timed out.

you should do:

# mount NFS_server:/data /new_directory/data

so if the mount point hang, you can still do listing of root.

# ls -al /

would work fine just that

# ls -al /new_directory

would fail.

bash history - ignore duplicate & commands

put these lines in your .bash_profile to ignore duplicate history or to ignore certain commands.
and it also keep your history short :)

# ignore duplicate
HISTCONTROL=ignoreboth

# ignore these commands in history file
HISTIGNORE=ls:ll:la:l:cd:pwd:exit:mc:su:df:clear

recover root password

previous post i mentioned about only hp-ux will ask old root password if you want to change the root password. in case you lost the root password, here is a list of a good compilation on how to recover the root password on various flavor of unixes:

Solaris
how do I log into this machine - with a twist... (http://www.unix.com/showthread.php?s=&threadid=1927 )
Lost password on a SPARCstation Voyager (http://www.unix.com/showthread.php?s=&threadid=13702 )
How do I restore the etc/shadow file (http://www.unix.com/showthread.php?s=&threadid=3895 )
Root Password Lost !!! (http://www.unix.com/showthread.php?s=&threadid=9488 )
root locked out (http://www.unix.com/showthread.php?s=&threadid=2492)
sc password (http://www.unix.com/showthread.php?t=22079)

HP-UX
root pwd (http://www.unix.com/showthread.php?s=&threadid=13882)
lost root password using (SAM) trusted security (http://www.unix.com/showthread.php?s=&threadid=2084 )
I lost my password root (http://www.unix.com/showthread.php?s=&threadid=12466 )
passwd file corrupted (http://www.unix.com/showthread.php?t=17175)
Boot interaction ( http://www.unix.com/showthread.php?t=22534 )
Re-enabling a locked root account on Trusted HP-UX (http://www.unix.com/showthread.php?t=25319 )

Linux
root and all but 2 users locked out (http://www.unix.com/showthread.php?s=&threadid=12419 )
root password (http://www.unix.com/showthread.php?s=&threadid=5224)
how to retrieve root paswd ( http://www.unix.com/showthread.php?t=1844)
Help! passwd file corrupted (http://www.unix.com/showthread.php?t=18908)

AIX
root account has been locked (http://www.unix.com/showthread.php?s=&threadid=13658 )
Boot in Single-Mode (http://www.unix.com/showthread.php?t=18760)

Tru64
How to Enable locked root account (http://www.unix.com/showthread.php?s=&threadid=8360 )

SCO
Fortgot root password SCO 5.0. (http://www.unix.com/showthread.php?t=20345)

Ultrix
I neede help!!! (http://www.unix.com/showthread.php?s=&threadid=3208)

NCR Unix
single user mode - user accounts passwords (http://www.unix.com/showthread.php?t=17435)

Windows/XP
Beginners Guides: Forgotten Passwords & Recovery Methods (http://www.unix.com/showthread.php?s=&threadid=16216 )

source: http://www.unix.com/showthread.php?t=14327

ooops, they got for windows as well :)
i think on linux is the easiest one, so make sure you don't lost it :)
or maybe it time to use unix password manager

change root password

i just notice only hp-ux will ask old password if you want to change the root password

$ uname -a
HP-UX hostname B.11.00 U 9000/800 146901507 unlimited-user license
$ passwd
Changing password for root
Old password:

very good security practise i must say.
the rest, linux, solaris & aix, you can simply change the root password to new one without asking the old password.

# uname -a
Linux hostname 2.4.21-32.0.1.nfswan2 #1 SMP Thu Aug 4 11:02:01 CEST 2005 x86_64 x86_64 x86_64 GNU/Linux
# passwd
Changing password for user root.
Enter new UNIX password:

# uname -a
SunOS hostname 5.8 Generic_108528-13 sun4u sparc SUNW,Ultra-2
# passwd
passwd: Changing password for root
New password:

# uname -a
AIX hostname 1 5 0045C2CA4C00
# passwd
Changing password for "root"
root's New password:

unix/linux load average

do you know what it mean by load average when you execute these commands:

$ uptime
8:00pm up 238 day(s), 6:06, 39 users, load average: 0.60, 0.27, 0.26

$ w
8:01pm up 238 day(s), 6:07, 39 users, load average: 0.61, 0.30, 0.27

$ top
20:03:19 up 11 days, 5:23, 37 users, load average: 0.24, 0.14, 0.08

$ cat /proc/loadavg
0.12 0.09 0.09 1/253 532

from the man page, really i also not sure since it only says:

DESCRIPTION
The uptime command prints the current time, the length of
time the system has been up, and the average number of jobs
in the run queue over the last 1, 5 and 15 minutes. It is,
essentially, the first line of a w(1) command.

but i've found these articles that nicely explained about this. read it here:
http://www.lifeaftercoffee.com/2006/03/13/unix-load-averages-explained/
http://nosheep.net/story/defining-unix-load-average/
http://www.teamquest.com/resources/gunther/display/5/index.htm

vi / vim graphical cheat sheet

In an earlier entry, i mentioned about top 10 tips using vi, since it hard to remember it all, i used to have this vi graphical cheat sheet print out on my cubicle for quick reference.


today, while browsing the internet i found another useful & colorful vi/vim graphical cheat sheet. this is a single page describing the full vi/vim input model, the function of all keys, and all major features. you can see it as a compressed vi/vim manual. cool eh :)

source: http://www.viemu.com/vi-vim-cheat-sheet.gif

upgrade ssh server remotely

i've been looking for the most efficient way to upgrade ssh server remotely. this task is quite risky since if the upgrade not going as expected you'll lost your access to the box, unless you have console server connected to it, otherwise you'll need to ask somebody on site to log in locally & start the ssh daemon.

so far i've found several ways to do it:

1. install normal way
i found this useful article on how to upgrade ssh server remotely here
quite straight forward. just that you need to kill the ssh daemon manually & be careful not to kill your current session.

2. using at command
you can set specific time using at command to start the ssh daemon after you've installed it.
or something like:

# /etc/init.d/sshd stop; make install; /etc/init.d/sshd start

but i must say that this is quite risky since you are not sure whether the daemon can start up without any problem after the new installation.
Read more...

3. use different port
some say you better use other port for the new ssh installation. if you can connect to the new ssh server using those port then you can revert it back to the normal port which is 22. low risk but extra steps.

4. enable the backdoor
enable the other remote access which is telnet. i think this is the most safest way to do it even though it is security risk to enable telnet but at least you can still access your box if the ssh kaput!

but how about you need to upgrade a ton of servers? which way you should go?
for me i'll go for no 4. write a script to do:
1) enable telnet & make sure you can access using telnet
2) install/upgrade ssh server & test the installation
with 1 condition, all your servers must have the same configuration. otherwise things gonna mess up pretty bad! good luck :)
or maybe i'll come up with the script later...

unix tip: find a character position in word

somebody asked on how to find a character position in a word/line on the unix.com forum

the question:
how to find a character position in a word/line?
eg.
Unix forum is the best site.

now in above line i want to find the character 'x' position
i.e 4

this is what i came up with:

$ echo "Unix forum is the best site." | grep -o "[-_a-zA-Z0-9\.]" | grep -n x
4:x

update:
one of the users has a very simple solution:

$ expr index "Unix forum is the best site." x
4

unix tip: sendmail - email not been sent out but queued

ever come accross with this problem? when you try to sent out email, the email can't be sent out but get queued instead!!!

# cat filename | /usr/sbin/sendmail -v yourname@domain.com
yourname@domain.com... queued

somehow, sendmail by default will check the system load average
before sent out the email. if the load average exceeds QueueLA (default 8),
it stops delivering the mail, if load average exceeds RefuseLA (default 12)
it stops accepting the mail. both of the settings are located in sendmail.cf

even you increased the value, sendmail still not works as its will give an error below.

# ps auxw | grep sendmail
root 6519 0.0 0.0 6252 2040 ? S 12:52 0:00 sendmail: rejecting connections on daemon MTA: load average: 77

Read more...

so you need to set the value higher, but i'm not so sure about this.
this is my system load average: quite high huh!!!

# uptime
3:01pm up 96 days, 23:09, 15 users, load average: 78.10, 77.86, 77.57

of course you can manually flush/sent out the emails from the queue by using command below:

# /usr/sbin/sendmail -OQueueLA=80 -q 0 -v

but in this case i suggest you take a look at your system load & if possible try to decrease the load.

it took me a few days to find out this thing since there was no error message about the load average, but if you're lucky you'll get this "sendmail: rejecting connections on daemon MTA: load average:" in your log files which make easier to google :)

update:
QueueLA default values is (8 * numproc) where numproc is the number of processors online (if that can be determined).
http://www.sendmail.org/m4/tweaking_config.html