Things to make sysadmin work easier

Thought I would put together a list of things time saving tips and best practices that can make Linux system administration easier. The list isnt in any particular order. Just put it down as it came to mind.

 

Symlinks

Think before you use them. If the application has a  configuration directive for a different file location use that. Example don’t symlink  /var/lib/mysql/. Use the datadir option in mysql to move it. I think the problem is most admins find out about symlink and think its a band aid for everything. Symlink isn’t a band aid its a tool make sure you dont have a better way of doing it before you create a symlink jungle.

Logs

Dont have multiple logs logging the same stuff in different formats. I see this with Apache. You don’t need 6 logs for each site. You should have 4 logs per vhost max and you should log duplicate data in them.  One log per vhost is probably all you really need.

Configuration management tool

Puppet is your best friend. It has a steep learning curve but if you learn puppet in and out it will save you many weeks of work over the years.

At jobs

I’m guilty of not using this one enough. If you need to run a job one time at 1 am at could help you get a full night of sleep.  Dont use cron for this because if you forget its going to run again. If you use cron you have to touch that command twice where as with at its a fire and forget.

Bash loops

Don’t be afraid of the loop. One I use a lot is the for loop. It can look something like this for X in $(seq 7); do dig -x 127.0.0.$X +short; done Another handy one is the while loop.  while [ $(date +%k) != “20” ]; do echo “something”; sleep 5 ; done this one will run till the hour is 20

Bashrc functions and aliases

If you need to run it more than once on demand but don’t want it to be its own script add it to .bashrc as a function or alias. I also use .bashrc for some of my common typos like alias shs=’ssh’

Bash, VI and Termanal shortcuts

Learn shortcuts like Shift+Ctrl+c and Shift+Ctrl+v for copy and paste in gnome term. For some vi shortcuts check out D&C GNU/Linux User Group and http://www.keyxl.com.

Adding this to  your .bash_profile will allow you to search your bash history with the up and down arrows

bind '"\e[A"':history-search-backward
bind '"\e[B"':history-search-forward

Bash completion

Get tab happy with bash completion and stop typing everything out. Bash completion is easy to install and is part of the base install on most distributions.

Scripting

If you need to do it more than once script it.

Monitoring

Monitoring shouldn’t only monitor it should help solve problems. . If you find yourself doing X after you get a alert then use the monitoring system to automate X. It will save you some sleep and reduce downtime.  A good example of this is have the monitoring system check the java logs for out of memory errors. If the monitoring system finds that error it restarts the application and send a email that it fixed it.  If you do this I suggest you put in some thresholds so the monitoring system will only restart the application every X number of hours.

RPL

RPL is one of those have to have installed system tools. Its like sed but it can do so much more. RPL is as easy as  $rpl “old stuff” “new stuff” file. RPL also will recurse into subdirectories and has a ton of options including ignore case.

Don’t hoard disk

Some administrators like to keep disk in reserve with a volume manager. I dont see the point unless you think its job security because you look important adding that extra 5GB the space when the application runs out of the original 10GB you gave it even though you know it will one day need 100G and you have 500Gb hoarded. If you have 500GB of local disk present it all to the application, OS and users. Ext might even perform better having a little extra space to do its thing.

Clusterssh

You might find this useful. I used clusterssh a lot in the past but I now use puppet in its place.

DU

Du is the best way to find out whats using all the disk space. A good example of its usage is $du -h –max-depth=1 / That will give you the directory size of each directory in /.

 

 

I hope this gets you thinking about ways to save time not only now but in the future. I think this list is a good start. Something to be conscious of as you’re making changes is to think about how hard will this be to trouble shoot and what problems could this cause.  If I could only have one tool from the above list it would be puppet. With puppet you can accomplish alot of the task the other tools do and have it replace any day to day task you currently have.

Leave a Reply

Your email address will not be published. Required fields are marked *

*


*