However, though most of us use SSH daily (probably hundreds of times per day for TLF guys) the SSH client configuration options are curiously underused in most situations.   Normally kept in ~/.ssh/config along with your private and public keys, the myriad of configurable tweaks that can be made to SSH can help improve your efficiency in your day-to-day work.

To get you started, I present a few examples of options and settings I use on my own account.  Of course we encourage you to read the man page for ssh_config (usually ‘man 5 ssh_config’ on most Linuxes) and delve into your own customizations too!

ServerAliveInterval 90

What It Does:  Tells the SSH client when to consider the connection “dead” after not receiving any data from the server.  When a “dead” connection is determined, the client will disconnect from the server   The value is in seconds.

Why I Use It:  Though there is also a separate configuration for TCPKeepAlive, I’ve found using this SSH-level server message keepalive in conjunction helps keep connections healthy when things blink.   Example:  I can usually upgrade SSHD and restart the daemon remotely, and still remain connected.   Near magic!

StrictHostKeyChecking no

What It Does: Sets the behavior of SSH when connecting to a host with new or changed host keys.

Why I Use It: Though the setting sounds insecure, it isn’t.  There is no way to tell SSH to completely disable host keys since that would inherently make it insecure.  Rather, a ‘no’ value here tells SSH to automatically add new host keys to ~/.ssh/known_hosts, skipping  the yes or no prompt which you’re forced to answer.   Existing keys in known_hosts that have changed will continue to prompt, which is the primary reason of maintaining that file in the first place.  Minor time and annoyance saver.

ControlMaster no

What It Does: Modifies the behavior of connection re-use by the SSH server and client.

Why I Use It: By default, SSH will re-use a single control connection for multiple SSH sessions to the same server.   Bet you didn’t know that!   Try opening multiple terminals and connect to the same server with them.  You should only see one socket used if a ‘netstat -an’ is run.    While that’s a great idea and saves resources, in some cases (for sysadmin purposes), I want it to run in several separate network connections.   This is purely a way for me to eliminate SSH as a problem in some troubleshooting situations.

NumberOfPasswordPrompts 5

What It Does: Controls how many password attempts are allowed before SSH dumps you back to a shell prompt in failure.  Defaults to 3.

Why I Use It: Some days are better typing days than others.  I’ll be the first to admit that my worse typing days outnumber my good ones.  For years I thought the SSHD daemon controlled the password attempt count, so imagine my chagrin when I found out I could simply set a value and have it ask me a few more times instead!

There you have it!  A few selections from my own ~/.ssh/config file.    It doesn’t end here!  There are methods to customize settings for a certain host or subset of hosts, set custom user names for specific servers so you don’t have to remember them,  customize which local environment variables get replicated to the destination server and more.

Over the years, OpenSSH has proven itself to be an invaluable tool.   Make the most of it!

As any regular Unix user can tell you, there are three primary “streams” in which programs are fed and output data.    STDIN, which by default is the keyboard.   STDOUT, which by default is your terminal/screen, and STDERR, which by default is also your terminal/screen.   In addition to their textual reference, these three also have file descriptor numbers assigned (you’ll find out why in a moment).  Good Unix shell users regularly take full advantage of all of these.   In fact, if you’ve done any Unix or Linux work you’ve likely used them without even knowing.   For example.

cat ~/textfile.txt | more

You’ve certainly used the pipe (‘|’) character before, but do you know what it is actually doing behind-the-scenes?   Effectively, the pipe tells the shell to “stitch” the STDOUT from the ‘cat’ command to STDIN of the ‘more’ command.    In Unix-land this is referred to as creating a pipeline.

Master of redirection

In magician terms, redirection means to draw away the focus of the audience so something sneaky and magical can be occurring elsewhere.   While there’s very little sneaky going on in the Unix shell, when used correctly STDIN and STDOUT redirection can be pretty magical.   For instance:

/usr/bin/somepossiblybrokenprogram.sh  1> /var/log/myapp.log  2> /var/log/myapp-error.log

Stiches the “regular” output of your possiblybrokenprogram to STDIN of a log file, while the errors go to STDIN of a log file aptly named myapp-error.log.    Remember I mentioned the file descriptor numbers?   Now you know what they’re for, and here’s how they’re assigned:

STDIN  - 0
STDOUT - 1
STDERR - 2

Now you should be able to see how we did that bit of “magic” above.   Redirection of output can be darn handy when debugging or extremely useful when running things as scheduled (cron) jobs.   Also note: by default the ’1′ when redirecting STDOUT is unnecessary, since using a bare “>” assumes you mean STDOUT.  We specified it above for clarity’s sake, and you’re welcome to do it as well.  Either way works.

There is a downside to this however–you can only redirect any given file descriptor once per command. …but what if you want to log things to a file and see it on the screen at the same time?  Impossible right?   NAY!

Line up to the ‘tee’

That’s exactly where the ‘tee’ utility comes into play.    Don’t think of ‘tee’ as in a golf tee, but rather like a ‘tee’ in plumbing terms.  In other words, it takes the stream and splits it in two, three, or even more:    STDOUT and STDIN simultaneously.  Even STDOUT and STDIN multiple times.   I know you’re already scratching your head, so how about a few examples.  First the simplest:

/usr/bin/somepossiblybrokenprogram.sh | tee /var/log/myapp.log

Hopefully you see what’s going on here.   Instead of wholly redirecting STDOUT to a single file, we’ve instead utilized the ‘tee’ command to split it to both the screen and a file named /var/log/myapp.log. Pretty handy huh?   But wait, there’s more!

/usr/bin/somepossiblybrokenprogram.sh | tee /var/log/myapp.log /var/log/other.log /var/log/joe.sh.log

You see what it’s doing?  Not only is it displaying to the screen, but it simultaneously wrote the output to three separate log files!

Finally the ‘tee’ command has an option to append to files instead of overwriting them each time it is invoked–which would be a bummer if you were debugging something.   You do that like so:

/usr/bin/somepossiblybrokenprogram.sh | tee -a /var/log/myapp.log /var/log/other.log /var/log/joe.sh.log

There you have it!  The very tiny, but immensely useful ‘tee’ command.

Learning Linux Commands is part of a blog series that highlights useful Linux/Unix commands for our web hosting clients.  Keep our blog RSS feed refreshed for new entries–we’ll be adding many more soon!  

So most people these days have heard of  .com.   You’ve probably used  .org and .net as well.   But have you heard of  .info, .mobi or .tv yet?   If not, you will!  All of these are valid “top-level” domains (TLD) gaining in popularity and I’d like to provide a quick history lesson on how these generic codes came to be.

A long time ago when the Internet was barely more than a U.S. Department of Defense research project, only a single TLD called .arpa existed.    ARPA stands for “Advanced Research Projects Agency“, or as they’re more commonly known: “the group of smart guys that figured out how to build the Internet”.   This aptly-named .arpa TLD was used to migrate the first domain names off the old non-hierachical ARPANET to the shiny new Internet.   Luckily, new TLDs were eventually introduced, because “ebay.arpa” and “google.arpa” just don’t have the same ring to them.  Nonetheless .arpa is still in use today by computer network geeks like us even though we changed what it stands for (address routing & parameter area—told you it was geeky!)

Throughout the 1970′s and 80′s, the responsibility of managing the new Internet’s domain names fell loosely to university labs in California, the U.S. Military, and eventually a company called Network Solutions.   During this tumultuous time, the standards for requesting new TLDs and keep track of who was using what was a bit shoddy, as valid TLDs were loosely grouped into three categories:  Countries, Categories, and Multiorganizations.   Not surprisingly things got messy.  For instance, in the early 1980′s, NATO was upset that there wasn’t a sufficiently international-themed TLD for their organization, so .nato was created.   In fact during the 1970′s – 1980′s, the various managing organizations were creating TLD’s that weren’t so generic.   Can you imagine having a “.ibm” and “.yahoo” and so on today?   Clearly there needed to be a standard solution as commercial interest in the Internet was growing.

In order to fix the problem the U.S. National Science Foundation decided to hold a bidding competition in the early 1990′s for three different aspects of  managing domain name data:

• Registration services, so people could “sign up” for domains in a standard way
• Information services, so there was an organized method to know who owned what
• Directory and Database services, so there was a way to store and look up that data.

The contract was awarded to Network Solutions, General Atomics, and AT&T respectively  and the collaberative organization known as InterNIC was born.  This was a huge leap for today’s Internet, as it made it much easier for people and companies to “get online” with their own unique domain name since things were more organized under InterNIC.   Many old-timers  (I’m that old) remember a time when you had no choice in registering a domain.  Everyone had to use Network Solutions!

Finally in 1998, a few folks at AT&T forgot to look at the expiration dates on their contracts and AT&T bowed out of managing their piece of the puzzle.   This was a big turning point because the U.S. Government also wasn’t very interested in managing the Internet any longer.  So instead of the U.S. Government taking on the job on directly,  InterNIC was folded under a new non-profit company called ICANN.   Contracted by the U.S Commerce Department, ICANN decides things like what new TLDs are deemed worthy.   This is currently how it stands today, and InterNIC still provides that service as a subsidiary of ICANN.

Also under the management of ICANN, another big change occurred.   Now any organization with sufficient worthiness (and many papers signed) could register domain names under the existing and established TLDs.    This opened the doors for places like The Linux Fix to register domain names on behalf of their customers and make the whole process much easier for the average person.   Obviously this has had a sweeping effect on the Internet by putting a globally-accessible domain name within reach of anyone.

And there you have it!   The brief and amazing history of the Internet domain name.   Oh, and perhaps you’re now wondering what all the valid TLDs are?  Here you go!

Much of PHP’s success can be attributed to it being free and open, able to run on nearly every operating system, and being easy to use and learn.   Yet the same things that have made it so successful in the open source community have historically hindered its commercial success.   But why?

Quite simply PHP is a scripted language.   Unlike C programs which need pre-compilation to run, PHP is compiled and executed on the fly by the PHP engine.  Perl, JavaScript, and Shell script all execute in a similar manner.  This makes PHP very flexible and friendly to open source and hobbyist developers, but it essentially requires you to “give away” your source code in order to distribute your program.

Of course commercial developers attempting to make a profit on their work won’t  want to give away their code, not if they expect to make money! But the PHP market was huge and lucrative, so unsurprisingly a company stepped in to offer a solution.  That solution is IonCube, and here’s how it works.

First, the commercial PHP developer will write their application.   When it’s finished, the developer will use the IonCube program to encode the raw, human-readable PHP source code in a proprietary binary format.  This protects the underlying source code from prying eyes by turning it into a jumble of unreadable goobleygook.

Next,  the application is purchased by an end user.  After the purchase, the application developer provides the customer with a special key which is used to “unlock” the encoded source.

Finally, the customer installs the IonCube Loader on their web server, which acts as a on-the-fly decoder for the coded application.    At no time is the user ever able to view the PHP source code:  all the decoding happens inside the web server via the IonCube loader.   This keeps the application code a secret for the developer, as well as allowing fine control over the licensing of their application via the unlock key.

Though IonCube had a rocky start at its debut in 2002 (thanks somewhat to vocal opposition by strong open source proponents), it has slowly become the de-facto standard for distributing non-free PHP applications.   Though not all web hosts support it, The Linux Fix offers the IonCube Loader with every web hosting plan we provide.  We hope that it gives you the freedom of using whatever PHP application you choose, without much fuss!

Nevertheless, when it comes down to scripting and programming, I try to take my time.   When I’m in the process of writing code, especially when that particular bit of code may involve external processing (say reading from a URI), I like to think out all the possible ways it could go wrong.   Not only is this good programming practice for debugging purposes, but writing it in such a way allows you to present very relevant and accurate error messages to your users.

Now, let me explain why this is important.

Earlier this week I was offering help to my housing overlords with a computer problem they happen to use for their point-of-sale system.   Whenever they tried to print a receipt on their printer, an error message would pop up stating “Insufficient disk space to print”.

The computer had 300GB of free disk space.

To make a long story short, the program lied.   The problem was that “C:\windows\temp” had the read-only attribute turned on.   The disk wasn’t full at all, but since the printer driver program couldn’t write to that path, it assumed the drive was full.  We all know what “assume”-ing makes us…

One silly coding choice by a programmer burned several hours of my life away.   If the programmer had simply chosen to test a few conditions before attempting to write the file, namely:

• Does the directory exist?
• What are the modes on the directory?
• Is there a file named the same as I’m about to write?
• … and so on

If you’re looking to virtualize a server of your own or to pick up a VPS, you’ll probably be picking between these two staples of the virtualized world.  Each technology has its pros and cons, and your decision may change based on your needs as a sysadmin or a user (Spoilers: We chose OpenVZ).  Xen and OpenVZ differ in how they virtualize servers, manage memory, and more!  I’ll try to give a little more in-depth analysis of the differences to help you decide which method is right for you.

The first big difference comes right down to fundamental design goals.  They are different down to the core!  Neither system is full virtualization (virtualized bios, hardware, os, everything). Xen is a hypervisor (or paravirtualization), OpenVZ is container (or OS level) virtualization.  The simple explanation is that a Xen hypervisor runs an OS that knows it’s being virtualized.  Each user is seperated and running their own OS/system, all on top of the master virtual controller (or Hypervisor).  OpenVZ uses the same underlying OS for each of the users.  The containers are all isolated and separated from each other, but using the same underlying OS.  It creates a kind of OS level ‘jail’.  If you’re unfamiliar with these concepts, don’t worry, it’s not as bad as it sounds!

This intro is getting wordy.  Time to break out some headers!

Virtualization (The Underbelly):

Xen’s hypervisor basically sits over the hardware, and acts like a hardware virtual manager.  The ‘guest systems’ that run on the hypervisor know they’re on a virtualized system, but are still able to function as wholly separate entities.  This means you can run just about any OS or setup regardless of what the base level system is running.  This lets you do fun stuff like running a Windows VM on a Linux box, or run a different kernel in each guest.

OpenVZ containers don’t try to virtualize the hardware.  Instead, it uses a special kernel to isolate processes and resources.   Every guest runs on top of the same identical system, but they are fully separated from each other (still have their own filesystems/processes).  Everyone has the same kernel, and you can’t run different OS types on the same machine (can’t support both Linux and Windows VMs).

From this limited information, Xen is clearly more versatile.  But wait, I don’t use Xen.  ‘Why not?’, you ask in a haughty voice.   I’m getting to it.

Overhead (Isolation vs Ease of Use):

To keep things simple (and relevant), I’ll just be talking about 64-bit systems.  Xen does some pretty complicated stuff to make sure x86 architecture works just the way they wanted it to, and it’s impressive enough, but it adds more overhead to the process.  Not wanting to hold that against them, I’ll keep my discussions relevant to x86_64!  Oh, you only care about 64-bit architecture?  Good on you, mate!

Here personal preference is king.  Xen is much more virtualized, which comes with some nice pros to its cons.   It isolates its resources for you (more on this later).  This also allows custom/different kernels, since each guest is basically a wholly different environment.

But there is a cost in doing all of this.  Xen uses significantly more resources in overhead to create/maintain seperate instances for each VM.  OS level virtualization means many basic components exist once on the machine, and are used by all guests (like the identical kernel).  OpenVZ also uses one filesystem with chrooted environments, instead of fully separated Xen file systems.

If you’ll need a system with a custom kernel or want to load kernel modules,  stop reading now.  That decision pretty much makes itself, you’re a Xen fanboy now.  Get your hat on your way out of the lecture.

But with a single kernel, and a more manageable file system making backup/recovery easier, I find OpenVZ more manageable as a sysadmin.

Memory Management (The Gamble):

Here is the real meat and potatoes of this debate.  Xen’s dedicated memory and swap space, versus OpenVZ’s density and burstable memory.

It comes down to Xen’s swap space use versus OpenVZ killing processes.

With Xen’s isolation, you get your own section of memory.  It is quartered off for you, and nobody else has write privileges to it.  If you max out your memory, it’ll dump over into swap space so nothing crashes.  The problem on a shared server is that even though disk filesystems and swap spaces are isolated, disk I/O is not.  When people start bursting into swap space, it can cause the exact problem that Xen was intended to isolate around; bad VMs affecting the performance of other VMs.  Sure, your Apache will keep on kicking, but it’s going to run terribly, and may cause problems for the rest of the machine.

With OpenVZ, you are given a guaranteed amount of memory.  Since it’s not fully isolated, it’s a part of shared memory on the server.  Everyone has their guaranteed memory in the same pool.  There’s also (typically) burstable memory.  If memory isn’t being utilized, others who need it for a short time can burst up into extra memory.  If guaranteed memory is required somewhere else, this optional burst memory will fail away, and the memory will be allocated to fill a guaranteed quota.  But if you suddenly have no memory where memory once was, things die.  The problem is, they sometimes don’t die nicely.   No swap space with OpenVZ, so things die, and you don’t get a lot of control over what dies.

So we have a stability versus performance issue!  Or do we….

Conclusions (tl;dr):

I like OpenVZ.  I like the low overhead on containers and the performance is great.  The problem with OpenVZ is over-allocating memory.  You may notice on our VPS plan descriptions that we don’t talk about burstable memory.  That’s because we don’t allocate burstable memory.  We take a stability approach to our VPS.  OpenVZ won’t allocate memory it doesn’t have.  It’ll throw an error and stop new processes from spawning, but you will know about the problem before it accidently kills off Apache or MySQL.  It won’t bring down the whole house of cards, nor will it put the burden on everyone.

As an admin, I like using OpenVZ for it’s natural performance benefits, and managing memory with a more Xen approach.  It’s served me, and our customers, very well.

As a user, I suggest two good options.

• Use Xen because you need it (custom kernel, different OS, etc)
• Find an OpenVZ vendor who doesn’t oversell their resources (I suggest The Linux Fix!)

DNS: Not just for network nerds any more!

As recently as ten years ago, the inner workings of the Internet remained a mystery for all but the most foolhardy of people.    Fast forward today, and almost everyone has a mobile phone with Internet connectivity, a blog, a Facebook page, and maybe a dedicated website for themselves and family.

Along with any new technology comes the frustration when something doesn’t work right.  The Internet has become prevalent in nearly everything we do (who watches the Weather Channel now?  Anyone?) and is, in fact, a part of everyday life.   So then, a general understanding of “how things work” isn’t such a bad idea.   After all, you probably know how to check the oil on your car or truck–certainly fill it up with gas–even though you may not understand how the rest of the engine works.

Getting Started with DiG

Being that DNS is involved in nearly every bit of communication on the Internet, it’s a great place to start troubleshooting.   If a web site doesn’t come up, is it because that server is down?  Or is your DNS acting up…maybe theirs?   Well read on!  We’ll show you how to find out.

The primary tool to use in troubleshooting DNS is called DiG, short for the “DNS Information Groper” (seriously).  It is part of the BIND DNS Server software package.   Other than the slightly wonky name, its use as an investigative DNS tool is invaluable.   If you are using an Apple Mac or some type of Linux/Unix variant, DiG is likely already installed (on a Mac, open the Terminal to use it).   If you’re on a Windows host, you can download DiG for Windows, which is roughly the same tool compiled for the Microsoft platform.

Lets fire up dig and perform a quick query to determine if it’s working.   Open up your terminal or command line box, and type the following:

dig google.com. NS

After a very short period of time, you should get results which look similar to the following:

[bdowney@tlfmgt1 ~]$ dig google.com NS

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5 <<>> google.com NS
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52302
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;google.com.            IN    NS

;; ANSWER SECTION:
google.com.        172781    IN    NS    ns2.google.com.
google.com.        172781    IN    NS    ns1.google.com.
google.com.        172781    IN    NS    ns3.google.com.
google.com.        172781    IN    NS    ns4.google.com.

;; Query time: 0 msec
;; SERVER: 172.16.65.90#53(172.16.65.90)
;; WHEN: Tue Sep 13 12:48:38 2011
;; MSG SIZE  rcvd: 100

So what exactly is this telling us?   First, the header shows the DiG version, as well as some basic information.   Above, the header  shows that we performed a Query, there was No Error, as well as your query’s ID.   Furthermore, it shows the flags passed to DiG when it was run (the ones printed are the default),  and finally the enumeration of queries and results returned.   In this case, we sent one query: NS, (“Give me the DNS servers for google.com”) and it returned four distinct answers  (ns1, 2, 3, and 4.google.com).  Finally the bottom portion shows how long the query took (pretty fast in our case!) as well as the DNS server that it obtained the information from.

In our next post, we’ll show you how to use dig to look up different types of DNS “records”, some of which will let you even troubleshoot email problems.

CentOS 5 supports PHP 5.1.6, and will not ever officially support the newer versions of PHP.  As stated in the CentOS wiki “As with each other program in CentOS, the version numbers of released software will not change over the life time of a CentOS product.”  So in short, the yum repos we all know and love have no intention of including the new PHP software.

But what can we do?  Well, we’re Linux users.  We can do whatever we want!

Here at The Linux Fix we have a talented bloke named Greg who has worked on this problem in the past when he created a yum repo for PHP 5.3.5 for use with CentOS.  Well, what’s changed since then?  Not much actually!  In fact, most of the info here will just be the same helpful tips he wrote back in February.  Except now it’s with an updated repo for 5.3.6, with additional information for TLF clients!  (oh look, he beat me to this post!)

So first things first, you’ll need to add our TLF branded repo to your yum repository list.  We keep all that tidy PHP goodness over here at yum.thelinuxfix.com (with appropriate rpms for 32 and 64-bit systems).

Install Instructions:

As root run:

64-bit:
rpm -Uvh http://yum.thelinuxfix.com/RHEL/5/x86_64/tlf-release-1-2.noarch.rpm

32-bit:
rpm -Uvh http://yum.thelinuxfix.com/RHEL/5/i386/tlf-release-1-2.noarch.rpm

This will add our rpm repository to your yum list of approved repos!  We’re all excited about it.
So what’s left?  Updating(or installing) PHP of course!  You have a few options here.  If you’ve already got PHP installed (probably 5.1.6) then a simple

 yum update php

will do. This should nail the basics, like php; php-common; php-cli; (php and it’s immediate dependancies), as well as any additional php modules you may be using (like php-pdo; php-odbc; php-mysql; etc). This won’t always catch everything, so be thorough in making sure you update everything you need (case in point: This won’t catch php-pear, which is available in our repo).

zVPS™ Users:

But wait! You’re a customer using one of our zVPS™ Virtual Servers? Well, I’ve got good news!

As you (hopefully) already know, we have pre-designed single button install templates for many useful purposes.  One of these templates (VPS Template Web Server) has all the CentOS 5 latest rpms for php and a number of relevant mods.  And since I like you guys so much (almost as much as I like parenthesis), I wrote you a short little gift.

As root run:

wget -O - http://www.thelinuxfix.com/script/tlfphp53.sh|bash

If you’ve already installed our web server template, this will automatically set up and install the tlf-repo and update php and it’s friends to 5.3.6!

–Steven

As the conversation continued, he had mentioned how the web hosting industry was now a commodity, and they were smashing all kinds of record profits by offering super cheap hosting and doing huge volume.   As he was talking, I kept glancing over his shoulder at their equipment:   Clone PC’s running hundreds of websites each,  $40 Linksys switches,  OfficeMax power strips, and twist ties.   Lots and lots of twist ties.

We bid farewall to each other and I made mental notes of what I found interesting:

• He didn’t mention his customers once.
• He took pride in how cheap their infrastructure was.   In fact, he made a joke about a noisy fan that had been failing for six months.
• His focus on their profits dominated the discussion

Though he seemed like a great guy I wouldn’t want to be his customer.   He may be offering something for a low price point, but he’s doing so at the expense of service and quality.    The Linux Fix has always believed that with great customer service comes success.   Our web hosting customers can’t succeed if we don’t provide quality equipment and service.   We want to be known for our service and support, not for our profit margin.

In fact we believe if we take care of our customers and clients, the business will take care of itself.   So far it has worked pretty well!

This shouldn’t effect any of your services, and any personal domain names you may have pointing to your dedicated server remain unchanged.