As any regular Unix user can tell you, there are three primary “streams” in which programs are fed and output data.    STDIN, which by default is the keyboard.   STDOUT, which by default is your terminal/screen, and STDERR, which by default is also your terminal/screen.   In addition to their textual reference, these three also have file descriptor numbers assigned (you’ll find out why in a moment).  Good Unix shell users regularly take full advantage of all of these.   In fact, if you’ve done any Unix or Linux work you’ve likely used them without even knowing.   For example.

cat ~/textfile.txt | more

You’ve certainly used the pipe (‘|’) character before, but do you know what it is actually doing behind-the-scenes?   Effectively, the pipe tells the shell to “stitch” the STDOUT from the ‘cat’ command to STDIN of the ‘more’ command.    In Unix-land this is referred to as creating a pipeline.

Master of redirection

In magician terms, redirection means to draw away the focus of the audience so something sneaky and magical can be occurring elsewhere.   While there’s very little sneaky going on in the Unix shell, when used correctly STDIN and STDOUT redirection can be pretty magical.   For instance:

/usr/bin/somepossiblybrokenprogram.sh  1> /var/log/myapp.log  2> /var/log/myapp-error.log

Stiches the “regular” output of your possiblybrokenprogram to STDIN of a log file, while the errors go to STDIN of a log file aptly named myapp-error.log.    Remember I mentioned the file descriptor numbers?   Now you know what they’re for, and here’s how they’re assigned:

STDIN  - 0
STDOUT - 1
STDERR - 2

Now you should be able to see how we did that bit of “magic” above.   Redirection of output can be darn handy when debugging or extremely useful when running things as scheduled (cron) jobs.   Also note: by default the ’1′ when redirecting STDOUT is unnecessary, since using a bare “>” assumes you mean STDOUT.  We specified it above for clarity’s sake, and you’re welcome to do it as well.  Either way works.

There is a downside to this however–you can only redirect any given file descriptor once per command. …but what if you want to log things to a file and see it on the screen at the same time?  Impossible right?   NAY!

Line up to the ‘tee’

That’s exactly where the ‘tee’ utility comes into play.    Don’t think of ‘tee’ as in a golf tee, but rather like a ‘tee’ in plumbing terms.  In other words, it takes the stream and splits it in two, three, or even more:    STDOUT and STDIN simultaneously.  Even STDOUT and STDIN multiple times.   I know you’re already scratching your head, so how about a few examples.  First the simplest:

/usr/bin/somepossiblybrokenprogram.sh | tee /var/log/myapp.log

Hopefully you see what’s going on here.   Instead of wholly redirecting STDOUT to a single file, we’ve instead utilized the ‘tee’ command to split it to both the screen and a file named /var/log/myapp.log. Pretty handy huh?   But wait, there’s more!

/usr/bin/somepossiblybrokenprogram.sh | tee /var/log/myapp.log /var/log/other.log /var/log/joe.sh.log

You see what it’s doing?  Not only is it displaying to the screen, but it simultaneously wrote the output to three separate log files!

Finally the ‘tee’ command has an option to append to files instead of overwriting them each time it is invoked–which would be a bummer if you were debugging something.   You do that like so:

/usr/bin/somepossiblybrokenprogram.sh | tee -a /var/log/myapp.log /var/log/other.log /var/log/joe.sh.log

There you have it!  The very tiny, but immensely useful ‘tee’ command.

Learning Linux Commands is part of a blog series that highlights useful Linux/Unix commands for our web hosting clients.  Keep our blog RSS feed refreshed for new entries–we’ll be adding many more soon!  

So most people these days have heard of  .com.   You’ve probably used  .org and .net as well.   But have you heard of  .info, .mobi or .tv yet?   If not, you will!  All of these are valid “top-level” domains (TLD) gaining in popularity and I’d like to provide a quick history lesson on how these generic codes came to be.

A long time ago when the Internet was barely more than a U.S. Department of Defense research project, only a single TLD called .arpa existed.    ARPA stands for “Advanced Research Projects Agency“, or as they’re more commonly known: “the group of smart guys that figured out how to build the Internet”.   This aptly-named .arpa TLD was used to migrate the first domain names off the old non-hierachical ARPANET to the shiny new Internet.   Luckily, new TLDs were eventually introduced, because “ebay.arpa” and “google.arpa” just don’t have the same ring to them.  Nonetheless .arpa is still in use today by computer network geeks like us even though we changed what it stands for (address routing & parameter area—told you it was geeky!)

Throughout the 1970′s and 80′s, the responsibility of managing the new Internet’s domain names fell loosely to university labs in California, the U.S. Military, and eventually a company called Network Solutions.   During this tumultuous time, the standards for requesting new TLDs and keep track of who was using what was a bit shoddy, as valid TLDs were loosely grouped into three categories:  Countries, Categories, and Multiorganizations.   Not surprisingly things got messy.  For instance, in the early 1980′s, NATO was upset that there wasn’t a sufficiently international-themed TLD for their organization, so .nato was created.   In fact during the 1970′s – 1980′s, the various managing organizations were creating TLD’s that weren’t so generic.   Can you imagine having a “.ibm” and “.yahoo” and so on today?   Clearly there needed to be a standard solution as commercial interest in the Internet was growing.

In order to fix the problem the U.S. National Science Foundation decided to hold a bidding competition in the early 1990′s for three different aspects of  managing domain name data:

• Registration services, so people could “sign up” for domains in a standard way
• Information services, so there was an organized method to know who owned what
• Directory and Database services, so there was a way to store and look up that data.

The contract was awarded to Network Solutions, General Atomics, and AT&T respectively  and the collaberative organization known as InterNIC was born.  This was a huge leap for today’s Internet, as it made it much easier for people and companies to “get online” with their own unique domain name since things were more organized under InterNIC.   Many old-timers  (I’m that old) remember a time when you had no choice in registering a domain.  Everyone had to use Network Solutions!

Finally in 1998, a few folks at AT&T forgot to look at the expiration dates on their contracts and AT&T bowed out of managing their piece of the puzzle.   This was a big turning point because the U.S. Government also wasn’t very interested in managing the Internet any longer.  So instead of the U.S. Government taking on the job on directly,  InterNIC was folded under a new non-profit company called ICANN.   Contracted by the U.S Commerce Department, ICANN decides things like what new TLDs are deemed worthy.   This is currently how it stands today, and InterNIC still provides that service as a subsidiary of ICANN.

Also under the management of ICANN, another big change occurred.   Now any organization with sufficient worthiness (and many papers signed) could register domain names under the existing and established TLDs.    This opened the doors for places like The Linux Fix to register domain names on behalf of their customers and make the whole process much easier for the average person.   Obviously this has had a sweeping effect on the Internet by putting a globally-accessible domain name within reach of anyone.

And there you have it!   The brief and amazing history of the Internet domain name.   Oh, and perhaps you’re now wondering what all the valid TLDs are?  Here you go!

Much of PHP’s success can be attributed to it being free and open, able to run on nearly every operating system, and being easy to use and learn.   Yet the same things that have made it so successful in the open source community have historically hindered its commercial success.   But why?

Quite simply PHP is a scripted language.   Unlike C programs which need pre-compilation to run, PHP is compiled and executed on the fly by the PHP engine.  Perl, JavaScript, and Shell script all execute in a similar manner.  This makes PHP very flexible and friendly to open source and hobbyist developers, but it essentially requires you to “give away” your source code in order to distribute your program.

Of course commercial developers attempting to make a profit on their work won’t  want to give away their code, not if they expect to make money! But the PHP market was huge and lucrative, so unsurprisingly a company stepped in to offer a solution.  That solution is IonCube, and here’s how it works.

First, the commercial PHP developer will write their application.   When it’s finished, the developer will use the IonCube program to encode the raw, human-readable PHP source code in a proprietary binary format.  This protects the underlying source code from prying eyes by turning it into a jumble of unreadable goobleygook.

Next,  the application is purchased by an end user.  After the purchase, the application developer provides the customer with a special key which is used to “unlock” the encoded source.

Finally, the customer installs the IonCube Loader on their web server, which acts as a on-the-fly decoder for the coded application.    At no time is the user ever able to view the PHP source code:  all the decoding happens inside the web server via the IonCube loader.   This keeps the application code a secret for the developer, as well as allowing fine control over the licensing of their application via the unlock key.

Though IonCube had a rocky start at its debut in 2002 (thanks somewhat to vocal opposition by strong open source proponents), it has slowly become the de-facto standard for distributing non-free PHP applications.   Though not all web hosts support it, The Linux Fix offers the IonCube Loader with every web hosting plan we provide.  We hope that it gives you the freedom of using whatever PHP application you choose, without much fuss!

If you’re looking to virtualize a server of your own or to pick up a VPS, you’ll probably be picking between these two staples of the virtualized world.  Each technology has its pros and cons, and your decision may change based on your needs as a sysadmin or a user (Spoilers: We chose OpenVZ).  Xen and OpenVZ differ in how they virtualize servers, manage memory, and more!  I’ll try to give a little more in-depth analysis of the differences to help you decide which method is right for you.

The first big difference comes right down to fundamental design goals.  They are different down to the core!  Neither system is full virtualization (virtualized bios, hardware, os, everything). Xen is a hypervisor (or paravirtualization), OpenVZ is container (or OS level) virtualization.  The simple explanation is that a Xen hypervisor runs an OS that knows it’s being virtualized.  Each user is seperated and running their own OS/system, all on top of the master virtual controller (or Hypervisor).  OpenVZ uses the same underlying OS for each of the users.  The containers are all isolated and separated from each other, but using the same underlying OS.  It creates a kind of OS level ‘jail’.  If you’re unfamiliar with these concepts, don’t worry, it’s not as bad as it sounds!

This intro is getting wordy.  Time to break out some headers!

Virtualization (The Underbelly):

Xen’s hypervisor basically sits over the hardware, and acts like a hardware virtual manager.  The ‘guest systems’ that run on the hypervisor know they’re on a virtualized system, but are still able to function as wholly separate entities.  This means you can run just about any OS or setup regardless of what the base level system is running.  This lets you do fun stuff like running a Windows VM on a Linux box, or run a different kernel in each guest.

OpenVZ containers don’t try to virtualize the hardware.  Instead, it uses a special kernel to isolate processes and resources.   Every guest runs on top of the same identical system, but they are fully separated from each other (still have their own filesystems/processes).  Everyone has the same kernel, and you can’t run different OS types on the same machine (can’t support both Linux and Windows VMs).

From this limited information, Xen is clearly more versatile.  But wait, I don’t use Xen.  ‘Why not?’, you ask in a haughty voice.   I’m getting to it.

Overhead (Isolation vs Ease of Use):

To keep things simple (and relevant), I’ll just be talking about 64-bit systems.  Xen does some pretty complicated stuff to make sure x86 architecture works just the way they wanted it to, and it’s impressive enough, but it adds more overhead to the process.  Not wanting to hold that against them, I’ll keep my discussions relevant to x86_64!  Oh, you only care about 64-bit architecture?  Good on you, mate!

Here personal preference is king.  Xen is much more virtualized, which comes with some nice pros to its cons.   It isolates its resources for you (more on this later).  This also allows custom/different kernels, since each guest is basically a wholly different environment.

But there is a cost in doing all of this.  Xen uses significantly more resources in overhead to create/maintain seperate instances for each VM.  OS level virtualization means many basic components exist once on the machine, and are used by all guests (like the identical kernel).  OpenVZ also uses one filesystem with chrooted environments, instead of fully separated Xen file systems.

If you’ll need a system with a custom kernel or want to load kernel modules,  stop reading now.  That decision pretty much makes itself, you’re a Xen fanboy now.  Get your hat on your way out of the lecture.

But with a single kernel, and a more manageable file system making backup/recovery easier, I find OpenVZ more manageable as a sysadmin.

Memory Management (The Gamble):

Here is the real meat and potatoes of this debate.  Xen’s dedicated memory and swap space, versus OpenVZ’s density and burstable memory.

It comes down to Xen’s swap space use versus OpenVZ killing processes.

With Xen’s isolation, you get your own section of memory.  It is quartered off for you, and nobody else has write privileges to it.  If you max out your memory, it’ll dump over into swap space so nothing crashes.  The problem on a shared server is that even though disk filesystems and swap spaces are isolated, disk I/O is not.  When people start bursting into swap space, it can cause the exact problem that Xen was intended to isolate around; bad VMs affecting the performance of other VMs.  Sure, your Apache will keep on kicking, but it’s going to run terribly, and may cause problems for the rest of the machine.

With OpenVZ, you are given a guaranteed amount of memory.  Since it’s not fully isolated, it’s a part of shared memory on the server.  Everyone has their guaranteed memory in the same pool.  There’s also (typically) burstable memory.  If memory isn’t being utilized, others who need it for a short time can burst up into extra memory.  If guaranteed memory is required somewhere else, this optional burst memory will fail away, and the memory will be allocated to fill a guaranteed quota.  But if you suddenly have no memory where memory once was, things die.  The problem is, they sometimes don’t die nicely.   No swap space with OpenVZ, so things die, and you don’t get a lot of control over what dies.

So we have a stability versus performance issue!  Or do we….

Conclusions (tl;dr):

I like OpenVZ.  I like the low overhead on containers and the performance is great.  The problem with OpenVZ is over-allocating memory.  You may notice on our VPS plan descriptions that we don’t talk about burstable memory.  That’s because we don’t allocate burstable memory.  We take a stability approach to our VPS.  OpenVZ won’t allocate memory it doesn’t have.  It’ll throw an error and stop new processes from spawning, but you will know about the problem before it accidently kills off Apache or MySQL.  It won’t bring down the whole house of cards, nor will it put the burden on everyone.

As an admin, I like using OpenVZ for it’s natural performance benefits, and managing memory with a more Xen approach.  It’s served me, and our customers, very well.

As a user, I suggest two good options.

• Use Xen because you need it (custom kernel, different OS, etc)
• Find an OpenVZ vendor who doesn’t oversell their resources (I suggest The Linux Fix!)

This shouldn’t effect any of your services, and any personal domain names you may have pointing to your dedicated server remain unchanged.