Environment Blog Archive
Adventures with using Ruby 2.0 and libreadline
I was asked to develop a prototype app for one of our clients lately. The basis for this app was an old Rails app:
- Rails 3.2.8
- RailsAdmin
- MySQL
- rbenv + ruby-build
I wanted to upgrade the stack to work with latest toys all cool kids are so thrilled about. I also didn’t have Rails console facility at my disposal since the Ruby version installed on the development machine hadn’t been compiled against libreadline.
Not having root or sudo access on the machine I embarked on a sligthly hacky journey to make myself a better working environment.
Ruby 2.0
After reading Mike Farmer’s blog post about Ruby 2.0 and tons of other material about it on the Internet, I wanted to get a feeling of how faster & greater the new Ruby is. It’s always great also to stay up-to-date with latest technologies. It’s great for me as a developer, and more importantly - it’s great for our clients.
Importance of libreadline in development with Ruby
To be productive developing any Rails-based application, we have to have Rails-console available at any moment. It serves a multitude of purposes. It’s also a great scratch-pad when developing methods.
While you don’t need your Ruby to support libreadline for basic uses of irb, you need it when using with Rails.
Installing Ruby 2.0.0 with rbenv (ruby-build)
If you’ve installed ruby-build some time ago, chances are that you need to update it in order to be able to install latest build of Ruby 2.0.0
To do it:
cd ~/.rbenv/plugins/ruby-build git pull
And you should be able now to have available latest Ruby build to install:
rbenv install 2.0.0-p195
If you want to install Ruby compiled with support for libreadline, you have to have it installed in your system before compiling the build with rbenv install.
If you have access to root or sudo on your system, the easiest way is to e. g:
on Debian-related Linuxes:
apt-get install libreadline-dev
or on Fedora:
yum install readline-devel
Installing libreadline from sources
In my case - I had to download sources and compile them myself. Luckily the system had all needed essential packages installed for building it.
wget "ftp://ftp.cwru.edu/pub/bash/readline-6.2.tar.gz" tar xvf readline-6.2.tar.gz cd readline-6.2 ./configure --prefix=/home/kamil/libs make make install
I had to specify –prefix option pointing at the path where I wanted the libreadline library to be installed after compilation.
Then, I was able to actually build Ruby with readline support “on”:
CONFIGURE_OPTS="--with-readline-dir=/home/kamil/libs" rbenv install 2.0.0-p195
Notice: I was making myself a development environment and compiling from sources was my last resort. It is not a good practice for production environments.
Last thing I needed to do was to get rb-readline working with the project I was working on.
It turnes out that latest rb-readline doesn’t play well with latest Ruby. Also, when using Ruby 2.0.0 one have to explicitely specify it in the Gemfile, or else it won’t be loaded for the console.
Gemfile:
gem 'rb-readline', '~> 0.4.2'
This still isn’t perfect
While this setup works, it won’t let you use arrow keys. The irb process crashes quickly after even first try to navigate through the text.
For some reason, after upgrading Ruby, the RailsAdmin stylesheets stopped working. I noticed that they are being served with comments which should be replaced by other stylesheets like:
/* ... *= require_self *= require_tree . */
I had to update Rails version in the Gemfile to have my admin back:
Gemfile:
gem 'rails', '3.2.13'
Console:
bundle
Last thing I wanted to do, was to try if I could upgrade Rails even further and have a working Rails4 setup. This was impossible unfortunately since RailsAdmin isn’t yet compatible with it as stated here.
I conclude that latest Ruby is quite usable right now. If you don't mind the quirks with the readline - you're pretty safe to upgrade. This assumes though that your app doesn't use any incompatible elements.
The main Ruby site describes them like so:
There are five notable incompatibilities we know of:
Test Web Sites with Internet Explorer for Free
Browser Testing
While many Web Developers prefer to build sites and web applications with browsers like Chrome or Firefox it's important for us to keep an eye on the browser market share for all web users. Internet Explorer (IE) still owns a large piece of this pie and because of this, it is important to test sites and applications to ensure they work properly when viewed in IE. This poses a potential problem for developers who do not use Windows.
Although I use OS X on my desktop, I have Windows virtual machines with IE 6,7,8,9 and 10 installed. I also have a Linux virtual machine running Ubuntu so I can check out Chrome/Chromium and Firefox on that platform. In the past I had tried solutions like MultipleIEs but wasn't satisfied with them. In my experience I've found that the best way to see what users are seeing is to have a virtual machine running the same software they are.
I did some IE8 testing for a colleague a short time ago and suggested she should give VirtualBox a shot. Her response was "You should write a blog post about that!". So here we are.
Free Tools
VirtualBox is a free virtualization application similar to Parallels or VMWare. These applications aim to solve this problem by allowing us to run different "guest" operating systems on the "host" operating system of our choice. For example, I run several versions of IE and Windows on my computer which is running OS X.
Microsoft have begun in recent years to release Virtual PC (VPC) images for free to enable Web Developers to test their sites and applications with various versions of Internet Explorer. The most recent images were released on July 30, 2012 and you can check them out here. Although these images are created to run on Microsoft's Virtual PC application, with a little bit of effort we can also use them on Linux, OS X or Windows via VirtualBox.
For our purposes, we will be building a Windows 7 virtual machine running Internet Explorer 8. However, there are several images available with various Windows OS and Internet Explorer version combinations:
- Windows 7 / IE8
- Windows 7 / IE9
- Windows Vista / IE7
- Windows XP / IE6
Linux and OS X Users
In researching this article I came across a very helpful project on GitHub. The ievms project by xdissent is a bash script that automates this process. I was able to run the script and run my newly created Windows 7 / IE virtual machine shortly afterward.
Requirements
To start we'll need to download and install VirtualBox. Be sure you have the most recent version available as it gets updated quite often. We'll also need to install the VirtualBox Extension Pack. Both are available on the VirtualBox Downloads page.
Because the VPC images are quite large (~2.6GB in our case) they have been split up into several files. For the Windows 7 / IE8 image we'll download the following files:
Once the files have been extracted we have created our virtual machine the disk will take up close to 10 GB. Make sure you have enough disk space available. I run my virtual machines on an external USB hard drive which works well if your local hard drive is starved for space.
Extract the VPC Files
Once you have all of the the VPC image files downloaded you'll need to extract them:
Windows users: double-click the Windows_7_IE8.part01.exe file
Linux / OS X users: You will need to install unrar. I did so on OS X with Homebrew and there are packages available for Ubuntu as well.
Once you have unrar installed, issue the following command in the directory containing all of the VPC image files:
`unrar e -y Windows_7_IE8.part01.exe`
This will extract the files and combine them into a single .vhd file. The -y flag tells unrar to say "yes" to the EULA from Microsoft so be aware of that.
Create a new Machine
Once the .vhd file is ready we can create a new machine in VirtualBox. Start up VirtualBox and click the "New" button:
VM Name and OS Type
We'll enter a name (IE8 / Windows 7) and select "Microsoft" and "Windows 7" in the Operating System and Version drop-downs:
Click "Continue" to proceed.
Memory
Specify how much memory (RAM) you'd like to allocate to the virtual machine. This really depends on how much you have available but it is best to go with the minimum at the very least: 
Click "Continue" to proceed.
Disk
Select the "Do not add a virtual hard drive" option.: 
The reason we skip this for now is because we want to add the hard disk as an IDE drive. For some reason VirtualBox adds the disk as a SATA drive which causes the blue screen of death (BSOD) to appear when the virtual machine is booted up.
Click "Continue" to proceed and click "Continue" again when presented with the warning about not attaching a hard disk.
Review Settings and Create the VM
At this point you can review the settings you've made and click "Create" to complete the process:
Configure Storage Settings
Highlight your newly created virtual machine and click the "Settings" button. From there, click the "Storage" tab and add a hard disk to the IDE controller: 
Absolute Pointing Device
In the Settings, click on the "System" tab and uncheck "Enable absolute pointing device". I had to do this in order to to get the mouse working properly.
Start it Up
We're almost there: next we need to start up the VM by clicking the "Start" button. You'll see Administrator and IEUser accounts and the password for both is "Password1". Log in as Administrator and check it out! It's important to note when the virtual machine has the keyboard and mouse focus. When you click inside the virtual machine window, it will capture the input from your mouse and keyboard. To return focus back to your computer you will need to press the "host key" sequence on your computer:
The "host-key" sequence can be configured in the VirtualBox preferences. On my computer, the host-key sequence is the left CTRL key (displayed as Left ^). When the arrow icon is illuminated, the guest computer – Windows 7 in this case, has the focus. When I press the left CTRL key on my computer the arrow icon goes tray to indicate focus has been returned to my computer.
Activation
As soon as you log in you will be presented with the Windows Activation wizard. Follow the steps to activate this copy of Windows. Once that is complete you will be prompted to restart your computer. Go ahead and do that.
The screen resolution in the virtual machine will be quite low at first — I believe 800x600 is the default. Don't worry about this as it will be addressed in the next step.
Guest Additions
Log in as Administrator (Password1) and Install the VirtualBox Guest Additions. To do this, navigate to Devices in the VirtualBox menu on your computer (the "host" machine"). Choose the menu option to "Install Guest Additions". Each time you are prompted with the "Would you like to install..." dialog, choose "Install":
This will install some drivers optimized by VirtualBox which allow you to change the screen resolution, improve the interaction between your host computer and the Windows guest operating system. Once the install process is complete you will be prompted to restart your virtual machine once more.
Finished Product
With Windows activated and the Guest Additions installed you should be able to log in to your very own Windows 7 / IE8 testing machine!
This free copy of Windows will operate for 30 days and the trial period can be extended twice by running the following at the command prompt:
slmgr –rearm
Check out the documentation at Microsoft for more details.
GNU Screen + SSH_AUTH_SOCK; my new approach
Over the years, I've played around with several different methods of keeping my forwarded SSH-Agent authentication socket up-to-date in long-running screen sessions (referenced via the $SSH_AUTH_SOCK shell variable). The basic issue here is that Screen sees the process environment at the time it was initially launched, not that which exists when reattaching in a subsequent login session. This means that the $SSH_AUTH_SOCK variable as screen sees it will refer to a socket which no longer exists (as it was removed when you logged out after detaching on the initial login when starting screen).
Some of my previous methods have included a hard-coded variable for the socket itself (downsides: if it's a predictable name you're potentially opening some security issues, plus if you open multiple sessions to the same account, you kill the latest socket), symlinking the latest $SSH_AUTH_SOCK to a hard-coded value on login (similar issues), dumping $SSH_AUTH_SOCK to a file, and aliasing ssh and scp to first source said file to populate the local window's enviroment (doesn't work in scripts, too much manual setup when adapting to a new system/environment, won't work with any other subsystem not already explicitly handled, etc).
Recently though, I've come up with a simple approach using screen's -X option to execute a screen command outside of screen and just added the following to my .bashrc:
screen -X setenv SSH_AUTH_SOCK "$SSH_AUTH_SOCK"
While not perfect, in my opinion this is a bit of an improvement for the following reasons:
- It's dirt-simple. No complicated scripts to adjust/maintain, just a command that's almost completely self-explanatory.
- It doesn't kill the environment for existing screen windows, just adjusts the $SSH_AUTH_SOCK variable for new screen windows. This ends up matching my workflow almost every time, as unless a connection dies, I leave the screen window open indeterminately.
- If you have multiple sessions open to the same account (even if not running both in screen), you're not stomping on your existing socket.
- Did I mention it's dirt-simple?
There are presumably a number of other environment variables that would be useful to propagate in this way. Any suggestions or alternate takes on this issue?
Managing Perl environments with perlbrew
As a Perl hobbyist, I've gotten used to the methodical evolution of Perl 5 over the years. Perl has always been a reliable language, not without its faults, but with a high level of flexibility in syntactical expression and even deployment options. Even neophytes quickly learn how to install their own Perl distribution and CPAN libraries in $HOME. But the process can become unwieldy, particularly if you want to test across a variety of Perl versions.
To contrast, Ruby core development frequently experiences ABI breakages, even between minor releases. In spite of the wide adoption of Ruby as a Web development language (thanks to Ruby on Rails), Ruby developers are able to plod along unconcerned, where these incompatibilities would almost certainly lead to major bickering within the Perl or PHP communities. How do they do it? The Ruby Version Manager.
Ruby Version Manager (RVM) allows users to install Ruby and RubyGems within their own self-contained environment. This allows each user to install all (or only) the software that their particular application requires. Particularly for Ruby developers, this provides them with the flexibility to quickly test upgrades for regressions, ABI changes and enhancements without impacting system-wide stability. Thankfully a lot of the ideas in RVM have made their way over to the Perl landscape, in the form of perlbrew.
Perlbrew offers many of the same features found in RVM for Ruby. It's easy to install. It isolates different Perl versions and CPAN installations in your $HOME and helps you switch between them. It automates your environment setup and teardown. And most importantly, using perlbrew means not having to clutter your default system Perl with application-specific CPAN dependencies.
Getting started with perlbrew couldn't be easier. A quick one-liner is all it takes to install perlbrew in your home directory.
$ curl -L http://xrl.us/perlbrewinstall | bash
If you need to install perlbrew somewhere other than your home directory, just download the installer and pass it the PERLBREW_ROOT environment variable.
$ curl -LO http://xrl.us/perlbrew $ chmod +x perlbrew $ PERLBREW_ROOT=/mnt/perlbrew ./perlbrew install
Follow the instructions on screen and you'll be ready to use perlbrew in no time. The perlbrew binary will be installed in ~/perl5/perlbrew/bin, so make sure to adjust your login $PATH accordingly.
Once you're done installing perlbrew there are a couple commands you'll want to run before installing your own Perl versions or CPAN modules. The perlbrew init command is mandatory; this initializes your perlbrew directory. It can also be used later if you need to modify your PERLBREW_ROOT setting. The perlbrew mirror is optional (but recommended) to help you select a preferred CPAN mirror.
$ perlbrew init $ perlbrew mirror
Next comes the fun part. Start off by verifying the Perl version(s) that perlbrew sees.
$ perlbrew list * /usr/bin/perl (5.10.1)
Install a newer version of Perl.
$ perlbrew install 5.12.3
Now switch to the newer Perl.
$ perlbrew list * /usr/bin/perl (5.10.1) perl-5.12.3 $ perlbrew switch perl-5.12.3 $ perlbrew list /usr/bin/perl (5.10.1) * perl-5.12.3 $ perl -v This is perl 5, version 12, subversion 3 (v5.12.3) built for x86_64-linux Copyright 1987-2010, Larry Wall Perl may be copied only under the terms of either the Artistic License or the GNU General Public License, which may be found in the Perl 5 source kit. Complete documentation for Perl, including FAQ lists, should be found on this system using "man perl" or "perldoc perl". If you have access to the Internet, point your browser at http://www.perl.org/, the Perl Home Page.
Alternatively, if you only want to test a different Perl version, try the perlbrew use command (note: this only works in bash and zsh). Unlike the switch command, use is only active for the current shell.
$ perlbrew use system $ perlbrew list * /usr/bin/perl (5.10.1) perl-5.12.3
A quick peek behind the curtain reveals much of the simplicity behind perlbrew.
$ ls -l ~/perl5/perlbrew/ total 2680 -rw-r--r-- 1 testy users 408 Feb 10 23:58 Conf.pm drwxr-xr-x 2 testy users 512 Feb 10 23:46 bin drwxr-xr-x 4 testy users 512 Feb 11 09:59 build -rw-r--r-- 1 testy users 1333196 Feb 11 10:33 build.log drwxr-xr-x 2 testy users 512 Feb 11 09:59 dists drwxr-xr-x 2 testy users 512 Feb 10 23:47 etc drwxr-xr-x 4 testy users 512 Feb 11 10:32 perls $ ls -l ~/perl5/perlbrew/perls/ total 8 drwxr-xr-x 5 testy users 512 Feb 11 00:38 perl-5.12.3 drwxr-xr-x 5 testy users 512 Feb 11 10:32 perl-5.13.6
If you're a Perl developer, the perlbrew project may help alleviate a lot of the pain associated with team development or multi-tenant programming environments. Suddenly it becomes much easier to manage your own software requirements, resulting in faster development and testing cycles for you, and fewer headaches for your System Administrators.
Visit at DistribuTECH
I had the chance to attend DistribuTECH in San Diego, CA this past week. DistribuTECH is billed as the utility industry's leader in smart grid conference and exposition. End Point was present at the conference on behalf of Silver Spring Networks. Silver Spring Networks contracted with us to provide a Liquid Galaxy installation for their exhibit.
The Liquid Galaxy did its job from what I could tell. The exhibit was consistently surrounded with conference goers both interested in listening and watching the tours that were being presented as well as wanting to see what the Liquid Galaxy was all about. This was the first time I had seen the Liquid Galaxy and was quite impressed with how well it worked. I saw many people moving their bodies in sync with what was being displayed on the screen, showing that they felt immersed while within the galaxy. One gentleman knelt down while attempting to look under a graph that was being presented on the screen. This same person had returned to the exhibit several times, bringing colleagues back each time to "show off" what he had found.
I spent some time on the conference floor, checking out what was being displayed and seeing how others were getting the attention of the attendees. I could not find anything that compared to the Liquid Galaxy in both wow factor and usability. The fine folks at Silver Spring Networks also seemed impressed with the reaction they were receiving.
One product that I found interesting while walking the floor was a large unit that would freeze itself at night, when power is less expensive, the draw on the grid is less, and it is cooler outside and then use the ice as a coolant to be pumped into the AC unit during the day resulting in a reduction of cost and energy usage to cool a building.
I also took a look at the Silver Spring Networks products on display. One that interested me was their home portal that allowed customers that had Silver Spring smart meters installed by their utility to visit a web portal and view several things including their current usage and compare their usage to that of their neighbors. I can see how someone concerned with the environment could use this information to lessen their power usage.
Keep an eye out here for a few blog posts from Adam on his experience with tour development for the Liquid Galaxy.
Debugging jQuery
A recent reskin project for a client requires that we convert some of their old Prototype code to jQuery as well as create new jQuery code. I have not done much with converting Prototype to jQuery and I felt like my debugging tools for JavaScript were under par. So this morning I set out to find what was available for jQuery and I found this article on the subject.
I've used Firebug for some time now, but was unaware of some of the supporting plugins that would certainly help with debugging JavaScript. Some of the plugins and methods found in the article that I found immediately helpful were:
- FireFinder: Makes it quite easy to verify that the selector values in your code are correct and that the proper elements are returned. I was able to immediately pinpoint problems with my selectors and this brought to light why certain events were not firing.
- Firebug Console: Using the console.log function allowed me to check values without littering my code with alert statements.
- FireQuery: At a glance this plugin for Firebug shows which elements have event handlers bound to them.
- Firebug Breakpoints: Setting breakpoints and watch statements in your code makes it easier to see what is happening in the JavaScript code as it is executed instead of trying to figure out what happened after the code has run its course.
Thanks to the author of the article, Elijah Manor, for the in-depth information on debugging jQuery code.
Guidelines for Interchange site migrations
I'm involved at End Point often with Interchange site migrations. These migrations can be due to a new client coming to us and needing hosting or migrating from one server to another within our own infrastructure.
There are many different ways to do a migration, in the end though we need to hit on certain points to make sure that the migration goes smoothly. Below you will find steps which you can adapt for your specific migration.
During the start of the migration it might be a good time to introduce git for source control. You can do this by creating the repository and cloning it to /home/account/live, setting up .gitignore files for logs, counter files, gdbm files. Then commit the changes back to the repo and you've now introduced source control without much effort, improving the ability to make changes to the site in the future. This is also helpful to document the changes you make to the code base along the way during the migration in case you need to merge changes from the current production site before completing the migration.
- Export all of the gdbm databases to their text file equivalents on the production server
- Take a backup from production of the database, catalog, interchange server, htdocs
- Setup an account
- Create the database and user
- Restore the database, catalog, interchange server and htdocs
- Update the paths in interchange/bin for each script to point at the new location
- Grep the restored code for hard coded paths and update those paths to the new locations. Better yet move these paths out to a catalog_local.cfg where environment specific information can go.
- Grep the restored code for hard coded urls and use the [area] tag to generate the urls
- Update the urls in products/variable.txt to point at the test domain
- Update the sql settings in products/variable.txt to point at the new database using the new user
- Remove the gdbm databases so they will be recreated on startup from the source text files
- Install a local Perl if it's not already installed (./configure -des will compile and install Perl locally)
- Install Bundle::InterchangeKitchenSink
- Install the DBD module for MySQL or PostgreSQL
- Review the code base looking for use statements in custom code and Require module settings in interchange.cfg. Install the Perl modules found into the local Perl.
- Setup a non ssl and ssl virtual host using a temporary domain. Configure the temporary domain to use the SSL certificate from the production domain.
- Firewall or password protect the virtual host so it is not accessible to the public
- Generate a vlink using interchange/bin/compile and copy it into the cgi-bin directory and name it properly
- Startup the new Interchange
- Review error messages and resolve until Interchange will start properly
- Test the site thoroughly, resolving issues as they appear. Make sure that checkout, charging credit cards, sending of emails, using the admin, etc all function.
- Migrate any cron jobs running on the current production site, such as session expiration scripts
- Setup logrotation for the new logs that will be created
- Verify that you have access to make DNS changes
- Set the TTL for the domain to a low value such as 5 minutes
- Modify the new production site to respond to the production url, test by updating your hosts file to manually set the IP address of the domain
- Shutdown the new Interchange
- Restore a copy of the original backup for Interchange, the catalog and htdocs to /tmp on the production server
- Shutdown the production Interchange, put up a maintenance note on the production site.
- Take a backup of the production database and restore on the new server
- Diff the Interchange, catalog and htdocs directory between /tmp and the current production locations, making note of the files that have changed since we took the original copy.
- Copy the files that have changed, making sure to merge with any changes we have made on the new production site. Making sure to copy over all .counter and .autonumber files to the new production site.
- Start Interchange on the new production server
- Test the site thoroughly on the new production server, using the production url. Make sure that checkout with charging the credit card functions properly.
- Resolve any remaining issues found during the testing
- Setup the Interchange daemon to start at boot for this site in /etc/rc.d/rc.local or in cron using @reboot
- Update DNS to point at the new production IP address
- Update the TTL of the domain to a longer value
- Open the site to the public by opening the firewall or removing the password protection
- Keep an eye on the error logs for any issues that might crop up
This will hopefully give you a solid guide for performing an Interchange site migration from one server to another and some of the things to watch out for that might cause issues during the migrations.
Why is my load average so high?
One of the most common ways people notice there's a problem with their server is when Nagios, or some other monitoring tool, starts complaining about a high load average. Unfortunately this complaint carries with it very little information about what might be causing the problem. But there are ways around that. On Linux, where I spend most of my time, the load average represents the average number of process in either the "run" or "uninterruptible sleep" states. This code snippet will display all such processes, including their process ID and parent process ID, current state, and the process command line:
#!/bin/sh
ps -eo pid,ppid,state,cmd |\
awk '$3 ~ /[RD]/ { print $0 }'
Most of the time, this script has simply confirmed what I already anticipated, such as, "PostgreSQL is trying to service 20 times as many simultaneous queries as normal." On occasion, however, it's very useful, such as when it points out that a backup job is running far longer than normal, or when it finds lots of "[pdflush]" operations in process, indicating that the system was working overtime to write dirty pages to disk. I hope it can be similarly useful to others.
