Ruby On Rails Blog Archive

Adventures with using Ruby 2.0 and libreadline

I was asked to develop a prototype app for one of our clients lately. The basis for this app was an old Rails app:

• Rails 3.2.8
• RailsAdmin
• MySQL
• rbenv + ruby-build

I wanted to upgrade the stack to work with latest toys all cool kids are so thrilled about. I also didn’t have Rails console facility at my disposal since the Ruby version installed on the development machine hadn’t been compiled against libreadline.

Not having root or sudo access on the machine I embarked on a sligthly hacky journey to make myself a better working environment.

Ruby 2.0

After reading Mike Farmer’s blog post about Ruby 2.0 and tons of other material about it on the Internet, I wanted to get a feeling of how faster & greater the new Ruby is. It’s always great also to stay up-to-date with latest technologies. It’s great for me as a developer, and more importantly - it’s great for our clients.

Importance of libreadline in development with Ruby

To be productive developing any Rails-based application, we have to have Rails-console available at any moment. It serves a multitude of purposes. It’s also a great scratch-pad when developing methods.

While you don’t need your Ruby to support libreadline for basic uses of irb, you need it when using with Rails.

Installing Ruby 2.0.0 with rbenv (ruby-build)

If you’ve installed ruby-build some time ago, chances are that you need to update it in order to be able to install latest build of Ruby 2.0.0

To do it:

cd ~/.rbenv/plugins/ruby-build
git pull

And you should be able now to have available latest Ruby build to install:

rbenv install 2.0.0-p195

If you want to install Ruby compiled with support for libreadline, you have to have it installed in your system before compiling the build with rbenv install.

If you have access to root or sudo on your system, the easiest way is to e. g:

on Debian-related Linuxes:

apt-get install libreadline-dev

or on Fedora:

yum install readline-devel

Installing libreadline from sources

In my case - I had to download sources and compile them myself. Luckily the system had all needed essential packages installed for building it.

wget "ftp://ftp.cwru.edu/pub/bash/readline-6.2.tar.gz"
tar xvf readline-6.2.tar.gz
cd readline-6.2
./configure --prefix=/home/kamil/libs
make
make install

I had to specify –prefix option pointing at the path where I wanted the libreadline library to be installed after compilation.

Then, I was able to actually build Ruby with readline support “on”:

CONFIGURE_OPTS="--with-readline-dir=/home/kamil/libs" rbenv install 2.0.0-p195

Notice: I was making myself a development environment and compiling from sources was my last resort. It is not a good practice for production environments.

Last thing I needed to do was to get rb-readline working with the project I was working on.

It turnes out that latest rb-readline doesn’t play well with latest Ruby. Also, when using Ruby 2.0.0 one have to explicitely specify it in the Gemfile, or else it won’t be loaded for the console.

Gemfile:

gem 'rb-readline', '~> 0.4.2'

This still isn’t perfect

While this setup works, it won’t let you use arrow keys. The irb process crashes quickly after even first try to navigate through the text.

For some reason, after upgrading Ruby, the RailsAdmin stylesheets stopped working. I noticed that they are being served with comments which should be replaced by other stylesheets like:

/* ...
*= require_self
*= require_tree .
*/

I had to update Rails version in the Gemfile to have my admin back:

Gemfile:

gem 'rails', '3.2.13'

Console:

bundle

Last thing I wanted to do, was to try if I could upgrade Rails even further and have a working Rails4 setup. This was impossible unfortunately since RailsAdmin isn’t yet compatible with it as stated here.

I conclude that latest Ruby is quite usable right now. If you don't mind the quirks with the readline - you're pretty safe to upgrade. This assumes though that your app doesn't use any incompatible elements.

The main Ruby site describes them like so:

There are five notable incompatibilities we know of:

• The default encoding for ruby scripts is now UTF-8 [#6679]. Some people report that it affects existing programs, such as some benchmark programs becoming very slow [ruby-dev:46547].
• Iconv was removed, which had already been deprecated when M17N was introduced in ruby 1.9. Use String#encode, etc. instead.
• There is ABI breakage [ruby-core:48984]. We think that normal users can/should just reinstall extension libraries. You should be aware: DO NOT COPY .so OR .bundle FILES FROM 1.9.
• #lines, #chars, #codepoints, #bytes now returns an Array instead of an Enumerator [#6670]. This change allows you to avoid the common idiom “lines.to_a”. Use #each_line, etc. to get an Enumerator.
• Object#inspect does always return a string like #<ClassName:0x…> instead of delegating to #to_s. [#2152]
• There are some comparatively small incompatibilities. [ruby-core:49119]

Debugging Localization in Rails

Rails offers a very extensive library for handling localization using the rails-i18n gem. If you've done any localization using Rails, you know that it can be difficult to keep track of every string on your web application that needs translation. During a recent project, I was looking for an easy way to visually see translation issues while browsing through the UI in our application.

Missing Translations

I've known for some time that when I was missing a translation for a given text that it would appear in the HTML with a special <span> tag with the class .translation_missing. So in my global CSS I added the following:

.translation_missing { background-color: red; color: white !important; }

Now when I viewed any page where I had forgotten to add a translation to my en.yml file I could see the text marked in bright red. This served as a great reminder that I needed to add the translation.

I'm an avid watcher of my rails log while I develop. I almost always have a terminal window right next to my browser that is tailing out the log so I can catch any weirdnesses that may crop up. One thing that I thought would be nice is if the log showed any translation issues on the page it was loading. After some research, I found that the I18n gem used by Rails doesn't give you any obvious hooks for translation errors. There is, however, a section in the Rails I18n Guide for adding custom exception handlers for translation issues and this is where I started with something like this in an initializer:

module I18n
  def self.just_raise_that_exception(*args)
    Rails.logger.debug args.join("\n")
    raise args.first
  end
end

I18n.exception_handler = :just_raise_that_exception

This got things started for me. After some small modifications to the logger line, I had the output that I wanted and to make things really stick out, I added some color using the term-ansicolor gem and came up with this:

def self.just_raise_that_exception(*args)
  require 'term/ansicolor'
  include Term::ANSIColor
  Rails.logger.debug red("--- I18n Missing Translation: #{args.inspect} ---")
  raise args.first
end

I18n Fallbacks

After using this solution for a while I ran into another issue. When I would change the locale in my app to something like es (Spanish) and look for translation issues, I was noticing some text that wasn't being translated but it wasn't being marked as red by my CSS. When I looked at the text further I noticed that it wasn't even being surrounded by the <span class="translation_missing> tag. As it turned out, I did have a translation for that text in English, but not in the language that was set to my current locale. The reason for this is what I18n refers to as Fallbacks. Fallbacks tell the I18n what alternate, or default, locale to "fall back" to if a translation isn't available for the current locale. By default, the fallback locale is en. Since I had a translation for that text in my en.yml there was no indicator that I had "fallen back" even though it was obvious that there was a translation issue.

Coming up with a way to be notified of fallbacks wasn't nearly as simple as my initial solution. After a lot of googling and reading in StackOverflow, I found that perhaps the best solution was to override the translate method in the I18n gem. This sounds really scary, but it actually wasn't too bad. Back in my initializer, I removed the solution that I had originally come up with and added the following:

module I18n::Backend
  module Fallbacks

    # To liven things up a bit
    require 'term/ansicolor'
    include Term::ANSIColor

    def translate(locale, key, options = {})
      # First things first, if there is an option sent, decorate it and send it back.
      return fallback_message(locale, key, super) if options[:fallback]

      default = extract_non_symbol_default!(options) if options[:default]

      options[:fallback] = true
      I18n.fallbacks[locale].each do |fallback|
        catch(:exception) do
          # Get the original translation
          translation = super(fallback, key, options)

          # return the translation if we didn't need to fall back
          return translation if fallback == locale

          # return the decorated fallback message if we did actually fall back.
          result = fallback_message(locale, key, translation)
          return result unless result.nil?
        end
      end
      options.delete(:fallback)

      return super(locale, nil, options.merge(:default => default)) if default

      # If we get here, then no translation was found.
      Rails.logger.debug red("--- I18n Missing Translation: #{locale}::#{key} ---")
      throw(:exception, I18n::MissingTranslation.new(locale, key, options))
    end

    # Added this method to log the fallback and decorate the text with a <span> tag.
    def fallback_message(locale, key, fallback_text)
      return nil if fallback_text.nil?
      keys = key.to_s.split(".")
      return fallback_text if keys.first == 'platform_ui'

      Rails.logger.debug yellow("--- I18n Fallback: #{locale}::#{key} ---")
      %(<span class="translation_fallback" title="translation missing: #{locale}, #{key}">#{fallback_text}</span>).html_safe
    end

  end
end

With this initializer in place, I added the following CSS to my global stylesheet:

.translation_fallback { background-color: green; color: white !important; }

Now all missing translation errors and fallbacks were noticeable while browsing the UI and all translation issues were logged to my development log. Now obviously, I don't want these things showing up in production so I wrapped the whole thing in a if Rails.env.development? conditional. I could also think of times, like while working on implementing design, that I don't really care about translation issues as well, so I added a flag for turning it off in development as well.

You can see my entire solution on a gist over at github. It was suggested by a fellow colleague that I should turn this into a gem, but I didn't see the need if all you had to do is copy and paste this initializer. If a gem is something you'd like to see this in, please let me know and I'll consider packaging it up.

jQuery Performance Tips: Slice, Filter, parentsUntil

I recently wrote about working with an intensive jQuery UI interface to emulate highlighting text. During this work, I experimented with and worked with jQuery optimization quite a bit. In the previous blog article, I mentioned that in some cases, the number of DOM elements that I was traversing at times exceeded 44,000, which caused significant performance issues in all browsers. Here are a few things I was reminded of, or learned throughout the project.

• console.profile, console.time, and the Chrome timeline are all tools that I used during the project to some extent. I typically used console.time the most to identify which methods were taking the most time.
• Caching elements is a valuable performance tool, as it's typically faster to run jQuery calls on a cached jQuery selector rather than reselecting the elements. Here's an example:
  

  Slower	Faster
  //Later in the code $('.items').do_something();	//On page load var cached_items = $('.items'); //Later in the code cached_items.do_something();

• The jQuery .filter operator came in handy, and gave a bit of a performance bump in some cases.
  

  Slower	Faster
  $('.highlighted');	cached_items.filter('.highlighted');

• jQuery slicing from a cached selection was typically much faster than reselecting or selecting those elements by class. If retrieving slice boundaries is inexpensive and there are a lot of elements, slice was extremely valuable.
  

  Slower	Faster
  cached_items.filter('.highlighted');	cached_items.slice(10, 100);

• Storing data on elements was typically a faster alternative than parsing the id from the HTML markup (class or id). If it's inexpensive to add data values to the HTML markup, this was a valuable performance tool. Note that it's important to test if the jQuery version in use automatically parses the data value to an integer.
  

  Slower	Faster
  //given var slice_start = parseInt($('tt#r123') .attr('id') .replace(/^r/, ''));	//given var slice_start = $('tt#r123') .data('id');

• Advanced jQuery selectors offered performance gain as opposed to jQuery iterators. In the example below, it's faster to use selectors :has and :not combined rather than iterating through each parent.
  

  Slower	Faster
  $.each($('p.parent'), function(i, el) { //if el does not have any visible spans // do_something() });	$('p.parent:not(:has(span:visible))') .do_something();

• The jQuery method parentsUntil was a valuable tool instead of looking at the entire document or a large set of elements. In the cases where the children were already defined in a subset selection, I used the parentsUntil method to select all parents until a specific DOM element.
  

  Slower	Faster
  $('#some_div p.parent:not(:has(span:visible))') .do_something();	subset_of_items .parentsUntil('#some_div') .filter(':not(:has(span:visible))') .do_something();

The best takeaway I can offer here is that it was almost always more efficient to work with as precise set of selected elements as possible, rather than reselecting from the whole document. The various methods such as filter, slice, and parentsUntil helped define the precise set of elements.

How to Apply a Rails Security Patch

With the announcement of CVE-2013-0333, it's time again to secure your Rails installation. (Didn't we just do this?) If you are unable to upgrade to the latest, secure release of Rails, this post will help you apply a Rail security patch, using CVE-2013-0333 as an example.

Fork Rails, Patch

The CVE-2013-0333 patches so kindly released by Michael Koziarski are intended for use with folks who have forked the Rails repository. If you are unable to keep up with the latest releases, a forked repo can help you manage divergences and make it easy to apply security patches. Unfortunately, you cannot use wget to download the attached patches directly from Google Groups, so you'll have to do this in the browser and put the patch into the root of your forked Rails repo. To apply the patch:

cd $RAILS_FORK_PATH
git checkout $RAILS_VERSION
# Download attachment from announcement in browser, sorry no wget!
git am < $CVE.patch

You should see the newly committed patch(es) at the HEAD of your branch. Push out to GitHub and then bundle update rails on your servers.

Patching without Forks

If you are in the unfortunate case where there have been modifications or patches applied informally outside version control or you are otherwise compelled to modify the Rails source on your server directly, you are still able to use the provided patches.

Before begining, take a look at the diffstat at the top of the patch:

 .../lib/active_support/json/backends/okjson.rb     |  644 ++++++++++++++++++++
 .../lib/active_support/json/backends/yaml.rb       |   71 +---
 activesupport/lib/active_support/json/decoding.rb  |    2 +-
 activesupport/test/json/decoding_test.rb           |    4 +-

As you can see the base path of the diff is "activesupport". (The triple dots are simply there to truncate the paths so the diffstats line up nicely.) However, when the activesupport gem is installed on your system, the version number is appended in the path. This means we need to use the -p2 argument for patch to "strip the smallest prefix containing num leading slashes from each file name found in the patch file." We'll see how to do this in just a second, but first, let's find the source files we need to patch.

Locating Rails Gems

To find the installed location of your Rails gems, make sure you are using the desired RVM installation@gemset (check with rvm current), and then run "gem env" and look for the "GEM PATHS" section. If you're using the user-based installation of RVM it might look something like this:

/home/$USER/.rvm/gems/ree-1.8.7-2012.02

Now that we know where the installed gems are, we need to get our patch and apply.

cd /home/$USER/.rvm/gems/ree-1.8.7-2012.0/gems/activesupport-2.3.15
# Download attachment from announcement in browser, sorry no wget!
patch -p2 < $CVE.patch

Often times these patches will include changes to tests which are not included in the ActiveSupport gem installations. You may get an error like this while patching CVE-2013-0333:

patch -p2 < cve-2013-0333.patch                                                 
patching file lib/active_support/json/backends/okjson.rb
patching file lib/active_support/json/backends/yaml.rb
patching file lib/active_support/json/decoding.rb
can't find file to patch at input line 768
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/activesupport/test/json/decoding_test.rb b/activesupport/test/json/decoding_test.rb
|index e45851e..a7f7b46 100644
|--- a/activesupport/test/json/decoding_test.rb
|+++ b/activesupport/test/json/decoding_test.rb
--------------------------
File to patch: 
Skip this patch? [y] y
Skipping patch.
1 out of 1 hunk ignored

This error is just saying it cannot find the file test/json/decoding_test.rb. It's OK to skip this patch, because the file doesn't exist to patch.

Verify the Patch is Installed

When doing any kind of security patching it is essential that you have confidence your actions were applied successfully. The strategies on doing this will verify based on the type of changes made. For CVE-2013-0333, it's a fairly simple check.

# Before applying patch
script/console
Loading development environment (Rails 2.3.15)
>> ActiveSupport::JSON::DECODERS
=> ["Yajl", "Yaml"]

# After applying patch
script/console
Loading development environment (Rails 2.3.15)
>> ActiveSupport::JSON::DECODERS
=> ["Yajl", "OkJson"]

JavaScript-driven Interactive Highlighting

One project I've been involved in for almost two years here at End Point is the H2O project. The Ruby on Rails web application behind H2O serves as a platform for creating, editing, organizing, consuming and sharing course materials that is used by professors and their students.

One of the most interesting UI elements of this project is the requirement to allow highlighting and annotating text interactively. For example, when one reads a physical textbook for a college course, they may highlight and mark it up in various ways with different colors and add annotated text. They may also highlight a section that is particularly important for an upcoming exam, or they may highlight another section with a different color and notes that may be needed for a paper.

An example of highlighted text, by sergis on Flickr

The H2O project has required support for digitizing interactive highlighting and annotating. Since individual text is not selectable as a DOM element, each word is wrapped into an individual DOM element that is selectable, hoverable, and has DOM properties that we can assign it. For example, we have the following text:

The cow jumped over the moon.

Which is manipulated to create individual DOM elements by word:

<span>The </span>
<span>cow </span>
<span>jumped </span>
<span>over </span>
<span>the </span>
<span>moon.</span>

And an id is assigned to each element:

<span id="e1">The </span>
<span id="e2">cow </span>
<span id="e3">jumped </span>
<span id="e4">over </span>
<span id="e5">the </span>
<span id="e6">moon.</span>

This markup is the foundation of digitizing highlighting behavior: it allows the starting and ending boundaries of a highlighted section to be selected. It also allows additional highlighting boundary elements to be created before and after highlighted text, as well as provide the ability to interactively toggle highlighting. Without this, we can't easily parse or identify starting and ending points other than trying to determine it by other methods such as substring-ing text or using positioning details to identify the current word.

How do we Highlight?

With our example, I'll describe the history of highlighting functionality that I've been involved in on the project. For this post, let's say our desired emulated highlighted behavior is the following, where "cow jumped over" is highlighted in pink, and "over the" is highlighted in blue:

The cow jumped over the moon.

And our HTML markup may look like this to indicate the highlighted layers:

<span id="e1">The </span>
<span id="e2" class="pink">cow </span>
<span id="e3" class="pink">jumped </span>
<span id="e4" class="pink blue">over </span>
<span id="e5" class="blue">the </span>
<span id="e6">moon.</span>

Highlighting Iteration 1

One of the challenges with emulating highlighting is that a DOM element can only have one background color. We can't easily layer pink and blue highlights over a specific word to give a layered highlighted effect. So in our example, the pink and blue highlighted words will show up fine, but no color combinations will show up because a node cannot have multiple background colors. During my first iteration on this functionality, I implemented behavior to track the history of our highlights per textual node. The following steps are an example of a use case that demonstrates the highlight overlap:

• Starting from unhighlighted text, first a user highlights pink:
  The cow jumped over the moon.
• Next, a user highlightes blue:
  The cow jumped over the moon.
• Next, a user unhighlights blue:
  The cow jumped over the moon.
• Finally, a user unhighlights pink:
  The cow jumped over the moon.

Or another simple use case:

• Starting from unhighlighted text, first a user highlights pink:
  The cow jumped over the moon.
• Next, a user highlightes blue:
  The cow jumped over the moon.
• Next, a user unhighlights pink:
  The cow jumped over the moon.
• Finally, a user unhighlights pink:
  The cow jumped over the moon.

Programmatically, this method required that a history of highlights be stored to each text node in the form of an array. When something was highlighted or unhighlighted, the array was manipulated to remove or add highlights and the last highlight applied to the text node. While this method was fairly simply to implement, it did not allow users to visualize the overlapping highlighted sections. This iteration was in place for over a year, but ultimately it did not appropriately demonstrate overlapped highlights.

Highlighting Iteration 2

In the next iteration, I attempted to implement a method that added opaque, absolute and fixed positioned elements underneath the text, similar to how one might see layers in Photoshop:

In the second iteration, additional colored & opaque nodes were created under the text to provide a layered highlighting effect.

In the above markup, the following should be noted:

• Each span.highlights node is absolutely positioned with a width and height matching the text node.
• Each span.highlights span node (e.g. a node with the highlight_pink class) has a width and height of 100%, a background color defined in CSS, and an opacity that is scaled based on the number of highlights.
• Whenever highlights are toggled in the text, the children nodes of span.highlights are manipulated (added or removed), as well as the opacity.

While this functionality provides a nice highlighted layering effect, absolute positioning is probably my least favorite thing to work with in cross browser development, and it did not always behave as expected. Specifically, IE and Chrome behaved somewhat as expected, but Firefox did not. This absolute positioning also caused problems with other absolutely positioned elements on the page.

Additionally, this markup was subject to cause significant performance issues. In one case, content with 44,000 words (and text nodes) alone caused performance implications, but additional highlighting layering caused extreme performance pain, to the point that Chrome would not load the content and the content would take more than 30 seconds to load in Firefox, so I went searching for a better solution.

Highlighting Iteration 3

Finally, in the most recent iteration, after identifying that Iteration 2 produced significant performance issues, after more research, I came across the jQuery xColor plugin. This plugin allows you to do mathematical operations on colors, such as combining colors. While the plugin itself only lets you combine 2 colors at one time, I created a method to combine multiple opaque layers:

$.each($('span.' + highlighted_class), function(i, el) {
    var current = $(el);
    var highlight_colors = current.data('highlight_colors');
    if(highlight_colors) {
      highlight_colors.push(hex);
    } else {
       highlight_colors = new Array(hex);
    }    
    var current_hex = '#FFFFFF';
    var opacity = 0.4 / highlight_colors.length;
    $.each(highlight_colors, function(i, color) {
        var new_color = $.xcolor.opacity(current_hex, color, opacity);
        current_hex = new_color.getHex();
     });
     current.css('background', current_hex);
     current.data('highlight_colors', highlight_colors);        
});

Step by step, the above code does the following:

• For each span node with a specific highlight:
  
  • Retrieve the array of highlights applied to that node, or create a new array with the new highlight color.
  • For each highlight, layer an opaque version of that highlight on top of the summation of colors.
  • Set the background to the final combination of layered highlights.
  • Store the new array of highlights applied to that node.

The markup is back to the original markup with no additional children elements per text node. Each highlighting interaction triggers a recalculation of the background color per text node based on the data stored to that node.

Conclusion

Significant limitations during this work have included the described performance limitations, as well as the inability to set multiple background images on a DOM element. Absolutely positioning, while valuable at times, proved to be quite challenging because of other absolute and fixed positioned elements on the page. In addition to emulating this highlighting behavior, there are additional interactive requirements included with the interactive highlighting.

Example of additional features needed in interactive highlighting tool.

Additional UI functional requirements include:

• Functionality to show paragraph numbers, and hide all paragraph numbers when they have no visible children (Hint: advanced jQuery selectors are used here).
• Ability to toggle display of unhighlighted text. The […] in the above image trigger the unhighlighted text in that section to display, while the left and right arrows trigger the unhighlighted text in that section to be hidden.
• Ability to toggle display of highlighted text, similar to the toggle of unhighlighted text.
• Ability to toggle between a "read" and "edit" mode for owners of the text, which allows for these users to interactively add additional highlights and dynamically modify the markup. In the edit mode, additional markup is added to identify these highlighted layers.
• Ability to toggle display of the annotation. In the above image, clicking on the green asterisk toggles this display. No asterisk is shown if there is no annotation.
• Ability for highlights to encompass HTML nodes that are not individual span elements. For example, highlighted sections may encompass multiple paragraphs and headers of <span> nodes, which is why simply adding a wrapping element to highlights will not work.

One might suggest we go to a better tool to manage content markup, but ultimately these types of markup tools do not provide the interactivity we seek, and they require that the end user have knowledge of HTML markup, which is not always the case.

Use Metasploit to Verify Rails is Secured from CVE-2013-0156

On January 8th, 2013 Aaron Patterson announced a major security vulnerability on the Rails security mailing list, affecting all releases of the Ruby on Rails framework. This vulnerability allows an unskilled attacker to execute commands remotely on any unpatched Rails web server. Unsurprisingly, it's getting a lot of attention; Ars Technica estimates more than 200,000 sites may be vulnerable. With all the hype, it's important to separate the facts from the fiction and use the attacker's own tools to verify your site is secure.

Within 36 hours of the announcement of CVE-2013-0156, the developers at Rapid7 released a metasploit exploit module. Metasploit lowers the barriers to entry for attackers, making the whole process a point and click affair with a slick web GUI. Fortunately, the Rails security team has provided many easy to implement mitigation options. But, how do *know* you've really closed the vulnerability, particularly to the most automated and unskilled attacks? No better way than to try and exploit yourself.

It's best to scan your unpatched site first so you can be certain the scan is working as expected and you don't end up with a false positive that you've eliminated the vulnerability. Here is the quick and dirty introduction to running Metasploit, and executing a scan:

UPDATE: I've changed the Metasploit instructions here a bit to include setting the VHOST option. Teammate Steph Skardal was using these instructions and together we found that without a VHOST set, the RHOSTS are resolved to an IP address. It's worth checking your Rails logs to verify a request is being received and processed. If you don't see anything there, check your nginx or Apache (or whatever) access logs for any possible 301 redirects.

git clone git://github.com/rapid7/metasploit-framework.git
cd metasploit-framework
./msfconsole
use auxiliary/scanner/http/rails_xml_yaml_scanner
set RHOSTS mycompany.com
set VHOST app.mycompany.com
set RPORT 80
set URIPATH /rails_app
set VERBOSE true
show options
run
[+] mycompany.com:80 is likely vulnerable due to a 500 reply for invalid YAML
[*] Scanned 1 of 1 hosts (100% complete)
[*] Auxiliary module execution completed

If you don't get back a "likely vulnerable" message, it's probably because you're still running Ruby 1.8.7. As of this writing the metasploit module exploit states:

The technique used by this module requires the target to be running a fairly version of Ruby 1.9 (since 2011 or so). Applications using Ruby 1.8 may still be exploitable using the init_with() method, but this has not been demonstrated.

It's only a matter of time before Ruby 1.8 becomes supported, but this does give some folks a bit more time. Now let's review the mitgation strategies provided by the announcement to show you just how easy it can be to secure yourself.

Disabling XML Entirely

The nature of the vulnerability is in parsing XML in request parameters. If you don't parse XML, you should disable XML parsing entirely by placing one of the following snippets inside an application initializer.

# Rails 3.2, 3.1 and 3.0 
ActionDispatch::ParamsParser::DEFAULT_PARSERS.delete(Mime::XML) 
# Rails 2.3 
ActionController::Base.param_parsers.delete(Mime::XML) 

Removing YAML and Symbol support from the XML parser

I couldn't say it better than Aaron than myself, so I'll give it to you straight from the announcement:

If your application must continue to parse XML you must disable the YAML and Symbol type conversion from the Rails XML parser. You should place one of the following code snippets in an application initializer to ensure your application isn't vulnerable. You should also consider greatly reducing the value of REXML::Document.entity_expansion_limit to limit the risk of entity explosion attacks.

To disable the YAML and Symbol type conversions for the Rails XML parser add these lines to an initializer:

#Rails 3.2, 3.1, 3.0 
ActiveSupport::XmlMini::PARSING.delete("symbol") 
ActiveSupport::XmlMini::PARSING.delete("yaml") 

#Rails 2.3 
ActiveSupport::CoreExtensions::Hash::Conversions::XML_PARSING.delete('symbol') 
ActiveSupport::CoreExtensions::Hash::Conversions::XML_PARSING.delete('yaml') 

Removing YAML Parameter Parsing

While it's *much* less common to parse YAML in request params than XML, Rails does support this, but not by default (except in version 1.1.0!). There is no fix for YAML params injection, so it must be disabled . The methods for doing this differ in Rails among rails versions.

# Rails 2.x: find and remove all instances
ActionController::Base.param_parsers[Mime::YAML] = :yaml

# Rails 3.x: add to initializer
ActionDispatch::ParamsParser::DEFAULT_PARSERS.delete(Mime::YAML)

# Rails 3.2, 3.1, 3.0: add to initializer
ActiveSupport::XmlMini::PARSING.delete("symbol")
ActiveSupport::XmlMini::PARSING.delete("yaml")

# Rails 2.3: add to initializer
ActiveSupport::CoreExtensions::Hash::Conversions::XML_PARSING.delete('symbol')
ActiveSupport::CoreExtensions::Hash::Conversions::XML_PARSING.delete('yaml')

Go Check your Site!

As you can see, with just a few lines of code, any site can manage their exposure to this risk. I strongly urge you to read the security announcement and avoid the hype. Then go patch your site!

Piggybak: End of Year Update

Over the last few months, my coworkers and I have shared several updates on Piggybak progress (October 2012 Piggybak Roadmap , November 2012 Piggybak Roadmap Status Update). Piggybak is an open source, mountable as a Rails Engine, Ruby on Rails ecommerce platform developed and maintained by End Point. Here's a brief background on Piggybak followed by an end of year update with some recent Piggybak news.

A Brief Background

Over the many years that End Point has been around, we've amassed a large amount of experience in working with various ecommerce frameworks, open source and proprietary. A large portion of End Point's recent development work (we also offer database, hosting, and Liquid Galaxy support) has been with Interchange, a Perl based open source ecommerce framework, and Spree, a Ruby on Rails based open sourced ecommerce framework. Things came together for Piggybak earlier this year when a new client project prompted the need for a more flexible and customizable Ruby on Rails ecommerce solution. Piggybak also leveraged earlier work that I did with light-weight Sinatra-based cart functionality.

Jump ahead a few months, and now Piggybak is a strong base for an ecommerce framework with several extensions to provide advanced ecommerce features. Some of the features built and already reported on were real-time shipping lookup (USPS, UPS, and FedEx support), improvement of the Piggybak installation process, gift certificate, discount, and bundle discount support.

Recent Progress

Since the last general update, we've tackled a number of additional changes:

• SSL support: The Piggybak core now supports SSL for the checkout, which leverages the lightweight Rails gem rack-ssl-enforcer. A Piggybak config variable specifying that checkout should be secure must be set to true in the main Rails application, which triggers that a specific set of pages should be secure. This configuration is not ideal to use if the main Rails application requires more complex management of secure pages.
• Minor bug fixes & cleanup: The updates below include minor refactoring and/or bug fixes to the Piggybak core:
  
  • Moved order confirmation outside of controller, to minimize failure of order processing if the email confirmation fails.
  • RailsAdmin DRY cleanup
  • Abilities (CanCan) cleanup to require less manual coding, which simplifies the code required in the CanCan model.
  • Breakdown of orders/submit.html.erb, which allows for easier override of checkout page elements.
  • Tax + coupons bug fixes.
  • RailsAdmin upgrade to updated recent versions.
• Heroku tutorial: Piggybak support in Piggybak was described in this blog article.
• Advanced taxonomy or product organization: An extension for advanced product organization (e.g. categories, subcategories) was released, but we still plan to add more documentation regarding its functionality and use.
• Bundle discount support: Another extension for bundle discount support was released. Bundle discount offers the ability to give customer discounts when a bundle or set of products has been added to the cart. Barrett shared his experiences in creating this extension in this article.
• Fancy jQuery tour: I wrote about a new Piggybak demo tour that I created for checking out the features of Piggybak.
• Advanced product optioning: Another extension for advanced product option support (e.g. size, color) was released a couple of months ago, but this recent article provides more documentation on its functionality and use.

What's Next?

At this point, one of our big goals is to grow the Piggybak portfolio and see many of the extensions in action. We'd also like to improve the Piggybak core and extension documentation to help get folks up and running on Piggybak quickly. In addition to documentation and portfolio growth, additional features we may focus on are:

• Product reviews & ratings support
• Saved address/address book support
• Wishlist, saved cart functionality

A few large features that are on our wishlist that may need client sponsorship for build-out are:

• Multiple shipping addresses per order: This allows for users to select multiple shipping addresses per order. I implemented this functionality for Paper Source just over a year ago. This would likely be developed in the form of an extension that requires several non-trivial Piggybak core overrides.
• Subscription support: The Piggybak Google Group has expressed interest in subscription support, which also is not trivial.
• Point-based credit support
• Multi-store architecture: End Point is very familiar with multi-store architecture, which allows multiple stores to be support via one code base. I shared some of the options in this blog article.
• One deal at a time support: This is another popular feature that End Point has been involved with for Backcountry.com sites Steep and Cheap, WhiskeyMilitia.com, and Chainlove.com.

Get Involved

If you are interested in helping develop Piggybak, don't hesitate to jump on the Piggybak google group or tackle one of the Piggybak GitHub issues.

If you are interested in getting your site up and running using Piggybak, contact End Point right now!

Announcing Ruby gem: email_verifier

How many times have you tried to provide a really nice validation solution for our fields containing user emails? Most of the time - the best we can come up with is some long and incomprehensible Regex we find on StackOverflow or somewhere else on the Internet.

But that's really only a partial solution. As much as email format correctness is a tricky thing to get right using regular expressions, it doesn't provide us with any assurance that user entered email address in reality exists.

But it does a great job at finding out some typos and misspellings.. right?

Yes - but I'd argue that it doesn't cover full range of that kind of data entry errors. The user could fill in 'whatever' and traditional validation through regexes would do a great job at finding out that it's not really an email address. But what I'm concerned with here are all those situations when I fat finger kaml@endpoint.com instead of kamil@endpoint.com.

Some would argue at this point that it's still recoverable since I can find out about the error on the next page in a submission workflow, but I don't want to spend another something-minutes on going through the whole process again (possibly filling out tens of form fields along the way).

And look at this issue from the point of view of a web application owner: You'd like to be sure that all those leads you have in your database point to some real people and that some percentage of them will end up paying you at some point real money, making you a living. What if even 10% of email addresses invalid (being valid email addresses but pointing to no real mailboxes) due to user error? What would that potentially mean to you in cash?

The Solution

Recently, I faced this email validation question for mobixa.com. (By the way. if you own a smart phone that you'd like to sell - there is no better place than mobixa.com to do it!)

The results of my work, I'd like to announce here and now. Please give a warm welcome to a newborn citizen of RubyGems society: email_verifier

How does it work?

Email verifier takes a different approach to email validation. Instead of checking just the format of given address in question - it actually tries to connect with a mail server and pretends to send a real mail message. We can call it 'asking mail server if recipient exists'.

How to use it?

Add this line to your application's Gemfile:

gem 'email_verifier'

And then execute:

$ bundle

Or install it yourself as:

$ gem install email_verifier

Some SMTP servers will not allow you to check if you will not present yourself as some real user. So first thing you'd need to set up is to put something like this either in initializer or in application.rb file:

EmailVerifier.config do |config|
  config.verifier_email = "realname@realdomain.com"
end

Then add this to your model:

validates_email_realness_of :email

Or - if you'd like to use it outside of your models:

EmailValidator.check(youremail)

This method will return true or false, or - will throw exception with nicely detailed info about what's wrong.

Read More about the extension at Email verifier RDoc or try to sell your smartphone back here at Mobixa.com.

SFTP virtual users with ProFTPD and Rails: Part 1

I recently worked on a Rails 3.2 project that used the sweet PLupload JavaScript/Flash upload tool to upload files to the web app. To make it easier for users to upload large and/or remote files to the app, we also wanted to let them upload via SFTP. The catch was, our users didn't have SFTP accounts on our server and we didn't want to get into the business of creating and managing SFTP accounts. Enter: ProFTPD and virtual users.

ProFTPD's virtual users concept allows you to point ProFTPD at a SQL database for your user and group authentication. This means SFTP logins don't need actual system logins (although you can mix and match if you want). Naturally, this is perfect for dynamically creating and destroying SFTP accounts. Give your web app the ability to create disposable SFTP credentials and automatically clean up after the user is done with them, and you have a self-maintaining system.

Starting from the inside-out, you need to configure ProFTPD to enable virtual users. Here are the relevant parts from our proftpd.conf:

##
# Begin proftpd.conf excerpt. For explanation of individual config directives, see the 
# great ProFTPD docs at http://www.proftpd.org/docs/directives/configuration_full.html
##
DefaultServer off
Umask 002
AllowOverwrite on

# Don't reference /etc/ftpusers
UseFtpUsers off



# Enable SFTP
SFTPEngine on

# Enable SQL based authentication
SQLAuthenticate on

# From http://www.proftpd.org/docs/howto/CreateHome.html
# Note that the CreateHome params are kind of touchy and easy to break.
CreateHome on 770 dirmode 770 uid ~ gid ~

# chroot them to their home directory
DefaultRoot ~

# Defines the expected format of the passwd database field contents. Hint: An
# encrypted password will look something like: {sha1}IRYEEXBUxvtZSx3j8n7hJmYR7vg=
SQLAuthTypes OpenSSL

# That '*' makes that module authoritative and prevents proftpd from
# falling through to system logins, etc
AuthOrder mod_sql.c*

# sftp_users and sftp_groups are the database tables that must be defined with
# the proceeding column names. You can have other columns in these tables and
# ProFTPD will leave them alone. The sftp_groups table can be empty, but it must exist.
SQLUserInfo sftp_users username passwd uid sftp_group_id homedir shell
SQLGroupInfo sftp_groups name id members

SFTPHostKey /etc/ssh/ssh_host_rsa_key
SFTPHostKey /etc/ssh/ssh_host_dsa_key

SFTPCompression delayed
SFTPAuthMethods password
RequireValidShell no

# SQLLogFile is very verbose, but helpful for debugging while you're getting this working
SQLLogFile /var/log/proftpd_sql.sql

## Customize these for production
SQLConnectInfo database@localhost:5432 dbuser dbpassword

# The UID and GID values here are set to match the user that runs our web app because our
# web app needs to read and delete files uploaded via SFTP. Naturally, that is outside
# the requirements of a basic virtual user setup. But in our case, our web app user needs
# to be able to cd into a virtual user's homedir, and run a `ls` in there. Also, note that
# setting these two IDs here (instead of in our sftp_users table) *only* makes sense if
# you are using the DefaultRoot directive to chroot virtual users.
SQLDefaultUID  510
SQLDefaultGID  500

The CreateHome piece was the trickiest to get working just right for our use-case. But there are two reasons for that; we needed our web app to be able to read/delete the uploaded files, and we wanted to make ProFTPD create those home directories itself. (And it only creates that home directory once a user successfully logs in via SFTP. That means you can be more liberal in your UI with generating credentials that may never get used without having to worry about a ton of empty home directories lying about.)

That's it for the introductory "Part 1" of this article. In Part 2, I'll show how we generate credentials, the workflow behind displaying those credentials, and our SftpUser ActiveRecord model that handles it all. In Part 3, I'll finish up by running through exactly how our web app accesses these files, and how it cleans up after it's done.

Advanced Product Options (Variants) in Piggybak

About a month ago, Tim and I developed and released a Piggybak extension piggybak_variants, which provides advanced product optioning (or variant) support in Piggybak. Piggybak is an open source Ruby on Rails ecommerce platform developed and maintained by End Point. Here, I discuss the background and basics of the extension.

Motivation & Background

The motivation for this extension was the common ecommerce need for product options (e.g. size, color), where each variation shares high-level product information such as a title and description, but variants have different options, quantities available, and prices. Having been intimately familiar with Spree, another open source Ruby on Rails ecommerce framework, we decided to borrow similarities of Spree's product optioning data model after seeing its success in flexibility over many projects. The resulting model is similar to Spree's data model, but a bit different due to the varied nature in Piggybak's mountability design.

Spree's data model for advanced product optioning. A product has many variants. Each variant has and belongs to many option values. A product also has many options, which define which option values can be assigned to it.

Piggybak Variants Data Model

Option configuration data model in Piggybak

The data model starts with option configurations, option configurations are created and specify which class they belong to. For example, a Shirt model may have options Size and Color, and this would be stored in the option configurations table. In this case, an option will have a name (e.g. Size and Color) and a position for sorting (e.g. 1 and 2). The option configuration will reference an option and assign a klass to that option (in this case Shirt). Another example of option configurations may be a Picture Frame, that has option configurations for Dimensions and Finish.

Option value configuration in Piggybak

After option configurations are defined, one will define option values for each option configuration. For example, option values will include Red, Blue, and Green for the option Color with position 1, 2, and 3. And option values will include Small, Medium, and Large with positions 1, 2, and 3 for the option Size.

After options, option configurations, and option values are defined, we are ready to create our variants. Per the above data model, a variant has and belongs to many option_values_variants (and must have one value per option). In our Shirt example, a variant must have one Color option value and one Size option value assigned to it through the option_values_variants table. A variant belongs to a specific sellable item (Shirt) through a polymorphic relationship, which is consistent with Piggybak's mountability design to allow different classes to be sellable items. Finally, a variant has_one piggybak_sellable and accepts piggybak_sellable attributes in a nested form, which means that a variant has one sellable which contains quantity, pricing, and cart description information. What this gives us is a sellable item (Shirt) with many variants where each variant has option values and each variant has sellable information such as quantity available, price, and description in cart. Below I'll provide a few screenshots of what this looks like in the admin and front-end interface.

How to Use the Plugin

To install the extension, the following steps must be applied:

1. Add the gem to the Gemfile and run bundle install
2. Install and run the extension rake piggybak_variants:install:migrations and rake db:migrate
3. Add acts_as_sellable_with_variants to any model that should have variants. You may need to add appropriate attr_accessible settings in your model as well, depending on your attribute accessibility settings.
4. In the admin, define option configurations and option values for each option, then create variants for your sellable instances.
5. Finally, add <%= variant_cart_form(@instance) %> to your sellable item's show page to render the cart form.

These steps are similar to Piggybak's core behavior for adding non-variant sellable items.

Screenshots

The Piggybak demo uses this extension for selling several product options of photography frames. The images and captions below represent the variants extension for this use case.

The Frame class has two options assigned to it (Frame Size and Frame Finish). Since Frame Size has a position equal to one and Frame Finish has a position equal to two, Frame Size will show as the first option on the product page.

The Frame Finish option is assigned to the Frame class and it has four option values (Black, Cherry, Bronze, and Iron).

On the Frame edit page, 8 variants are created to represent the combinations of 2 Frame Sizes and 4 Frame Finishes.
Each variant has pricing, quantity, and cart description information, as well as additional sellable fields.

And the product page shows the options and option values for that item, displayed based on Position and Size data.
When each option value is triggered, appropriate pricing information is displayed.

Conclusion

The goal of this extension was to provide variant functionality that is not necessarily required to be used with Piggybak. Piggybak can still be leveraged without this extension to provide simple single product option add to cart functionality. The Piggybak cart only examines what elements are in the cart based on the sellable_id and the quantity, which is the driving force of the core Piggybak architecture as well as this extension.

Stay tuned for additional updates to the Piggybak Ruby on Rails Ecommerce platform or contact End Point today to start leveraging our Ruby on Rails ecommerce expertise on your next project!

Piggybak Extensions: A Basic How-To Guide

This article outlines the steps to build an extension for Piggybak. Piggybak is an open-source Ruby on Rails ecommerce platform created and maintained by End Point. It is developed as a Rails Engine and is intended to be mounted on an existing Rails application. If you are interested in developing an extension for Piggybak, this article will help you identify the steps you need to take to have your extension leveraging the Piggybak gem, and integrating smoothly into your app.

Introduction

The Piggybak platform is lightweight and relies on Rails meta-programming practices to integrate new extensions. The best references to use alongside your development should be the previously developed extensions found here:

• Bundle Discounts
• Coupons
• Variants
• Taxonomy
• Real-time Shipping

It is likely that your extension will tie into the admin interface. Piggybak utilizes the RailsAdmin gem for its admin interface.

Setting up the Development Environment

A convenient way to start building out your extension is to develop against the demo app found here. The demo app utilizes the Piggybak gem and comes with sample data to populate the e-commerce store.

The Piggybak demo app sample data is exported for a PostgreSQL database. To use this data (suggested) you should be prepared to do one of the following:

• be using PostgreSQL and understand how to work with the existing data dump
• transform this data dump to another database format that fits your database flavor of choice
• ignore the sample data and create your own

Creating the Extension (Gem, Engine)

In a folder outside of the project utilizing the Piggybak gem, create a mountable rails engine:

$ rails plugin new [extension_name] --mountable

The "mountable" option makes you engine namespace-isolated.

Next, update your app's Gemfile to include the extension under development

gem "piggybak_new_extension", :path => "/the/path/to/the/extension"

Run bundle install to install the extension in your application and restart your application.

Special Engine Configuration

Your extension will rely on the engine.rb file to integrate with Piggybak. A sample engine.rb for the piggybak_bundle_discount can be found here. Let's go over this file to get a clue of how bundle discounts are served as an extension in Piggybak.

Make sure you are requiring any of your classes at the top of your engine.rb file, e.g.:

require 'piggybak_bundle_discounts/order_decorator'

The code below is decorating the Piggybak::Order class, which is a helpful pattern to use when you wish to enhance class capabilities across engines. In the bundle discount case, the decorator adds several active record callbacks.

config.to_prepare do
  Piggybak::Order.send(:include, ::PiggybakBundleDiscounts::OrderDecorator)
end

An order is comprised of many line items, which are used to calculate the balance due. More information on the line item architecture is described here. If your extension needs to register new line item types to the order, you may use something similar to the following code to set up the information regarding this new line item type.

config.before_initialize do
  Piggybak.config do |config|
    config.extra_secure_paths << "/apply_bundle_discount"
    config.line_item_types[:bundle_discount] = { 
      :visible => true,
      :allow_destroy => true,
      :fields => ["bundle_discount"],
      :class_name => "::PiggybakBundleDiscounts::BundleDiscount",
      :display_in_cart => "Bundle Discount",
      :sort => config.line_item_types[:payment][:sort]
    } 
    config.line_item_types[:payment][:sort] += 1
  end
end

Does your extension need client side support? Piggybak utilizes the asset pipeline so you will need to register your assets here to have them pre-compiled.

initializer "piggybak_bundle_discounts.precompile_hook" do |app|
  app.config.assets.precompile += ['piggybak_bundle_discounts/piggybak_bundle_discounts.js']
end

Finally, since Piggybak utilizes RailsAdmin for its admin system, we need to register the models as following the RailsAdmin documentation.

initializer "piggybak_bundle_discounts.rails_admin_config" do |app|
  RailsAdmin.config do |config|
    config.model PiggybakBundleDiscounts::BundleDiscount do
      navigation_label "Extensions"
      label "Bundle Discounts"

      edit do
        field :name
        field :multiply do 
          help "Optional"
        end 
        field :discount
        field :active_until
        field :bundle_discount_sellables do 
          active true
          label "Sellables"
          help "Required"
        end
      end
    end

    config.model PiggybakBundleDiscounts::BundleDiscountSellable do
      visible false
      edit do
        field :sellable do          
          label "Sellable"
          help "Required"
        end
      end
    end
  end
end

What else?

From here, extension development can follow standard Rails engine development, which allows for support of its own models, controllers, views, and additional configuration. Any database migrations inside an extension must be copied to the main Rails application to be applied.

You may also need to be aware of how Piggybak integrates with CanCan to ensure that CanCan permissions on your extension models are set correctly.

End Point created and maintains Piggybak project. Much of the inspiration for Piggybak comes from our expert engineers who have ecommerce experience working and contributing to platforms such as Spree, RoR-e, and Interchange. If you are interested in talking with us about your next ecommerce project, or have an ecommerce project that needs support, contact us today.

Interactive Piggybak Demo Tour

A new interactive tour of Piggybak and the Piggybak demo has been released at piggybak.org. Piggybak is an open source Ruby on Rails ecommerce framework built as a Rails 3 engine and intended to be mounted on existing Rails applications.

The tour leverages jTour (a jQuery plugin) and guides you through the homepage, navigation page, product page, cart and checkout pages, gift certificate page, advanced product option page, and WYSIWYG driven page.The tour also highlights several of the Piggybak plugins available and installed into the demo such as plugins that introduce advanced product navigation, advanced product optioning, and gift certificate functionality. Below are a few screenshots from the demo.

An interesting side note of developing this tour is that while I found many nice jQuery-driven tour plugins available for free or at a small cost, this jQuery plugin was the only plugin offering decent multi-page tour functionality.

Here is the starting point of Piggybak tour.	The Piggybak tour adds an item to the cart during the tour.
The Piggybak tour highlights advanced product navigation in the demo.	The Piggybak tour highlights features and functionality on the one-page checkout.

If you'd like to check out the interactive tour, visit the Piggybak demo page here and click on the link to begin the tour! Or contact End Point right now to get started on your next Ruby on Rails ecommerce project!

Mobixa: A Client Case Study

A few weeks ago we saw the official (and successful!) website launch for one of our clients, Mobixa. Mobixa will buy back your used iPhones and/or provide you with information about when you should upgrade your existing phone and sell it back. Right now, Mobixa is currently buying back iPhones and advising on iPhones and Androids. End Point has been working with Mobixa for several months now. This article outlines some of the interesting project notes and summarizes End Point's diverse skillset used for this particular website.

Initial Framework

Mobixa initially wanted a an initial proof of concept website without significant investment in development architecture because the long-term plan and success was somewhat unknown at the project unset. The initial framework comprised of basic HTML combined with a bit of logic driven by PHP. After a user submitted their phone information, data was sent to Wufoo via a Wufoo provided PHP-based API, and data was further handled from Wufoo. Wufoo is an online form builder that has nice export capabilities, and painlessly integrates with MailChimp.

This initial architecture was suitable for collecting user information, having minimal local database needs and allowing external systems (e.g. Wufoo, MailChimp) to handle much of the user logic. However, it became limiting when the idea of user persistence came into play – the long-term goal will be to allow users to modify previous submissions and look up their order information, essentially a need for basic user account management functionality. For that reason, we made a significant switch in the architecture, described below.

Framework #2: Rails 3

Because of the limiting nature of a database-less application with externally managed data and as business needs for users increased, we decided to make the move to Rails. End Point has a large team of Rails developers, Rails is a suitable framework for developing applications efficiently, and we are experienced in working with Rails plugins such as RailsAdmin, Devise, and CanCan, which immediately provide a configurable admin interface, user authentication, and user authentication to the application. In the process of moving to Rails, we eliminated the middle-man Wufoo to integrate with the shipping fulfillment center and MailChimp directly.

The current Mobixa site runs on Rails 3, Nginx and Unicorn backed by PostgreSQL, leverages End Point's DevCamps to allow multiple developers to simultaneously add features and maintain the site painlessly, and uses RailsAdmin, Devise, and CanCan. It features a responsive design and uses advanced jQuery techniques. The focus of the site is still a simple HTML page that passes user-entered information to the local database, but several user management features have been added as well as the ability to sell back multiple phones at a time.

MailChimp Integration

In my search for a decent Rails MailChimp integration gem, I found gibbon. Gibbon is fairly simple - it's an API wrapper for interacting with MailChimp. Any API capabilities and methods available in MailChimp can be called via Gibbon. The integration looks something like this:

# user model 
def update_mailchimp
  gb = Gibbon.new(*api_key*, { :timeout => 30 })

  info = gb.list_member_info({ :id => *list_id*, :email_address => self.email })

  if info["success"] == 1
    gb.listUpdateMember({ :id => *list_id*,
                          :email_address => self.email,
                          :merge_vars => self.mailchimp_data })
  else
    gb.list_subscribe({ :id => *list_id*,
                        :email_address => self.email,
                        # additional new user arguments #
                        :merge_vars => self.mailchimp_data })
  end
end

The above method instantiates a connection to Mailchimp and checks if the user is already subscribed to the Mailchimp list. If the user is subscribed, the listUpdateMember method is called to update the user subscription information. Otherwise, list_subscribe is called to add the user to the Mailchimp list.

What's Next?

In addition to expanding the product buyback capabilities, we expect to integrate additional features such as external-API driven address verification, social media integration, referral management, and more advanced user account management features. The project will continue to involve various members of our team such as Richard, Greg D., Tim, Kamil, Josh W. and me.

Rails: Devise and Email Capitalization

This week, I found a bug for one of our Rails clients that was worth a quick blog post. The client website runs on Rails 3.2.8 with ActiveRecord and PostgreSQL, uses RailsAdmin for an admin interface, Devise for user authentication, and CanCan for user authorization. Before we found the bug, our code looked something like this:

class SomeController < ApplicationController
  def some_method
    user = User.find_or_create_by_email(params[:email])
    # do some stuff with the user provided parameters
    if user.save
      render :json => {}
    else
      render :json => {}, :status => 500
    end
  end
end

It's important to note that the 500 error wasn't reported to the website visitor - there were no visible UI notes to indicate the process had failed. But besides that, this code looks sane, right? We are looking up or creating a user from the provided email, updating the user parameters, and then attempting to save. For the most part, this worked fine, until we came across a situation where the user data was not getting updated properly.

Looking through the logs, I found that the user experiencing the bug was entering mixed caps emails, for example, Steph@endpoint.com. Let's walk through the code in this scenario:

First, a new user is created because there is no user in the system with the exact email Steph@endpoint.com. However, a user does exist in the system tied to steph@endpoint.com.

user = User.find_or_create_by_email(params[:email]) # with "Steph@endpoint.com" 

No problems here:

# do some stuff with the user provided parameters

Below is where the issue is coming up. Devise, our user authentication gem, automatically downcases (lowercases) all emails when they are stored in the database. There is already a user tied to steph@endpoint.com, so user.save fails, a 500 error is thrown, but as an end-user, I don't see anything to indicate that my AJAX call failed.

if user.save

The moral of this story is that it's important to a) understand how plugins manipulate user data automatically (in this case Devise automatically filters the email) and b) test a variety of use cases (in this case, we hadn't considered testing mixed caps emails). Our updated code looks something like this, which downcases emails and upon failure, adds more to the logs for additional unexpected user update failures:

class SomeController < ApplicationController
  def some_method
    user = User.find_or_create_by_email(params[:email].downcase)
    # do some stuff with the user provided parameters
    if user.save
      render :json => {}
    else
      render :json => {}, :status => 500
      Rails.logger.warn "USER ERROR: #{user.errors.full_messages} #{user.attributes.inspect}"
    end
  end
end

Piggybak on Heroku

Several weeks ago, we were contacted through our website with a request for Heroku support on Piggybak. Piggybak is an open source Ruby on Rails ecommerce platform developed and maintained by End Point. Piggybak is similar to many other Rails gems in that it can be installed from Rubygems in any Rails application, and Heroku understands this requirement from the application’s Gemfile. This is a brief tutorial for getting a Rails application up and running with Piggybak. For the purpose of this tutorial, I’ll be using the existing Piggybak demo for deployment, instead of creating a Rails application from scratch.

a) First, clone the existing Piggybak demo. This will be your base application. On your development machine (local or other), you must run bundle install to get all the application’s dependencies.

b) Next, add config.assets.initialize_on_precompile = false to config/application.rb to allow your assets to be compiled without requiring creating a local database.

c) Next, compile the assets according to this Heroku article with the command RAILS_ENV=production bundle exec rake assets:precompile. This will generate all the application assets into the public/assets/ directory.

d) Next, add the assets to the repo by removing public/assets/ from .gitignore and committing all modified files. Heroku’s disk read-only limitation prohibits you from writing public/assets/ files on the fly, so this is a necessary step for Heroku deployment. It is not necessary for standard Rails deployments.

e) Next, assuming you have a Heroku account and have installed the Heroku toolbelt, run heroku create to create a new Heroku application.

f) Next, run git push heroku master to push your application to your new Heroku application. This will push the code and install the required dependencies in Heroku.

g) Next, run heroku pg:psql, followed by \i sample.psql to load the sample data to the Heroku application.

h) Finally, run heroku restart to restart your application. You can access your application through a browser by running heroku open.

That should be it. From there, you can manipulate and modify the demo to experiment with Piggybak functionality. The major difference between Heroku deployment and standard deployment is that all your compiled assets must be in the repository because Heroku cannot write them out on the fly. If you plan to deploy the application elsewhere, you will have to make modifications to the repository regarding public/assets.

A full set of commands for this tutorial includes:

# Clone and set up the demo app
git clone git://github.com/piggybak/demo.git
bundle install
# add config.assets.initialize_on_precompile = false
# to config/application.rb

# Precompile assets and add to repository
RAILS_ENV=production bundle exec rake assets:precompile
# edit .gitignore here to stop ignoring public/assets/
git add .
git commit -m "Heroku support commit."

# Deploy to Heroku
heroku create
git push heroku master
heroku pg:psql
>> \i sample.psql
heroku restart
heroku open

Piggybak: Roadmap Status Update

About a month ago, I shared an outline of the current Piggybak Roadmap. Piggybak is an open-source Ruby on Rails ecommerce platform created and maintained by End Point. It is developed as a Rails Engine and is intended to be mounted on an existing Rails application. Over the last month, Tim and I have been busy at work building out features in Piggybak, and completing refactoring that opens the door for better extension and feature development. Here's a summary of the changes that we've finished up.

Real-time Shipping Lookup

One of our Piggybak clients already had integrated USPS and UPS shipping, but we decided to extract this and combine it with FedEx, to offer real-time shipping lookup shipping in Piggybak. This extension leverages Shopify's open-source ActiveShipping Ruby gem. When you are ready to get your Piggybak store up and running, you can include this new extension and configure USPS, UPS, and FedEx real-time shipping lookup immediately.

Installer process

Tim updated the installation process to be more streamlined. The previous installation process was a bit crufty and required changes to your Gemfile, routes, layouts, and precompiled assets. Tim described the installation work in this article.

Rename Variant to Sellable

A minor but notable change that happened in the last month was the change of "variant" to "sellable". Any model in a Rails application, now can be extended with the class method acts_as_sellable, which will allow that item to be managed as a sellable item and be a purchaseable item.

Variants Extension

Tied directly to the variant to sellable change, we developed a new extension to provide advanced variant support in Piggybak. The advanced variant data model has similarities to Spree's data model, one that we have observed as a successful feature of Spree. The basic principles are that you assign specific options to sellable items (e.g. size and color), and then you assign option values to those options (e.g. red and blue for size, large and small for color). Then, for each sellable item, you can define many variants each with a different combination of options, each with a unique sku, quantity on hand, cart description, and price. The user sees these options on the product detail page, and selects option values to add items to the cart.

Advanced production optioning support in Piggybak: In this screenshot, options for frame size and frame finish are provided.
Each variant has individual pricing, quantity on hand, and a description in the cart.

Line Item Rearchitecture

I also spent a good amount of time rearchitecting line item associations to orders, where a line item now represents all monetary items in an order (sellable, payment, tax item, shipment, etc.). This results in a more simplified order total and balance due calculation, as well as allows for extensions to introduce custom line items that are included in order calculations without order processing code changes. This significant change is described in this article.

Piggybak Coupons

The line item rearchitecture work was done in tandem with development of a Piggybak coupon extension. The extension includes support for defining discount type (percent, dollar, or free shipping), discount amount (for percent and dollar), minimum cart total, expiration date, allowed number of uses.

Coupon support in Piggybak: Coupon application on the checkout happens via AJAX and
is displayed in the order totals calculations shown in the screenshot.

Gift Certificate Support

Finally, one of the recent extensions completed was development of a gift certificate extension. A gift certificate can be purchased at various increments and applied on the checkout page to an order via an AJAX call. Gift certificates may also be purchased and redeemed in the Piggybak admin.

Gift Certificate support in Piggybak: Gift certificate application on the checkout happens via AJAX and is displayed in the order totals calculations shown in the screenshot. In this case, the gift certificate covers the entire order.

Minor Bug Fixes, Refactoring and Feature Development

Several bug fixes and minor refactoring was applied during development of these features, including but not limited to:

• attr_accessible updates to support Rails 3 mass assignment attributes
• Improved inventory management on the admin side
• Minor refactoring to introduce Proxy Association extensions
• Removal of jeweler, and move to standard Rails engine architecture
• Added functionality to support copying a billing address to shipping address in admin.
• Added logic to enforce one payment method be added at a time via admin.
• JavaScript-based validation on the checkout.
• Stripe payment gateway support via an extension.

What's Next?

If we take a look the roadmap list a month ago, we can cross several items off from the list:

• Realtime Shipping with USPS, UPS, and Fedex support
• Improvement of Piggybak installation process
• Advanced Product Optioning Support
• Line Item Rearchitecture to support future work on Gift Certificates, Discounts
• Gift Certificate, Discount Support
• Advanced Taxonomy
• Reviews & Ratings

A few new things have recently been added to the list:

• Add SSL support in core
• Create Heroku deployment tutorial
• Saved cart, Wishlist support
• Saved address support

Our goal for the immediate future is to focus on development of the most common ecommerce features.

All of the features described in this article are active on the Piggybak demo. Check it out now! And of course, contact End Point today if you'd like to get started on a Ruby on Rails Piggybak project!

How to Build a Command Line Executable Installer with Rubygems and Thor

Gems for Rails often need the user to do something more for installation than just adding the gem to a Gemfile and running bundler install. Sometimes it's a simple matter of copying over some migration files and sometimes it's just setting up a config file, and most of the time these simple installation steps are best handled with a well written installation section in the README file. When the installation process is more complex a long README might not be so enticing to the potential gem user, in a world where everyone has a finger on the back button it's nice to be able to create an installer that allows the user to complete complex installation tasks by executing a one liner and that's where an installer made through Gem executables and Thor can come in handy.

We wanted to make it easier for new users of Piggybak to get started and decided that an installer was the best way to do that. Creating a binary installer that is installed by Rubygems is one of those esoteric things that may not be thought of as one of the core strengths of Rubygems and Rails but it's a bonus to be able to do something like this without a whole lot of fuss.

Creating an installer with Rubygems, and Thor:

1. In your Rails app, create a file in your lib directory that inherits from Thor, this file will house all of your command line actions. Thor is already included as a part of Rails so you don't need to add it to your Gemfile.
2. Inside your Thor subclass, define methods which will in turn become invokable actions from the command line. Installers usually need to copy files around and execute commands, Thor provides a library the covers the most common cases which can be added to your class by including Thor::Actions (A list of the included actions). Have a look at the Piggybak installer class, and you'll see that the Thor actions are not too complicated to understand.
3. Create a bin directory in your Rails directory that will be used to start your Thor class, add a file with the name of your executable which starts your Thor class (details below)
4. Add an "executables" entry for the file in your bin directory to your gemspec file

Add a file to your bin folder than starts Thor

The code below shows the file located in the bin directory and it could act as a template for your own executable. The things to note are the inclusion of the ruby shebang, and the requiring of the piggybak cli class. Finally at the end of the file the start method is sent to the Thor class.

#!/usr/bin/env ruby

require 'rubygems'

begin
  require 'piggybak/cli'
rescue LoadError => e
warn 'Could not load "piggybak/cli"'
  exit -1
end

Piggybak::CLI.start

Add an entry to your gemspec for the executable

Rubygems expects the executable to be in a directory called bin which is in the same directory as the gemspec, if you want to place the executable in a different location you'll need to specify that in your gemspec with a "bindir" entry. (Check the Rubygem docs for a more detailed explanation.)

spec.executables << 'piggybak'

Once that's done your gem is ready to go and can be included inside the Gemfile of a Rails app. When the gem is installed, Rubygems will place a file in your Ruby bin directory that can be invoked via the command line.

$ piggybak install

Association Extensions in Rails for Piggybak

I recently had a problem with Rails named scopes while working on minor refactoring in Piggybak, an open source Ruby on Rails ecommerce platform that End Point created and maintains. The problem was that I found that named scopes were not returning uncommitted or new records. Named scopes allow you to specify ActiveRecord query conditions and can be combined with joins and includes to query associated data. For example, based on recent line item rearchitecture, I wanted order.line_items.sellables, order.line_items.taxes, order.line_items.shipments to return all line items where line_item_type was sellable, tax, or shipment, respectively. With named scopes, this might look like:

class Piggybak::LineItem < ActiveRecord::Base
    scope :sellables, where(:line_item_type => "sellable")
    scope :taxes, where(:line_item_type => "tax")
    scope :shipments, where(:line_item_type => "payment")
    scope :payments, where(:line_item_type => "payment")
  end

However, while processing an order, any uncommited or new records would not be returned when using these named scopes. To work around this, I added the Enumerable select method to iterate over the line items, e.g.:

# Reviewing shipments in an order
order.line_items.select { |li| li.line_item_type == "shipment" }.all? { |s| s.shipment.status == "shipped" }

# Get number of new payments
order.line_items.select { |li| li.new_record? && li.line_item_type == "payment" }.size

Association Extensions

I felt that the above workaround was crufty and not very readable and sent out a request to my coworkers in hopes that there was a solution for improving the readability and clarity of the code. Kamil confirmed that named scopes do not return uncommitted records, and Tim offered an alternative solution by suggesting association extensions. An association extension allows you to add new finders, creators or methods that are only used as part of the association. After some investigation, I settled on the following code to extend the line_items association:

class Piggybak::Order < ActiveRecord::Base
  has_many :line_items, do 
    def sellables
      proxy_association.proxy.select { |li| li.ilne_item_type == "sellable" }
    end
    def taxes
      proxy_association.proxy.select { |li| li.ilne_item_type == "tax" }
    end
    def shipments
      proxy_association.proxy.select { |li| li.ilne_item_type == "shipment" }
    end
    def payments
      proxy_association.proxy.select { |li| li.ilne_item_type == "payment" }
    end
  end
end

The above code allows us to call order.line_items.sellables, order.line_items.taxes, order.line_items.shipments, and order.line_items.payments, which will return all new and existing line item records. These custom finder methods are used during order preprocessing which occurs during the ActiveRecord before_save callback before an order is finalized.

Dynamic Creation

Of course, the Piggybak code takes this a step further because additional custom line item types can be added to the code via Piggybak extensions (e.g. coupons, gift certificates, adjustments). To address this, association extensions are created dynamically in the Piggybak engine instantiation:

Piggybak::Order.class_eval do
  has_many :line_items, do
    Piggybak.config.line_item_types.each do |k, v|
      # k is sellable, tax, shipment, payment, etc.
      define_method "#{k.to_s.pluralize}" do
        proxy_association.proxy.select { |li| li.line_item_type == "#{k}" }
      end
    end
  end
end

Conclusion

The disadvantage to association extensions versus named scopes are that association extensions are not chainable, which means you cannot add methods to the association extension. For example, a named scope may allow you to query order.line_items.sellables.price_greater_than_50 to return committed line items with a price greater than 50, but this functionality would not be possible with association extensions. This is not a limitation in the current code base, but it may become a limitation in the future.

Postgres system triggers error: permission denied

This mystifying Postgres error popped up for one of my coworkers lately while using Ruby on Rails:

ERROR:  permission denied: "RI_ConstraintTrigger_16410" is a system trigger

On PostgreSQL version 9.2 and newer, the error may look like this:

ERROR:  permission denied: "RI_ConstraintTrigger_a_32778" is a system trigger

ERROR:  permission denied: "RI_ConstraintTrigger_c_32780" is a system trigger

I labelled this as mystifying because, while Postgres' error system is generally well designed and gives clear messages, this one stinks. A better one would be something similar to:

ERROR:  Cannot disable triggers on a table containing foreign keys unless superuser

As you can now guess, this error is caused by a non-superuser trying to disable triggers on a table that is used in a foreign key relationship, via the SQL command:

ALTER TABLE foobar DISABLE TRIGGERS ALL;

Because Postgres enforces foreign keys through the use of triggers, and because data integrity is very important to Postgres, one must be a superuser to perform such an action and bypass the foreign keys. (A superuser is a Postgres role that has "do anything" privileges). We'll look at an example of this in action, and then discuss solutions and workarounds.

Note that if you are not a superuser *and* you are not the owner of the table, you will get a much better error message when you try to disable all the triggers:

ERROR:  must be owner of relation foobar

To reproduce the original error, we will create two tables, and then link them together via a foreign key:

postgres=# create user alice;
CREATE ROLE

postgres=# \c postgres alice
You are now connected to database "postgres" as user "alice".

-- Verify that we are not a superuser
postgres=> select usename, usesuper from pg_user where usename = (select current_user);
 usename | usesuper 
---------+----------
 alice   | f

postgres=> create table foo(a int unique);
NOTICE:  CREATE TABLE / UNIQUE will create implicit index "foo_a_key" for table "foo"
CREATE TABLE

postgres=> create table bar(b int);
CREATE TABLE

postgres=> alter table bar add constraint baz foreign key (b) references foo(a);
ALTER TABLE

Let's take a look at both tables, and then try to disable triggers on each one. Because the triggers enforcing the foreign key are internal, they will not show up when we do a \d:

postgres=> \d foo
      Table "public.foo"
 Column |  Type   | Modifiers 
--------+---------+-----------
 a      | integer | 
Indexes:
    "foo_a_key" UNIQUE CONSTRAINT, btree (a)
Referenced by:
    TABLE "bar" CONSTRAINT "baz" FOREIGN KEY (b) REFERENCES foo(a)

postgres=> \d bar
      Table "public.bar"
 Column |  Type   | Modifiers 
--------+---------+-----------
 b      | integer | 
Foreign-key constraints:
    "baz" FOREIGN KEY (b) REFERENCES foo(a)


postgres=> alter table foo disable trigger all;
ERROR:  permission denied: "RI_ConstraintTrigger_41047" is a system trigger

postgres=> alter table bar disable trigger all;
ERROR:  permission denied: "RI_ConstraintTrigger_41049" is a system trigger

If we try the same thing as a superuser, we have no problem:

postgres=# \c postgres postgres
You are now connected to database "postgres" as user "postgres".

postgres=# select usename, usesuper from pg_user where usename = (select current_user);
 usename  | usesuper 
----------+----------
 postgres | t

postgres=# alter table foo disable trigger all;
ALTER TABLE

postgres=# alter table bar disable trigger all;
ALTER TABLE

-- Don't forget to re-enable the triggers!

postgres=# alter table foo enable trigger all;
ALTER TABLE

postgres=# alter table bar enable trigger all;
ALTER TABLE

So, this error has happened to you - now what? Well, it depends on exactly what you are trying to do, and how much control over your environment you have. If you are using Ruby on Rails, for example, you may not be able to change anything except the running user. As you may imagine, this is the most obvious solution: become a superuser and run the command, as in the example above.

If you do have the ability to run as a superuser however, it is usually much easier to adjust the session_replication_role. In short, this disables *all* triggers and rules, on all tables, until it is switched back again. Do NOT forget to switch it back again! Usage is like this:

postgres=# \c postgres postgres
You are now connected to database "postgres" as user "postgres".

postgres=# set session_replication_role to replica;
SET

-- Do what you need to do - triggers and rules will not fire!

postgres=# set session_replication_role to default;
SET

Note: while you can do "SET LOCAL" to limit the changes to the current transaction, I always feel safer to explicitly set it before and after the changes, rather than relying on the implicit change back via commit and rollback.

It may be that you are simply trying to disable one or more of the "normal" triggers that appear on the table. In which case, you can simply disable user triggers manually rather than use 'all':

postgres=# \c postgres alice
You are now connected to database "postgres" as user "alice".

postgres=> \d bar
      Table "public.bar"
 Column |  Type   | Modifiers 
--------+---------+-----------
 b      | integer | 
Foreign-key constraints:
    "baz" FOREIGN KEY (b) REFERENCES foo(a)
Triggers:
    trunk AFTER INSERT ON bar FOR EACH STATEMENT EXECUTE PROCEDURE funk()
    vupd BEFORE UPDATE ON bar FOR EACH ROW EXECUTE PROCEDURE verify_update();

postgres=> alter table bar disable trigger trunk;
ALTER TABLE

postgres=> alter table bar disable trigger vupd;
ALTER TABLE

-- Do what you need to do, then:

postgres=> alter table bar enable trigger trunk;
ALTER TABLE

postgres=> alter table bar enable trigger vupd;
ALTER TABLE

Another option for a regular user (in other words, a non super-user) is to remove the foreign key relationship yourself. You cannot disable the trigger, but you can drop the foreign key that created it in the first place. Of course, you have to add it back in as well:

postgres=# \c postgres alice
You are now connected to database "postgres" as user "alice".

postgres=> alter table bar drop constraint baz;
ALTER TABLE

-- Do what you need to do then:

postgres=> alter table bar add constraint baz foreign key (b) references foo(a);
ALTER TABLE

The final solution is to work around the problem. Do you really need to disable triggers on this table? Then you can simply not disable any triggers. Perhaps the action you are ultimately trying to do (e.g. update/delete/insert to the table) can be performed some other way.

All of these solutions have their advantages and disadvantages. And that's what charts are good for!:

For the rest of this article, we will tie up two loose ends. First, how can we see the triggers if \d will not show them? Second, what's up with the crappy trigger name?

As seen above, the output of \d in the psql program shows us the triggers on a table, but not the internal system triggers, such as those created by foreign keys. Here is how triggers normally appear:

postgres=# \c postgres postgres
You are now connected to database "postgres" as user "postgres".

postgres=# create language plperl;
CREATE LANGUAGE

postgres=# create function funk() returns trigger language plperl as $$ return undef; $$;
CREATE FUNCTION

postgres=# create trigger trunk after insert on bar for each statement execute procedure funk();
CREATE TRIGGER

postgres=# \d bar
      Table "public.bar"
 Column |  Type   | Modifiers 
--------+---------+-----------
 b      | integer | 
Foreign-key constraints:
    "baz" FOREIGN KEY (b) REFERENCES foo(a)
Triggers:
    trunk AFTER INSERT ON bar FOR EACH STATEMENT EXECUTE PROCEDURE funk()

postgres=# alter table bar disable trigger all;
ALTER TABLE

postgres=# \d bar
      Table "public.bar"
 Column |  Type   | Modifiers 
--------+---------+-----------
 b      | integer | 
Foreign-key constraints:
    "baz" FOREIGN KEY (b) REFERENCES foo(a)
Disabled triggers:
    trunk AFTER INSERT ON bar FOR EACH STATEMENT EXECUTE PROCEDURE funk()

Warning: Versions older than 8.3 will not tell you in the \d output that the trigger is disabled! Yet another reason to upgrade as soon as possible because 8.2 and earlier are end of life.

If you want to see all the triggers on a table, even the internal ones, you will need to look at the pg_trigger table directly. Here is the query that psql uses when generating a list of triggers on a table. Note the exclusion based on the tgisinternal column:

SELECT t.tgname, pg_catalog.pg_get_triggerdef(t.oid, true), t.tgenabled
FROM pg_catalog.pg_trigger t
WHERE t.tgrelid = '32774' AND NOT t.tgisinternal
ORDER BY 1;

So in our example table above, we should find the trigger we created, as well as the two triggers created by the foreign key. All of them are enabled. Disabled triggers will show as a 'D' in the tgenabled column. (O stands for origin, and has to do with session_replication_role).

postgres=# select tgname,tgenabled,tgisinternal from pg_trigger 
postgres-#  where tgrelid = 'bar'::regclass;
            tgname            | tgenabled | tgisinternal 
------------------------------+-----------+--------------
 RI_ConstraintTrigger_c_32780 | D         | t
 RI_ConstraintTrigger_c_32781 | D         | t
 trunk                        | D         | f

postgres=# alter table bar enable trigger all;
ALTER TABLE

postgres=# select tgname,tgenabled,tgisinternal from pg_trigger
postgres-#  where tgrelid = 'bar'::regclass;
            tgname            | tgenabled | tgisinternal 
------------------------------+-----------+--------------
 RI_ConstraintTrigger_c_32780 | O         | t
 RI_ConstraintTrigger_c_32781 | O         | t
 trunk                        | O         | f

As you recall, the original error - with the system trigger that had a rather non-intuitive named - looked like this:

ERROR:  permission denied: "RI_ConstraintTrigger_16509" is a system trigger

We can break it apart to see what it is doing. The "RI" is short for "Referential Integrity", and anyone who manages to figure that out can probably make a good guess as to what it does. The "Constraint" means it is a constraint on the table - okay, simple enough. The "Trigger" is a little redundant, as it is extraordinarily unlikely you will ever come across this trigger without some context (such as the error message above) that tells you it is a trigger. The final number is simply the oid of the trigger itself. Stick them all together and you get a fairly obscure trigger name that is hopefully not as mysterious now!

Piggybak Update: Line Item Rearchitecture

Over the last couple of weeks, I’ve been involved in doing significant rearchitecture of Piggybak’s line items data model. Piggybak is an open-source mountable Ruby on Rails ecommerce solution created and maintained by End Point. A few months ago after observing a few complications with Piggybak’s order model and it’s interaction with various nested elements (product line items, shipments, payments, adjustments) and calculations, and after reviewing and discussing these complications with a couple of my expert coworkers, we decided to go in the direction of a uniform line item data model based on our success with this model for other ecommerce clients over the years (whoa, that was a long sentence!). Here, I’ll discuss some of the motiivations and an overview of the technical aspects of this rearchitecture.

Motivation

The biggest drivers of this change were a) to enable more simplified order total calculations based on uniform line items representing products, shipments, payments, etc. and b) to enable easier extensibility or hookability into the order architecture without requiring invasive overrides. For example, the code before for order totals may looked something like this:

order.subtotal = sum of items + sum of adjustments + sum of credits + sum of shipments
order.total_due = sum of items + sum of adjustments + sum of payments + sum of credits + sum of shipments

And after the order calculation, a more simplified version of order total calculation looks like this:

self.subtotal = sum of line item prices that aren't payments
self.total_due = sum of all line items

A related motivation that helped drive this change was to develop several credit-based features for Piggybak such as gift certificates, coupons and bundle discounts to grow the feature set of Piggybak. Rather than requiring complex overrides to incorporate these custom credits to orders, a consistent line item interface supports integration of additional custom line item types.

Data Model Changes

Prior to the rearchitecture, the data model looked like this:

Piggybak data model prior to line item rearchitecture.

Some important notes on this are:

• Line items, payments, shipments and adjustments belong to the order. An order can have many of these elements.
• During order processing, all of these elements had to be processed independently without uniform consistency. An order balance due represented the sum of various charge related elements (items, shipments, tax) minus any payments or adjustments.
• In the case of adjustments, this was a bit tricky because an adjustment could be in the form of a negative or positive amount. This introduced complications in the order calculation process.
• Line items represented products only.

With the rearchitecture, the data model now looks like this:

Piggybak data model after line item rearchitecture.

Important notes on this are:

• In the core Piggybak data model, line item types represent sellable (product), payment, shipment, adjustment, and tax entries.
• Line items can still be related to other elements, such as payment and shipment, but the line item has uniform information such as price and description.
• Line item types are controlled by a Piggybak configuration variable, which allows for Piggybak extensions and the main Rails application to incorporate additional custom line item types.
• Because various calculation methods are applied on each line item type (e.g. payments are charged against credit card, shipping is recalculated with a shipping calculator) the line item order model is amenable to custom preprocessing and processing per line item type. This takes advantage Ruby’s respond_to? method to determine if specific preprocessing or postprocessing methods exist.
• The new architecture also takes advantage of metaprogramming by defining methods dynamically against the line item types. For example, order instance methods "shipping_charge" and "tax_charge" are dynamically created which return the sum of line item prices where the line item type is shipping or tax, respectively.

Coupon Support in Piggybak

Much of the line item rearchitecture work was done in tandem with development of a Piggybak coupon extension, so I’m excited to announce that with this change, we now have another Piggybak extension piggybak_coupons available for use with Piggybak. The piggybak_coupon extensions includes support for defining discount type (percent, dollar, or free shipping), discount amoount (for percent and dollar), minimum cart total, expiration date, allowed number of uses. A coupon may be applied on the checkout page via an AJAX lookup. The piggybak_coupons extension is similar to piggybak in that it must be installed as a gem into the application. It includes it's own migration, model, controller, view, and decorator files.

What's Next?

Introducing this new architecture gives us the abiliity to incorporate new and custom line item processing functionality. Popular line item types that correspond to popular ecommerce features include:

• refunds
• gift certificates
• coupons
• bundle discounts

Less common, but still possible with this new architecture might include:

• custom discounts (e.g. buy one get one free)
• payment via purchase order
• payment via check
• donations

The future for the Piggybak team includes further development of extensions to support some of the common line item type features.

Naturally, there may be a few follow-up incremental improvements since this was a significant change. All of this work is included in the Piggybak gem release version 0.6.2. Read more about Piggybak and check out the demo here, or contact End Point today to get us involved in your next ecommerce project!

How to pick a Ruby gem

RubyGems are one of the big benefits of developing in the Ruby environment as they can provide you with a powerful set of building blocks that were created by some great developers. Earlier in my Ruby career I used to think of RubyGems as a quick way to get some "free" code into my applications and I would spend a tremendous amount of time trying to see what kind of apps I could concoct by stacking gem on top of gem. In practice this turned out to be foolish because rather than gaining a stack of "free" code what I was instead doing was "paying" for each gem by having to learn how each of these gems worked and what kind of assumptions and gotchas they were bringing into my apps. I changed my ideas about gems and now I opt by default to avoid adding gems to my projects, but when I do decide that a piece of functionality might be better served through a gem, I make sure to put potential candidates through a rigorous vetting process.

When looking for a gem the question I keep in mind is, "Does adding this gem to my project benefit me more than just writing these features by hand?" I measure the gem up against some criteria and if the answer remains yes then I'll add the gem.

Criteria for gem evaluation

The first thing I like to do when I find a gem is to check out the README on Github and see if Rails version information is included with the gem installation instructions. This gives me an idea about how up to date the gem is.

2. When was the last commit and how much recent activity does the gem have?

Again I'm trying to make a decision if the gem is still fresh or if it's exceeded its expiration date. The Rails world changes pretty fast and if a gem hasn't been touched in recent months then that's a good indication that the gem is out of date. Unfortunately Rubygems, especially gems for Rails have unwritten expiration dates and an old gem past its prime doesn't automatically get dropped off of github or rubygems.org, instead it just sits around without any activity.

3. Does the code look like what I expect it to?

Generally I can come up with a quick idea about how I think the gem should work and then I'll quickly scan through the gem's contents to see how my idea of the gem and its actual internals match up. If my understanding of how the gem should be and its actual implementation are too far off then this is an indicator that the gem might be more expensive to add in terms of maintenance and the learning I'll need to invest. If the gem's code is extremely complex and I barely understand how it works then that is a red flag that maybe this gem is going to take me on a wild ride.

4. Does it pass its unit tests and do the tests show me how the gem works?

I like to use unit tests as documentation that can show me quickly how a gem is intended to be used. I'll run the unit tests and if they pass that's kind of a gold bond measurement for me that the gem is worth using. I'll still consider using a gem even if the tests fail because sometimes a gem's tests will fail or be difficult to execute and that still won't be a deal breaker for me so long as I can reasonably follow why the tests might be failing.

After a gem has passed its vetting process I'll still keep an awareness about how the gem feels in my app and how much time I'm spending integrating it with what I'm trying to do. If I get the feeling that I'm spending too much time fighting with the gem and that it's not quite fitting with what I'm doing then I will consider pulling the plug and either using a different gem or writing the functionality by hand.

If you liked this discussion of picking out Ruby gems then I encourage you to check out the always excellent Railscasts by RyanB who recently posted a video about this very same topic.

Don't Sleep on Rails 3 SQL Injection Vulnerabilities

SQL injection is a problem that every web developer needs to be aware of when accepting parameters that will during the life of the request be converted into SQL statements. Rails historically has done what it can to mitigate this risk for the developer by providing vehicles for sanitizing parameter inputs at the points when they are being converted for use inside of a SQL statement, however with Rails 3 there are numerous ways to execute a SQL statement against the database and some of these methods are safer than others.

Consider two cases where valid Rails code is vulnerable to SQL injection:

#user inputed parameters
params[:query] = "'robert'; DROP TABLE students; ##"

#CASE 1 - find_by_sql
User.find_by_sql("SELECT * FROM users WHERE (name = '#{params[:query]}'")  ##(BAD BAD BAD)

#generated SQL
SELECT  `users`.* FROM `users`  WHERE (email = 'Robert'); DROP TABLE STUDENTS; ##') ##(THIS STATEMENT WILL DROP TABLE STUDENTS)

The example above shows how find_by_sql can allow parameters submitted by a user to be directly entered into a SQL statement and how an attacker might use the vulnerability to wreak havoc. These types of find_by_sql statements used to be more commonly used in earlier versions of Rails (1.0 - 2.0) and it was through these statements that the Rails community realized that SQL injection was a problem that needed addressing. Here's another example prominent in the early Rails days:

User.find :first, :conditions => "(name = '#{params[:query]}')" ##(BAD BAD BAD)
produces this SQL statement:

#generated SQL
SELECT  `users`.* FROM `users`  WHERE (email = 'Robert'); DROP TABLE STUDENTS; ##') ##(THIS STATEMENT WILL DROP TABLE STUDENTS)

The above example shows a common Rails idiom for performing an ActiveRecord query, as with the previous find_by_sql example the find query here is piping the param in directly and generating the exact same tainted SQL.

Fortunately, Rails core decided to make it easier to just do the right thing and provided ways to pass in parameters by using built-in filters that handle special SQL characters, which will escape ’ , " , NULL character and line breaks. Instead of passing in the parameter directly as a raw string, you can pass in an array to sanitize the tainted strings using the built-in filters:

User.find_by_sql(["SELECT * FROM users WHERE (name = ?)", params])  ##(GOOD)
User.find :first, :conditions => ["(name = '?')", params] ##(GOOD)
#generated SQL
SELECT * FROM users WHERE (name = 'Robert\'); DROP TABLE STUDENTS; ##') (RETURNS NIL)

The distinction in the filtered SQL statement is the escaped single quote right after the t in Robert which prevents the name parameter from terminating and allowing the DROP TABLE STUDENTS from being executed since it remains a part of the string parameter. Additionally, Rails also included these built-in filters automatically when AR queries were called from find_by_something or a conditions hash:

User.find_by_name(params[:query]) ##(GOOD)
#generated SQL
SELECT * FROM `users` WHERE `users`.`name` = 'Robert\'); DROP TABLE STUDENTS; ##' (RETURNS NIL)
User.find :first, :conditions => {:name => params} ##(GOOD)
SELECT `users`.* FROM `users` WHERE `users`.`name` = 'Robert\'); DROP TABLE STUDENTS; ##' (RETURNS NIL)

Rails 3 introduced AREL, which is another way to perform ActiveRecord queries and with it came as well, another way to make the exact same SQL injection mistakes that are already listed above. However having gotten accustomed to looking for SQL injection vulnerabilities in the AR query formats above you might be lulled into thinking that the new and improved ActiveRecord query methods would just magically handle the tainted strings for you and you'd be dead wrong:

User.where("name = '#{params}'") (BAD)
SELECT `users`.* FROM `users`  WHERE (name = 'Robert'); DROP TABLE STUDENTS; ##') ##(THIS STATEMENT WILL DROP TABLE STUDENTS)

The nice thing is that the same fix can also be applied:

User.where(["name = ?", params])
SELECT `users`.* FROM `users`  WHERE (name = ''Robert\'); DROP TABLE STUDENTS; ##'')  ##(RETURNS NIL)

Feature Isolation, an Overview

Yesterday, Brian Buchalter blogged about a recent presentation I did for End Point Rails developers.

While the blog article did a great job of capturing some of the nitty gritty detail from the presentation, I'd like to just followup with a quick overview statement about Feature Isolation. I've also made my slides available for anyone who is interested.

Feature Isolation is what I'm calling a development strategy for adding new features to existing applications. In Rails, I'm utilizing cucumber, a tool for transforming use-case language into ruby code, to specify the requirements and then execute them outside of the Rails environment and away from the complexity of the rest of the application.

Using stubbing and a minimal mock of ActiveRecord (FastModel) I can then begin to design my feature from a more object oriented approach than is typical in Rails development. I can bring in any models, new or existing, that I will need and stub out the interface to the database. Likewise, I can design my classes and their public interface. Getting all my tests to pass from a high level without actually developing the behavior itself allows me to make design decisions quickly and ensure I'm capturing all the requirements in my design.

From there, it's just a matter of removing the stubs and mocks from the tests and then building them out in the application ensuring that I'm still passing from my outer cucumber tests as I go. Eventually, the cucumber tests will drive a browser (using a terminal headless browser called capybara-webkit).

This approach has really disciplined me in how I approach a new feature and helped me to stay focused on building what is needed instead of trying to do too much. It's also centralized my business logic in objects within the application instead of Rails itself.

I'm hoping to get some feedback from the Rails community to improve the process and the tools some more, but having gone through the process several times, I believe it can really help -- especially when dealing with existing complex applications.

Feature Isolation with Mike Farmer

My brilliant co-worker Mike Farmer gave a presentation today talking about a development strategy he calls "Feature Isolation." It involves developing new features on the fringe of your application, isolating it from the complexity of existing code. This allows greater focus on ensuring that your feature is well designed from an object-oriented perspective and that you don't build more than you need.

In order to truly isolate the feature, Mike put together some cucumber tools to allow you to run cucumber without Rails and to create what he calls a "FastModel". The models are fast for two reasons. First, you don't need to load ActiveRecord to get functionality like specifying field names, specifying relationships, or emulating saving records. Second, it let's you to sketch out a design for your class while the cost of change is very very low.

An Example: Product Variants

Here's an example of a tight little feature and step set for showing shoppers a comparison of product variants.

Feature: As a shopper, I want to compare the variants of a product

  Background:
    Given there is a product named "Product A"
    And it has some variants with various options
    And I am on the comparison page for "Product A"


  Scenario: The shopper sees a comparison chart of Variant A
    When "Variant A" has options "a,b,c"
    Then the comparison chart header should have options "a,b,c"
    And the comparison grid should have 3 checkboxes

When a feature request comes in you write out your cucumber scenarios and do the usual sign-off from the product manager. Nothing new or exciting here. Next, implement failing step files; yes of course they should fail first, this is TDD right? Mike has a strong opinion about what your features and steps should look like: short and specific. He's found that if you're writing too much setup, your feature is not well isolated or that perhaps this feature needs to be broken into several smaller ones.

Now that we have the feature in place, we can create the steps.

Given /^there is a product named "([^"]*)"$/ do |product_name|
    @product = Product.create(:name => product_name)
  end

  Given /^it has some variants with various options$/ do
    @variant_a = Variant.create(:sku => 'Variant A', product => @product)
    VariantOption.create(:name => "Option A", :variant => @variant_a)
    #... same thing for B, C, D
  end

  Given /^I am on the comparison page for "([^"]*)"$/ do |product|
    # UI Action
    # visit product_path(@product.permalink)
    @current_product = Product.find product
  end

  When /^"([^"]*)" has options "([^"]*)"$/ do |variant, options|
    @current_variant = Variant.find variant
    opts = options.split(",")
    #... find the options
    @options = [@option_a, @option_b, @option_c, @option_d]
  end

  #... more steps

Keep in mind that at first, that these steps are going to be powered by FastModel and not Rails. Let's see how we use cucumber_tools to do this.

Magic Sauce: Rapid Class Design

After compeleting a RED TDD cycle, Mike's strategy really takes center stage. Instead of creating classes, he uses FastModel to define the models, including existing models, fields and relationships needed for his feature. He goes even farther and stubs out the interface which is needed to satisify the cucumber test. Let's look at a specific example.

class Product < FastModel
    fields :name
    has_many :variants
  end

  class Variant < FastModel
    fields :name
    has_many :variant_options
    belongs_to :product
  end

  class VariantOption < FastModel
    fields :name
    belongs_to :variant
  end

  class VariantCompareChart; end

FastModel

Without writing migrations, or even loading Rails, you can use FastModel to think about some of the basics of your ActiveRecord models. This let's you focus on the design of your classes, by iterarting and testing quickly. These FastModels can actually go in the step file making it easier to stay focused on what matters: class design.

Why stub in a cucumber feature?

It might seem strange to start stubbing out your class interface inside a cucumber feature. Mike understands this and just like the FastModels above, he isn't expecting these stubs to stay. He's using it as a tool to drive very quick class interface modeling. It's a great way to stay inside one test file and keep the cost of iterating through the class design low. This is a central part of the magic of his strategy. By keeping the design of class' interface directly linked to the design of the feature's steps, we ensure we only design and test our public interface. Let's see it in action:

Then /^the comparison chart header should have options "([^"]*)"$/ do |options|
    opts = options.split(",")
    header = stub(:header) { opts }
    chart = stub(:chart, :header => header)
    VariantComparisonChart.stub(:build_for) { chart }

    chart = VariantComparisonChart.build_for(@current_product)
    opts.each do |option|
      chart.header.should include option
    end
  end

Outside: Green. Time to move inside!

With passing cucumber tests backed by FastModels and stubs, our class design is complete. It's time to actually start building out the real, but still isolated classes, outside Rails. Using your FastModels and stubs as guides you can TDD up a well unit tested class that does the job. Then swap out the stubs and the FastModels with your real classes and confirm the cucumber cycle is still green.

How do I use it?

FastModel and the stubbing library are included as part of cucumber_tool's no_rails.rb which can be called when executing your cucumber spec:

cucumber -r no_rails.rb path/to/feature.feature

Mike recommends avoiding the standard env.rb usage of cucumber to avoid loading Rails until you need it later. Mike says these tools were thrown together very quickly to capture the concept of "Feature Isolation" so any pull requests would be greatly appreciated. For more details, please visit Mike's follow up article.

Piggybak: The Roadmap

Over the last couple of weeks, a few of us at End Point have had some discussion about the future direction of Piggybak. Piggybak is an open source mountable ecommerce framework written in Ruby on Rails supported and developed by End Point. It introduces core ecommerce functionality into a Rails application, but is intended to allow the mounted Rails application to maintain control over some architecture elements.

Pros of Piggybak

Until now, the advantage of Piggybak is that it's a fairly lightweight approach. It leverages the power of RailsAdmin rather than creating it's own admin. It allows the mounted Rails application to make decisions on what types of items are sellable and how these items are found (i.e. product finding methods, SSL configuration). Piggybak also has streamlined integration of ActiveMerchant, which immediately provides support of over 40 popular payment gateways. Piggybak has a cookie-based cart and an AJAX-driven one-page checkout.

Cons of Piggybak Approach

Because Piggybak has a lightweight approach, the major disadvantage is that it cannot compete with existing ecommerce frameworks as an out of the box solution with a full ecommerce feature set. When compared with more feature-rich ecommerce platforms like Spree and Magento these other ecommerce platforms may have more features out of the box. This is a disadvantage because the abstraction, code cleanliness and maintainability provided by Piggybak is not necessarily as strong of a selling point to the feature list to a potential website owner.

The Roadmap

In looking towards the future of Piggybak, we've decided to build out some features of Piggybak, but will try to maintain a balance between having a good feature set while still maintaining the lightweightedness of Piggybak. Some of our goals in the future include:

• Realtime Shipping with USPS, UPS, and Fedex support [as extension]
• Improvement of Piggybak installation process [core]
• Advanced Product Optioning Support [as extension]
• Line Item Rearchitecture to support future work on Gift Certificates, Discounts [core]
• Gift Certificate, Discount Support [core]
• Advanced Taxonomy [as extension]
• Reviews & Ratings [as extension]

Follow the Piggybak github user here, check out the website and demo here and keep an eye out for future blog posts on the progress of Piggybak. Also, contact End Point now if you are ready to get started or need support on your current Ruby on Rails ecommerce platform.

Setting user ownership of nginx and Passenger processes

Do this now on all your production Rails app servers:

ps ux | grep Rails

The first column in the results of that command show which user runs your Rails and Passenger processes. If this is a privileged user (sudoer, or worse yet password-less sudoer), then this article is for you.

Assumptions Check

There are several different strategies for modifying which user your Rails app runs as. By default the owner of config/environment.rb is the user which Passenger will run your application as. For some, simply changing the ownership of this file is sufficient, but in some cases, we may want to force Passenger to always use a particular user.

This article assumes you are running nginx compiled with Passenger support and that you have configured an unprivileged user named rails-app. This configuration has been tested with nginx version 0.7.67 and Passenger version 2.2.15. (Dated I know, but now that you can't find the docs for these old versions, this article is extra helpful.)

Modifying nginx.conf

The changes required in nginx are very straight forward.

# Added in the main, top-level section
user rails-app;

# Added in the appropriate http section among your other Passenger related options
passenger_user_switching off;
passenger_default_user rails-app;

The first directive tells nginx to run it's worker processes as the rails-app user. It's not completely clear to me why this was required, but failing to include this resulted in the following error. Bonus points to any one who can help me understand this one.

[error] 1085#0: *1 connect() to unix:/tmp/passenger.1064/master/helper_server.sock failed (111: Connection refused) while connecting to upstream, client: XXX, server: XXX, request: "GET XXX HTTP/1.0", upstream: "passenger://unix:/tmp/passenger.1064/master/helper_server.sock:", host: "XXX"

The second directive, passenger_user_switching off, tells Passenger to ignore the ownership of config/environment.rb and instead use the user specified in the passenger_default_user directive. Pretty straight forward!

Log File Permissions Gotcha

Presumably you're not storing your production log files in your apps log directory, but instead in /var/log/app_name and using logrotate to archive and compress your logs nightly. Make sure you update the configuration of logrotate to create the new log files with the appropriate user. Additionally, make sure you change the ownership of the current log file so that Passenger can write your applications logs!

Piggybak: An Update on End Point's Ruby on Rails Ecommerce Engine

With the recent release of one of our client sites running on Piggybak, Piggybak saw quite a few iterations, both for bug fixes and new feature development. Here are a few updates to Piggybak since its announcement earlier this year.

Admin: Continues to Leverage RailsAdmin

Piggybak continues to leverage RailsAdmin. RailsAdmin is a customizable admin interface that automagically hooks into your application models. In the case of the recent project completion, the admin was customized to add new features and customize the appearance, which can be done in RailsAdmin with ease.

As much as I enjoy working with RailsAdmin, I think it would be great in the future to expand the admin support to include other popular Rails admin tools such as ActiveAdmin, which has also gained popularity in the Rails space.

Refund Adjustments

When Piggybak first came out, there was little in the way to allow orders to be monetarily adjusted in the admin after an order was placed. One requirement that came out of client-driven development was the need for recording refund adjustments. A new model for "Adjustments" is now included in Piggybak. An arbitrary adjustment can be entered in the admin, which is tied to a specific user, an amount, and a recorded note. This functionality allows for site administrators to record adjustments given against orders.

At the moment, the creation of an adjustment is not tied to a payment gateway to refund against the original transaction, because ActiveMerchant payment gateways have varied refund support.

AJAX Queueing for Shipping Requests on One-Page Checkout

Discussed in this article, AJAX queuing was added for shipping method generation on the one-page AJAX-driven checkout.

Order Notes

Another feature that originated from client needs was the need to record order changes over time. Now included in Piggybak are "Order Notes". Order notes are automatically created when attributes or nested attributes are changed on the order:

And arbitrary order notes can be added to record data not represented in attribute changes:

All order notes have a created_at attribute and belong to a user, which is tied to the administrator who made the change to the order.

Masked CC Number Storage

While we don't want to store unencrypted credit card numbers in the database per PCI compliance, Piggybak now includes storage of the masked credit card number to the payments table. This allows the site administrators to examine the credit card type and reference a particular card if needed. This was accomplished using the following method added to the String class:

class String
  def mask_cc_number
    masked = ''

    if self.gsub(/\D+/i, '').match(/^(\d\d)(.+)(\d\d\d\d)$/)
      masked = $1 + $2.length.times.inject('') { |s, i| "#{s}*" } + $3
    end

    masked
  end
end

Upgrade to Rails 3.2.8

The Piggybak demo has recently been updated to Rails 3.2.8. No changes were required, as this was considered a minor Rails update.

Conclusion

It's exciting to see the progress of Piggybak over the last several months, as well as to see Piggybak launch for a site with complex custom needs. I believe it accomplishes the vision I had in mind for it (summarized in this presentation), without requiring fighting against assumptions that traditional monolithic ecommerce platforms make. Granted, Piggybak still makes specific ecommerce assumptions, but they are limited to cart, checkout, and order functionality rather than systematic application-level behavior.

The future likely holds incremental improvements to Piggybak, but there are no plans to change Engine-like structure or principles of Piggybak.

Rails 4 Highlights

I watched this recent video What to Expect in Rails 4.0 presented by Prem Sichanugrist to the Boston Ruby Group. Here are a few high-level topics he covered in the talk:

• StrongParameters: replaces attr_accessor, attr_protected, moves param filtering concern to the controller rather than the model. Moving param filtering concern to the controller allows you to more easily modify user attribute change-ability in controllers (e.g. customer-facing vs admin).
• ActiveSupport::Queue: Discussed at RailsConf, add queueing support to Rails, e.g.:
  

  # Add to queue
  Rails.queue.push UserRegistrationMailerJob(@user.id)
  
  # Control queue configuration (asynchronous, synchronous or resque, e.g.
  config.queue = [:asynchronous, :synchronous, :resque]
  

• Cache Digests: Rails 4.0 introduces cache key generation based on an item and its dependencies, so nested cache elements properly expire when an item is updated.
• PATCH verb support: Support of HTTP PATCH method (_method equals "patch"), which will map to your update action is introduced in Rails 4.0.
• Routing Concern: Rails 4.0 introduces some methods to help clean up your duplicate routes.
• Improvements to ActiveRecord::Relation
  
  • Relation.all: returns ActiveRecord::relation object. User.all => User.to_a
  • Relation.none: Returns ActiveRecord::NullRelation, still chainable
  • Relation.___!: mutates current relation, e.g. @users.where!, @users.include!
• Deprecations
  
  • AR::Base.scoped
  • Dynamic Finder Methods: e.g. find_all_by_*
  • Hash-based Finders: e.g. User.find(:first)
  • Eager Evaluated Scope: scope will require a lambda
  • ActiveRecord::SessionStore
  • ActiveResource
  • Rails::Plugin
• New Deprecation Policy: Many of the above deprecations will still work in Rails 4.0 and included as gem dependencies, but will be removed in the jump to Rails 4.1. This means that the upgrade to Rails 4.1 may be more painful than the upgrade to Rails 4.0.

Check out the video or read the official current Rails 4.0 release notes here. Also, check out this post I came across about PostgreSQL array support in Rails 4, which may be pretty interesting to our PostgreSQL experts.

AJAX Queuing in Piggybak

AJAX is inherently asynchronous; for the most part, this works fine in web development, but sometimes it can cause problem if you have multiple related AJAX calls that are asynchronous to eachother, such as the use case described in this article.

In Piggybak, a Ruby on Rails open source shopping cart module developed and maintained by End Point, the one page checkout uses AJAX to generate shipping options. Whenever state and zip options change, the shipping address information is sent via AJAX and valid shipping methods are returned and rendered in a select dropdown.

Event listeners on the state and zip code inputs trigger to generate shipping options via AJAX.

While working on development for a client using Piggybak, I came across a scenario where AJAX asynchronous-ity was problematic. Here's how the problematic behavior looked on a timeline, picking up as the user enters their shipping address:

• 0 seconds: User changes state, triggers AJAX shipping lookup with state value, but no zip code entered (Let's refer to this as AJAX REQUEST 1).
• 1 second: User changes zip code, triggers AJAX shipping lookup with state and zip value present (Let's refer to this as AJAX REQUEST 2).
• 2 seconds: AJAX REQUEST 2 returns valid shipping options.
• 3 seconds: AJAX REQUEST 1 returns invalid shipping options, because no zip code provided in this data set, and overwrites shipping options returned by AJAX REQUEST 2.

The result is that the user has finished entering a valid shipping address, but sees that no valid shipping options can be chosen:

To address this issue, I researched AJAX queuing, with the requirement that AJAX requests should be performed synchronously and existing AJAX requests could be aborted if needed. After experimenting with a few different plugins, I found the most success with the jQuery-ajaxq plugin. It's simple to use:

• To append a new AJAX call on to a queue, you call $.ajaxq(queue_name, options), where options includes the standard AJAX arguments.
• To cancel or abort AJAX calls currently running in a queue, you call $.ajaxq(queue_name).

The event listener on state and zip changes now looks like the code shown below, in simplified form. The piggybak.update_shipping_options cancels AJAX requests on the shipping_options queue and then adds a new request to the queue which will execute immediately. This does not affect other asynchronous AJAX requests on the page.

var piggybak = {
    update_shipping_options: function() {
        $.ajaxq("shipping_options");
        $.ajaxq("shipping_options", {
            url: ...,
            cached: false,
            data: ...,
            dataType: "JSON",
            beforeSend: function() {
                 ...
            },
            success: function(data) {
                ...
            }
        });
    }
}

Here's how this looks on a timeline:

• 0 seconds: User changes state, triggers AJAX shipping lookup with state value, but no zip code entered (Let's refer to this as AJAX REQUEST 1).
• 1 second: User changes zip code, triggers AJAX shipping lookup with state and zip value present (Let's refer to this as AJAX REQUEST 2). This signals to abort all current AJAX requeusts on the shipping_options queue.
• 2 seconds: AJAX REQUEST 2 returns valid shipping options.

This was an interesting technical problem to solve, but the needs were not surprising. The jQuery-ajaxq plugin offers a simple, elegant solution for handling AJAX queing in jQuery.

Company Presentation: Ecommerce as an Engine

Today, I gave the presentation to my coworkers entitled "Puppies & Ecommerce as an Engine". The presentation is strongly coupled with my work on Piggybak, and includes a discussion of traditional ecommerce platforms versus a lightweight ecommerce approach through modularity (Rails Engine). It also provides some code examples as how this does work in Piggybak.

Below are a few more related articles to my work on Piggybak. Check them out!

• Recent Launch of Musica Russica on Piggybak
• Introduction of Piggybak
• Leveraging ActiveRecord Callbacks in Ecommerce
• Shopping Cart Options (applied to a Sinatra application)

Three Things: Times Two

It’s been a while since I’ve written up a “Three Things” article where I share a few featured web development tidbits picked up recently. So I made this a double episode!

1. event.stopPropagation() and event.stopImmediatePropagation()

I recently came across these two methods in jQuery, described here and here. Both of these methods [prevent the event from bubbling up the DOM tree, preventing any parent handlers from being notified of the event]. In my web application, my $('html') element had a listener on it, but I added specific listeners to children elements that when clicked on calls event.stopPropagation to cancel the event on the $('html') element. See the code below for a simplified example:

jQuery(function() {
    jQuery('html').click(function() {
        jQuery.hideSomething();
    });
    jQuery('.popup').click(function(event) {
        event.stopPropagation();
    });
})

2. alias_attribute

The alias method in Rails is one that I use frequently. But I recently came across the alias_attribute method as well. This might make the most sense to use when using shared views for multiple models with varying attributes.

3. Excel behavior in jQuery

Recently, there was a bit of discussion about jQuery tools that emulate spreadsheet behavior. A couple of the tools that came up were Handsontable and DataTables. They are worth checking out if you are looking to add Excel-like behavior to your web application!

4. Rack::SslEnforcer

I recently had a need on a Rails application to force some pages as secure, but have other pages be forced as non-secure. Instead of the common practice of adding controller before filters to force a redirect, this was included via the Gemfile, bundle install, and then configured in config/application.rb. Here’s an example configuration setup that I’m using:

config.middleware.use Rack::SslEnforcer,
        :only => [/\/checkout\/$/, /\/users$/, ‘/admin’],
        :strict => true

The one interesting caveat I found in working with this is that you absolutely must have all CSS and JavaScript assets precompiled in order for them to be served via SSL. The JS and CSS assets would not be forced to SSL, so they must exist in the Rails public directory via the precompiling, or this gem will redirect https requests on a secure page to http, resulting in the browser reporting that some non-secure elements are being served from a secure page.

5. Setting a viewport

I was recently troubleshooting a CSS issue for a client who was examining their website on an iPad. The iPad was setting the width of the viewport to a value that resulted in mis-alignment of floating elements. After a bit of research, I found that setting the viewport to the desired static width of my page fixed this issue. Here is a nice overview of the viewport attribute.

6. Line Specific Substitution in vi

Several of my coworkers are vi experts. When I’m in a shared screen with them, I pick up small tips for improving efficiency. One that I picked up on a few months ago and practiced enough to remember was line-specific substitution, e.g. 4,10s/moo/meow/g will substitute all occurrences of “moo” to “meow” in lines 4 through 10. I use this technique frequently.

Cannot parse Cookie header in Ruby on Rails

Yesterday I resolved a client emergency for a Ruby on Rails site that continues to leave me scratching my head, even with follow-up investigation. In short, the emergency came up after an email marketing campaign was sent out in the morning, and resulted in server (HTTP 500 Status Code) errors for every customer that clicked on the email links. Despite the fact that Rails exception emails are sent to the client and me, the errors were never reaching the exception email code, so I was unaware of the emergency until the client contacted me.

Upon jumping on the server, I saw this in the production log repeatedly:

ArgumentError (cannot parse Cookie header: invalid %-encoding (...)):
ArgumentError (cannot parse Cookie header: invalid %-encoding (...)):
ArgumentError (cannot parse Cookie header: invalid %-encoding (...)):

The URLs that the production log was complaining about had a bunch of Google Analytics tracking variables:

• utmcmd=Email
• utmcct=customeremail
• utmccn=New Site Sale 70% off
• etc.

After a user visits the site, these variables are typically stored as cookies for Google Analytics tracking. Upon initial investigation, the issue appeared to be triggered from any Google campaign variable that contained a '%' character.

After follow-up investigation today, the more complete story looks like this:

1. Email blast sent
2. User clicks on link in email. That link goes to the email marketing company first for tracking, then is redirected to the website.
3. According to the email marketing campaign (after chatting with them today), Google Analytics tacks on their own tracking here, which is the source of the non-parseable URLs.
4. Rack receives the request and tries to parse the query, utilizing the Ruby URI module:
   

   def self.decode_www_form_component(str, enc=Encoding::UTF_8)
       if TBLDECWWWCOMP_.empty?
         tbl = {} 
         256.times do |i|
           h, l = i>>4, i&15 
           tbl['%%%X%X' % [h, l]] = i.chr
           tbl['%%%x%X' % [h, l]] = i.chr
           tbl['%%%X%x' % [h, l]] = i.chr
           tbl['%%%x%x' % [h, l]] = i.chr
         end  
         tbl['+'] = ' '
         begin
           TBLDECWWWCOMP_.replace(tbl)
           TBLDECWWWCOMP_.freeze
         rescue
         end  
       end  
       raise ArgumentError, "invalid %-encoding (#{str})" unless /\A[^%]*(?:%\h\h[^%]*)*\z/ =~ str
       str.gsub(/\+|%\h\h/, TBLDECWWWCOMP_).force_encoding(enc)
     end 
   

5. The argument error on line 18 trickles up the pipeline in rack, and is not handled elegantly, so a rack-originated server (HTTP 500 Status Code) error is triggered. Again, the '%' character in the URL appears to be the problem here likely based on the regexp match on line 18 — the error is not triggered when the Google variable does not contain a '%' character.
6. Customer sees server error page and is unhappy :(

At the time of the emergency we tried solving the problem on multiple avenues:

• Investigated removal of Google Analytics tracking URLs from email blast links, but this wouldn't help all the customers who already received the emails.
• Remove CGI parameters or sanitize them via nginx.
• Make web application changes to ignore or handle the ArgumentError.

Ultimately, I ended up added a begin/rescue statement to the rack code to skip escaping URLs where decode_www_form_component was raising an error:

def unescape(s, encoding = Encoding::UTF_8)
  begin
    URI.decode_www_form_component(s, encoding)
  rescue
    Rails.logger.warn "DECODING on #{s.inspect} with #{encoding.inspect} FAILING."
  end 
end 

While this is a reasonable fix, I'm still puzzled for a number of reasons:

• The email marketing company (via chatting) claims that Google Analytics is tacking on the tracking variables, however, there is only one redirect from the email marketing company to the website. I don't understand the mechanism for which Google Analytics tracking variables are added to the URL, and if this process can be cleaned up to ensure proper URL encoding.
• I'm not sure if the issue happens immediately upon a customer landing on the site, or after a cookie is stored.
• At the moment, I'm not able to reproduce this issue in development mode, which makes it difficult to troubleshoot on my development instance.
• When I use the URI module directly in a console, no ArgumentError is raised:
  

  >> URI.decode_www_form_component("url_with_google_campaign_variables")
  >> #happy dance
  

My best advice at this point is to tell the client not to use '%' character in the Google Campaign ID, but I'm still putting all the pieces together in the virtual code map in my head. I think a fix is more likely needed on the Ruby and rack side to handle URL parameters with the '%' character, and to elegantly handle situations where the URI.decode_www_form_components method dies.

Musica Russica Launches with Piggybak

The new home page for Musica Russica.

Last week, we launched a new site for Musica Russica. The old site was running on an outdated version of Lasso and Filemaker and was approximately 15 years old. Although it was still chugging along, finding hosting support and developers for an outdated platform becomes increasingly challenging as time goes on. The new site runs on Ruby on Rails 3 with Nginx and Unicorn and uses open source Rails gems RailsAdmin, Piggybak, CanCan and Devise. RailsAdmin is a great open source Rails Admin tool that I've blogged about before (here, here, and here). Piggybak is End Point's home grown light-weight ecommerce platform, also blogged about several times (here, here, and here). Below are a few more details on the site:

• The site includes Rails 3 goodness such as an elegant and thorough MVC architecture, advanced routing to encourage clean, user-friendly URLs, the ability to integrate modular elements (Piggybak, RailsAdmin) with ease, and several built-in performance options. The site also features a few other popular Rails gems such as Prawn (for printing order and packing slip PDFs), Rack-SSL-Enforcer (a nice tool for enforcing SSL pages), exception_notification (a small tool to configure sending emails on Rails exceptions), and Paper Trail (a gem to track changes on your models).
• The site features complex, feature-rich search run by Sphinx with the popular Rails gem ThinkingSphinx. The search includes standard features such as selecting items listed per page, sort by various product attributes and wildcard text search. Although we've used Solr on several recent Rails projects, we wanted to go with Sphinx here to avoid the Tomcat and Java requirements. After working with both in depth, I'd conclude that both Sphinx and Solr offer very similar features.
• The site has a very custom data model. With the use of Piggybak, any Rails model can become a sellable item, which gives us the ability to customize each data model without affecting all sellable products. This opportunity suits Musica Russica well because they offer very different product types such as books, sheet music, CDs, and sheet music collections. Additional features (taxonomy, cross-sell, upsell) in the application are not constrained to assumptions that traditional monolithic ecommerce applications make.
• The site introduces downloadable products, described more in this article. A user purchasing downloadable products is required to register, and then has access to their purchased downloadables after purchase. Downloadable orders also include free shipping. Piggybak needed minor customization to support downloadable products.
• The site also features integration with the Elavon payment gateway. This is the first time that End Point has worked with Elavon. Elavon is supported by ActiveMerchant, so adding support for Elavon simply required hooking into ActiveMerchant's Elavon module.

It was exciting to build a custom Rails ecommerce site from the ground-up, and the site certainly showcases End Point's wide range of ecommerce expertise ranging from hosting with modern cutting-edge servers to development of popular front-end ecommerce features with advanced open source tools. Don't hesitate to contact us today to get started on development of your Rails application!

Rails 3 ActiveRecord caching bug ahoy!

Sometimes bugs in other people's code makes me think I might be crazy. I’m not talking Walter Sobchak gun-in-the-air-and-a-Pomeranian-in-a-cat-carrier crazy, but “I must be doing something incredibly wrong here” crazy. I recently ran into a Rails 3 ActiveRecord caching bug that made me feel this kind of crazy. Check out this pretty simple caching setup and the bug I encountered and tell me; Am I wrong?

I have two models with a simple parent/child relationship defined with has_many and belongs_to ActiveRecord associations, respectively. Here are the pertinent bits of each:

class MimeTypeCategory < ActiveRecord::Base
  # parent class
  has_many :mime_types

  def self.all
    Rails.cache.fetch("mime_type_categories") do
    MimeTypeCategory.find(:all, :include => :mime_types)
  end
end

class MimeType < ActiveRecord::Base
  # child class
  belongs_to :mime_type_category
end

Notice how in MimeTypeCategory.all, we are eager loading each MimeTypeCategory’s children MimeTypes because our app tends to use those MimeTypes any time we need a MimeTypeCategory. Then, we cache that entire data structure because it’s a good candidate for caching and we like our app to be fast.

Now, to reproduce this Rails caching bug, I clear my app’s cache using 'Rails.cache.clear' in the rails console, then load any page in my app that calls MimeTypeCategory.all. The page loads successfully and shows no errors. Doesn’t sound like a bug so far, right? If I load that same page a second time, I will get the standard Rails error page with:

undefined class/module MimeType
...
(app/models/mime_type_category.rb:17:in 'all')

Crazy, right? Why does it *appear* that one cannot cache model instances in Rails, and why did it work for exactly one page request after the Rails cache was cleared? Well, the former obviously cannot be true, and the latter is due to how Rails.cache.fetch handles cache misses and cache hits. For a cache miss, Rails.cache.fetch executes its block, serializes the return value, saves it to your cache store, then returns the block’s return value directly. For a cache hit, it reads the cached block from your cache store, deserializes it into whatever objects it identifies itself as, and returns that.

This is all well and good until you’re going along, innocently working on your app in the development Rails environment with config.cache_classes = false (which forces your app to lazy-load requested classes for each page request.) In that situation, Rails will try to deserialize the cached data structure that had references to the MimeType class. But, Rails may not have loaded the MimeType class at that point, so deserialization will fail and produce the error we see there. If you have other code paths in your app that do happen to load the child class before this type of cached parent/child class data structure, you might not hit the bug. Now you’ve entered a world of debugging pain.

I’m not about to give up on automatic class reloading in my development environment, and I don’t want to remove the cached eager loading of my child MimeTypes class because it’s sweet. So, after some digging, I discovered a solution: require_association. Adding “require_association ‘mime_type’ to my parent MimeTypeCategory class forces Rails to load the MimeType model when it loads the MimeTypeCategory model such that it can always deserialize the cached data structure successfully. I’ve used require_association in the same way for other instances of the same caching bug in our app as well.

Hopefully this explanation helps people avoid some of the pain I experienced while trying to determine if it was a Rails bug/feature or if I had finally gone insane. I should point out that some of the reading I’ve done suggests “require_dependency” is the more appropriate solution for this problem. I’ve verified that require_association works in all my cases, but to avoid “programming by coincidence,” I am going to snoop around the Rails core to understand the difference between the two.

Lastly, please remember: You can’t board a Pomeranian - they get upset and their hair falls out.

Ruby on Rails software developer needed

We're looking for another talented Ruby on Rails developer to consult with our clients and develop their Ruby on Rails web applications. If you like to focus on solving business problems and can take responsibility for getting a job done well without intensive oversight, please read on!

End Point is a 17-year-old web consulting company based in New York city, with 28 full-time employees working mostly remotely from home offices. Our team is made up of strong ecommerce, database, and system administration talent, working together using ssh, Screen and tmux, IRC, Google+ Hangouts, Skype, and good old phones.

We serve over 200 clients ranging from small family businesses to large corporations, using a variety of open source technologies including Ruby, Python, Perl, Git, PostgreSQL, MySQL, RHEL/CentOS, Debian, and Ubuntu.

What is in it for you?

• Work from your home office
• Flexible full-time work hours
• Annual bonus opportunity
• Health insurance benefit
• 401(k) retirement savings plan
• Ability to move without being tied to your job location

What you will be doing:

• Consult with clients to determine their web application needs
• Build, test, release, and maintain web applications for our clients
• Work with open source tools and contribute back as opportunity arises
• Use your desktop platform of choice: Linux, Mac OS X, Windows

What you will need:

• Professional experience building reliable Ruby on Rails apps
• Good front-end web skills with HTML, CSS, and JavaScript
• Experience with PostgreSQL, MySQL, or another relational database
• A customer-centered focus
• A passion for building flexible and, where needed, scalable web applications
• Strong verbal and written communication skills
• Experience directing your own work, and working from home
• Ability to learn new technologies

Bonus points for experience:

• Building and supporting ecommerce systems, including with Spree
• Working with other languages and web app frameworks
• Contributing to gems or other open source projects
• Handling system administration and deployment

Please email us an introduction to jobs@endpoint.com to apply. Include a resume, your GitHub or LinkedIn URLs, or whatever else that would help us get to know you. We look forward to hearing from you!

Changing Passenger Nginx Timeouts

It may frighten you to know that there are applications which take longer than Passenger's default timeout of 10 minutes. Well, it's true. And yes, those application owners know they have bigger fish to fry. But when a customer needs that report run *today* being able to lengthen a timeout is a welcomed stopgap.

Tracing the timeout

There are many different layers at which a timeout can occur, although these may not be immediately obvious to your users. Typically they receive a 504 and an ugly "Gateway Time-out" message from Nginx. Review the Nginx error logs both at the reverse proxy and application server, you might see a message like this:

upstream timed out (110: Connection timed out) while reading response header from upstream

If you're seeing this message on the reverse proxy, the solution is fairly straight forward. Update the proxy_read_timeout setting in your nginx.conf and restart. However, it's more likely you've already tried that and found it ineffective. If you expand your reading of the Nginx error you might notice another clue.

upstream timed out (110: Connection timed out) while reading response header from upstream, 
upstream: "passenger://unix:/tmp/passenger.3940/master/helper_server.sock:"

This is the kind of error message you'd see on the Nginx application server when a Passenger process takes longer than the default timeout of 10 minutes. If you're seeing this message, it'd be wise to review the Rails logs to get a sense for how long this process actually takes to complete so you can make a sane adjustment to the timeout. Additionally, it's good to see what task is actually taking so long so you can offload the job into the background eventually.

Changing nginx-passenger module's timeout

If you're unable to address the slow Rails process problem and must extend the length of the time out, you'll need to modify the Passenger gem's Nginx configuration. Start by locating the Passenger gem's Nginx config with locate nginx/Configuration.c and edit the following lines:

ngx_conf_merge_msec_value(conf->upstream.read_timeout,
                              prev->upstream.read_timeout, 60000);

Improving Error Pages

Another lesson worth addressing here is that Nginx error pages are ugly and unhelpful. Even if you have a Rails plugin like exception_notification installed, these kind of Nginx errors will be missed, unless you use the error_page directive. In other applications I've setup explicit routes to test exception_notification properly sends an email by creating a controller action that simple raises an error. Using Nginx's error_page directive, you can call an exception controller action and pass useful information along to yourself as well as present the user with a consistent error experience.

Respecting API Call Limit with Radian6

A common practice in the online API world is to enforce the call limit. Twitter allows 150 API calls per hour. Shopify has 500 API calls per 5 minutes limit. You may learn how to work with Shopify call limit from this great article.

One of our projects was built around interaction with Radian6 platform. Radian6 is a new and growing data mining service with the default limit of 100 calls per hour. I will describe our take on the call limit implementation.

Introducing the call counter

First, we need to know how many calls have been executed in the current hour. Every external call increments the counter field on a special model until the counter reaches the limit. The counter is reset back to zero at the beginning of every hour.

script/rails g model RadianCallCount

In the migration:

class CreateRadianCallCounts < ActiveRecord::Migration
  def change
    create_table :radian_call_counts do |t|
      t.integer :count
      t.timestamps
    end
  end
end

In db/seeds.rb file

puts "Initializing Radian6 counter"
RadianCallCount.delete_all
RadianCallCount.create(:count => 0)

Let's roll the counter!

rake db:migrate
rake db:seed

Scheduling the counter reset

It is necessary to reset the counter back to zero in the beginning of each hour otherwise the subsequent calls will not be executed. The excellent 'whenever' gem will take care of this.

gem install whenever
cd your_webapp
wheneverize .

In the model:

class RadianCallCount < ActiveRecord::Base
  def self.reset
    RadianCallCount.first.update_attribute(:count, 0)
  end
end

In config/schedule.rb:

every :hour do 
   runner "RadianCallCount.reset"
end

Tracking call count

We will now use the rcapture gem to intercept a call to external API and increment the counter with it.

gem install rcapture

In the module containing all Radian-specific methods:

require 'rcapture'
API_LIMIT = 100
def self.included(base)
  base.extend RCapture::Interceptable
  base.capture_pre :methods => [:authenticate,:tweet_stats] do |cs|
    RadianCallCount.transaction do 
      calls_per_hour = RadianCallCount.first.count 
      allowed = (calls_per_hour < Radian::API_LIMIT)
      cs.predicate = allowed
      cs.return = false 
      RadianCallCount.first.increment!(:count) if allowed
    end
  end
end

The code introduces a simple check before 'authenticate' and 'tweets_stats' methods. If call count exceeds the allowed limit the method is not executed and the method returns false. Otherwise, the counter increments after the successful method execution. We wrap the code in transaction because the actual count in the database may increase while we are making the API_LIMIT comparison.

Making the limit-aware call

Everything is ready to make the non-blocking API call. I scheduled a twitter statistics update to run every 3 hours:

every 3.hours do
   runner "Article.tweet_stats"
end

The non-blocking calls are suitable for most situations. Sometimes there is a need to just keep trying...

def call_with_timeout(&block)
  timeout = 0.minutes 
  results = false 
  while !results do
    results = block.call
    if !results
      break if timeout >= Radian::API_MAX_TIMEOUT
      Rails.logger.info("Sleeping for 5 minutes")
      sleep(Radian::API_CALL_TIMEOUT)
      timeout += Radian::API_CALL_TIMEOUT
    end
  end
  results 
end
...
call_with_timeout { tweet_stats }

That is all for today. Thanks for your attention. Hope the post was useful.

KISS: Slurping up File Attachments

I've been heavily involved in an ecommerce project running on Rails 3, using Piggybak, RailsAdmin, Paperclip for file attachment management, nginx and unicorn. One thing that we've struggled with is handling large file uploads both in the RailsAdmin import process as well as from the standard RailsAdmin edit page. Nginx is configured to limit request size and duration, which is a problem for some of the large files that are uploaded, which are large purchasable, downloadable files.

To allow these uploads, I brainstormed how to decouple the file upload from the import and update process. Phunk recently worked on integration of Resque, a popular Rails queueing tool which worked nicely. However, I ultimately decided that I wanted to go down a simpler route. The implementation is described below.

Upload Status

First, I created an UploadStatus model, to track the status of any file uploads. With RailsAdmin, there's an automagic CRUD interface connected to this model. Here's what the migration looked like:

class CreateUploadStatuses < ActiveRecord::Migration
  def change
    create_table :upload_statuses do |t|
      t.string :filename, :nil => false
      t.boolean :success, :nil => false, :default => false
      t.string :message

      t.timestamps
    end
  end
end

RailsAdmin also leverages CanCan, so I updated my ability class to allow list, reads, and delete on the UploadStatus table only, since there is no need to edit these records:

      cannot [:create, :export, :edit], UploadStatus
      can [:delete, :read], UploadStatus

KISS Script

Here's the simplified rake task that I used for the process:

namespace :upload_files do
  task :run => :environment do
    files = Dir.glob("#{Rails.root}/to_upload/*.*")
    files.each do |full_filename|
      begin
        ext = File.extname(full_filename)
        name = File.basename(full_filename, ext)

        (klass_name, field, id) = name.split(':')
        klass = klass_name.classify.constantize
        item = klass.find(id)  

        if item.nil?
          UploadStatus.create(:filename => "#{name}#{ext}", :message => "Could not find item from #{id}.")
          next
        end

        item.send("#{field}=", File.open(full_filename))

        if item.save
          FileUtils.rm(full_filename)
          UploadStatus.create(:filename => "#{name}#{ext}", :success => true)
        end
      rescue Exception => e
        UploadStatus.create(:filename => "#{name}#{ext}", :message => "#{e.inspect}")
      end
    end
  end
end

And here's how the process breaks down:

1. The script iterates through files in the #{Rails.root}/to_upload directory (lines 3-4).
2. Based on the filename, in the format "class_name:field:id.extension", the item to be updated is retrieved (line 11).
3. If the item does not exist, an upload_status record is created with a message that notes the item could not be found (lines 13-16).
4. If the file exists and the update occurs, the original file is deleted, and a successful upload status is recorded (lines 18-23).
5. If the process fails anywhere, the exception is logged in a new upload status record (lines 24-26).

This rake task is then called via a nightly cron job to slurp up the files. The simple script eliminates the requirement to upload large files via the admin interface, and decouples the upload from Paperclip/database management. It also has the added benefit of reporting the status to the administrators by leveraging RailsAdmin. Many features can be added to it, but it does the job that we need without much development overhead.

Simple Example of Dependency Injection with Rails

Today I came across a great opportunity to illustrate dependency injection in a simple context. I had a Rails partial that was duplicated across two subclasses. The partial was responsible for displaying options to create a new record from the data of the current record. It also offered two types of copy, shallow and deep. The shallow copy used a button to POST data, while the deep copy offered a form with some additional options. The only difference between the partials was the path to post data to. Let's see this in code.

#app/views/fun_event/_copy_options.html.erb
button_to(t("create_and_edit_shallow_copy"), fun_event_path(:from_event => @event.id, :return => true), :    id => "shallow_copy_btn")

form_tag(fun_event_path(:return => true)) do
  #form code
end


#app/views/boring_event/_copy_options.html.erb
button_to(t("create_and_edit_shallow_copy"), boring_event_path(:from_event => @event.id, :return => true), :    id => "shallow_copy_btn")

form_tag(boring_event_path(:return => true)) do
  #form code
end

The first, failed iteration

To remove the duplication, I passed in a path option into the partial, replacing specific references with the generic.

#app/views/fun_events/copy.html.erb
<%= render :partial => "events/copy_options", :event_path => fun_event_path %>


#app/views/boring_events/copy.html.erb
<%= render :partial => "events/copy_options, :event_path => boring_event_path %>

#app/views/events/_copy_options.html.erb
button_to(t("create_and_edit_shallow_copy"), event_path(:from_event => @event.id, :return => true), :    id => "shallow_copy_btn")

form_tag(event_path(:return => true)) do
  #form code
end

Can you guess where this led?

undefined method `event_path' for ActionView::Base:0xd6acf18

Dude! Inject the dependency!

Obviously the event_path variable I was passing was a string, not a method. I needed the method so I could pass in the appropriate arguments to construct the URL I needed. Had there not been two different calls to the routes, I would likely have just passed in the string needed in each context. But in this case, I was forced to think outside the box. Here's what I ended up with.

#app/views/fun_events/copy.html.erb
<%= render :partial => "events/copy_options", :event_path => method(:fun_event_path) %>


#app/views/boring_events/copy.html.erb
<%= render :partial => "events/copy_options, :event_path => method(:boring_event_path) %>

#app/views/events/_copy_options.html.erb
button_to(t("create_and_edit_shallow_copy"), event_path.call(:from_event => @event.id, :return => true), :    id => "shallow_copy_btn")

form_tag(event_path.call(:return => true)) do
  #form code
end

The changes are really quite subtle, but we use Object's method method to pass the reference to the method we want to call, and simply pass in the arguments when needed. Mind == Blown

.rbenv and Passenger: Working through an Upgrade

Yesterday, I worked on upgrading the Piggybak demo application, which runs on Piggybak, an open source Ruby on Rails ecommerce plugin developed and maintained by End Point. The demo was running on Ruby 1.8.7 and Rails 3.1.3, but I wanted to update it to Ruby 1.9.* and Rails 3.2.6 to take advantage of improved performance in Ruby and the recent Rails security updates. I also wanted to update the Piggybak version, since there have been several recent bug fixes and commits.

One of the constraints with the upgrade was that I wanted to upgrade via .rbenv, because End Point has been happily using .rbenv recently. Below are the steps Richard and I went through for the upgrade, as well as a minor Passenger issue.

Step 1: .rbenv Installation

First, I followed the instructions here to install rbenv and Ruby 1.9.3 locally under the user that Piggybak runs under (let's call it the steph user). I set the local Ruby version to my local install. I also installed bundler using the local Ruby version.

Step 2: bundle update

Next, I blew away the existing bundle config for my application, as well as the installed bundler gem files for the application. I followed the standard steps to install and update the new gems with the local updated Ruby and updated Rails. Then I restarted the app.

Step 3: Fail

At this point, my application would not restart, and the backtrace complained of a Passenger issue, and it referenced Ruby 1.8. Richard and I investigated the errors and concluced that the application's Passenger configuration was still referencing the system Ruby install and the outdated Passenger installation.

Here's where I hit the catch 22: I needed root access to update the passenger.conf as well as to install Passenger against Ruby 1.9.3. This defeated the purpose of using .rbenv and working with a local Ruby install only.

Step 4: Local Passenger Installation

To install Passenger against the local Ruby version, I decided to install it as the steph user. First, I installed the gem:

gem install passenger

Then, I went to the local installed version of Passenger to run the installation:

cd /home/steph/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/passenger-3.0.13/bin
./passenger-install-apache2-module

Next, I copied the passenger installation output to the passenger.conf file:

   LoadModule passenger_module /home/steph/.rbenv/versions/1.9.3-p194/lib/ruby/gems/\
     1.9.1/gems/passenger-3.0.13/ext/apache2/mod_passenger.so
   PassengerRoot /home/steph/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/passenger-3.0.13
   PassengerRuby /home/steph/.rbenv/versions/1.9.3-p194/bin/ruby

With a server restart, the Piggybak demo was up and running on updated Ruby and Rails!

Conclusion

Retrospectively, I could have avoided the Passenger issue by installing Ruby 1.9.3 on the server as root, because there isn't much else on the server. But I like using .rbenv and it's possible that a Passenger upgrade won't be required with every Ruby update, so the new Passenger configuration is acceptable [to me, for now].

Developing a Spree Application

Mike Farmer presented one of the projects he worked on. He gave us a very detailed overview of integration of a Spree application into Facebook for a large client.

He also described the changes that were made in the Spree project to customize it to the client's needs.

The most interesting thing in this Spree shop usage is that the company's clients make extensive use of the admin panel and they can sell their own products for other clients.

Mike showed us a great overview of writing better rails applications using better object oriented code along with TDD and extensive SASS usage. He also described the great tools he uses including Screen and Vagrant.

Mike also talked about the things he learned during working on this project, especially about Spree and Ruby.

TriSano and Pentaho at our NYC company meeting

Josh Tolley just spoke to us about the TriSano open source project he works on. It helps track and report on public health events, using data at least partly gathered from doctors following special legal reporting requirements to look for epidemics.

A lot of this is about data warehousing. Public health officials collect a lot of data and want to easily report on it. Typically they use SPSS. Need to filter the data before doing analysis.

And what is a data warehouse? Store all your data in a different way that's efficient for querying broken down by time (OLAP). Such queries don't usually work very well in normal transactional (OLTP) database.

Dimension tables: E.g. different public health departments, sex, disease. Fact tables: Contains the numbers, facts that you may do math aggregation against, and links to dimension tables. The key to the whole process is deciding what you want to track.

Pentaho is what we use for the query interface. To get done what we need to do, we have to make use of unpublished APIs, using JRuby to interface between Pentaho (Java) and TriSano, a Rails app. Postgres is the database.

Brian Buchalter then took over and delved more into the TriSano Rails side of the project.

Let's talk about the challenge of maintaining Rails applications. Getting them to work today is fine. Getting them to be flexible and change over time is a challenge. The early benefit of Rails was organizational: Having empty directories to put things into. But starting to use standard classic object oriented patterns is finally happening more in Rails.

Brian recommended Sandi Metz's talk about the path to better design, and her upcoming book, Practical Object Oriented Design in Ruby.

Ruby lets you open a class anytime, anywhere, and do whatever you want to it: "monkey patching." For example, plugins hijacking public methods. It can cause trouble. He strongly recommends watching the video Capability vs. Suitability by Gary Bernhardt, presented at MountainWest Ruby Conf 2012.

Practical limits unveil over time as people customize in ways you don't expect. For example, a 1300-question form that takes 4 minutes to load. Add arbitrary limits up front so that people understand there's a design limit rather than a "bug" of bad performance or UI due to unforeseen use cases.

Software is supposed to be easy to change. That's what people like, and that's what our employment is all about. But antipatterns emerge that make change harder over time.

How do we work with our clients when there's technical debt accumulated that will make further development increasingly expensive and troublesome? How do we estimate with imperfect information? Important to keep yourself from losing focus by diving into implementation details.

If you see part of your app that needs to be implemented but has no classes, just helper calls, double your estimate.

If you see lots of conditional blocks in your code, double your estimate. All the additional logic paths greatly increase complexity and make it harder to test.

Don't Repeat Yourself (DRY) is of course needed. But refactoring code is not (just) about removing duplication: It's about making code easier to modify forever.

Refactoring needs to be built into the development cycle, or else we end up forever dealing with first-implementation code, simply moving on to the next project.

Comprehensive tests that take 75 minutes to run won't get run by developers very often! The project automatically runs tests on check-in, but getting the result back takes too long. Would be good to speed up tests.

Lots of good topics to follow up on here.

Points of Interest

It's been a fairly straight forward week at work, but I have stumbled a few interesting finds along the way this week.

Vim Adventures

Finally! A game based approach to learning Vim keyboard commands. I was hoping someone would do this. It's just getting started (only two levels) and sadly, it looks like it'll be charging money to unlock higher levels. However, some things are worth paying for. I've found just playing the first two levels a few times have helped retrain my brain to not take my fingers off the home row. It's still quite buggy and seems to only work in Chrome. I found several times I needed to close all my Chrome windows after playing. Also, incognito mode seems to help with the bugs, as it disables all extensions you may have installed.

MySQL query comments in Rails

Ever wanted to know where that slow query was being called from? Well, if you're using MySQL with your Rails 2.3.x or 3.x.x app, you can get debug information about what controller's action made the call. Check out 37Signals new marginalia gem.

How to use EC2 as a web proxy

Kevin Burke provides a very detailed HOWTO article for working around restrictions you may experience in the course of an Internet based life. Pretty amazing what Amazon's free usage tier puts out there; of course it's only free for 12 months.

Include PIDs in your Logs

For many Rails developers we get comfortable looking at development log files. Sometimes when I have to investigate a customer issue on a production server using logs, I wished I had the level of detail the development logger had. While that's a wish, I'm finding it mandatory to include PID numbers in my production logs. In production systems with multiple requests being handled simultaneously, Rails logs start to become unusable. It's not clear which log lines are from which requests. Adding a PID in front of the time stamp can help untangle the mess. Here are some example approaches to this for Rails 3.x.x and Rails 2.3.x. Also, if you're really a log-lover and manage a lot of servers, check out Papertrail, it looks very impressive for $7/mo.

Spectrum Shortages - Why it's happening and what can be done

Telecom is not an area I have much familiarity with, but I found this article to be an interesting read. For example did you know that that largest owner of spectrum licenses are "under-capitalized or unwilling to build out networks" to use the spectrum? So while AT&T and Verizon struggle to meet the iPhone 4S's data demands (twice as much as iPhone 4!), "there are some companies that have spectrum, but they're struggling financially. Or they aren't quite sure what to do with the spectrum. And others that have the money and business model, but need the spectrum." It seems the way out of the mess is 4G, offering to improve the efficiency of spectrum use by 700 percent.

Three Things: Rails, JOIN tip, and Responsiveness

Here's another entry in my Three Things series, where I share a few small tips I've picked up lately.

1. Rails and Dramas

Sometimes I think that since Rails allows you write code efficiently, [a few] members of the Rails community have time to overdramatize incidents that otherwise would go relatively unnoticed :) Someone with a good sense of humor created this website to track these dramas. While it's probably a waste of time to get caught up on the personal aspects of the drama, some of the dramas have interesting technical aspects which are fiercely defended.

2. JOIN with concat

Recently I needed to perform a JOIN on a partial string match in MySQL. After some investigation, I found that I had use the CONCAT method in a conditional (in an implicit inner JOIN), which looked like this:

SELECT * FROM products p, related_items ri WHERE concat(p.sku, '%') = ri.id

In modern MVC frameworks with ORMs, databases are typically not designed to include data associations in this manner. However, in this situation, data returned from a third party service in a non-MVC, ORM-less application was only a substring of the original data. There may be alternative ways to perform this type of JOIN, and perhaps my fellow database experts will comment on the post with additional techniques ;)

3. Responsiveness

Responsive web design is all the rage lately from the increase in mobile web browsing and tablets. Here is a fun tool that that the ecommerce director at Paper Source pointed out to me recently. The website allows you to render a single URL at various widths for a quick review of the UI.

Instance Variable Collision with ActsAsTaggableOn

As developers, a lot of what we do is essentially problem solving. Sometimes it's a problem of how to implement a specific feature. Sometimes it's a problem *with* a specific feature. Last week, I ran into a case of the latter in some relatively mature code in the Rails app I was working on.

I was getting a sporadic exception while trying to save an instance of my StoredFile model. I encountered the problem while implementing a pretty trivial boolean field in my model, while I was playing around with it in the rails console. This is where it gets a little weird.

The exception message:

#<NoMethodError: undefined method 'uniq' for "":String>
#Backtrace:
... acts-as-taggable-on-2.2.2/lib/acts_as_taggable_on/acts_as_taggable_on/core.rb:264:in 'block in save_tags'
...rest of backtrace...

Note that none of my work was related to my model's use of acts_as_taggable_on. I looked briefly at line 264 and its cohorts in core.rb, but nothing jumped out as "a giant, obvious bug in acts-as-taggable-on" (which I wouldn't expect.) Also, the actual error is a bit suspicious. I love duck typing (and ducks) as much as anyone, but it's pretty rare to see well traveled code try to call uniq() on a String. An empty array, sure, no problem. I'll call uniq() on an empty array all day - but a String? Madness.

In the absence of any obvious answers, I switched to serious debug mode. I remembered that I ran into this same issue briefly about two weeks ago, but it had fixed itself. Then I remembered that this app is not Skynet and is therefore incapable of "fixing itself." OK, so there's been a bug lurking for a while. I was going to need a code debugging montage to get to the bottom of this. I fired up some techno music and the blue mood lighting in my office, and got down to debugging.

By the end of the debugging montage, I had determined that my StoredFile model was overriding the "tag_list" method supplied by ActsAsTaggableOn, in order to return the tags for this StoredFile instance regardless of which User owned the tags. Here's our entire method:

def tag_list
    @tag_list ||= self.anonymous_tag_list(:tags)
end

Can you guess where the issue might be with this code? Hint: It's the @tag_list instance variable. We're actually only using it here as a lazy way to cache the return value of self.anonymous_tag_list(:tags) during the lifespan of this single instance. I came to discover that ActsAsTaggableOn defines and uses an instance variable of the same exact name within the scope of my model. So, my tag_list method was assigning "" to the @tag_list instance variable for a StoredFile instance with zero tags, and that was trampling the [] that ActsAsTaggableOn would expect in that scenario. Hence, the ill-fated attempt to call "".uniq().

As a bonus, because it's an instance variable assigned inside a method, it would only get populated/trampled by my StoredFile model if an instance's tag_list was examined in any way. A sort of reverse Heisenbug, if you will. Renaming @tag_list to something else fixed the bug.

It was a problem and I solved it, which is cool. But, is this injection of instance variables considered poor behavior on ActsAsTaggableOn's part? What do you think? I'm still not sure how I feel about it, but I do know how I felt when I figured it out and fixed it:

Profile Ruby with ruby-prof and KCachegrind

This week I was asked to isolate some serious performance problems in a Rails application. I went down quite a few paths to determine how to best isolate the issue. In this post I want to document what tools worked most quickly to help find offending code.

Benchmarks

Before any work begins finding how to speed things up, we need to set a performance baseline so we can know if we are improving things, and by how much. This is done with Ruby's Benchmark class and some of Rail's Benchmark class.

The Rails guides would have you setup performance tests, but I found this cumbersome on this Rails 2.3.5 application I was dealing with. Initial attempts to set it up were unfruitful, taking time away from the task at hand. In my case, the process of setting up the test environment to reflect the production environment was prohibitively expensive, but if you can automate the benchmarks, do it. If not, use the logs to measure your performance, and keep track in a spreadsheet. Regardless of benchmarking manually or automatically, you'll want to keep some kind of log of the results keeping notes about what changed in each iteration.

Isolating the Problem

As always, start with your logs. In Rails, you get some basic performance information for free. Profiling code slows down runtime a lot. By reviewing the logs you can hopefully make a first cut at what needs to be profiled, reducing already long profile runs. For example, instead of having to profile an entire controller method, by reading the logs you might notice that it's just a particular partial which is rendering slowly.

Taking a baseline benchmark

Once you've got a sense of where the pain is, it's easy to get a benchmark for that slow code as a baseline.

module SlowModule
  def slow_method
    benchmark "SlowModule#slow_method" do
      #my slow code
    end
  end
end

Look to your log files to see results. If for some reason, you're outside your Rails enviornment, you can use Ruby's Benchmark class directly.

require 'benchmark'
result = Benchmark.ms do
  #slow code
end
puts result

This will tell you the process time in milliseconds and give you a precise measurement to compare against.

Profiling with ruby-prof

First, setup ruby-prof. Once installed, you can add these kinds of blocks around your code.

require 'ruby-prof'

module SlowModule
  def slow_method
    benchmark "SlowModule#slow_method" do
      RubyProf.start
      # your slow code here
      results = RubyProf.end
    end
    File.open "#{RAILS_ROOT}/tmp/SlowModule#slow_method_#{Time.now}", 'w' do |file|
      RubyProf::CallTreePrinter.new(results).print(file)
    end
  end
end

Keep in mind that profiling code will really slow things down. Make sure to collect your baseline both with profiling and without, to make sure you're doing apples-to-apples comparison.

By default ruby-prof measures process time, which is the time used by a process between any two moments. It is unaffected by other processes concurrently running on the system. You can review the ruby-prof README for other types of measurements including memory usage, object allocations and garbage collection time.

If you choose to measure any of these options, make sure your Ruby installation has the tools a profiler needs to collect data. Please see the Rails guides for guidance on compiling and patching Ruby.

Interpreting the Data with KCachegrind

At this point you should have a general sense of what code is slow having reviewed the logs. You've got a benchmark log setup with baseline measurements to compare to. If you're going to Benchmark while your profiling, make sure your baseline includes the profile code; it will be much slower! Remember we want an apples-to-apples comparison! You're ready to start profiling and identifying the root source of the performance problems.

After manually or automatically running your troubled code with the profiling block above, you can open up the output from ruby-prof and quickly find it not to be human friendly. Fortunately, KCachegrind turns that mess into something very useful. I found that my Ubuntu installation had a package for it already built, so installation was a breeze. Hopefully things are as easy for you. Next simply open your result files and start reviewing there results.

The image above shows what's called a "call graph" with the percentages representing the relative amount of time that method uses for the duration of the profile run. The CacheableTree#children method calls Array#collect and takes up more then 90% of the runtime. The subsequent child calls are relatively modest in proportion. It's clear we can't modify Array#collect so let's look at CacheableTree#children.

module CacheableTree
  def children(element = @root_element)
    full_set.collect { |node| if (node.parent_id == element.id)
      node
    }.compact
  end
end

Defined elsewhere, full_set is an array of Ruby objects. This is common performance optimization in Rails; collecting data looping through arrays works well with a small data set, but quickly becomes painful with a large one. It turned out in this case that full_set had 4200+ elements. Worse yet the children method was being called recusrively on each of them. Yikes!

At this point I had to decide how to optimize. I could go for broke and completely break the API and try and clean up the mess, or I could see if I could collect the data more quickly, some other way. I looked at how the full_set was defined and found I could modify that query to return a subset of elements rather easily.

module CacheableTree
  def children(element = @root_element)
    FormElement.find_by_sql(...) #the details aren't important
  end
end

By collecting the data directly via a SQL call, I was able to cut my benchmark by about 20%. Not bad for a single line change! Let's see what the next profile told us.

The above is another view of the profile KCachegrind provides. It's essentially the same information, but in table format. There were a few indicators that my optimization was helpful:

• The total process_time cost had dropped
• The amount of time spent in each function seemed to better distributed - I didn't have a single method soaking up all the process time
• Most of the time was spent in code wasn't mine!

module CacheableTree
  def children(element = @root_element)
    @children ||= {}
    @children[element] ||= FormElement.find_by_sql(...) #the details aren't important
  end
end

This is called memoization and let's us reuse this expensive method's results within the page load. This method took another 10% off the clock against the baseline. Yay!

Knowing When to Stop

Performance optimization can be really fun, especially once all the infrastructure is in place. However, unless you have unlimited budget and time, you have to know when to stop. For a few lines of code changed, the client would see ~30% performance improvement. It was up to them to decide how much further to take it.

If allowed, my next step would be to make use of the applications existing dependence on redis, and add the Redis-Cacheable gem. It allows you to marshal Ruby objects in and out of a redis server. The application already makes extensive use of caching, and this page was no exception, but when the user modified the page in a way that expired the cache, we would hit this expensive method again, unnecessarily. Based on the call graph above, we could eliminate another ~66% of the call time, and perhaps, by pre-warming this cache, could help the user to minimize the chances of experiencing the pain of slow browsing!

RailsConf 2012: What's Next in Rails?

I tend to walk away from every RailsConf feeling that there was a trend for each conference. In 2009, I thought the trend was Passenger, in 2010, NoSQL and Rails 3 was all the rage, and in 2011, many of the talks were related to CoffeeScript and SASS. While there was some of these technical specifics at this conference, I felt that the overarching trend this year is that several folks that are extremely involved in Rails (DHH, Yehuda Katz, Aaron Patterson, etc.) are very passionately involved in the course correction needed to keep Rails relevent.

As I mentioned in my blog article on Day 1, DHH talked about progress. DHH focused on the limitations of progress (loss aversion) and how and why people move from curious to suspicious when working with technology. He didn't delve into any of the Rails 4 details, but simply ended by sharing that Rails 4 will break and challenge things.

On Day 2, Aaron Patterson covered a discussion about what it means to incur technical debt, and how developers have different thresholds of how much technical debt is tolerated.

He talked about the types of refactoring needed to reduce technical debt. He ended his talk with a slide that said, "be prepared", which I understood to mean that there's a lot of movement, deep consideration, and no doubt passion. Aaron also participated in a Rails core panel on Day 2. While it was thoroughly entertaining (coloring book contest included!), there was some discussion over where Rails is going and the various perspectives. Aaron talked about how he is critical of Rails because he cares a lot, and the critical analysis of Rails is happening to keep Rails relevent in the future.

On Day 3, Yehuda Katz presented his talk on Rails: The Next Five Years. Katz noted that his perspective is not representative of the core team nor of his employer, that he comes from the perspective of developing very rich client-side web applications (that may also translate to mobile applications), and that part of the justification of his talk was to evangelize some of the technical features that he believes should be included in the core to keep Rails relevant.

For him, every time his company starts a new project and reevaluates whether Rails meets their needs means that something about Rails isn't working right now. Yehuda covered his examination of why Rails was great in the first place. And the underlying answer to that is convention over configuration. Rails eliminates trivial choices like code organization, assets, naming, routing, etc. and there is a shared benefit in this solution with convention. Gaining deep knowledge about hard problems is hard (e.g. CSRF prevention) and Rails makes these decisions for us with convention. Yehuda's perspective is [now] that any convention is better than no convention, because adding any convention upfront will allow something to be built out and improved incrementally over time rather than introducing divergent paths for trivial decisions.

<sidebar>

Yehuda took a short time in his talk to discuss justification for getting things in the core, even if they need follow-up iterations:

1. Getting new features into core gets a lot of eyeballs on it.
2. More importantly, getting new features into core gets a lot of eyeballs attached to people who have a lot of domain expertise on the problem that the feature is solving.
3. Getting new features into core helps reveal conceptual blindspots and additional required functionality.

</sidebar>

After talking about Rails and his desire to get new relevent features into Rails from at a high level, Yehuda dug into a few specific wants:

• serialization: This took up a large proportion of the talk. I can't give justice to everything covered here, so check out the slides and I'll post a link to the video later.
• bulk editing
• better convention for application structure (e.g. assets)
• API between JS framework and Rails, as the popularity of JS frameworks grows

Coding with convention (middle) allows developers* to avoid making diverging trivial decisions (left). Conventions provide the ability to share resources and domain expertise and add new features quickly, even if course correction (right) is needed on those conventions.
* not to be confused with sperm :)

What Now?

I think what Yehuda said during QnA resonated with me most: Decoupling the data from the representation is really important because the data then becomes represented in a variety of ways. Because of the extremely quick growth in technology and tools of various media consumption devices (browser, mobile web, native client apps), data is represented in an increasing number of ways and with varied degrees of interactivity, so what I took away from the conference is the view-level is where there may be some serious surgery in Rails. Obviously, there are a lot of core members looking at a variety of things in the stack, but I perceive that Rails needs to solve the problem of offering a strong convention for data representation to keep it relevant to new and growing technologies, while allowing Rails to continue to flourish.

RailsConf 2012: Day Three

It's Day 3 of RailsConf 2012 here in Austin Texas. Check out my posts from Day 1 and Day 2.

RailsConf 5k

To kick off the day, I ran the low-key RailsConf 5k along the lake here in Austin. I'm glad that this event was organized — I think it's good to take a mental break from all the conference activities.

RubyRogues

This morning kicked off with a RubyRogues live session. The topic for today's podcast was a discussion about what Rails developers care about and/or should care about. I thought that the answers apply to developers in general. Below are some of the topics covered in the talk.

• James Edward Gray II
  • James focused on the point that experimentation is important and to perceive all code as experimental.
• Avdi Grimm
  • Avdi's main argument was a suggestion to practice deliberate and mindful coding. While he believes that there is a place for the "shut up and code" mentality in some projects depending on risk, it's not acceptable to have this perspective when it comes to community sharing of knowledge.
  • Avdi recommends that exercising introspection by reviewing your code and explaining it to someone else (how and why) will make you a better programmer and communicator.
• David Brady
  • David shared that Ruby is not a block oriented lexically scoped language.
  • He suggests that you should a) learn Ruby as if it's unlike the other languages you know b) learn CoffeeScript because it writes better JavaScript than you will c) join the community and d) don't be a douchebag (translation: be humble, not arrogant) because d-bags make a bad stereotype for Rails programmers
• Josh Susser
  • Josh suggests to exercise prudence, or acting with or showing care and thought for the future by building a strong foundation.
  • He also discussed how while he agrees with comments of previous keynotes (loss aversion is pillar of conservatism, experienced developers have a lower tolerance for technical debt), that he also has the perspective that he has a limited budget for risk in projects. From his experience, bundler was a huge win in minimizing risk, but asset pipeline (in its current state) spent all of his risk budget.
• Charles Max Wood
  • Charles focused on how the Rails community should try to encourage and bring other developers to the community by evangelizing it. We should reach out to other communities and groups to encourage them to try Rails.

Lightning Talks

The last session I attended at RailsConf was the Lightning Talk session. I know I've abused lists in my RailsConf posts a bit, but Lightning Talks are meant to be presented in list form! Here's a quick rundown of some of the topics presented in the Lightning Talks, which were a mix of 1-minute and 5-minute talks. I'll update with more links as they become available.

• NSRails is a gem that bridges objective-C and rails, and will give Objective-C a bunch of methods similar to Rails methods (.all,.find(x), CRUD behavior)
• WindTunnel is a JavaScript testing framework without hassle.
• config.threadsafe is a setting in configuration that allows threading. Tony Arcieri provided a lot more details — I'll update this post to include a link to the slides later.
• Worker is a threading JavaScript tool.
• iwanttolearnruby.com is a resource for Ruby/Rails learning resources
• @marksim talked about how private teams should work like open source by a) doing more code reviews and b) using distributed communication tools like HipChat, Campfire, IRC, message boards, mailing lists, internal blogs, Google+ Hangout, and Skype to have success like open source does.
• sidekiq is a distributed message queue (like resque)
• erd is a gem that was written during RailsConf to allow visualization and changes of the database. I'll post a link to it when it becomes available.
• Rowe is a development technique that means trading results for money (as opposed to time).
• The job_interview gem is a silly (and extremely entertaining) gem created during RailsConf for helping you answer those tough interview questions.
• In Rails Engines, routes should a) always be drawn on the engine, and b) the engine should call isolate_namespace.
• Tokaido is a kickstarter project that Yehuda's been working on.
• railcar is an isolated, compiler-optional Rails environment (arcturo/Railcar).
• ninjascript is a JavaScript tool for unobtrusively adding event listeners, and more.
• tddium is a continuous testing integration tool.
• wicked is a gem for rendering wizards.
• flash_s3_rails is a gem for embedding a multi-file upload widget into your application.
• javascript-and-friends is a Google Group for discussing and acting on embedding JavaScript in Ruby.
• graphite is a scalable real-time monitoring tool.

A Little Bit More...

I also attended Yehuda Katz's talk on Rails: The Next Five Years. I'm saving a write-up on this for another post, because I'd like to wrap it in with some of the other keynotes discussions.

RailsConf 2012: Day Two

I'm here in Austin for Day 2 of RailsConf 2012. Here's a quick run-down of a few talks I attended today. Read about a couple of sessions I attended on Day 1 here.

Let's Make the Web Faster

One of the talks I attended on Day 2 was Let's Make the Web Faster — tips from the trenches @ Google by Ilya Grigorik. Ilya works on the MTWF (make the web faster) team at Google; one of their goals is not only to make Google's web applications faster, but also to make a bigger impact on web performance by developing and contributing to tools. I found a few main takeaways from Ilya's talk:

• Navigation Timing Spec: The navigation timing spec defining an interface for web applications to access timing information related to navigation and elements. You can look at this timing information by checking out the performance object in a console:
  
  Example output of navigation timing spec data.
• Site Speed Tools in Google Analytics: Google Analytics now has elements of the page speed broken down that allows you to compare page load times and examine what the bottlenecks in the system are. To get there, you go to Content => Site Speed => Page Timings in Google Analytics. You can also measure and examine this data using Segments and Advanced segments.
  Example metrics shown on Google Site Speed breakdown.
• WebPageTest.org is a tool that we've blogged about before a few times. Now it also provides a video of how the page loads to examine how the page load is perceived.
• Speed Index: Speed index is a new metric for measuring "visual completeness" of a page loading. This is an interesting/valuable metric to look at when you want to try to understand and improve user perceived latency. For example, can you force loading some images before peripheral scripts to improve the user perceived latency?
• A few Google tools that Ilya reviewed were PageSpeed Online and its API, PageSpeed SDK, and PageSpeed Service, which is in beta mode.
• Ilya's recap was that you should a) measure performance b) optimize performance (with specific attention to improve user perceived latency) and c) use tools to automate performance.

Check out the full presentation here.

Presenters and Decorators: A Code Tour

Another talk I attended was Presenters and Decorators: A Code Tour by Mike Moore. He started with a brief explanation of the justification (more later) for presenters — as a "pain driven developer", Mike tries to reduce the pain of advanced view logic with presenters. Mike then went through a complex example of rendering which mapped through a view to a helper to an app/component file to a class in the library back to the app to a haml view; this is painful to dissect, maintain, and test, so presenters are a good option here to avoid this spaghetti code.

Mike then went through coding examples and thought processes that lead to using a presenter-like solution. He presented an example where a view contained view-specific logic and instantiation and use of variables. The natural first step for refactoring the view here would be to pull the logic into the model, but the argument against this is that because it's view specific (as opposed to domain specific), it should not live in the model. Another progression step in pulling the logic out of the view would be to create a class that's initialized in the view, and the various instance methods on that class are called in the view. Here Mike introduced the ActiveDecorator plugin. This plugin automagicallly includes class decorators for instances in the view, which eliminates the need for instantiation of the presenter object. Mike also reviewed a standard serialization example, where a JSON-ified view can have logic that should probably live elsewhere. One option here is to create a class serializer option which has methods to render data to json. Another option here is to use ActiveModel::Serializers which looks for the rendering of JSON to remove logic from the JSON rendering view. These examples and another thorough code example are best viewed in the slides.

Mike explained that it's difficult to define Presenters and that he perceives presenters on a spectrum between the model and view which is an object that represents state and behavior of the view. The justification for presenters is to write easier to read, easier to maintain, easier to hand off to a designer, and easier to test code. Mike also noted that this isn't something that a Rails noob should get into and also that if it's a problem that you don't see in your code base, it's not a problem you necessarily need to solve.

Ten 42 Things You Didn't Know Rails Could Do

Another fun talk I attended on Day 2 was Ten Things You Didn't Know Rails Could Do by James Edward Gray II. In list-presentation style, James reviewed 42 things you can do with Rails that you might not know about [which work in Rails 3.2.3 and may or may not work in previous versions of Rails]. This talk reminded me of some of the protips I picked up from taking the Rails Best Practices course over at Code School because there were so many usable tips. I summarized the presentation into my own list of takeaways, but note that I left out some of the advanced tips that required extensive code examples. View the slides here.

• The command rake notes will output all your FIXME, TODO, and OPTIMIZE notes in your Rails application. You can also pass in custom annotations to output here as well.
• In the console, you can call helper methods (e.g. helper.number_to_currency(100)
• You can automagically add indexes via migration shorthand (e.g. rails g resource user name:index). And you can automagically add associations (e.g. rails g resource article user:references or rails g resource article:belongs_to) via migration shorthand.
• rake db:migrate:status will tell you where your migration is at
• You can use the pluck command instead of the map command. (e.g. User.pluck(:email), User.uniq.pluck(:email))
• You can override association methods to "hook" in custom behavior.
• You can use limitless strings in PostgreSQL. This requires a bit of code shared in the presentation. I know, my fellow PostgreSQL expert coworkers are going to balk that this is on the list, but unfortunately Rails doesn't allow this without some code.
• You can utilize PostgreSQL's full text search with a bit of code.
• You can use a different database for each user. This one could be great for multi-domain ecommerce, which I recently blogged about.
• You can merge nested hashes via the deep_merge method.
• You can remove a specific key from a hash via the except method.
• You can add defaults to a hash without overriding via the reverse_merge method.
• You can override form helpers, and a lot of other view stuff, but this was a little hard to encapsulate in a blog post, so it'll be best viewed in the slides.
• You can route exceptions (e.g. match "/404", to "errors#not_found")
• You can route to Sinatra. The example James gave here was to allow display of the Resque web interface inside a Rails application, but we have another client who would benefit from this technique as well.

Ta Da

Hopefully this article was more Rails-ey than yesterday's RailsConf 2012: Day One post which focused on Backbone.js and CoffeeScript. I'm looking forward to a couple of tomorrow's sessions — and hopefully they will be blogworthy. Stay tuned!

Using CarrierWave without an ORM in Spree

I was recently adding some settings to Spree (an open source Ruby on Rails ecommerce framework) using Spree's built-in preferences system and found myself needing to upload an image as part of the settings. Our application was already using CarrierWave to handle some other uploads and wanted to use it to handle the uploading for my current image as well. Since the only setting I was setting was the image and there would only ever be one image, I didn't want to tie the setting to ActiveRecord. So I did some reading and found that you can tie a CarrierWave to a class without an ORM. The documentation on this is very sparse so I resorted to reading through the code to figure out how to get it working. After hours of playing with it I finally got it working. With the current movement in the Rails community to move more toward an Object Oriented approach to building applications, I've decided to document this here so that others will be able to quickly tie CarrierWave uploaders to their non-persisted classes.

The uploader I'll be working on is to add a banner to the checkout process in Spree (0.60.x). The idea is to store the image url in Spree::Config (the general preferences mechanism within Spree). For those who unfamiliar, Spree::Config acts very similar to a Hash object that is persisted to the preferences table in the database.

The RSpec

Here are the tests that I will be using for the class. Hopefully these are fairly self explanatory.

# spec/checkout_view_setting_spec.rb
require 'spec_helper'
require File.join(Rails.root, 'lib', 'checkout_view_setting.rb')

describe CheckoutViewSetting do
  include ActionDispatch::TestProcess
  before do 
    @config = mock("Config")
    @cv = CheckoutViewSetting.new(@config)
    @uploaded_file = Rack::Test::UploadedFile.new(Rails.root.join('spec/support/test_image.png'), 'image/png')
  end

  it "should store the image in the config" do
    @config.should_receive(:[])
      .with(CheckoutViewSetting::IMAGE_ID_CONFIG_KEY)
      .and_return nil
    @config.should_receive(:set)
      .with({CheckoutViewSetting::IMAGE_ID_CONFIG_KEY => @uploaded_file.original_filename})

    @cv.image = @uploaded_file
    @cv.image_cache.include?(@uploaded_file.original_filename).should be_true
    @cv.write_image_identifier
    @cv.store_image!
  end

  it "should be able to set the image as form params" do
    @config.should_receive(:[])
      .with(CheckoutViewSetting::IMAGE_ID_CONFIG_KEY)
      .and_return nil

    @config.should_receive(:set)
      .with({CheckoutViewSetting::IMAGE_ID_CONFIG_KEY => @uploaded_file.original_filename})
      .and_return(@uploaded_file.original_filename)

    @cv.update_config(:image => @uploaded_file)
  end

  it "should be able to remove the image as form params" do
    @config.should_receive(:[])
      .with(CheckoutViewSetting::IMAGE_ID_CONFIG_KEY)
      .and_return nil
    @config.should_receive(:set)
      .with({CheckoutViewSetting::IMAGE_ID_CONFIG_KEY => nil})
      .and_return(nil)

    @cv.update_config(:remove_image => "1")
  end
end

The first thing I do here is mock the Spree config so that I can watch when the configuration gets written to make sure that works ok. I'm going to trust CarrierWave to do its thing as far as storing and removing the image. But what I'm really interested in for my tests is that the correct identifiers get read and set in the Spree::Config object. I'm also using the exact uploaded file object that will be passed to a Rails controller during the form submission so I don't have to worry about whether the image is the proper format.

In the next test, I want to make sure that I can use the standard conventions for setting the image straight from the class. The next two test cases I want to test the ability to pass a set of params as they would come from a form submission to save and remove an image. This matches the behavior you get when attaching CarrierWave to an ActiveRecord object.

Building the Class

One of the really nice things about CarrierWave is the design of the library. The record that is going to hold the information about the uploaded file can be any class so long as it implements a few methods for handling the reading and the writing of a serialized identifier. In ActiveRecord, this is a string field. For our example here, we are going to use the Spree::Config object. But it could be any method you choose to implement.

In its simplest form, our class is going to look like this:

# lib/checkout_view_setting.rb
require 'carrierwave/mount'

class CheckoutViewSetting
  extend CarrierWave::Mount
  mount_uploader :image do
    def default_url
      ""
    end
  end
  
  IMAGE_ID_CONFIG_KEY = :checkout_view_setting_image_id
  
  attr_reader :config
  
  def initialize(config)
    @config = config
  end

  def configured?
    ! @config[IMAGE_ID_CONFIG_KEY].nil?
  end

  # called to read the serialized identifier
  def read_uploader(column)
    @config[IMAGE_ID_CONFIG_KEY]
  end

  # called to write the serialized uploader
  # In Spree 0.60.x you cannot set attributes directly through []= like you can in the newer versions. 
  # So I use the set method to assign the value
  def write_uploader(column, identifier)
    @config.set IMAGE_ID_CONFIG_KEY => identifier
  end

  def write_remove_image_identifier
    write_uploader(:image, nil)
  end

  def store_remove_image!
    remove_image!
  end
end

The code above will allow our first test to pass. There's a bunch of stuff going on here, so I'll walk through the important parts:

• require 'carrierwave/mount' just loads the CarrierWave mount library.
• extend CarrierWave::Mount will add the mount_uploader method to our class so that we can designate which uploader we are going to use. Each file that is uploaded requires an uploader, so if our class were going to upload a banner and footer image, we would need an uploader for both of those.
• IMAGE_ID_CONFIG_KEY = :checkout_view_setting_image_id is a unique key in the Spree::Config preferences system that is going to contain the image identifier.

The two most important methods here are the write_uploader and read_uploader. These methods will be called by CarrierWave to get and set your identifier. Here, I'm just setting those in the config. The configured? method is just for convenience to see if the value is set. The other two methods are used when removing the image.

When mount_uploader is called, a bunch of methods are automatically defined in your class and they will all have the name of your uploader in the method name. It also takes a configuration block where you can configure your uploader. Another option is to create your uploader class like you would in a typical CarrierWave setup, but I chose to use an anonymous uploader here for brevity. Oh, and all those methods that were defined? Here's a list of methods that were defined in my class:

• image
• image=
• image?
• image_url
• image_cache
• image_cache=
• remote_image_url
• remote_image_url=
• remove_image
• remove_image!
• remove_image=
• remove_image?
• store_image!
• image_integrity_error
• image_processing_error
• write_image_identifier
• image_identifier
• store_previous_model_for_image
• find_previous_model_for_image
• remove_previously_stored_image

Pretty nice! You can see how this is done by checking out the source code on GitHub. Now that these methods are in place, my class will act just like an ActiveRecord object with CarrierWave!

Well, almost. CarrierWave lets you do some pretty cool stuff with ActiveRecord's update_attributes method. For example, setting a value of :remove_image => 1 in the params with a checkbox will call the remove_image= method on your class. Since my class doesn't have an update_attributes method, I'm going to implement something similar with a method called update_config that will accept params from a form and act in a similar fashion to ActiveRecord. Here's the code I came up with (borrowed heavily from ActiveRecord).

# stores the setting similar update_attributes in ActiveRecord
# if a remove_image param is present, the image is removed instead of added.
def update_config(params)
  return @config if params.empty?
  begin
    safe_params = params.reject {|k,v| ! %w(image remove_image).include?(k.to_s) }
    safe_params.each do |k,v| 
      send("#{k}=", v) 
      send("write_#{k}_identifier")
      send("store_#{k}!")
    end
  rescue => e
    Rails.logger.error "Error uploading Checkout View Setting: #{e.message}"
    return false
  end
  return @config
end

First, I do some parameter checking so that only params that I want to processed are actually sent. Since I'm using metaprogramming here, I don't want to introduce a case where someone could run various methods by passing bad params (much like mass-assignment vulnerabilities). Then I loop through the params and run the commands needed to save the image.

The Implementation

To use my class in my controller I just setup the following update method:

class Admin::CheckoutViewSettingsController < Admin::BaseController
  before_filter :get_checkout_view_settings
  def update
    config = @cv.update_config(params[:checkout_view_settings])
    if config
      flash[:notice] = t("admin.success")
    else
      flash[:error] = t("admin.fail")
    end
    redirect_to admin_checkout_view_settings_path
  end

  private
  def get_checkout_view_settings
    @cv = CheckoutViewSetting.new(Spree::Config)
  end

end

And then in my form I have the following (haml):

= form_tag admin_checkout_view_settings_path, :method => :put, :multipart => true do
  #formTop
    %fieldset
      .field
        = label_tag :image, t(".image.label")
        %br/
        = file_field_tag 'checkout_view_settings[image]', :accept => 'image/png,image/gif,image/jpeg'
    %fieldset
      .field
        = check_box_tag "checkout_view_settings[remove_image]"
        = label_tag "checkout_view_settings[remove_image]", t(".remove_image.label"), :style => "display: inline"
  #formBottom
    = image_tag(@cv.image.url) if @cv.configured?
    %br/
    = submit_tag

CarrierWave is an awesome library and a very suitable replacement for the more popular Paperclip library. I couldn't find anywhere online for showing a complete solution for implementing CarrierWave in an non-ActiveRecord class so I hope this helps anyone else that wants to go this route.

Read more about End Point's Spree development and consulting services here.

RailsConf 2012: Day One

Today kicks off the official first day of RailsConf 2012, in Austin, Texas (yesterday RailsConf unofficially kicked off with Ignite Rails). This is my fourth RailsConf, and I've found that I get increasingly more out of each one as my time working with Rails has accumulated.

The morning started with a keynote by DHH. DHH talked about progress and keeping an interest in something because it continues to make progress. Initially, the Rails community had a negative reaction to progress like Ruby 1.9, Rails 3, the Asset Pipeline and CoffeeScript, but after the initial learning curve, the new tools and progress was appreciated by the community. DHH talked about how there have been many studies on what prevents people from progressing, and one of those causes is loss aversion, meaning that people hate losing or failing enough that it prevents them from advancing. To me, the point of his talk was to prepare everyone for Rails 4 and how it will challenge and break things. Rails 4 is admittedly not coming out anytime soon, so he didn't touch on any specifics.

Using Backbone.js with Rails

One session talk I attended was Using Backbone.js with Rails by Sarah Mei of Pivotal Labs. Backbone is not an MVC framework in JavaScript, but it is lumped into traditional JavaScript MVC frameworks. It serves as a lightweight infrastructure for organizing code like templates, interactive elements, and events. Because Backbone is a relatively immature framework, like other JavaScript frameworks, there isn't a standard way of writing code, so the implementation of Backbone varies widely.

Sarah talked about the love story between JS and Rails:

She touched on the basics of Backbone from a Rails perspective:

• Models: Backbone and Rails models are very similar. An application will likely have Backbone models that mirror Rails models.
• Templates: Backbone templates are akin to Rails views or a mustache template.
• Views: Views are similar to Rails controllers. They set a template, class name, and define events.

Next, Sarah talked about two methods of implementation using Backbone.js and went through some code examples:

• Greenfield App
  • This approach tends to be more API driven and makes sense for an API-driven application that also requires buildout of native mobile apps.
  • In this approach, the server side returns [JSON, etc.] data and doesn't render HTML.
  • Sarah noted that the amount of code required here can be substantial because the framework is lightweight, so she would recommended considering alternatives (such as ember.js).
• "Backbone as Frosting" Implementation
  • In this implementation, there is a mixture of server-side and client-side rendering.
  • Sarah noted here that the lightweightedness of Backbone works nicely.

I haven't used Backbone.js in any client projects, but I've looked into it recently out of interest. I like the general idea of improving JavaScript code organization via the frosting approach described by Sarah. A few code examples were shared during the presentation — check out the slides here.

CoffeeScript for the Rubyist

Another talk I attended today was CoffeeScript for the Rubyist by Mark Bates. CoffeeScript is a JS meta language that compiles into JS and can easily integrate with your current JavaScript. It's easy to read, write, and maintain. I took the CoffeeScript lesson over at Code School recently (highly recommended!). Mark went through a brief history of JavaScript and explained that like the evolution to early assembly languages to C to Ruby (with a few missing steps), CoffeeScript's aim is to be an improved tool for writing readable and maintainable code. The syntax of CoffeeScript is a hybrid of Ruby and Python which is simple, uses no semicolons, typically no curly braces, no "function" keyword, relaxed parenthesis, and significant whitespace.

Mark then went into a review of Ruby-like CoffeeScript behavior. Below are some of the important topics in CoffeeScript that he covered, as well as links to the documentation:

• conditionals similarity (inline, postfix)
• objects and hashes are similar (can't screw up last comma)
• ranges
• string interpolation
• heredocs, (""" works)
• functions Ruby 1.9 Lamda syntax is similar to CoffeeScript version
• default arguments
• splats (e.g. first, second, others...)
• loops and comprehensions (for number in numbers)
• combination of loops and comprehensions and conditionals
• classes
• inheritence (e.g. class Manager extends Employee)
• Bound functions (with a fat arrow)
• Existential operator (e.g. if foo?, if console?.log "foo")

In addition to the talks on Backbone.js and CoffeeScript, I'll post the links to slides and video for the other talks I attended when they become available.

What's Next?

It's funny (but not surprising) how many of the talks I attended on Day One weren't actually about Rails or the specifics of Rails. Perhaps I'll try to pick a few more Rails-centric talks in the next couple of days of the conference and report on those. Stay tuned!

MWRC Highlights Part 2 of 2

This is Part 2 of my 2012 Mountain West Ruby Conference Highlights article I posted the week of the conference. To date, I still have a ton of TODO reading from the conference. Here are some of the things mentioned during Day 2 of the conference that are on that list.

Rollout: a gem that enables/disables sets of features for certain conditions, users, etc.

• Rollout is a very slick way to activate or deactivate features within your web app programmatically. Perhaps you want to deploy a new feature in production, but only for internal IP addresses, or certain @users, or a percentage of your @users. Perhaps you do and that feature blows up in production, but it's no big deal because with Rollout it only takes one line of code to turn that feature off for everyone. Perhaps it doesn't "blow up" as much as "melt down" and you'd like your app to turn off that feature automagically before you get an angry call from your CTO? Check out the Degrade gem for that, Rollout's awesome superhero sidekick.
• By James Golick
• Get it: https://github.com/jamesgolick/rollout
• Mentioned by: Matt White in his "Continuous Deployment" talk

Vagrant: an open source tool that manages virtual machines and their development environments to facilitate, well, development

• There are a lot of good reasons for checking out Vagrant. Fully working dev environments are portable between team members. They are reproducible. They are compartmentalized and switch-off-able.
  
  The Rock, leading the charge.
  Spend a few weeks working on client B's project with its specific environment needs, halt or suspend-to-disk that VM and instantly fire up client A's VM and its customized stack and pick up where you left off on that project.
• Behind the scenes, it uses Virtualbox and Chef or Puppet for VM creation and provisioning, so it’s not reinventing the wheel. Smart.
• I started developing Ruby on Rails at End Point, so I've always had the luxury of developing on a SpreeCamps server or at least in a DevCamps environment on a client's server. (At the risk of shilling for my employer on their own blog, I'll point out that those are both End Point creations and they are frickin' sweet.) Previous to working at End Point, I have taken the ride on that emotional roller coaster that is setting up a local dev environment (for other languages) with a bunch of dependencies . I've done it more than once, in fact. If you're like me and you had to do it without The Rock leading the way, you know it's no fun. If for some reason I couldn't use SpreeCamps or DevCamps, I'd definitely go with Vagrant.
• By Mitchell Hashimoto and John Bender
• Get it: vagrantup.com
• Recommended by: Mitchell Hashimoto during his "Rack Middleware as a General Purpose Abstraction" talk

New Relic

• If you're thinking I must be the last person on earth to hear about New Relic, you're wrong. I checked and my mother has also never heard of it and that woman is a saint with a heart of gold so you show some respect. Also she reads my blog. New Relic is a free* service that provides some pretty deep and valuable analytics and monitoring for your web app. Create an account on their site, install their gem and a YAML file with your account key, and restart your app. Their site has a great UI for presenting data on application code, database, front-end performance, etc. While I have signed up for a free New Relic account, I haven't started using it yet so I can't speak to how well it all works in reality. But, more than one MWRC speaker made an informal endorsement, so it's definitely worth checking out, especially for free.
• *Their Free account provides basic functionality and their Standard and Pro accounts are Non-Free, but offer additional functionality. If you’re running a Standard or Pro account, please let me know what you think about the additional features and their bang for the buck.
• By New Relic
• Get it: Surprise! New Relic
• Mentioned by BJ Clarke and others

Hstore data-type in Postgres

• Much like New Relic, Postgres's hstore data-type is not "new" in the traditional sense, but more in a "new to me" sense. Simply put, the hstore data-type in Postgres allows you to store structured key/value data in an indexable column such that the structure itself is intelligently queryable in Postgres.
• Described by Will Leinweber as a solution for data that would be pain to store and work with in strict third normal form. To be honest, I've only abandoned third normal form in a small number of cases and it was for performance reasons. So using hstore columns seems a little "wild, wild west" to me. Nevertheless, after seeing Will's examples of using hstore columns, I'm going to keep the idea in mind for the nastier schema challenges out there.
• By Postgres
• Get it: It's been a Postgres extension since Postgres 8.2.
• Mentioned by Will Leinweber during his "Schemaless SQL - The Best of Both Worlds" talk

This concludes my personal list of highlights for the Mountain West Ruby Conference. I urge you to check out the tools and people I've mentioned here and in Part I. Both Part I and Part II actually took me quite a while to write, as there was so much interesting reading to be had once I hit these people's blogs and GitHub pages. I would definitely invite you to, as Bruce Dickinson once said, "really explore the space." Last but not least, here's a shout out to Mike Farmer's excellent review of the conference's various discussions on dealing with Rails application complexity.

Web Development for HeARTs Speak

Many of my colleagues know that I'm fairly involved in animal rescue as a photographer and more recently as a foster. I recently became involved in HeARTs Speak, a non-profit, volunteer-driven organization that brings together artists (photographers and more) who volunteer at animal rescue organizations.	Frank the Foster
I worked with them to help launch a new website with a design from Enso Blue, which brings us to the point of the article. Given a choice of many platforms and tools, what tools did I use in development of a new site with the only restriction being how much time I was able to put into development and maintenance? Here's a quick rundown the tools I used for the new website.

CanCan and RailsAdmin in Ecommerce

I've written about Rails Admin a lot lately. One thing I haven't written about is how it can leverage CanCan for user authorization. Once a user is authenticated via Devise, CanCan adds an additional layer for you to control how users can access and interact with data from the admin.

A Simple Example

CanCan requires that you create an Ability class where all user permissions are defined. A very basic example in RailsAdmin ecommerce context might look like this:

class Ability
  include CanCan::Ability
  if user && user.is_admin?
    can :access, :rails_admin
    can :dashboard
    can :manage, [User,
                  Product,
                  Order]
  end
end

Note that in the above code, a user that is admin (where is_admin? returns a truthy reponse) has access to RailsAdmin, and the admin user can manage (create, read, update, destroy) all users, products, and orders.

Multi-Merchant Solution

Let's go a little deeper. Multi-merchant solutions are a frequent request in ecommerce. Let's say we have the following over-simplified data model, where users own and manage products and products are displayed by category:

The Ability class might look like:

class Ability
  include CanCan::Ability
  if user && user.is_store_owner?
    can :access, :rails_admin
    can :dashboard
    can :manage, Product, :store_owner_id => user.id
    can :read, Category, :visible => true
  end
  if user && user.is_admin?
    can :access, :rails_admin
    can :dashboard
    can :manage, [User,
                  Category,
                  Product,
                  Order]
  end
end

With the above Ability definition, nothing changes for an admin user. A store owner can create products and manage (read, update, destroy) those same products. A store owner can also read categories where the category visible attribute is true. As you can see, conditions can be passed in for ability definitions. Directly from the CanCan documentation: "Anything that you can pass to a hash of conditions in Active Record will work here. The only exception is working with model ids. You can't pass in the model objects directly, you must pass in the ids."

Custom Abilities

In addition to the CRUD methods (create, read, update, destroy), CanCan gives you the ability to define additional custom abilities. RailsAdmin has special abilities (:history, :show_in_app, :dashboard), but you can create custom actions with RailsAdmin where CanCan abilities can be managed. For example, Marina wrote about how you might create a custom task to approve reviews here. And I described how to create a custom action to import data here, where CanCan is used to define which models can be imported and which users can do the import.

Conclusion

CanCan is decoupled from authentication and the notion of roles, which yields quite a bit of flexibility in its use. Combine RailsAdmin, CanCan, Devise, and our mountable ecommerce Rails Engine, and you've got a powerful set of tools for development of a custom Ruby on Rails ecommerce application with a powerful admin interface, flexible user authorization and authentication, and an extensible ecommerce solution.

MWRC Highlights Part 1 of 2

I attended the 2012 Mountain West Ruby Conference in Salt Lake City last week. There were a lot of cool topics presented at this conference, but I've been suffering serious analysis paralysis trying to pick one topic to focus on here. So, I'm taking the easy way out. I'm going to run through some of the projects and tools mentioned during the conference that I flagged in my notes to check out further.

Sidekiq, a multi-threaded drop-in replacement for Resque

• Sidekiq requires *much* less RAM to do the same amount of work as the single-threaded Resque, while providing a very similar API to Resque. It’s been designed to behave like Resque as much as possible and as such, would be a very easy transition for anyone that’s used Resque before.
• By Mike Perham
• Get it at: mperham.github.com/sidekiq
• Recommended by: Tony Arcieri

Book: Growing Object-Oriented Software Guided by Tests (aka GOOS)

• The "GOOS book" was recommended by a number of speakers at the conference. One downside (for me) is that the Auction Sniper app developed in the book is done in Java. However, there are now ports of that code available in Ruby, Scala, C# and Groovy. Check out the book's website. The table of contents is very detailed, so you know what you're getting into.
• By Steve Freeman and Nat Pryce
• Get it at www.growing-object-oriented-software.com
• Recommended by: Multiple presenters

PRY, a very powerful irb replacement and Rails debugging tool

• It's difficult to describe PRY without using the word "powerful" in every sentence. It really is, well, that powerful. It speeds up some of the things you already do when writing Ruby code. It also enables you to do things you wish you could do when writing Ruby and Rails code. The 8-minute Ryan Bates Railscast on PRY shows it better than I can describe it.
• Core team: John Mair, Conrad Irwin and Ryan Fitzgerald
• Get it at pry.github.com
• Recommended by: Corey Woodcox

"Ruby is a tool that gets out of your way. Build that for your clients."

• More an ethos than a project or tool, this is a relatively direct quote from Jamis Buck in his talk entitled, "It's the Little Things." Part of what he talked about were the features or patterns of Ruby that tickle part of our brain and just feel right, natural or easy-flowing. The tools or apps that we build for our clients should have for that same sense of flow. I think achieving that requires a perspective shift that's not easy for many developers, but it's absolutely something worth aspiring to. Our clients deserve that.
• As said by Jamis Buck from 37Signals.com

Attending this conference was honestly pretty inspiring. It was my first Ruby conference, and it was exciting to hear some very knowledgeable people speak passionately about what they work on and how they work. I definitely have a lot more reading and investigation to do. Part 2 of this article will cover the remaining conference topics that I felt deserved a shout out. I'll probably do that in the next day or two.

A Cache Expiration Strategy in RailsAdmin

I've been blogging about RailsAdmin a lot lately. You might think that I think it's the best thing since sliced bread. It's a great configurable administrative interface compatible with Ruby on Rails 3. It provides a configurable architecture for CRUD (create, update, delete, view) management of resources with many additional user-friendly features like search, pagination, and a flexible navigation. It integrates nicely with CanCan, an authorization library. RailsAdmin also allows you to introduce custom actions such as import, and approving items.

Whenever you are working with a gem that introduces admin functionality (RailsAdmin, ActiveAdmin, etc.), the controllers that provide resource management do not live in your code base. In Rails, typically you will see cache expirations in the controller that provides the CRUD functionality. For example, in the code below, a PagesController will specify caching and sweeping of the page which expires when a page is updated or destroyed:

class PagesController < AdminController
  caches_action :index, :show
  cache_sweeper :page_sweeper, :only => [ :update, :destroy ]

  ...
end

While working with RailsAdmin, I've come up with a different solution for expiring caches without extending the RailsAdmin functionality. Here are a couple of examples:

Page Caching

On the front-end, I have standard full page caching on static pages. In this case, the config/routes.rb maps wildcard paths to the pages controller and show action.

match '*path' => 'pages#show'

The controller calls the standard caches_page method:

class PagesController < ApplicationController
  caches_page :show

  def show
    @page = Page.find_by_slug(params[:path])
    
    ...
  end
end

A simple ActiveRecord callback is added to clear the page cache:

class Page < ActiveRecord::Base
  ...

  after_update :clear_cache

  def clear_cache
    ActionController::Base.expire_page("/#{self.slug}")
  end
end

Fragment Caching

When a page can't be fully cached, I might cache a view shared across the application. In the example below, the shared view is included in the layout – it's generated dynamically but the data does not change often, which makes it suitable for fragment caching.

<% cache "navigation" do -%>
  <% Category.each do |category| -%>
    <%= link_to category.name, category_url(category) %>
  <% end -%>
<% end -%>

Inside the model, I add the following to clear the fragment cache when a category is created, updated, or destroyed:

class Category < ActiveRecord::Base
  after_create :clear_cache
  after_update :clear_cache
  before_destroy :clear_cache

  def clear_cache
    ActionController::Base.new.expire_fragment("navigation")
  end
end

Conclusion

One thing that's noteworthy is that expire_page requires a class method on ActionController::Base while expire_fragment requires an instance method (see here versus here). Action cache expiration with ActiveRecord callbacks should work similarly with action caching, as a class method (reference).

An alternative approach here would be to extend the generic RailsAdmin admin controller to introduce a generic sweeper. However, the sweeper would have to determine what model was modified and what to expire it. This can be implemented and abstracted elegantly, but in my application I preferred to use simple ActiveRecord callbacks because the caching was limited to a small number of models.

Multi-store Architecture for Ecommerce

Something that never seems to go out of style in ecommerce development is the request for multi-site or multi-store architecture running on a given platform. Usually there is interest in this type of setup to encourage build-out and branding of unique stores that have shared functionality.

A few of Backcountry.com's stores driven by a multi-store architecture, developed with End Point support.

End Point has developed several multi-store architectures on open source ecommerce platforms, including Backcountry.com (Interchange/Perl), College District (Interchange/Perl), and Fantegrate (Spree/Rails). Here's an outline of several approaches and the advantages and disadvantages for each method.

Option #1: Copy of Code Base and Database for Every Site

This option requires multiple copies of the ecommerce platform code base, and multiple database instances connected to each code base. The stores could even be installed on different servers. This solution isn't a true multi-store architecture, but it's certainly the first stop for a quick and dirty approach to deploy multiple stores.

The advantages to this method are:

• Special template logic doesn't have to be written per site – the templates would simply follow the ecommerce platform's template pattern.
• Relative to Option #3 described below, no custom database development is required.
• Custom business logic may be more easily applied to a set of the stores, without affecting the other stores.

The disadvantages to this method are:

• Maintenance of the applications can be time consuming, as changes must be applied to all instances.
• Custom changes must be applied to all multi-store instances.
• Users and administrator accounts are not shared across multiple stores.

Option #2: Single Code Base, Single Database In this method, there is one copy of the source code that interacts with one database. The single database would be modified to contain a store specific id per product, order, and peripheral tables. The code base would also have to be modified to be able to limit the visible products described here. In this method, the individual store may be identified by the domain or subdomain. With this method, there may also be code customization that allows for custom templates per store. The advantages to this method are: Relative to Option #1, maintenance for one code base is relatively simple. User and administrator accounts are shared across multiple stores. Super administrators may view and manage data from one backend administrative interface. The disadvantages to this method are: Rights and role management can be complicated. Development is required for code and database customization. Development is required for coding to handle flexible templating across stores.

A second option in multi-store architecture may use a data model with store specific entries in various tables, described here.

Option #3: Single Code Base, Single Database with Schemas or Views Per Store

In this method, there is one copy of the source code that interacts with a database that has views specific to that store, or a schema specific to that store. In this case, the code base would not necessarily need customization since the data model it accesses should follow the conventions of the ecommerce platform. However, moderate database customization is required in this method. With this method, there may also be code customization that allows for custom templates per store.

The advantages to this method are:

• Relative to Option #1, maintenance for one code base is relatively simple.
• Relative to option #2, code base changes are minimal.
• User accounts may or may not be shared across stores.
• Relative to option #2, there may be a potential performance gain by removing time spent limiting data to the current store instance.

The disadvantage to this method is:

• Customization and development is required for database configuration and management of multi-store database schemas.

A tangential variation on the methods above are two different codebases and functionality attached to one back-end web service and backing database, such as the architecture we implemented for Locate Express. And a similar tangential variation I've investigated before is one that might use a Sinatra driven front-end and a Rails backed admin, such as RailsAdmin used in Piggybak.

College District has a collection of stores driven by a multi-store architecture, developed with End Point support.

Conclusion

In most cases for our clients, there is cost-benefit analysis that drives the decision between the three options described above. Option #1 might be an acceptable solution for someone interested in building out two or three stores, but the latter two options would be more suitable for someone interested in spinning up many additional instances quickly with lower long term maintenance costs.

Rails 3 remote delete link handlers with Unobtrusive Javascript

I recently encountered a bug in a Rails 3 application that used a remote link_to tag to create a Facebook-style "delete comment" link using unobtrusive javascript. I had never worked with remote delete links like this before, so I figured I’d run through how I debugged the issue.

Here are the relevant parts of the models we're dealing with:

class StoredFile < ActiveRecord::Base
  has_many :comments, :dependent => :destroy
end
class Comment < ActiveRecord::Base
  belongs_to :user
  belongs_to :stored_file
end

Here's the partial that renders a single Comment (from the show.html.erb view for a StoredFile) along with a delete link if the current_user owns that single Comment:

<%= comment.content %> -<%= comment.user.first_name %>
<% if comment.user == current_user >
  <%= link_to 'X', stored_file_comment_path(@stored_file, comment), :remote => true, :method => :delete, :class => 'delete-comment' >
<% end ->

Here’s a mockup of the view with 3 comments:

At first, the bug seemed to be that the "X" wasn’t actually a link, and therefore, didn't do anything. Clicking the "X" with Firebug enabled told a different story. There was a link there (hidden by sneaky CSS,) and the Firebug console showed that it was sending the appropriate request to the correct url: /stored_files/78/comments/25

The development.log file on the server corraborated the story and showed a successful delete:

Started DELETE "/stored_files/78/comments/25"
  Processing by CommentsController#destroy as JS
  Parameters: {"stored_file_id"=>"78", "id"=>"25"}
  SQL (0.6ms)  DELETE FROM "comments" WHERE "comments"."id" = 25
  Completed 200 OK in 156ms

So far, so good. I know that our client code is making the correct request and the Rails app is handling it appropriately. I knew that the existing code "worked," but still didn’t provide UI feedback to inform the user. I needed to write some jQuery to handle the successful (HTTP 200) server response to our unobtrusive javascript call.

Normally, when writing my own handler (e.g. to handle a button click) to initiate an Ajax call with jQuery, I’d use $.ajax or $.post and use its built-in success handler. Something like this:

$.ajax({
    type: 'POST',
    url: 'my_url',
    data: { param_name1: 'param_value1'},
    success: function(data){ 
        alert('Successfully did the needful!');
    }
});

It turns out that you still define a event handler when handling server responses to unobtrusive javascript, it’s just that the syntax is very different. I needed to bind the ‘ajax:success’ event when it’s fired by any of my comment delete links (where class=”delete-comment”, as specified in my link_to call).

$(document).on('ajax:success', '.delete-comment', function() {
    // .parent() is the div containing this "X" delete link
    $(this).parent().slideUp();
    }
);

(Note that I happen to be using the newer jQuery 1.7+ .on() method and argument list instead of the older, more common .live() style. In this case, they are functionally equivalent, but the methods that .on() replaces are deprecated in jQuery 1.7.)

Now, when the user clicks the "X" to delete one of their own comments, the successful unobtrusive javascript call is detected and the div containing that single comment is neatly hidden with the help of jQuery's slideUp() method. 

Much better!

DevCamps setup with Ruby 1.9.3, rbenv, Nginx and Unicorn

I was working with Steph Skardal on the setup of a new DevCamps installation that was going to need to use Ruby 1.9.3, Rails 3, Unicorn and Nginx. This setup was going to be much different than a standard setup due to the different application stack that was required.

The first trick for this was going to get Ruby 1.9.3 on the server. We were using Debian Squeeze but that still only comes with Ruby 1.9.1. We wanted Ruby 1.9.3 for the increased overall speed and significant speed increase with Rails 3. We decided on using rbenv for this task. It's a very easy to setup utility that allows you to maintain multiple version of Ruby in your system user account without the headache of adjusting anything but the PATH environment variable. It takes advantage of another easy to setup utility called ruby build to handle the actual installation of the Ruby source code.

git clone git://github.com/sstephenson/rbenv.git .rbenv

echo 'export PATH=$HOME/.rbenv/shims:$HOME/.rbenv/bin:$PATH' >> ~/.bash_profile

rbenv install 1.9.3-p0

rbenv global 1.9.3-p0

gem install bundler

rbenv rehash

Now you are ready to use the bundler gem to install any other gems required for the application.

The normal camps setup assumes you are going to be using Apache for the web server. In this case, we wanted to use Nginx due to memory constraints. We decided to use the proxy capability and just proxy through to Unicorn instead of having to build our own version Nginx to use Passenger. To do this, we had to use a feature in the local-config file in camps that allows you to skip the Apache setup and use your own commands to start, stop and restart your web server and application. Here is the example from our local-config that controlls Nginx and Unicorn. This approach could also be used with Interchange or any other application if you need other services started when mkcamp is run.

skip_apache:1
httpd_start:/usr/sbin/nginx -c __CAMP_PATH__/nginx/nginx.conf
httpd_stop:pid=`cat __CAMP_PATH__/var/run/nginx.pid 2>/dev/null` && kill $pid
httpd_restart:pid=`cat __CAMP_PATH__/var/run/nginx.pid 2>/dev/null` && kill -HUP $pid || /usr/sbin/nginx -c __CAMP_PATH__/nginx/nginx.conf
app_start:__CAMP_PATH__/bin/start-app
app_stop:pid=`cat __CAMP_PATH__/var/run/unicorn.pid 2>/dev/null` && kill $pid
app_restart:pid=`cat __CAMP_PATH__/var/run/unicorn.pid 2>/dev/null` && kill $pid ; sleep 5 ;  __CAMP_PATH__/bin/start-app

cd __CAMP_PATH__ && bundle exec unicorn_rails -c __CAMP_PATH__/config/unicorn.conf.rb -D

You could create one script that handles all aspects of start, stop and restart if you wanted. This setup really wasn't much harder than a normal Ruby on Rails setup. The added time here required to set up rbenv per camp user is offset by the fact that users can manage and try multiple versions of ruby.

Download Functionality for Rails Ecommerce

I recently had to build out downloadable product support for a client project running on Piggybak (a Ruby on Rails Ecommerce engine) with extensive use of RailsAdmin. Piggybak's core functionality does not support downloadable products, but it was not difficult to extend. Here are some steps I went through to add this functionality. While the code examples apply specifically to a Ruby on Rails application using paperclip for managing attachments, the general steps here would apply across languages and frameworks.

Data Migration

Piggybak is a pluggable ecommerce engine. To make any models inside your application "sellable", the class method acts_as_variant must be called for any class. This provides a nice flexibility in defining various sellable models throughout the application. Given that I will sell tracks in this example, my first step to supporting downloadable content is adding an is_downloadable boolean and attached file fields to the migration for a sellable item. The migration looks like this:

class CreateTracks < ActiveRecord::Migration
  def change
    create_table :tracks do |t|
      # a bunch of fields specific to tracks

      t.boolean :is_downloadable, :nil => false, :default => false

      t.string :downloadable_file_name
      t.string :downloadable_content_type
      t.string :downloadable_file_size
      t.string :downloadable_updated_at
    end
  end
end

Class Definitions

Next, I update my class definition to make tracks sellable and hook in paperclip functionality:

class Track < ActiveRecord::Base
  acts_as_variant

  has_attached_file :downloadable,
                    :path => ":rails_root/downloads/:id/:basename.:extension",
                    :url => "downloads/:id/:basename.:extension"
end

The important thing to note here is that the attached downloadable files must not be stored in the public root. Why? Because we don't want users to access the files via a URL through the public root. Downloadable files will be served via the send_file call, discussed below.

Shipping

Piggybak's order model has_many shipments. In the case of an order that contains only downloadables, shipments can be empty. To accomplish this, I extend the Piggybak::Cart model using ActiveSupport::Concern to check whether or not an order is downloadable, with the following instance method:

module CartDecorator
  extend ActiveSupport::Concern

  module InstanceMethods
    def is_downloadable?
      items = self.items.collect { |li| li[:variant].item }
      items.all? { |i| i.is_downloadable }
    end
  end
end

Piggybak::Cart.send(:include, CartDecorator)

If all of the cart items are downloadable, the order is considered downloadable and no shipment is generated for this order. With this cart method, I show the FREE! value on the checkout page under shipping methods.

Forcing Log In

The next step for adding downloadable support is to add code to enforce user log in. In this particular project, I assume that downloads are not included as attachments in files since the files may be extremely large. I add a has_downloadable method used to enforce log in:

module CartDecorator
  extend ActiveSupport::Concern

  module InstanceMethods
    ...

    def has_downloadable?
      items = self.items.collect { |li| li[:variant].item }
      items.any? { |i| i.is_downloadable }
    end
  end
end

Piggybak::Cart.send(:include, CartDecorator)

On the checkout page, a user is forced to log in if cart.has_downloadable?. After log in, the user bounces back to the checkout page.

Download List Page

After a user has purchased downloadable products, they'll need a way to access these files. Next, I create a downloads page which lists orders and their downloads:

With a user instance method (current_user.downloads_by_order), the download index page iterates through orders with downloads to display orders and their downloads. The user method for generating orders and downloads shown here:

class User < ActiveRecord::Base
  ...
  def downloads_by_order
    self.piggybak_orders.inject([]) do |arr, order|
      downloads = []
      order.line_items.each do |line_item|
        downloads << line_item.variant.item if line_item.variant.item.is_downloadable?
      end

      arr << {
          :order => order,
          :downloads => downloads
      } if downloads.any?
      arr
    end
  end
end

The above method would be a good candidate for Rails low-level caching or alternative caching which should be cleared after user purchases to minimize download lookup.

Sending Files

As I mentioned above, download files should not be stored in the public directory for public accessibility. From the download list page, the "Download Now" link maps to the following method in the downloads controller:

class DownloadsController < ApplicationController
  def show
    item = ProductType.find(params[:id])

    if current_user.downloads.include?(item)
      send_file "#{Rails.root}/#{item.downloadable.url(:default, false)}"
    else
      redirect_to(root_url, :notice => "You do not have access to this content.")
    end
  end
end

Note that there is additional verification here to check if the current user's downloads includes the download requested. The .url(:default, false) bit hides paperclip's cache buster (e.g. "?123456789") from the url in order to send the file.

Conclusion

This straightforward code accomplished the major updates required for download support: storing and sending the file, enforcing login, and handling shipping. In some cases, download support functionality may be more advanced, but the elements described here make up the most basic building blocks.

If you are interested in this project, check out these related articles:

• Introducing Piggybak: A Mountable Ruby on Rails Ecommerce Engine
• ActiveRecord Callbacks for Order Processing in Ecommerce Applications
• Importing into RailsAdmin: Part 1
• Importing into RailsAdmin: Part 2

Or read more about End Point's web development and consulting services!

RailsAdmin Import: Part 2

I recently wrote about importing data in RailsAdmin. RailsAdmin is a Rails engine that provides a nice admin interface for managing your data, which comes packed with configuration options.

In a recent Ruby on Rails ecommerce project, I've been using RailsAdmin, Piggybak (a Rails ecommerce gem supported by End Point), and have been building out custom front-end features such as advanced search and downloadable product support. When this client came to End Point with the project, we offered several options for handling data migration from a legacy system to the new Rails application:

1. Create a standard migration file, which migrates data from the existing legacy database to the new data architecture. The advantage with this method is that it requires virtually no manual interaction for the migration process. The disadvantage with this is that it's basically a one-off solution and would never be useful again.
2. Have the client manually enter data. This was a reasonable solution for several of the models that required 10 or less entries, but not feasible for the tables containing thousands of entries.
3. Develop import functionality to plug into RailsAdmin which imports from CSV files. The advantage to this method is that it could be reused in the future. The disadvantage with ths method is that data exported from the legacy system would have to be cleaned up and formatted for import.

The client preferred option #3. Using a quick script for generating custom actions for RailsAdmin, I developed a new gem called rails_admin_import to handle import that could be plugged into RailsAdmin. Below are some technical details on the generic import solution.

ActiveSupport::Concern

Using ActiveSupport::Concern, the rails_admin_import gem extends ActiveRecord::Base to add the following class methods:

• import_fields: Returns an array of fields that will be included in the import, excluding :id, :created_at, and :updated_at, belongs_to fields, and file fields.
• belongs_to_fields: Returns an array of fields with belongs_to relationships to other models.
• many_to_many_fields: Returns an array of fields with has_and_belongs_to_many relationships to other models.
• file_fields: Returns an array of fields that represent data for Paperclip attached files.
• run_import: Method for running the actual import, receives request params.

And the following instance methods:

• import_files: sets attached files for object
• import_belongs_to_data: sets belongs_to associated data for object
• import_many_to_many_data: sets many_to_many associated data for object

The general approach here is that the import of files, belongs_to, many_to_many relationships, and standard fields makes up the import process for a single object. The run_import method collects success and failure messages for each object import attempt and those results are presented to the user. A regular ActiveRecord save method is called on the object, so the existing validation of objects during each save applies.

Working with Associated Data

One of the tricky parts here is how to handle import of fields representing associations. Given a user model that belongs to a state, country, and has many roles, how would one decide what state, country, or role value to include in the import?

I've solved this by including a dropdown to select the attribute used for mapping in the form. Each of the dropdowns contains a list of model attributes that are used for association mapping. A user can then select the associated mappings when they upload a file. In a real-life situation, I may import the state data via abbreviation, country via display name (e.g. "United States", "Canada") and role via the role name (e.g. "admin"). My data import file might look like this:

Many to Many Relationships

Many to many relationships are handled by allowing multiple columns in the CSV to correspond to the imported data. For example, there may be two columns for role on the user import, where users may be assigned to multiple roles. This may not be suitable for data with a large number of many to many assignments.

Import of File Fields

In this scenario, I've chosen to use open-uri to request existing files from a URL. The CSV must contain the URL for that file to be imported. The import process downloads the file and attaches it to the imported object.

self.class.file_fields.each do |key|
  if map[key] && !row[map[key]].nil?
    begin
      row[map[key]] = row[map[key]].gsub(/\s+/, "")
      format = row[map[key]].match(/[a-z0-9]+$/)
      open("#{Rails.root}/tmp/uploads/#{self.permalink}.#{format}", 'wb') { |file| file << open(row[map[key]]).read }
      self.send("#{key}=", File.open("#{Rails.root}/tmp/uploads/#{self.permalink}.#{format}"))
    rescue Exception => e
      self.errors.add(:base, "Import error: #{e.inspect}")
    end
  end
end

If the file request fails, an error is added to the object and presented to the user. This method may not be suitable for handling files that do not currently exist on a web server, but it was suitable for migrating a legacy application.

Configuration: Display

Following RailsAdmin's example for setting configurations, I've added the ability to allow the import display to be set for each model.

config.model User do
  label :name
end

The above configuration will yield success and error messages with the user.name, e.g.:

Configuration: Excluded Fields

In addition to allowing a configurable display option, I've added the configuration for excluding fields.

config.model User do
  excluded_fields do
    [:reset_password_token, :reset_password_sent_at, :remember_created_at,
      :sign_in_count, :current_sign_in_at, :last_sign_in_at, :current_sign_in_ip,
      :last_sign_in_ip]
  end
end

The above configuration will exclude the specified fields during the import, and they will not display on the import page.

Configuration: Additional Fields and Additional Processing

Another piece of functionality that I found necessary for various imports was to hook in additional import functionality. Any model can have an instance method before_import_save that accepts the row of CSV data and map of CSV keys to perform additional tasks. For example:

def before_import_save(row, map)
  self.created_nested_items(row, map)
end

The above method will create nested items during the import process. This simple extensibility allows for additional data to be handled upon import outside the realm of has_and_belongs_to and belongs_to relationships.

Fields for additional nested data can be defined with the extra_fields configuration, and are shown on the import page.

config.model User do
  extra_fields do
    [:field1, :field2, :field3, :field4]
  end
end

Hooking into RailsAdmin

As I mentioned above, I used a script to generate this Engine. Using RailsAdmin configurable actions, import must be added as an action:

config.actions do
  dashboard
  index
  ...
  import
end

And CanCan settings must be updated to allow for import if applicable, e.g.:

cannot :import, :all
can :import, User

Conclusion

My goal in developing this tool was to produce reusable functionality that could easily be applied to multiple models with different import needs, and to use this tool across Rails applications. I've already used this gem in another Rails 3.1 project to quickly import data that would otherwise be difficult to deal with manually. The combination of association mapping and configurability produces a flexibility that encourages reusability.

Feel free to review or check out the code here, or read more about End Point's services here.

Importing Data with RailsAdmin

Update #1: Read an update to this functionality here.

Update #2: This article was written in January of 2012, and the code related to the RailsAdmin actions no longer applies to the current release. Please make sure to read the RailsAdmin documentation regarding current action implementation.

I've blogged about RailsAdmin a few times lately. I've now used it for several projects, and have included it as a based for the Admin interface my recent released Ruby on Rails Ecommerce Engine (Piggybak). One thing that I found lacking in RailsAdmin is the ability to import data. However, it has come up in the RailsAdmin Google Group and it may be examined in the future. One problem with developing import functionality is that it's tightly coupled to the data and application logic, so building out generic import functionality may need more thought to allow for elegant extensibility.

For a recent ecommerce project using RailsAdmin and Piggybak, I was required to build out import functionality. The client preferred this method to writing a simple migration to migrate their data from a legacy app to the new app, because this import functionality would be reusable in the future. Here are the steps that I went through to add Import functionality:

#1: Create Controller

class CustomAdminController < RailsAdmin::MainController
  def import
    # TODO
  end
end

First, I created a custom admin controller for my application in the app/controllers/ directory that inherits from RailsAdmin::MainController. This RailsAdmin controller has several before filters to set the required RailsAdmin variables, and defines the correct layout.

#2: Add import route

match "/admin/:model_name/import" => "custom_admin#import" , :as => "import", :via => [:get, :post]
mount RailsAdmin::Engine => '/admin', :as => 'rails_admin'

In my routes file, I introduced a new named route for import to point to the new custom controller. This action will be a get or a post.

#3: Override Rails Admin View

Next, I copied over the RailsAdmin app/views/rails_admin/_model_links.html.haml view to my application to override RailsAdmin's view. I made the following addition to this file:

...
- can_import = authorized? :import, abstract_model

...
%li{:class => (params[:action] == 'import' && 'active')}= link_to "Import", main_app.import_path(model_name) if can_import

With this logic, the Import tab shows only if the user has import access on the model. Note that the named route for the import must be prefixed with "main_app.", because it belongs to the main application and not RailsAdmin.

#4: CanCan Settings

My application uses CanCan with RailsAdmin, so I leveraged CanCan to control which models are importable. The CanCan Ability class (app/models/ability.rb) was updated to contain the following, to allow exclude import on all models, and then allow import on several specific models.

if user && user.is_admin?
  cannot :import, :all
  can :import, [Book, SomeModel1, SomeModel2, SomeModel3]
end

I now see an Import tab in the admin:

#5: Create View

Next, I created a view for displaying the import form. Here's a generic example to display the set of fields that can be imported, and the form:

<h1>Import</h1>
<h2>Fields</h2>
<ul>
    <% @abstract_model::IMPORT_FIELDS.each do |attr| -%>
    <li><%= attr %></li>
    <% end -%>
</ul>

<%= form_tag "/admin/#{@abstract_model.to_param}/import", :multipart => true do |f| -%>
    <%= file_field_tag :file %>
    <%= submit_tag "Upload", :disable_with => "Uploading..." %>
<% end -%>

This will look something like this:

#6: Import Functionality

Finally, the code for the import looks something like this:

def import
  if request.post?
    response = { :errors => [], :success => [] }
    file = CSV.new(params[:file].tempfile)

    # Build map of attributes based on first row
    map {}
    file.readline.each_with_index { |key, i| map[key.to_sym] = i }

    file.each do |row|
      # Build hash of attributes
      new_attrs = @abstract_model.model::IMPORT_FIELDS.inject({}) { |hash, a| hash[a] = row[map[a]] if map[a] }

      # Instantiate object
      object = @abstract_model.model.new(new_attrs)

      # Additional special stuff here

      # Save
      if object.save
        response[:success] << "Created: #{object.title}"
      else
        response[:error] << "Failed to create: #{object.title}. Errors: #{object.errors.full_messages.join(', ')}."
      end
    end
  end
end

Note that a hash of keys and locations is created to map keys to the columns in the imported file. This allows for flexibility in column ordering of imported files. Later, I'd like to to re-examine the CSV documentation to identify if there is a more elegant way to handle this.

#7: View updates to show errors

Finally, I update my view to show both success and error messages, which looks sorta like this in the view:

Conclusion and Discussion

It was pretty straightforward to get this figured out. The only disadvantage I see to this method is that overriding the rails_admin view requires recopying or manual updates to the view over during upgrades of the gem. For example, if any part of the rails_admin view has changes, those changes must also be applied to the custom view. Everything else should be smooth sailing :)

In reality, my application has several additional complexities, which make it less suitable for generic application:

• Several of the models include attached files via paperclip. Using open-uri, these files are retrieved and added to the objects.
• Several of the models include relationships to existing models. The import functionality requires lookup of these associated models (e.g. an imported book belongs_to an existing author), and reports and error if the associated objects can not be found.
• Several of the models require creation of a special nested object. This was model specific.
• Because of this model specific behavior, the import method is moved out of the controller into model-specific class methods. For example, CompactDisc.import is different from Book.import which is different from Track.import. Pulling the import into a class method also makes for a skinnier controller here.

Read more about End Point's Ruby on Rails development and consulting services or contact us to help you out with a Rails project today!

Using Disqus and Ruby on Rails

Recently, I posted about how to import comments from a Ruby on Rails app to Disqus. This is a follow up to that post where I outline the implementation of Disqus in a Ruby on Rails site. Disqus provides what it calls Universal Code which can be added to any site. This universal code is just JavaScript, which asynchronously loads the Disqus thread based on one of two unique identifiers Disqus uses.

Disqus in a development environment

Before we get started, I'd recommend that you have two Disqus "sites"; one for development and one for production. This will allow you to see real content and experiment with how things will really behave once you're in production. Ideally, your development server would be publicly accessible to allow you to fully use the Disqus moderation interface, but it isn't required. Simply register another Disqus site, and make sure that you have your shortname configured by environment. Feel free to use whatever method you prefer for defining these kinds of application preferences. If you're looking for an easy way, considering checking out my article on Working with Constants in Ruby. It might look something like this:

# app/models/article.rb

DISQUS_SHORTNAME = Rails.env == "development" ? "dev_shortname".freeze : "production_shortname".freeze

Disqus Identifiers

Each time you load the universal code, you need to specify a few configuration variables so that the correct thread is loaded:

• disqus_shortname: tells Disqus which website account (called a forum on Disqus) this system belongs to.
• disqus_identifier: tells Disqus how to uniquely identify the current page.
• disqus_url: tells Disqus the location of the page for permalinking purposes.

# app/views/disqus/_thread.html.erb
# assumes you've passed in the local variable 'article' into this partial
# from http://docs.disqus.com/developers/universal/

<div id="disqus_thread"></div>
<script type="text/javascript">

    var disqus_shortname = '<%= Article::DISQUS_SHORTNAME %>';
    var disqus_identifier = '<%= article.id %>';
    var disqus_url = '<%= url_for(article, :only_path => false) %>';

    /* * * DON'T EDIT BELOW THIS LINE * * */
    (function() {
        var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
        dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
        (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
    })();
</script>

The above code will populate the div#disqus_thread with the correct content based on your disqus_identifier. By setting up a single partial that will always render your threads, it becomes very easy to adjust this code if needed.

Disqus Identifier Gotcha

We found during our testing a surprising and unexpected behavior in how Disqus associates a thread to a URL. In our application, the landing page was designed to show the newest article as well as the Disqus comments thread. We found that once a new article was posted, the comments from the previous article were still shown! It seems Disqus ignored the unique disqus_identifier we had specified and instead associated the thread with the landing page URL. In our case, a simple routing change allowed us to forward the user to the unique URL for that content and thread. In your case, there may not be such an easy work around, so be certain you include both the disqus_identifier and disqus_url JavaScript configuration variables above to minimize the assumptions Disqus will make. When at all possible, always use unique URLs for displaying Disqus comments.

Comment Counters

Often an index page will want to display a count of how many comments are in a particular thread. Disqus uses the same asynchronous approach to loading comment counts. Comment counts are shown by adding code such as the following where you want to display your count:

# HTML
<a href="http://example.com/article1.html#disqus_thread" 
   data-disqus-identifier="<%=@article.id%>">
This will be replaced by the comment count
</a>

# Rails helper
<%= link_to "This will be replaced by the comment count", 
    article_path(@article, :anchor => "disqus_thread"), 
    :"data-disqus-identifer" => @article.id %>

At first this seemed strange, but it is the exact same pattern used to display the thread. It would likely be best to remove the link text so nothing is shown until the comment count is loaded, but I felt for my example, having some meaning to the test would help understanding. Additionally, you'll need to add the following JavaScript to your page.

# app/view/disqus/_comment_count_javascript.html.erb
# from http://docs.disqus.com/developers/universal/
# add once per page, just above </body>

<script type="text/javascript">
   
    var disqus_shortname = '<%= Article::DISQUS_SHORTNAME %>';

    /* * * DON'T EDIT BELOW THIS LINE * * */
    (function () {
        var s = document.createElement('script'); s.async = true;
        s.type = 'text/javascript';
        s.src = 'http://' + disqus_shortname + '.disqus.com/count.js';
        (document.getElementsByTagName('HEAD')[0] || document.getElementsByTagName('BODY')[0]).appendChild(s);
    }());
</script>

Disqus recommends adding it just before the closing </body> tag. You only need to add this code ONCE per page, even if you're planning on showing multiple comment counts on a page. You will need this code on any page with a comment count, so I do recommend putting it in a partial. If you wanted, you could even include it in a layout.

Styling Comment Counts

Disqus provides extensive CSS documentation for its threads, but NONE for its comment counters. In our application, we had some very particular style requirements for these comment counts. I found that in Settings > Appearance, I could add HTML tags around the output of the comments.

This allowed me to style my comments as needed, although these fields are pretty small, so make sure to compress your HTML as much as possible.

ActiveRecord Callbacks for Order Processing in Ecommerce Applications

As I recently blogged about, I introduced a new Ruby on Rails Ecommerce Engine. The gem relies on RailsAdmin, a Ruby on Rails engine that provides a nice interface for managing data. Because the RailsAdmin gem drives order creation on the backend in the context of a standard but configurable CRUD interface, and because I didn't want to hack at the RailsAdmin controllers, much of the order processing logic leverages ActiveRecord callbacks for processing. In this blog article, I'll cover the process that happens when an order is saved.

Order Data Model

The first thing to note is the data model and the use of nested attributes. Here's how the order model relates to its associated models:

class Order < ActiveRecord::Base
  has_many :line_items, :inverse_of => :order
  has_many :payments, :inverse_of => :order
  has_many :shipments, :inverse_of => :order
  has_many :credits, :inverse_of => :order

  belongs_to :billing_address, :class_name => "Piggybak::Address"
  belongs_to :shipping_address, :class_name => "Piggybak::Address"
  belongs_to :user
  
  accepts_nested_attributes_for :billing_address, :allow_destroy => true
  accepts_nested_attributes_for :shipping_address, :allow_destroy => true
  accepts_nested_attributes_for :shipments, :allow_destroy => true
  accepts_nested_attributes_for :line_items, :allow_destroy => true
  accepts_nested_attributes_for :payments
end

An order has many line items, payments, shipments and credits. It belongs to [one] billing and [one] shipping address. It can accept nested attributes for the billing address, shipping address, multiple shipments, line items, and payments. It cannot destroy payments (they can only be marked as refunded). In terms of using ActiveRecord callbacks for an order save, this means that all the nested attributes will also be validated during the save. Validation fails if any nested model data is not valid.

Step #1: user enters data, and clicks submit

Step #2: before_validation

Using a before_validation ActiveRecord callback, a few things happen on the order:

• Some order defaults are set
• The order total is reset
• The order total due is reset

Step #3: validation

This happens without a callback. This method will execute validation on both order attributes (email, phone) and nested element attributes (address fields, shipment information, payment information, line_item information).

Payments have a special validation step here. A custom validation method on the payment attributes is performed to confirm validity of the credit card:

validates_each :payment_method_id do |record, attr, value|
  if record.new_record?
    credit_card = ActiveMerchant::Billing::CreditCard.new(record.credit_card)
    
    if !credit_card.valid?
      credit_card.errors.each do |key, value|
        if value.any? && !["first_name", "last_name", "type"].include?(key)
          record.errors.add key, value
        end
      end
    end
  end
end

This bit of code uses ActiveMerchant's functionality to avoid reproducing business logic for credit card validation. The errors are added on the payment attributes (e.g. card_number, verification_code, expiration date) and presented to the user.

Step #4: after_validation

Next, the after_validation callback is used to update totals. It does a few things here:

• Calculates shipping costs for new shipments only.
• Calculates tax charge on the order.
• Subtracts credits on the order, if they exist.
• Calculates total_due, to be used by payment

While these calculations could be performed before_validation, after_validation is a bit more performance-friendly since tax and shipping calculations could in theory be expensive (e.g. shipping calculations could require calling an external API for real-time shipping lookup). These calculations are saved until after the order is confirmed to be valid.

Step #5: before_save part 1

Next, a before_save callback handles payment (credit card) processing. This must happen after validation has passed, and it can not happen after the order has saved because the user must be notified if it fails. If any before_save method returns false, the entire transaction fails. So in this case, after all validation has passed, and before the order saves, the payment must process successfully.

Examples of failures here include:

• Credit card transaction denied for a number of reasons
• Payment gateway down
• Payment gateway API information incorrect

Step #6: before_save part 2

After the payment processes, another before_save method is called to update the status of the order based on the totals paid. I initially tried placing this in an after_save method, but you tend to experience infinite loops if you try to save inside and after_save callback :)

Step #7: Save

Finally, if everything's gone through, the order is saved.

Summary

As I mentioned above, the RailsAdmin controllers were not extended or overridden to handle backroom order processing. All of the order processing is represented in the Order model in these active record callbacks. This also allows for the frontend order processing controller to be fairly lightweight, which is a standard practice for writing clean MVC code.

Check out the full list of ActiveRecord callbacks here. And check out the Order model for Piggybak here.

Take a snapshot in Cucumber and sync it with Dropbox!

In a previous post I talked about running cucumber using capybara-webkit. In a recent project using this setup I noticed that I couldn't use capybara in connection with launchy to open up a page in the browser for debugging tests. The "save and open page" step is one that I used a lot when I was developing locally. But now that I'm developing on a server, I don't have any way save the page or open it for review.

The solution I found to this comes in two parts. First, create a "take a snapshot" cucumber step that drops a snapshot of the HTML and a PNG of the page in a temp directory. Second, add that temp directory to dropbox so that it gets synced to my desktop automatically when it is created.

Wait, seriously? Dropbox?

Yes, absolutely. Dropbox.

I often develop inside of my dropbox folder because A) all my code is automatically backed up, even with versions and B) because it's really simple to sync my code to other computers. I'll admit that one problem I had early on was that log files were using an awful amount of bandwidth getting copied up constantly, but I solved this by adding the log directory to an exclusions list. I'll show you how to do that below.

Step 1: Create a "take a snapshot" step.

The first thing we need to do is setup our take a snapshot step. For our app, it made the most sense to put this in web_steps.rb but you can add it to any of your step_definitions files. The step looks like this:

Then /take a snapshot/ do
  # save a html snapshot
  html_snapshot = save_page
  puts "Snapshot saved: \n#{html_snapshot}"

  # save a png snapshot
  png_snapshot = html_snapshot.gsub(/html$/, "png")
  page.driver.render png_snapshot
  puts "Snapshot saved: \n#{png_snapshot}"

end

The first line there is fairly simple. Capybara provides a save_page method by default which is going to save the page to tmp/capybara off the root of your app. The file will look something like this: capybara-20111228210921591550991.html. You can see the source code on the rdoc page for more information on how this works. If you want to customize the file name, you can do that by calling Capybara.save_page directly like this:

Capybara.save_page(body, 'my_custom_file.html')

This reveals what save_page is actually doing. Every cucumber step has available to it by default Capybara::Session. The source html of the current page is stored in Capybara::Session#body. The save_page method is just writing the source html to a file.

The next block of code saves the page as a PNG file. This comes from the capybara-webkit driver so this will only work if you are using that driver specifically. (You can explore the code on github to get more information, but basically it's calling the "Render" command on webkit and then storing the image as a PNG.)

All I'm doing here is changing out the html file extension for png so that the files will be easy to find. You can also pass width and height options if you'd like with a hash {:width => 1000, :height => 10}.

Step 2: Setup Dropbox.

I didn't know this until recently but you can run dropbox on a linux server without a UI. It is available as a binary for Ubuntu (.deb), Fedora (.rpm), Debian (.deb). You can also compile from source if you'd like. Since I'm doing my development on a server, however, I wanted a little bit more of an isolated installation and luckily Dropbox has the answer for me. It's called their command line installation and it works great. Here are the instructions from the web site:

For 32 bit:

cd ~ && wget -O - http://www.dropbox.com/download?plat=lnx.x86 | tar xzf -

or 64 bit:

cd ~ && wget -O - http://www.dropbox.com/download?plat=lnx.x86_64 | tar xzf -

You'll also need a small python script for working with the daemon. You can download it from the web site or from here.

The dropbox cli will walk you through a simple authentication process and then start downloading your dropbox folder in your ~/Dropbox directory when you run dropbox.py start for the first time. If your dropbox folder is like mine, this is going to download way more stuff than you'd probably want on your server so you'll need to add some exceptions. I created a folder in Dropbox for my screenshots called ~/Dropbox/cuke_snapshots and then excluded everything else. Here's how I did it with the dropbox.py file (I renamed dropbox.py to just dropbox for ease and clarity. It's also helpful to put it in a directory that's in your PATH):

cd ~/Dropbox
dropbox exlcude add Public Photos

That adds the Public and Photos folders to the exclusion list and Dropbox deletes them from the system for you. The nice thing is that you can continue to add folder names to the end of that command so you can get rid of stuff really quick. There are a bunch of options using the dropbox cli that make working with dropbox on the server very simple and flexible.

status       get current status of the dropboxd
help         provide help
puburl       get public url of a file in your dropbox
stop         stop dropboxd
running      return whether dropbox is running
start        start dropboxd
filestatus   get current sync status of one or more files
ls           list directory contents with current sync status
autostart    automatically start dropbox at login
exclude      ignores/excludes a directory from syncing
lansync      enables or disables LAN sync

The last step is to create a symbolic link from your app into your dropbox folder. I did this with the following command:

mkdir -p ~/Dropbox/cuke_snapshots/my_app/capybara
cd ~/rails_apps/my_app/tmp
rm -rf capybara   (if it already exists)
ln -s ~/Dropbox/cuke_snapshots/my_app/capybara

Of course there are many ways you could set this up, but this will get the job done.

Using "take a snapshot"

Once you have everything setup, all you need to do is call the cucumber step from within your scenarios. Here's a contrived example:

@javascript   # may not be needed if everything is using webkit-capybara
Scenario: A shopper wants to checkout
  When I go to the address step
  And I fill in the address information
  And I follow "Next"
  Then I should be on the delivery step
  And take a snapshot

When the scenario runs, it will drop the html and png file in your dropbox directory which will immediately be synced to your local machine. In my experience, by the time I open up the folder on my local machine, the file is there ready for inspection.

Importing Comments into Disqus using Rails

It seems everything is going to the cloud, even comment systems for blogs. Disqus is a platform for offloading the ever growing feature set users expect from commenting systems. Their website boasts over a million sites using their platform and offers a robust feature set and good performance. But before you can drink the Kool-Aid, you've got to get your data into their system.

If you're using one of the common blog platforms such as WordPress or Blogger, there are fairly direct routes Disqus makes available for automatically importing your existing comment content. For those with an unsupported platform or a hand-rolled blog, you are left with exporting your comments into XML using WordPress's WXR standard.

Disqus leaves a lot up to the exporter, providing only one page in there knowledge base for using what they describe as a Custom XML Import Format. In my experience the import error messages were cryptic and my email support request is still unanswered 5 days later. (Ok, so it was Christmas weekend!)

So let's get into the nitty gritty details. First, the sample code provided in this article is based on Rails 3.0.x, but should work with Rails 3.1.x as well. Rails 2.x would work just as well by modifying the way the Rails environment is booted in the first lines. I chose to create a script to dump the output to standard output which could be piped in to a file for upload. Let's see some of the setup work.

Setting up a Rails script

I choose to place the script in the RAILS_ROOT/scripts directory and named it wxr_export.rb. This would allow me to call the script with the Rails 2.x style syntax (ahh, the nostalgia):

script/wxr_export.rb > comments.xml

This would fire up the full Rails enviornment, execute our Ruby code, and pipe the standard output to a file called comments.xml. Pretty straightforward, but it's not that often Rails developers think about creating these kind of scripts, so it's worth discussing to see the setup mechanics.

#!/usr/bin/env ruby
require File.expand_path('../../config/boot', __FILE__)
require File.expand_path('../../config/environment', __FILE__)

I think the first line is best explained by this excerpt from Ruby Programming:

First, we use the env command in the shebang line to search for the ruby executable in your PATH and execute it. This way, you will not need to change the shebang line on all your Ruby scripts if you move them to a computer with Ruby installed a different directory.

The next two lines are essentially asking the script to boot the correct Rails environment (development, testing, production). It's worth briefly offering an explanation of the syntax of these two somewhat cryptic lines. File#expand_path converts a pathname to an absolute pathname. If passed only the first string, it would use the current working path to evaluate, but since we pass __FILE__ we are asking it to use the current file's path as the starting point.

The config/boot.rb file is well documented in the Rails guides which explains that boot.rb defines the location of your Gemfile, hooks up Bundler, which adds the dependencies of the application (including Rails) to the load path, making them available for the application to load.

The config/enviornment.rb file is also well documented and effectively loads the Rails packages you've specified, such as ActiveModel, ActiveSupport, etc...

Exporting WXR content

Having finally loaded our Rails enviornment in a way we can use it, we are ready to actually build the XML we need. First, let's setup our XML and the gerenal format we'll use to popualate our file:

# script/wxr_export.rb

xml = Builder::XmlMarkup.new(:target => STDOUT, :indent => 2)

xml.instruct! :xml, :version=>"1.0", :encoding=>"UTF-8"

xml.rss 'version' => "2.0",
        'xmlns:content' => "http://purl.org/rss/1.0/modules/content/",
        'xmlns:dsq' => "http://www.disqus.com/",
        'xmlns:dc' => "http://purl.org/dc/elements/1.1/",
        'xmlns:wp' => "http://wordpress.org/export/1.0/" do
 
  xml.channel do
    Articles.all.each do |article|
      if should_be_exported?(article)
        xml.item do

          #Article XML goes here

   article.comments.each do |comment|
      
     #Comments XML goes here

   end #article.comments.each
 end   #xml.item
      end     #if should_be_exported?
    end       #Articles.all.each
  end         #xml.channel
end           #xml.rss

This is the general form for the WXR format as described by Disqus's knowledge base article. Note that you need to nest the comments inside each specific Article's XML. I found that I needed to filter some of my output so I added a helper function called should_be_exported? which can be defined at the top of the script. This would allow you to exclude Articles without comments, or whatever criteria you might find helpful.

With our basic format in place, let's look at the syntax for exporting the Article fields. Keep in mind that the fields you'll want to pull from in your system will likely be different, but the intention is the same.

Inside the Article XML block

# script/wxr_export.rb

# Inside the Article XML block

xml.title article.title

xml.link create_url_for(article)

xml.content(:encoded) { |x| x << "" }

xml.dsq(:thread_identifier) { |x| x << article.id }

xml.wp(:post_date_gmt) { |x| x << article.created_at.utc.to_formatted_s(:db) }

xml.wp(:comment_status) { |x| x << "open" } #all comments open

Let's look at each of these fields one by one:

• xml.title: This is pretty straight forward, just the plain text tile of the blog article.
• xml.link: Disqus can use URLs for determining which comments to display on your page, so it asks you to provide a URL associated with this article. I found that for this particular app, it would be easier to write another helper function to generate the URLs then using the Rails routes. If you wish to use the Rails routes (and I suggest you do), then I suggest checking out this excellent post for using routes outside of views.
• xml.content(:encoded): The purpose of this field is clear, but the syntax is not. Hope this saves you some time and headache!
• xml.dsq(:thread_identifier): The other way Disqus can identify your article is by a unique identifier. This is strongly recommended over the use of a URL. We'll just use your unique identifier in the database.
• xml.wp(:post_date_gmt): The thing to keep in mind here is that we need the date in a very particular format. It needs to be in YYYY-MM-DD HH:MM:SS 24-hour format and adjusted to GMT which typically implies UTC. Rails 3 makes this very easy for us, bless their hearts.
• xml.wp(:comment_status): This app wanted to leave all comments open. You may have different requirements so consider adding a helper function.

Inside the Comment XML block

article.comments.each do |comment|
  
  xml.wp(:comment) do

    xml.wp(:comment_id) { |x| x << comment.id }

    xml.wp(:comment_author) do |x| 
      if comment.user.present? && comment.user.name.present?
        x << comment.user.name
      else
 x << ""
      end 
    end 
                  
    xml.wp(:comment_author_email) do |x| 
      if comment.user.present? && comment.user.email.present?
        x << comment.user.email
      else
        x << ""
      end 
    end 

    xml.wp(:comment_author_url) do |x|
      if comment.user.present? && comment.user.url.present?
        x << comment.user.url
      else
        x << ""
      end
    end

    xml.wp(:comment_author_IP) { |x| x << "255.255.255.255" }

    xml.wp(:comment_date_gmt) { |x| x << comment.created_at.utc.to_formatted_s(:db) }

    xml.wp(:comment_content) { |x| x << "" }

    xml.wp(:comment_approved) { |x| x << 1 } #approve all comments

    xml.wp(:comment_parent) { |x| x << 0 }

  end #xml.wp(:comment)
end #article.comments.each

Again, let's inspect this one field at a time:

• xml.wp(:comment_id): Straightforward, a simple unique identifier for the comment.
• xml.wp(:comment_author): Because some commentors may not have a user associated with them, I added some extra checks to make sure the author's user and name were present. I'm sure there's a way to shorten the number of lines used, but I was going for readability here. I'm not certain it was necessary to include the blank string, but after some of the trouble I had importing, I wanted to minimize the chance of strange XML syntax issues.
• xml.wp(:comment_author_email): More of the same safe guards of having empty data.
• xml.wp(:comment_author_url): More of the same safe guards of having empty data.
• xml.wp(:comment_author_IP): We were not collecting user IP data, so I put in some bogus data which Disqus did not seem to mind.
• xml.wp(:comment_date_gmt): See xml.wp(:post_date_gmt) above for comments about date/time format.
• xml.wp(:comment_content): See xml.content(:encoded) above for comments about encoding content.
• xml.wp(:comment_approved): Two options here, 0 or 1. Typically you'd want to automatically approve your existing comments, unless of course you wanted to give a moderator a huge backlog of work.
• xml.wp(:comment_parent): This little field turned out to be the cause of a lot of trouble for me. In the comments on Disqus's XML example, it says parent id (match up with wp:comment_id), so initially, I just put in the comment's ID in this field. This returned the very unhelpful error * url * URL is required to which I still have my unanswered supprot email in to Disqus. By trial error, I found that by just setting the comment_parent to zero, I could successfully upload my comment content. If you are using threaded comments, I suspect this field will be of more importance to you then it was to me. When I hear from Disqus, I will update this article with more information.

Rails Request-Based Routing Constraints in Spree

I recently adopted an unreleased ecommerce project running Spree 0.60.0 on Rails 3.0.9. The site used a Rails routing constraint and wildcard DNS to dynamically route subdomains to the “dispatch” action of the organizations_controller. If a request’s subdomain component matched that regular expression, it was routed to the dispatch method. Here's the original route:

match '/' => 'organizations#dispatch', :constraints => { :subdomain => /.+/ }

The business requirement driving this feature was that a User could register an Organization by submitting a form on the site. Once that Organization was marked "approved" by an admin, that Organization would become accessible at their own subdomain - no server configuration required.

For marketing reasons, we decided to switch from subdomains to top-level subdirectories. This meant RESTful routes (e.g. domain.com/organizations/143) wouldn’t cut it. In order to handle this, I created a routing constraint class called OrgConstraint. This routing constraint class works in tandem with a tweaked version of that original route.

match '*org_url' => 'organizations#show', :constraints => OrgConstraint.new

The :constraints param takes an instance of a class (not a class name) that responds to a matches? predicate method that returns true or false. If matches? returns true, the request will be routed to that controller#action. Else, that route is treated like any other non-matching route. Here’s the entire OrgConstraint class:

class OrgConstraint
  def matches?(request)
    Organization.valid_url? request.path_parameters[:org_url]
  end
end

Note how Rails automatically passes the request object to the matches? method. Also note how the relative url of the request is available via the :org_url symbol - the same identifier we used in the route definition. The Organization.valid_url? class method encapsulates the logic of examining a simple cached (via Rails.cache) hash consisting of organization urls as keys and true as their value.

The final step in this process is, of course, the organizations_controller’s show method. It now needs to look for that same :org_url param that the route definition creates, in the standard params hash we all know and love:

def show
  @organization = Organization.find(params[:id]) if params[:id]  
  # from routing constraint
  @organization ||= Organization.find_by_url(params[:org_url]) if params[:org_url]  
  ...
end

I should point out that Rails instantiates exactly one instance of your routing constraint class when it first loads your routes. This means you’ll want to ensure your class’s design will respond appropriately to changes in any underlying data. This is one of the reasons the Organization class caches the {$org_url => true} hash rather than using instance variables within the OrgConstraint class.

Modifying Models in Rails Migrations

As migrations have piled up in projects that I work on, one problem seems to come up fairly consistently. New changes to models can break migrations.

This can happen a number of different ways. One way is to break old migrations. Another is for the changes to be made to the file before the migration is run (timing issues with version control).

While these can be (and usually are) considered coordination rather than technical issues, sometimes you just need to handle them and move on.

One case I'd like to cover here is removing or changing associations. At the time the migration is expected to run, the file for the model class will have been updated already, so it is hard use that in the migration itself, even though it would be useful.

In this case I found myself with an even slightly trickier example. I have a model that contains some address info. Part of that is an association to an external table that lists the states. So part of the class definition was like so:

Class Contact 
 belongs_to :state
 ...
end

What I needed to do in the migration was to remove the association and introduce another field called "state" which would just be a varchar field representing the state part of the address. The two problems the migration would encounter were:

1. The state association would not exist at the time it ran
2. And even if it did, there would be a name conflict between it and the new column I wanted

To get around these restrictions I did this in my migration:

Contact.class_eval do
  belongs_to :orig_state,
             :class_name => "State",
             :foreign_key => "state_id"
end

This creates a different association named "orig_state" using the states table for the Contact class. I can now use my original migration code more-or-less as is, and still create a new state column. column.

Another problem I had was that the table had about 300 rows of data that failed one of the validations called "validate_names". I didn't feel like sorting it out, so I just added the following code to the above class_eval block:

define_method(:validate_names) do
  true
end

With these two modifications to the Contact class, I was able to use the simple migration with all of my Rails associations to do what I needed in the migration without resorting to hand crafting more complex SQL that would have been required in order to not have to refer to the model classes at all in the migration.

Nifty In-Button Confirmation

I've been working on a personal email client after work, called Warm Sunrise, that forces myself to keep a manageable inbox. One of the goals of the project was to get to a zero-inbox everyday, so I needed a 'Delete All' button that was easy-to-use without running the risk of accidentally deleting emails. I took a look at JavaScript's confirm, which is jarring, and jQuery's dblClick, which doesn't provide any feedback to the user after the first click, leaving the user to wonder why their emails weren't deleted.

Given these options, I built my own button using Rails 3.1, jQuery, and CoffeeScript, that better fit the goals I set out with. It requires a double click, but gives the user a confirmation in the button itself, without any sort of timeout. You can see a video of it in action here:

Starting with app/views/letters/index.html.erb, I generated the buttons using Rails helpers and Twitter's Bootstrap classes:

<%= link_to 'Write letter', new_letter_path, :class => "btn primary pull-right far-right" %>
<%= link_to 'Delete all', '#', :class => "btn pull-right no_danger", :id => "delete_all" %>
<%= link_to 'Are you sure?', delete_all_letters_path, :method => :destroy, :class =>"btn pull-right danger confirm", :id => "delete_all", :style => "display:none;" %>

Notice that the 'Delete all' button doesn't actually specify a url and the 'Are you sure?' link's style is set to "display:none"

Here's the relationship I set up in my models:

app/models/letter.rb

belongs to :user

app/models/user.rb

has_many :letters, :dependent => :destroy

I set up config/routes.rb to work with the explicit path I set in:

post 'delete_all_letters' => 'letters#delete_all'

Finally, I finished this lot by adding the delete_all action to my app/controllers/letters_controller.rb:

def delete_all 
    current_user.letters.delete_all

    respond_to do |format|
        format.html { redirect_to letters_url, notice: 'Successfully deleted all letters.' }
        format.json { head :ok }
    end 
end 

CoffeeScript is a beautiful language that compiles to JavaScript, which I prefer to JavaScript itself. You can read more about it here. Let's take a look at the CoffeeScript that makes this button work:

$('a#delete_all.no_danger').hover( ->
    $(this).addClass('danger')
    $(this).click( ->
        $('a#delete_all.no_danger').hide()
        $('a#delete_all.confirm').show()
    )   
)
$('a#delete_all.no_danger').mouseleave( ->
    $(this).removeClass('danger')
)
$('a#delete_all.danger').mouseleave( ->
    $(this).hide()
    $('a#delete_all.no_danger').show()
)

Since the button's text changes to a confirmation on the first click, makes it better for my purposes than Javascript's dblClick method. Check the video to see what it looks like in action.

Let's take a look at what this compiles to in plain JavaScript, too, since this is the only thing the browser sees:

$('a#delete_all.no_danger').hover(function() {
    $(this).addClass('danger');
    return $(this).click(function() {
        $('a#delete_all.no_danger').hide();
        return $('a#delete_all.confirm').show();
    });
});
$('a#delete_all.no_danger').mouseleave(function() {
    return $(this).removeClass('danger');
});
$('a#delete_all.danger').mouseleave(function() {
    $(this).hide();
    return $('a#delete_all.no_danger').show();
});

Not shown in the video, but I modified index.html.erb to only show the 'Delete all' button when the user has a zero-inbox.

<%= link_to 'Write letter', new_letter_path, :class => "btn primary pull-right far-right" %>
<% if !@letters.empty? %>
    <%= link_to 'Delete all', '#', :class => "btn pull-right no_danger", :id => "delete_all" %>
    <%= link_to 'Are you sure?', delete_all_letters_path, :method => :destroy, :class =>"btn pull-right danger confirm", :id => "delete_all", :style => "display:none;" %>
<% end %>

Sunspot, Solr, Rails: Working with Results

Having worked with Sunspot and Solr in several large Rails projects now, I've gained some knowledge about working with result sets optimally. Here's a brief explanation on working with results or hits from a search object.

MVC Setup

When working with Sunspot, searchable fields are defined in the model:

class Thing < ActiveRecord::Base
  searchable do
    text :field1, :stored => true
    text :field2
    string :field3, :stored => true
    integer :field4, :multiple => true
  end
end

The code block above will include field1, field2, field3, and field4 in the search index of things . A keyword or text search on things will search field1 and field2 for matches. field3 and field4 may be used for scoping, or limiting the search result set based to specific values of field3 or field4.

In your controller, a new search object is created with the appropriate scoping and keyword values, shown below. Pagination is also added inside the search block.

class ThingsController < ApplicationController
  def index
    @search = Sunspot.search(Thing) do
      #fulltext search
      fulltext params[:keyword]

      #scoping
      if params.has_key?(:field3)
        with :field3, params[:field3]
      end 
      if params.has_key?(:field4)
        with :field3, params[:field4]
      end

      paginate :page => params[:page], :per_page => 25
    end
    @search.execute!
  end
end

In the view, one can iterate through the result set, where results is an array of Thing instances.

<% @search.results.each do |result| -%>
<h2><%= result.field3 %></h2>
<%= result.field1 %>
<% end -%>

Working with Hits

The above code works. It works nicely until you display many results on one page where instantiation of things is not expensive. But the above code will call the query below for every search, and subsequently instantiate Ruby objects for each of the things found. This can become sluggish when the result set is large or the items themselves are expensive to instantiate.

# development.log
Thing Load (0.9ms)  SELECT "things".* FROM "things" WHERE "things"."id" IN (6, 12, 7, 13, 8, ...)

An optimized way to work with search results sets is working directly with hits. @search.hits is an array of Sunspot::Search::Hits, which represent the raw information returned by Solr for a single returned item. Hit objects provide access to stored field values, identified by the :stored option in the model's searchable definition. The model definition looks the same. The controller may now look like this:

class ThingsController < ApplicationController
  def index
    search = Sunspot.search(Thing) do
      #fulltext search
      fulltext params[:keyword]

      #scoping
      if params.has_key?(:field3)
        with :field3, params[:field3]
      end 
      if params.has_key?(:field4)
        with :field3, params[:field4]
      end
    end
    search.execute!

    @hits = search.hits.paginate :page => params[:page], :per_page => 25
  end
end

And working with the data in the view may look like this:

<% @hits.each do |result| -%>
<h2><%= hit.stored(:field3) %></h2>
<%= hit.stored(:field1) %>
<% end -%>

In some cases, you may want to introduce an additional piece of logic prior pagination, which is the case with the most recent Rails application I've been working on:

    ...
    search.execute!

    filtered_results = []

    search.hits.each do |hit|
      if hit.stored(:field3) == "some arbitrary value"
        filtered_results << hit
      elsif hit.stored(:field1) == "some other arbitrary value"
        filtered_results << hit
      end
    end
   
    @hits = filtered_results.paginate :page => params[:page], :per_page => 25

Sunspot and Solr are rich with functionality and features that can add value to a Rails application, but it's important to identify areas of the application where database calls can be minimized and lazy loading can be optimized for better performance. The standard log file and database log file are good places to start looking.

Running Integration Tests in Webkit Without a Browser

As your ruby web applications increase in UI complexity, they get harder to test using the standard Cucumber or even RSpec integration test suites. This is because of introduction of JavaScript in your UI. You've probably been there before. Here's the use case: You've written your integration tests for your Rails app and up to this point, you've been able to get away with not tagging your cucumber scenarios with the "@javascript" tag. Everything is going smoothly and then it's time to implement that one UI feature that is going to require an Ajax call or some javascript to hide or unhide a crucial piece of the user experience. This must be included in your integration tests.

So you go through the pain of setting up cucumber to work with selenium and tag your scenario as javascript so that it will run the test in the browser. At first, there's this thrill of excitement as you get to see Firefox load, and then run through a series of steps, executing your tests and then seeing them pass. Job done.

But maybe there's a different scenario at play here. What if you don't do your development in an environment that has a browser? At End Point, we are strong advocates of doing development on the same environment that your app is going to run on. It eliminates unexpected issues down the road. We believe in it so much, actually, that we've created DevCamps that allows you to setup development environments on a server.

Obviously, your selenium based tests are not going to work here without some work to get it to run headless.

The good folks at thoughtbot have come up with a great solution to this and it is called capybara-webkit. Capybara webkit assumes that you are using capybara for your testing framework. If you are using webrat, the transition is fairly smooth. You'll probably only need to change a few minor details in your tests.

What capybara-webkit does for you is enable you to run your tests inside of webkit. This will simulate an environment that will be very close to what you would see in Google Chrome or Safari as well as many mobile browsers. I've found that except for some edge cases, it covers Firefox and IE as well.

To install capybara-webkit you will need to install the Qt development toolkit. It's fairly straight forward so I'll just refer you to the github wiki page for instructions for the various platforms. In Ubuntu, I just ran the following:

sudo apt-get install libqt4-dev

If you are installing on a server environment, you'll also need to install Xvfb. You can do that in Ubuntu with the following command:

sudo apt-get install xvfb

It's a little outside the scope of this blog post to go into other little things you need to setup with xvfb. The important thing is that you set it up to run on display 99. Another important note, is that you don't have to set it up to run on boot. We will be starting it up when we run our tests if it isn't running.

The next step is to configure your cucumber tests to use the capybara-webkit driver. To do that, add

gem "capybara-webkit"

to your Gemfile in the development and test group. Then in your env.rb file for cucumber add the following lines:

Capybara.javascript_driver = :webkit

In some cases, I've found it helpful to also specify a server port and app_host as follows:

Capybara.server_port = '8000'
Capybara.app_host = 'http://localhost:8000'

Now your tests are setup to run in webkit. The final step is running the tests. To do this, you'll need to run them from within xvfb. You can do that with the following command:

xvfb-run bundle exec cucumber

I've created an alias for this and dropped it in my .bashrc file. Here's my entry, but you can set it up anyway you'd like.

alias xcuke="xvfb-run bundle exec cucumber"

Now running tests is a simple as running xcuke from the root of my Rails application.

There are a couple of big benefits to running capybara-webkit. First is speed. In my experience tests run much faster than they do in Selenium. Second, all JavaScript errors are dumped to STDOUT so you can see them in the output of your cucumber tests. Third, all of your tests are being run on webkit instead of rack so you get a test environment that acts more like a real browser would behave.

Thanks to the guys at thoughtbot for putting together this awesome gem.

Performing Bulk Edits in Rails: Part 2

This is the second article in the series on how to perform a bulk edit in Rails. Let's recap our user's story from Part 1.

• User makes a selection of records and clicks "Bulk Edit" button
• User works with the same form they would use for a regular edit, plus
• check boxes are added by each attribute to allow the user to indicate this variable should be affected by the bulk edit
• only attributes which are the same among selected records should be populated in the form

Part 1 addressed the first part of our user story. Now that we have our user's selection, we need to create an interface to allow them to select attributes affected by the bulk edit. Let's start with the form we'll use to POST our input.

# app/controllers/bulk_edits_controller.rb

def new
  @foos = Foo.find(params[:stored_file_ids]) #params collected by work done in Part 1
  @foo = Foo.new
end


# app/views/bulk_edit/new.html.erb

<%= form_for @foo, :url => "/bulk_edits" do |f| %>
  <% @foos.each do |foo| %>
    <%= hidden_field_tag "foo_ids[]", foo.id %>
  <% end %>
  <%= render "foos/form", :f => f %>
  <%= f.submit %>
<% end %>

Let's first look at how we formed our form_for tag. Although this is a form for a Foo object, we don't want to POST to foos_controller#create so we add :url => "/bulk_edits" which will POST to the bulk_edits_controller#create. Additionally, we need to send along the foo_ids we eventually want to bulk update. Finally, we don't want to re-create the form we already have for Foo. By modifying one master form, we'll make long term maintenance easier. Now that we've got our form posting to the right place, let's see what modifications will need to make to our standard form to allow the user to highlight attributes they want to modify.

# app/views/foos/_form.html.erb

<%= check_box_tag "bulk_edit[]", :bar %>
<%= f.label :bar %>
<%= f.text_field :bar %>

Bulk edit check boxes appear in front of field names to let users know which fields will be modified.

We've added another check_box_tag to the form to record which attributes the user will select for bulk updating. However, we only want to display this when we're doing a bulk edit. Let's tweak this a bit further.

# app/views/foos/_form.html.erb

<%= bulk_edit_tag :bar %>
<%= f.label :bar %>
<%= f.text_field :bar %>

# app/helpers/foos_helper.rb

def bulk_edit_tag(attr)
  check_box_tag("bulk_edit[]", attr) if bulk_edit?
end

def bulk_edit?
  params[:controller] == "bulk_edits"
end

With these modifications to the form in place, the user can now specify which fields are eligible for bulk editing. Now we need the logic to determine how to populate the bar attribute based on the user's selection. This way, the user will see that an attribute is the same across all selected items. Let's revise our bulk edit controller.

# app/controllers/bulk_edit_controller.rb

def new
  @foos = Foo.find(params[:foo_ids])
  matching_attributes = Foo.matching_attributes_from(@foos)
  @foo = Foo.new(matching_attributes)
end


# app/models/foo.rb

def self.matching_attributes_from(foos)

  matching = {}
  attriubtes_to_match = Foo.new.attribute_names  #see attribute_names for more details

  foos.each do |foo|

    attributes_to_match.each do |attribute|

      value = foo.__send__(attribute)  #see send, invokes the method identified by symbol, use underscore version to avoid namespace issues

      if matching[attribute].nil?
        matching[attribute] = value  #assume it's a match

      elsif matching[attribute] != value
        matching[attribute] = "" #on the first mismatch, empty the value, but don't make it nil

      end

    end

  end
end

Only fields which are the same across all selected records will be populated. Other fields will be left blank by default.

With Foo#matching_attributes_for generating a hash of matching attributes, the form will only populate fields which match across all of the user's selected items. With our form in place, the last step is to actually perform the bulk edit.

# app/controllers/bulk_edits_controller.rb
def create
  if params.has_key? :bulk_edit

    foos = Foo.find(params[:foo_ids])
    foos.each do |foo|

        eligible_params = {}
        params[:bulk_edit].each do |eligible_attr|

            #create hash of eligible_attributes and the user's value
            eligible_params.merge! { eligible_attr => params[:foo][eligible_attr] } 

        end

        #update each record, but only with eligible attributes
        foo.update_attributes(eligible_params)

    end
  end
end

We've now completed the entire user story. Users are able to use check boxes to identify which attributes should be bulk updated. They also get to see which attributes match across their selection. Things are, of course, always more involved with a real production application. Keep in mind this example does not make good use of mass assignment protection using attr_accessible and forcing an empty whitelist of attributes by using config.active_record.whitelist_attributes = true. This is a best practice that should be implemented anytime you need sever-side validation of your forms.

Additionally, there may be cases where you want to perform bulk edits of more complex attributes, such as nested attributes. Consider appending your additional attributes to the Foo.new.attribute_names array and then tweaking the eligible attributes logic. Also consider implementing a maximum number of records which are able to be bulk edited at a time; wouldn't want your server to time out. Good luck!

Performing Bulk Edits in Rails: Part 1

This will be the first article in a series, outlining how to implement a bulk edit in Rails 3.1.1 (although most any version of Rails will do). Today we'll be focusing on a simple user interface to allow the user to make a selection of records. But first, let's look at our user story.

The user story

• User makes a selection of records and clicks "Bulk Edit" button
• User works with the same form they would use for a regular edit, plus
• check boxes are added by each attribute to allow the user to indicate this variable should be affected by the bulk edit
• only attributes which are the same among selected records should be populated in the form

An example UI from Google's AdWords interface for
selecting multiple records for an action.

Sounds straight forward, right?  Well, there are a couple of gotcha's to be worked out along the way.

Capturing the user's selection

We'd like to offer the user a form with check boxes to click so when submitted, our controller gets an array of IDs we can pass to our ActiveRecord finder. It's best implemented using check_box_tag which means it's not auto-magically wired with an ActiveRecord object, which makes sense in this case because we don't want our form manipulating an active record object. We simply want to send our user's selection of records along to a new page.  Let's see what this looks like.

# app/views/search/_results.html

<% @foos.each do |foo| %>
  <%= check_box_tag "foo_ids[]", foo.id  %>
<% end %>

# when posted looks like
# "foo_ids"=>["4", "3", "2"]

# app/controller/bulk_edit_controller.rb

def new
  if params[:foo_ids].is_a?(Array) && params[:foo_ids].length > 1  #let's make sure we got what we expected
    @foos = Foo.find(params[:foo_ids])
  else
    redirect_to search_path
  end
end

Refining the UI with Javascript and CSS

It's not just enough to have these check boxes. We need our "Bulk Edit" button only to appear when the user has made an appropriate selection. Let's update our view code to give our tags some class.

# app/views/search/_results.html

<%= form_tag new_bulk_edit_path, :method => "GET", :id => "bulk-edit-form" do %>
  <%= submit_tag "Bulk Edit", :id => "bulk-edit-submit" %>
<% end %>

<div class="search_results">
  <% @foos.each do |foo| %>
    <%= check_box_tag "foo_ids[]", foo.id, false, :class => "downloadable"  %>
  <% end %>
</div>

# app/assets/stylesheets/search.css

#bulk-edit-submit {
  input { display: none; }
}

We've added the downloadable class tag to our check boxes, while adding a simple form to send data to the new_bulk_edit_path. This path corresponds to the new action, which typically, you don't post forms to (which is why we needed to be explicit about setting the GET method). However, in this case we need this information before we can proceed with a new bulk edit. We've also hidden the submit button by default. We'll need some Javascript to show and hide it.

# app/assets/javascripts/search.js

$('.downloadable').click(function() {     //when an element of class downloadable is clicked
  var check_count = $('.downloadable:checked').size();  //count the number of checked elements
  if( check_count > 1 ) {
    $("#bulk-edit-submit").show();
  } else {
    $("#bulk-edit-submit").hide();
  }
});

At this point, you might have noticed that we're submitting a form with no fields in it! While we could simply wrap our form_tag around our search results, but we may not always want this. For example, what if we need multiple forms to be able to send our selection to different controllers in our application? Right now we're working on a bulk edit, but you know the client is expecting a bulk download as well. We can't wrap the same search results partial in multiple forms. Let's see how we would populate the our form using more Javascript.

# app/assets/javascripts/search.js

$('#bulk-edit').submit(function() {  //When the bulk-edit form is submitted
  $('#bulk-edit input:checked').remove();  //clear all checked elements from form
  var selected_items = $('.downloadable:checked').clone();
  $('#bulk-edit').append(selected_items);
  return true;  //VERY IMPORTANT, needed to actually submit the form
});

This is a simple, unobtrusive way to give your forms a little more flexibility. It's also a good example of how to use :checked as a modifier on our jQuery selector.

Namespacing and Refactoring our Javascript

Knowing you'll need to implement a bulk-download form later in this same style, so let's refactor out this cloning functionality.

# app/assets/javascripts/search.js

$('#bulk-edit').submit(function() {
  MyAppName.clone_downloadable_checkboxes_to($(this));  //You MUST wrap "this" inside $()
  return true;
});

if(!window.MyAppName) {
  MyAppName = {};  //Initialize namespace for javascript functions
}

MyAppName.clone_downloadable_checkboxes_to = function(destination) {
  destination.children("input:checked").remove();
  var selected_items = $('.downloadable:checked').clone();
  destination.append(selected_items);
};

One of the big highlights here is namespacing our Javascript function. While the chances are low that someone out there is going to have clone_downloadable_checkboxes_to in the global namespace too, it's always best to use proper namespaces.

Well, we've made it through the first part of our user story. The user can now check their boxes, and submit a form to the appropriate Rails resource. Stay tuned to see how we implement the second half of our user's story.

Advanced Rights and Roles Management in Rails

I've been working with Phunk, Brian, and Evan on a large Rails 3.1 project that has included several unique challenges. One of these challenges is a complex rights, roles, and accessibility system, which I'll discuss here.

Before I wrote any code, I researched existing authorization systems, and came across this article which lists a few of the popular authorization gems in Rails. After reading through the documentation on several more advanced current authorization gems, I found that no gem offered the level of complexity we needed, where rights are layered on top of roles and can be mapped out to specific actions. Because the client and my team were most familiar with acl9, we chose to work with it and layer rights on top of the existing access control subsystem. Here's a look at the data model we were looking for:

The data model shows a has_and_belongs_to_many (or many-to-many) relationship between users and roles, and roles and rights. Things are an example model, which belong_to users. Rights map out to methods in the controller that can be performed on thing instances.

Implementation

Starting from the admin interface, a set of rights can be assigned to a role, a standard has_and_belongs_to_many relationship:

The admin interface includes ability to assign roles to users, another has_and_belongs_to_many relationship:

And the user model has an instance method to determine if the user's rights include the current method or right:

class User < ActiveRecord::Base
  ...
  def can_do_method?(method)
    self.rights.detect { |r| r.name == method }
  end 
  ...
end

At the controller level without abstraction, we use the access control system to determine if the user has the ability to do that particular action, by including a conditional on the rule. Note that in these examples, the user also must be logged in, which is connected to the application's authentication system (devise).

class ThingsController < ApplicationController
  ...
  access_control do
    allow logged_in, :to => :example_right1, :if => :allow_example_right1?
    allow logged_in, :to => :example_right2, :if => :allow_example_right2?
    allow logged_in, :to => :example_right3, :if => :allow_example_right3?
  end

  def allow_example_right1?
    current_user.can_do_method?("example_right1")
  end
  def example_right1
    # actual method on Thing instance
  end
  def allow_example_right2?
    current_user.can_do_method?("example_right2")
  end
  def example_right2
    # actual method on Thing instance
  end
  def allow_example_right3?
    current_user.can_do_method?("example_right3")
  end
  def example_right3
    # actual method on Thing instance
  end
  ...
end

The controller is simplified with the following abstraction. The access control statements do not need to be modified for each new potential method/right, but the method itself must be defined.

class ThingsController < ApplicationController
  ...
  access_control do
    allow logged_in, :to => :generic_method, :if => :allow_generic_method?
  end

  def allow_generic_method?
    current_user.can_do_method?(params[:method])
  end
  def generic_method
    self.send(params[:method])
  end

  def example_right1
    # actual method on Thing instance
  end
  def example_right2
    # actual method on Thing instance
  end
  def example_right3
    # actual method on Thing instance
  end
  ...
end

And don't forget the handler for Acl9::AccessDenied exceptions, inside the ApplicationController, which handles both JSON and HTML responses:

class ApplicationController < ActionController::Base
  ...
  # Rescuing from any Access denied messages, generic JSON response or redirect and flash message
  rescue_from Acl9::AccessDenied do |exception|
    respond_to do |format|
      format.json do  
        render :json => { :success => false, :message => "You do not have access to do this action." }
      end 
      format.html do
        flash[:error] = 'You do not have access to view this page.'
        redirect_to root_url
      end 
    end 
  end 
end

Conclusion

Note that in actuality, our application has additional complexities, such as:

• The relationship between rights and $subject is polymorphic, where $subject is a user or a role. This slightly complicates the has_and_belongs_to_many relationship between rights and users or roles. The can_do_method? predicate is updated to consider both user assigned rights and role assigned rights.
• Performance is a consideration in this application, so Rails low-level caching may be leveraged to minimize accessibility lookup.
• There is a notion of a global right and an ownership-level right, which means that a user with an ownership-level right may have the ability to do certain method only if they own the thing. A user with a global right has the ability to do the method regardless of ownership. This complicates our can_do_method? predicate further, to determine if the user has the global right or ownership-level right for that method on that thing.
• A few methods have more complex business logic which determine whether or not a user has the ability to do that method. In those cases, an additional access_control allow rule is created, and distinct conditional predicate is used to determine if the user can do that method (i.e. allow_generic_method? is not used for these actions).

Other than the additional complexities, leveraging acl9's access control subsystem makes for a clean rights and roles management solution. Stay tuned for a follow-up article on leveraging this data model in combination with Rails' attr_accessible functionality to create elegant server-side validation.

Double habtm Relationship Between Models

Oh, man! It's been a month since my last blog article. End Pointers Brian, Evan, Phunk and I have been working on a sizable Ruby on Rails project for a client. We've been excited to work with Rails 3.1 and work on a project that presents many unique and interesting web application challenges.

Today I wanted to write about the fairly simple task of defining two has and belongs to many (or many to many) associations between the same models, which is something I haven't seen often in Rails applications.

Data Model

First, let's look at the data model and discuss the business case for the data model. As shown above, the data model excerpt contains four tables. Users is the standard users table, which uses devise for user authentication. Groups are intended to be a group of users that will be allowed to do some combination of controller#action in our application. In our case, groups have many members (or users), but they also have many owners, who are allowed to manage the group. And obviously on the user side, users can exist as a member or an owner in many groups.

The Code

The groups_users relationship is a standard has and belongs to many relationship. The User class defines its relationship to groups:

class User < ActiveRecord::Base
  ....
  has_and_belongs_to_many :groups
  ...
end

And the Group class defines it's relationship to users:

class Group < ActiveRecord::Base
  ...
  has_and_belongs_to_many :users
  ...
end

Rails makes it fairly easy to define has_and_belongs_to_many associations and override the join table, class name, and foreign key, which is applicable to the groups_owners relationship. Here, the User class defines it's relationship to owned_groups, and specifies the join_table, class name, and foreign key:

class User < ActiveRecord::Base
  ....
  has_and_belongs_to_many :owned_groups, :class_name => "Group", :join_table => "groups_owners", :foreign_key => "owner_id"
  ...
end

And the Group model has similar overrides (except in this case, we override the association foreign key):

class Group < ActiveRecord::Base
  ..
  has_and_belongs_to_many :owners, :association_foreign_key => "owner_id", :join_table => "groups_owners", :class_name => "User"
  ..
end

And that's how to define the has and belong to many relationship between two of the same models! Obviously in our case, we can easily call and modify these associations, by calling some_user.groups, some_user.owned_groups, some_group.owners, and some_group.users.

Extras

Here I've also created a couple of instance methods on the Group and User model to make it easy to pull the aggregate of owners and users (Group) and owned_groups and groups (User):

class User < ActiveRecord::Base
  ...
  def all_groups
    (self.groups + self.owned_groups).uniq
  end
  ...
end

And:

class Group < ActiveRecord::Base
  ...
  def all_members
    (self.owners + self.users).uniq
  end
  ...
end

Performance techniques such as calling raw SQL or with Rails low-level caching can potentially be applied to these methods, since I would not expect them to be highly performing as they are shown above. Examples of raw SQL and Rails low-level caching are described here!

Rails Controllers and Transactions

Actions involving single objects in Rails are generally nicely and automatically handled. By handled, what I mean specifically, is that ActiveRecord will encapsulate saving and updating an object in a transaction and you can set up the various callbacks and validations to ensure that the object and its associations meet whatever requirements you have before you allow it to be committed to the database.

You are allowed some degree of latitude to say "Give it a shot, if it doesn't work, then it's no big deal." One of the upsides is that you can throw whatever random nonsense the UI passes you right on through, and, presumably, the model correctness validation code will do double duty as input validation as well.

Which is nice, as far as it goes, but it tends to be localized. Which is to say, that your objects generally only care about validating themselves. Sometimes, it turns out to be necessary to update the state of multiple (possibly unrelated) objects simultaneously and additionally to ensure that if any part of any of these updates fail, you roll the whole thing back.

The application in mind is in a controller, where we are getting input from the browser and we have access to a set of callbacks, but not the atomic-like transactional control we get around save/update actions in a model single.

I suspect the wording of the problem suggests the solution, but let's discuss the hard way to approach it for a bit first. You could enumerate all of the objects that you knew you were going to update, iterate through each of them, determine what state that you were going to attempt to transition them to, determine if that state is valid independently (easier), determine if that state is valid in concert with all of the other attempted transitions (harder), and then, if everything looked cool, make all of the transitions. You might notice that I left out secondary, tertiary, etc. cascading transitions caused by first order transitions that we were inspecting.

That approach starts looking likely to fail in pretty short order, even for small numbers of items. I believe the impetus for this blog post was about 3 items, and a complete lack of ability to track down all of the corner cases generated.

With that, the obvious solution is to wrap the entire operation in a transaction. It turns out the necessary parts have always been there, waiting. Here's one way to use them together.

Set up your own transactions

First, we need to be able to create our own transaction. We can do so by simply enclosing our code like so:

ActiveRecord::Base.transaction do
    ... code ...
end

We can use either a class object (as in this example), or an instance object. The documentation for ActiveRecord transactions is here. The advantage in this case for using the base class to set up the transaction is that the controller doesn't need to do any semi-mystical nonsense to try to guess the name of the object it might be related to.

How can we get our action into the middle of that transaction block?

The second part is how to get the transaction wrapped around our controller methods. The obvious in retrospect solution is the around_filter provided by ActionController. If you have a controller, then you could implement such a filter like so:

 
class MyObjectController < ApplicationController
    around_filter :transactions_filter

    def transactions_filter
        ActiveRecord::Base.transaction do
            yield
        end
    end
    ... the rest of the controller ...
    end

It probably only makes sense to add the around_filter to actions which are of the modification type. A definition like this might be more reasonable.

    around_filter :transactions_filter, :only => [:create, :update, :destroy]

Its also simple enough to add this to your ApplicationController definition if you want to have all of your controller classes inherit this functionality universally.

Rails 3.1: Upgrading a Simple App - Part 1

Here at End Point, I've worked with a few Rails 3 applications in production and a couple of Rails 3.1 apps in development, so I've become familiar with the new features and functionality including the Rails 3.1 Asset Pipeline that I mentioned earlier this year. I thought it was a good time to upgrade our website to Rails 3.1 and share the experience.

To start, here's a quick summary of our website:

• Simple Rails application running on Rails 2.1.2 with no database
• Static pages throughout the site, fully cached
• Rake tasks to generate partials throughout the site to display dynamic blog content
• Site uses a moderate amount of jQuery and jQuery plugins.
• Site is optimized in terms of asset serving (ETags, Expires headers, CSS sprites, etc.)

While I've worked with a few Rails 3 apps, I haven't been involved in the actual upgrade process myself. There are plenty of resources out there with upgrade advice, including a few RailsCasts (one, two, and three). My favorite resource was the rails_upgrade gem, a gem that is now officially supported by Rails to help with the upgrade process. I followed the instructions to install the gem (script/plugin install git://github.com/rails/rails_upgrade.git) to install it as a plugin in our site's application in a fresh git branch (on a camp, of course!).

The rails_upgrade provides a few new rake tasks for checking compatibility, upgrading the routes, creating a Gemfile, and upgrading configuration. For me, the most valuable task was the rake rails:upgrade:check task. Here's what the output looked like for this app:

Deprecated session secret setting
Previously, session secret was set directly on ActionController::Base; it's now config.secret_token.
More information: http://lindsaar.net/2010/4/7/rails_3_session_secret_and_session_store

The culprits: 
 - config/initializers/session_store.rb

Old router API
The router API has totally changed.
More information: http://yehudakatz.com/2009/12/26/the-rails-3-router-rack-it-up/

The culprits: 
 - config/routes.rb

New file needed: config/application.rb
You need to add a config/application.rb.
More information: http://omgbloglol.com/post/353978923/the-path-to-rails-3-approaching-the-upgrade

The culprits: 
 - config/application.rb

Deprecated constant(s)
Constants like RAILS_ENV, RAILS_ROOT, and RAILS_DEFAULT_LOGGER are now deprecated.
More information: http://litanyagainstfear.com/blog/2010/02/03/the-rails-module/

The culprits: 
 - app/views/layouts/application.rhtml
 - ...

Soon-to-be-deprecated ActiveRecord calls
Methods such as find(:all), find(:first), finds with conditions, and the :joins option will soon be deprecated.
More information: http://m.onkey.org/2010/1/22/active-record-query-interface

The culprits: 
 - app/views/blog_archive/_ruby_on_rails.html.erb
        - ...

Deprecated AJAX helper calls
AJAX javascript helpers have been switched to be unobtrusive and use :remote => true instead of having a seperate function to handle remote requests.
More information: http://www.themodestrubyist.com/2010/02/24/rails-3-ujs-and-csrf-meta-tags/

The culprits: 
 - app/views/blog_archive/_company.html.erb
        - ...

Deprecated ActionMailer API
You're using the old ActionMailer API to send e-mails in a controller, model, or observer.
More information: http://lindsaar.net/2010/1/26/new-actionmailer-api-in-rails-3

The culprits: 
 - app/controllers/contact_controller.rb

Old ActionMailer class API
You're using the old API in a mailer class.
More information: http://lindsaar.net/2010/1/26/new-actionmailer-api-in-rails-3

The culprits: 
 - app/models/contact_form.rb

As you can see, the upgrade check spits out a list of necessary and recommended upgrades and the corresponding *culprits*. It's also nice that the task provides documentation in the form of a link for each message. Studying the source of the plugin, I found additional examples of upgrade messages: named_scope updates, validate_on_* syntax, test_help path updates, gem bundling configuration, Rails generator API syntax updates, messaging on known broken plugins (e.g. searchlogic, cucumber, nifty-generators), and depracation on ERb helper and AJAX calls.

I went through and applied my updates, according to the checklist. Notable updates were:

Before

ActionController::Routing::Routes.draw do |map|
  map.root :controller => 'home', :action => 'index'
  map.connect 'contact/submit', :controller => 'contact', :action => 'submit'
  map.connect ':controller/:id'
  map.connect '*path', :controller => 'redirect' 
end

After

Endpoint::Application.routes.draw do
  root :to => 'home#index'
  match 'contact/submit' => 'contact#submit'
  match ':controller(/:id)', :action => :index
  match '*path' => 'redirect#index'
end

source 'http://rubygems.org'

gem 'rails', '3.1.0'
gem 'json'

# Gems used only for assets and not required
# in production environments by default.
group :assets do
  gem 'sass-rails', "  ~> 3.1.0"
  gem 'coffee-rails', "~> 3.1.0"
  gem 'uglifier'
end

gem 'jquery-rails'
gem 'fastercsv'
gem 'execjs'
gem 'therubyracer'
gem 'rake', '0.8.7'

Something that didn't come up in the rails upgrade check that is required to have a working app is renaming all rhtml files to html.erb, briefly described here.

To get the basic app working, I moved the public/stylesheets and public/javascripts to the new app/assets directories to start. I did not move the images out of the public/ directory because several of the images in the application are referenced by blog articles.

I followed the directions here combined with a bit of troubleshooting to configure a Rails 3.1 app that does not require a database.

The upgrade was a relatively painless process, although it still took a few hours for even the most basic application with only a handful of controllers, routes, and one mailer. My experience suggests that with a more complex application, the upgrade will take at least a few hours, if not much more. This simple app doesn't do much with remote forms and links, so I didn't spend any time upgrading the app to work with the jquery-ujs gem. Also, I obviously didn't mess around with Rails 3.1 ActiveRecord issues since the application is database-less. Both of these items may add significant overhead to the upgrade process.

I spent a significant amount of time working with the new asset pipeline and restructuring the assets, which I plan to describe in Part 2 of the upgrade. Stay tuned!

Ruby on Rails Performance Overview

Over the last few months, I've been involved in a Ruby on Rails (version 2.3) project that had a strong need to implement performance improvements using various methods. Here, I'll summarize some the methods and tools used for performance optimization on this application.

Fragment Caching

Before I started on the project, there was already a significant amount of fragment caching in use throughout the site. In it's most basic form, fragment caching wraps a cache method around existing view code:

<%= cache "product-meta-#{product.id}" %>
#insert view code
<% end %>

And Rails Sweepers are used to clear the cached fragments, which looks something like the code shown below. In our application, the Sweeper attaches cache clearing methods to object callbacks, such as after_save, after_create, before_update.

class ProductSweeper < ActionController::Caching::Sweeper
  observe Product

  def after_save(record)
    expire_fragment "product-meta-#{product.id}"
  end
end

Fragment caching is a good way to reuse small modular view components throughout the site. In this application, fragment caches tended to contain object meta data that was shown on various index list pages and single item show pages.

Page Caching

I did not initially add page caching to the application because the system has complex role management where users can have edit access at an object, class, or super level. However, later I investigated advanced techniques to leverage full page caching, described in depth here. The benefit gained here was that the application server was not hit during full page requests, and a quick AJAX request was made after the page loaded to determine user access level.

Raw SQL methods

Another performance technique I employed on this application was using raw SQL rather than use standard ActiveRecord methods to lookup association data. The application uses ActsAsTaggable, a gem that enables you to tag objects. The simplified data model looks like this, which includes a polymorphic relationship in the taggings table to the items tagged (products, categories):

In the application, the front-end required that we pull the most popular 25 tags for a specific class. Working with the objects and their associations, one might use the following code:

def self.tag_list
  b = Product.all.collect { |p| p.tag_list }.flatten
  c = b.inject({}) { |h, a| h[a] ||= 0; h[a]+=1; h } 
  c.sort_by { |k, v| v }.reverse[0..25]
end

However, this request is quite sluggish because it has to iterate through each object and it's tags. I wrote raw SQL to generate the Tag objects, which runs at least 10 times faster than using standard ActiveRecord association lookup:

def self.tag_list
  Tag.find_by_sql("SELECT ts.tag_id AS id, t.name FROM taggings ts
    JOIN tags t ON ts.tag_id = t.id
    WHERE taggable_type = 'Product'
    GROUP BY ts.tag_id, t.name
    ORDER BY COUNT(*) DESC LIMIT 25")
end

Typically, using ActiveRecord find methods and the item associations may yield more readable code and require minimal knowledge of the underlying database structure. But in this example, having an understanding of the database model and how to work with it gave a significant performance bump. This technique was also combined with fragment caching.

Rails Low Level Caching

Next up, there were several opportunities through the site to use Rails low level caching. Here's one example of a simple use of Rails low level caching, which pulls a list of products that the user has owner or creator rights to:

class User < ActiveRecord::Base 
  def products
    Rails.cache.fetch("user-products-#{self.id}") do
      self.roles
        .find(:all, :conditions => {:authorizable_type => 'Product', :name => ['owner','creator']})
        .collect(&:authorizable)
        .uniq
        .compact
        .sort_by{|a| a.updated_at}
    end
  end
end

Rails low level caching makes sense for data that's pulled throughout various actions but additional computations are applied to this data. We are unable to cache this at the page request or action level, but we can cache the data retrieved with low level caching. I also used Rails low level caching on the search index pages, which is described more in depth here.

HTML Asset related Performance

In addition to server-side optimization, I investigated several avenues of HTML asset related performance optimization:

• Extensive use of CSS Sprites
• Consolidation and minification of JS, CSS. Note that Rails 3.1 introduces new functionality to improve the process of serving minified and consolidated JS and CSS.
• HTML caching, gzipping, and Expires headers

Tools Used

Throughout performance tweaking, I used the following tools:

• Rails' standard logger
• PostgreSQL's ANALYZE
• Ruby's Benchmark Module
• WebPageTest.org

Conclusion

There are a few Rails caching techniques that I did not use in the application, such as action caching and SQL Caching. The Rails caching overview provides a great summary of caching techniques, but does not cover Rails low level caching. Another great resources for performance optimization is Yahoo's Best Practices for Speeding Up Your Web Site, but it focuses on asset related optimization opportunities. I typically recommend pursuing optimization on both the server-side and asset related fronts.

The rails_admin gem in Ecommerce

Update: Since writing this article, this gem has had several updates. All rails_admin configuration now must be pulled out of ActiveRecord models and into the initializer. The look and feel has also been updated. Read about it more here. Additionally, End Point has released a Ruby on Rails Ecommerce Engine with a sleek admin interface driven by rails_admin. Read more about it here.

I recently installed the rails_admin gem into a new Rails project. This particular site is currently running on Interchange, but it is not an ecommerce business, so Interchange is overkill for the site. The client has recently decided to make the switch to Rails (and DevCamps) with our help, and they have a moderate budget to do so. For the first increment, we considered installing phpMyAdmin for them to work directly with the data until a nice admin interface was built, but instead I spent a bit of time installing the rails_admin gem which has been on my mind for the last 6 months. The result: I'm extremely impressed with what rails_admin has to offer.

To show some examples of rails_admin in action, I'll use the data model from a Sinatra ecommerce setup I recently wrote about, because it has a basic ecommerce data model that should be universally understood by my coworkers and clients.

To create a new site, I went through the standard Rails 3.0 installation steps:

• gem install rails -v=3.0.9 to install Rails
• rails new mystore to create a new Rails application
• In my gemfile, I added:

  gem 'devise'
  gem 'rails_admin', :path => 'vendor/gems/rails_admin'
  gem 'ckeditor' # for WYSIWYG editing
  gem 'paperclip' # and installed imagemagick, for image attachments
  

• sudo bundle install to install the project gems
• rake rails_admin:install to install rails_admin, which also runs devise setup and migrations
• rails generate ckeditor:install to install ckeditor
• Copied my Sinatra data migrations and models over to my Rails application (except for the User, which is now replaced with devise's User)
• rake db:migrate to apply all migrations
• rails s to start my server
• Also, I created a user from the rails console

Note that I also tried to use Rails 3.1 and the master branch of rails_admin, but I was experiencing a few bugs, so I decided to stick with Rails 3.0.9 for this article. Coincidentally, the rails_admin master branch just jumped to Rails 3.1 very recently. There are a few Rails 3.1 things to be aware of with the Rails 3.1 Asset Pipeline.

Out of the Box

After I got everything up and running, my first look at /admin looks nice. It shows the number of records per model, recent history, and a list of models for navigation on the right side:

rails_admin has a nice DSL to modify the admin interface. I applied a few updates to my application, described below.

Remove Model / Tab

First, I wanted to hide Users from the /admin, which required the change below. Note that you can also pass a block to determine the visibility of a model, which may be valuable if you want to limit the visibility of models to specific roles or users.

class User < ActiveRecord::Base
  ...

  rails_admin do
    visible false # or visible { some block }
  end
end

Users Tab removed from navigation

List Views

Product List View, rails_admin Out of the Box

Next, I updated my products list view with the following changes, to limit the list view to the name and price field, sort by the name, and format the price:

class Product < ActiveRecord::Base
  rails_admin do
    list do
      field :name
      field :price
    end
  end
end

class Product < ActiveRecord::Base
  rails_admin do
    list do
      sort_by :name
      field :name
      field :price
    end
  end
end

class Product < ActiveRecord::Base
  rails_admin do
    list do
      sort_by :name
      field :name
      field :price do
        formatted_value do
          sprintf "$%.2f",
        end
      end
    end
  end
end

Here's a screenshot of the Products list view after these updates:

WYSIWIG-ing It

Next, I want to add a WYSIWYG editor for one of my fields, which can be accomplished with the following change:

class Page < ActiveRecord::Base
  rails_admin do
    edit do
      include_all_fields
      field :content, :text do
        ckeditor true
      end
    end
  end
end

ckeditor used for content field.

Paperclip Image Attachments

rails_admin also works nicely with Paperclip, a very popular image attachment Rails gem. Paperclip requires that imagemagick be installed on the server. I add the following code to my product model, which already had the Paperclip reference to an attached image, and the migration to introduce the Paperclip required attachment fields.

class Product < ActiveRecord::Base
  rails_admin do
    edit do
      include_all_fields
      field :image do
        thumb_method :thumb
      end
    end
  end
end

The products edit view is now a multitype form to allow for image upload.

And to update the show view, I made this change:

class Product < ActiveRecord::Base
  ...
  rails_admin do
    show do
      include_all_fields
      field :image do
        thumb_method :thumb
      end
    end
  end
end

Thumb image added to product show view.

Overriding Views

Another common update in rails_admin might be the need to override views to change the look of the backend interface. I accomplished this by copying the rails_admin partial views into my Rails application and updating them to include "My Store" branding:

# app/views/rails_admin/main/_title.html.haml
%h1.title
  = link_to rails_admin_dashboard_path do
    %span.red My
    %span.white Store

After overriding the header view, "My Store" now appears in the header.

And an override of app/views/layouts/rails_admin/main.html.haml removes "Rails Admin | 2011" from the bottom of the page.

Conclusion

There's tons(!) more you can do with the gem, and it's documented thoroughly here. rails_admin comes out of the box with export to CSV, JSON, and XML logic, which makes it a nice base for building simple APIs. It recognizes the Rails associations has_many, belongs_to, has_and_belongs_to_many, etc. It also includes user history and filtering of items and does authentication with devise, which has become a very popular user authentication choice in Rails. I found a few potential disadvantages:

• Some type of import functionality is missing (needed for my client)
• ActiveRecord is the only ORM supported, but that's fine with me.
• It's right on the bleeding edge of Rails, which makes it ideal for new Rails applications and can't always be used for older apps.
• I am curious to see if performance suffers on applications with a large number of records.

Despite the potential disadvantages, I've been extremely impressed with it's functionality and how much development time can be saved here to allow more time for custom business-centric functionality for a client.

Note that a couple of popular alternatives to rails_admin are ActiveAdmin and Typus

What next?

Since I've already built a Sinatra front-end ecommerce application, I might try to get my Rails admin running with a Sinatra frontend by following tips in this article. It'll be a little more complex here since I need to map user and admin and user routes to Rails and other routes to Sinatra, but the article covers the general idea for dispatching the routes. Why do it this way? Because you get the best of both worlds: a nice Rails backend for the CRUD interface and API management, and a speedy Sinatra driven frontend with simple paths to define product navigation, product pages, content pages, the cart and checkout process (which is not standard RESTful behavior). You can also leverage Ruby gem functionality in both Sinatra and Rails.

Rails Optimization: Digging Deeper

I recently wrote about raw caching performance in Rails and advanced Rails performance techniques. In the latter article, I explained how to use a Rails low-level cache to store lists of things during the index or list request. This technique works well for list pages, but it doesn't necessarily apply to requests to an individual thing, or what is commonly referred to as the "show" action in Rails applications.

In my application, the "show" action loaded at ~200 ms/request with low concurrency, with the use of Rails fragment caching. And with high concurrency, the requests shot up to around 2000 ms/request. This wasn't cutting it! So, I pursued implementing full-page caching with a follow-up AJAX request, outlined by this diagram:

First, the fully-cached is loaded (quickly). Next, an AJAX request is made to retrieve access information. The access information returns a JSON object with information on whether or not there is a user, and if that user has edit access to that thing. If there is no user, the page stays as is. If there is a user, but he does not have edit permissions, the log out button is shown and the username is populated. If there is a user and he has edit permissions, the log out button is shown, the username is populated, and additional buttons requiring edit access are shown.

The Code

To cache the full page, I use the caches_page method, and cache only on requests of HTML format (other formats are not cached):

class ThingsController < ApplicationController
 caches_page :show, :if => Proc.new { |c| c.request.format.html? }
 ...
end

My access level request looks something like this:

def accessibility
  respond_to do |format|
    format.json do
      render :json => {
        :logged_in => current_user ? current_user.to_json(:only => [:id, :username]) : false,
        :can_edit => current_user ? Thing.find(params[:id]).can_edit?(current_user) : false }
    end
  end 
end 

My HTML has some bits of code sprinkled throughout it:

...
<a href="#" id="edit_thing" class="requires_editability">Edit</a>
...
<a href="#" id="my_account" class="requires_logged_in"><!-- no username yet --></a>
...

My jQuery AJAX request looks something like the code shown below. Note that I remove elements that do not apply to the current request:

$.ajax({
  type: 'GET',
  cache: false,
  url: editability_path,  //editability_path is defined in the HTML (a JavaScript variable)
  dataType: "JSON",
  error: function(xhr){
    $('.require_editability,.require_loggged_in').remove();
  },
  success: function(results) {
    if(results.logged_in) {
      $('.require_logged_in').show();
      $('#my_account').html(results.logged_in.username);
      if(results.can_edit) {
        $('.require_editability').show();
      } else {
        $('.require_editability').remove();
      }
    } else {
      $('.require_editability,.require_loggged_in').remove();
    }
  }
});

And don't forget the sweeper to clear the fully cached page after edits (or other ActiveRecord callbacks):

class ThingSweeper < ActionController::Caching::Sweeper

  observe Thing

  def after_save(record)
    expire_page :controller => :things, :action => :show, :id => record.id
  end
end

Additional Notes

There are some additional notes to mention:

• If a user were to hack the AJAX or JavaScript, server-side validation is still being performed when an "edit" action is submitted. In other words, if a hacker somehow enabled an edit button to show up and post an edit, a server-side response would prohibit the update because the hacker does not have appropriate accessibility.
• HTML changes were made to accommodate this caching behavior, which was a bit tricky. HTML has to handle all potential use cases (no user, user & no edit access, user & edit access). jQuery itself can also be used to introduce new elements per use case.
• The access level AJAX request is also hitting more low-level Rails caches: For example, the array of things that a user has edit permissions is cached and the cache is cleared with standard Rails sweepers. With this additional caching component, the access level AJAX request is hitting the database minimally.
• Performance optimization scenarios such as this make an argument against inline editing of resources. If there were a backend admin interface to allow editing of things, full-page caching would be more straight-forward to implement.

Conclusion

With this functionality, fully cached pages are served with an average of less than 5 ms/request, and the AJAX accessibility request appears to be around 20 ms/request (although this is harder to test with simple command line tools). This is an improvement over the 200 ms/request initially implemented. Additionally, requests at a high concurrency don't bog down the system as much.

Company Presentation: jQuery and Rails

Yesterday, I gave a company presentation on jQuery and Rails. The talk covered details on how jQuery and Rails work together to build rich web applications, with a considerable amount of focus on AJAX methods. Check out the slides here:

One piece of knowledge I took away from the talk is how different the Rails 3 approach is for unobtrusive AJAX behavior using helpers like link_to_remote and remote_form_for. Mike Farmer made a recommendation to read the rails.js source here to see how onclick behavior is handled in Rails 3.

Rails Optimization: Advanced Techniques with Solr

Recently, I've been involved in optimization on a Rails 2.3 application. The application had pre-existing fragment caches throughout the views with the use of Rails sweepers. Fragment caches are used throughout the site (rather than action or page caches) because the application has a fairly complex role management system that manages edit access at the instance, class, and site level. In addition to server-side optimization with more fragment caching and query clean-up, I did significant asset-related optimization including extensive use of CSS sprites, combining JavaScript and CSS requests where ever applicable, and optimizing images with tools like pngcrush and jpegtran. Unfortunately, even with the server-side and client-side optimization, my response times were still sluggish, and the server response was the most time consuming part of the request for a certain type of page that's expected to be hit frequently:

A first stop in optimization was to investigate if memcached would speed up the site, described in this article Unfortunately, that did not improve the speed much.

Next, I re-examined the debug log to see what was taking so much time. The debug log looked like this (note that table names have been changed):

Processing ThingsController#index (for 174.111.14.48 at 2011-07-12 16:32:04) [GET]
  Parameters: {"action"=>"index", "controller"=>"things"}
Thing Load (441.2ms)  SELECT * FROM "things" WHERE ("things"."id" IN (22,6,23,7,35,24,36,25,14,9,37,26,15,...)) 
Rendering template within layouts/application
Rendering things/index
Cached fragment hit: views/all-tags (1.6ms)
Rendered things/_nav_search (3.3ms)
Rendered shared/_sort (0.2ms)
Cached fragment hit: ...
Cached fragment hit: ...
Cached fragment hit: ...
Rendered ...
Rendered ...
Completed in 821ms (View: 297, DB: 443) | 200 OK [http://www.mysite.com/things]

From the debug log, we can point out:

• The page loads in 821ms according to Rails, which is similar to the time reported in the waterfall shown above.
• The page is loading several cached fragments, which is good.
• The biggest time-suck of the page loading is a SELECT * FROM things ...

To rule out any database slowness due to missing indexes, I examined the query speed via console (note that this application runs on PostgreSQL):

=> EXPLAIN ANALYZE SELECT * FROM "things" WHERE ("things"."id" IN (22,6,23,7,35,24,36,25,14,9,37,26,15,...));
                                                 QUERY PLAN                                                 
------------------------------------------------------------------------------------------------------------
 Seq Scan on things  (cost=0.00..42.19 rows=24 width=760) (actual time=0.023..0.414 rows=25 loops=1)
   Filter: (id = ANY ('{22,6,23,7,35,24,36,25,14,9,37,26,15,...}'::integer[]))
 Total runtime: 0.452 ms
(3 rows)

The query here is on the scale of 1000 times faster than the loading of the objects from the ThingsController. It's well known that object instantiation in Ruby is slow. There's not much I can do to speed up the pure performance of object instantation except possibly 1) upgrade to Ruby 1.9 or 2) try something like JRuby or Rubinius, which are both out of the scope of this project.

My next best option is to investigate using Rails low-level caching here to cache my objects pulled from the database, but there are a few challenges with this:

• The object instantiation is happening as part of a Solr (via sunspot) query, not a standard ActiveRecord lookup.
• The Solr object that's retrieved is used for pagination with the will_paginate gem.
• Rails low-level caches can only store serializable objects. The Solr search object and WillPaginate:Collection object (a wrapper around an array of elements that can be paginated) are not serializable, so I must determine a suitable structure to store in the cache.

Controller

After troubleshooting, here's what I came up with:

@things = Rails.cache.fetch("things-search-#{params[:page]}-#{params[:tag]}-#{params[:sort]}") do  
  things = Sunspot.new_search(Thing)

  things.build do
    if params.has_key?(:tag)
      with :tag_list, CGI.unescape(params[:tag])
    end
    with :active, true
    paginate :page => params[:page], :per_page => 25
    order_by params[:sort].to_sym, :asc
  end
  things.execute!
  t = things.hits.inject([]) { |arr, h| arr.push(h.result); arr }
  { :results => t,  
    :count => things.total }
end 
@things = WillPaginate::Collection.create(params[:page], 25, @things[:count]) { |pager| pager.replace(@things[:results]) }

Here's how it breaks down:

• My cache key is based on the page #, tag information, and sort type, shown in the argument passed into the low-level cache build:
  

  @things = Rails.cache.fetch("things-search-#{params[:page]}-#{params[:tag]}-#{params[:sort]}") do  
  ###
  end 
  

• All this stuff creates a Solr object, sets the Solr object details, and builds the result set. In this particular Solr object, we are pulling things that have an :active value of true, may or may not have a specific tag, limiting the result set to 25, and ordering by the :sort parameter:
  

    things = Sunspot.new_search(Thing)
  
    things.build do
      if params.has_key?(:tag)
        with :tag_list, CGI.unescape(params[:tag])
      end
      with :active, true
      paginate :page => params[:page], :per_page => 25
      order_by params[:sort].to_sym, :asc
    end
    things.execute!
  

• things is my Sunspot/Solr object. I build an array of the Solr result set items and record the total number of things found. A hash that contains an array of "things" and a total count is my serializable cacheable object.
  

    t = things.hits.inject([]) { |arr, h| arr.push(h.result); arr }
    { :results => t,  
      :count => things.total }
  

• The tricky part here is building a WillPaginate::Collection object after pulling the cached data, since a WillPaginate object is also not serializable. This needs to know what the current page is, things per page, and total number of things found to correctly build the pagination links, but it doesn't require that you have all the other "things" available:
  

  @things = WillPaginate::Collection.create(params[:page], 25, @things[:count]) { |pager| pager.replace(@things[:results]) }
  

View

My view contains the standard will_paginate reference:

There are <%= pluralize @things.total_entries, 'Thing' %> Total
<%= will_paginate @things %>

And I pass the result set in a partial as a collection to display my listed items:

<%= render :partial => 'shared/single_thing', :collection => @things %>

Sweepers

Another thing to get right here is clearing the low-level cache with Rails sweepers. I have a fairly standard Sweeper setup similar to the one described here. I utilize two ActiveRecord callbacks (after_save, before_destroy) in my sweeper to clear the cache, shown below.

class ThingSweeper < ActionController::Caching::Sweeper
  observe Thing

  def after_save(record)
    Rails.cache.delete_matched(%r{things-search*})

    # expire_fragment ...
  end

  def before_destroy(record)
    Rails.cache.delete_matched(%r{things-search*})
 
    # expire_fragment ...
  end
end

With the changes described here (caching a serializable hash with the Solr results and total count, generating a WillPaginate:Collection object, and defining the Sweepers to clear the cache), I saw great improvements in performance. The standard "index" page request does not hit the database at all for users not logged in nor does it experience the sluggish object instantiation. My waterfall now looks like this:

And running 100 requests at a concurrency of 1 on the system (running in production on a development server) shows the requests are averaging 165ms, which is decent. After I wrote this post, I did a even more optimization on a different page type in the application that I hope to share in a future blog post.

Note: Ideally, it would be better to cache individual objects so that I would not have expire entire search caches on every save or delete. However, I could not find methods in Solr that allows us to pull a list of ids of the result set without building the result set.

Raw Caching Performance in Ruby/Rails

Last week, I set up memcached with a Rails application in hopes of further improving performance after getting a recommendation to pursue it. We're already using many Rails low-level caches and fragment caches throughout the application because of its complex role management system. Those are stored in NFS on a NetApp filer, and I was hoping switching to memcached would speed things up. Unfortunately, my http request performance tests (using ab) did not back this up: using file caching on NFS with the NetApp was about 20% faster than memcached from my tests.

I brought this up to Jon, who suggested we run performance tests on the caching mechanism only rather than testing caching via full http requests, given how many layers of the stack are involved and influence the overall performance number. From the console, I ran the following:

$ script/console   # This app is on Rails 2.3
> require 'benchmark'
> Rails.cache.delete("test")
> Rails.cache.fetch("test") { [SomeKlass.first, SomeKlass.last] }
> # to emulate what would potentially be stored with low-level cache
> Benchmark.bm(15) { |x| x.report("times:") { 10000.times do; Rails.cache.fetch("test"); end } } 

We ran the console test with a few different caching configurations, with the results shown below. The size of the cached data here was ~2KB.

I also ran the console test with a much larger cache size of 822KB, with much different results:

Conclusion

It's interesting to note here that the file-system caching outperformed memcached on the smaller cache, but memcached far outperformed the file-system caching on the larger cache. Ultimately, this difference is negligible compared to additional Rails optimization I applied after these tests, which I'll explain in a future blog post.

We are hiring: Ruby on Rails Developer

Job Description:
End Point is looking for a talented software developer who can consult with our clients and develop their Rails web applications. We need someone that focuses on the client and will deliver quality, tested code.

About End Point:
End Point is a 15-year-old web consulting company based in New York City, with 20 full-time employees working remotely from around the United States. Our team is made up of strong e-commerce, database, and system administration talent that leverage a variety of open source technologies.

We service over 200 clients ranging from small mom and pop shops to large corporations. End Point continues to grow this year and we're looking for intelligent and passionate people that want to join our team and make a difference. We prefer open source technology and do collaborative development with Git, GNU Screen, IRC, and voice.

What is in it for you?

• Work from your home office or our Manhattan based headquarters
• Flexible full-time work hours
• Strong balance of work and home life
• Bonus opportunities
• Health insurance benefits
• Ability to move without being tied to your job location

What you will be doing:

• Consulting with clients to determine their web application needs
• Building, testing, and releasing web applications for our clients
• Working with open source tools

What you will need:

• Experience building and testing Ruby on Rails applications
• Experience with PostgreSQL, MySQL or another relational database
• Customer-centric focus
• A passion for building flexible and scalable web applications
• Strong verbal and written communication skills
• Be motivated and able to work from home
• Ability and desire to learn new technologies

Bonus points for:

• Spree, Sinatra, DataMapper or NoSQL experience
• Prior consulting experience
• Building and working with e-commerce systems
• System administration and deployment
• Mobile and location-based technologies

Please email jobs@endpoint.com to apply.

Does this job description not quite fit what you do? Do you have other software development skills such as Perl, Python, or PHP, and think End Point is the place for you to work? We are always looking for talented people, so send us your resume and we'll let you know if there is another position that might be a better fit.

Using Set Operators with Ruby Arrays

Spree Performance Benchmarking

I see a lot of questions regarding Spree performance in the spree-user group, but they are rarely answered with metrics. I put together a quick script using the generic benchmark tool ab to review some data. Obviously, the answer to how well a site performs and scales is highly dependent on the host and the consumption of the web application, so the data here needs to be taken with a grain of salt. Another thing to note is that only two of the following use cases are running on Rails 3.0 — many of our current Spree clients are on Spree 0.11.2 or older. I also included one non-Spree Rails ecommerce application, in addition to a few non-Rails applications for comparison. All of the tests were run from my home network, so in theory there shouldn't be bias on performance tests for sites running on End Point servers.

Here's the same data in graphical form:

Requests per Second

Time Per Request (ms)

We expect to see high performance on some of the sites with significant performance optimization. On smaller VPS, we expect to see the the server choke with higher concurrency.

Coding Tips from RailsConf 2011

A couple of the sessions I attended on Day 1 of RailsConf 2011 were along the lines of how to write good Rails code: Keeping Rails on the Tracks by Mikel Lindsaar and Confident Code by Avdi Grimm. Here's a mishmash summary from these talks. Although the talks didn't focus on Ruby and Rails techniques, both talks had plenty of examples and tips for writing maintainable code that apply to my world of consulting and development.

In the first talk, Mikel talked about what he meant by keeping Rails on the Tracks and the balance that we might often experience between just working code and sexy code. Maintainable code lands in the middle there somewhere. And he briefly touched on the fact that budget matters in writing maintainable code, trade-offs are part of life, and that you are selling a solution and not code, which means that clients don't really care about the technology as long as it's maintainable.

Mikel's first pro tip for building a maintainable Rails app is to treat deployment as a first class citizen by having trivial deployment that takes advantage of existing tools like Chef and Puppet. Using Bundler will help you avoid additional pain, but be careful of avoiding locking on git repos that you don't own since you can't control these other repos. This really speaks to End Point's camps project — it's so easy to deploy changes on many of our campified Perl-based Interchange sites, but more difficult for clients that aren't on camps. The bottom line is that having trivial deployment saves time & money.

Mikel also mentioned several performance tips that make clients happy, listed below. I wasn't sure how these recommendations fit into the talk on how to keep Rails on the tracks by writing maintainable code, but nonetheless here they are:

• combining JS, CSS, CSS sprites, utilizing the Rails 3.1 asset pipeline
• caching optimization: fragment caching, action caching, page caching
• avoid a bloated session, and avoid storing objects in the session
• push things out to the browser if possible to minimize data and web-app load

Another topic that Mikel touched on was how being smart can be stupid. He recommends to not use before and after filters for object instantiation and to minimize their use to state modifications such as authentication, authorization or related to the user session. Mikel mentioned that while meta programming is sexy and has its place, that that place is not in your Rails app because it's harder for other developers and even yourself to understand what's automagically happening when you look at the code 3 months after you wrote it.

Mikel mentioned a few examples of using the right tools for the job. He discussed two examples where using simple SQL reduced a Ruby report run-time from 45 minutes down to 15 seconds and a implementing a PostgreSQL COPY statement that completed a data migration in 74 minutes down from 150 hours. Mikel also noted that Cucumber is not unit testing, so just write unit tests!

Confident Code

Next up, Avdi gave a nice talk about writing confident code and explained the standard narrative of a method or function:

When gathering input, Avdi recommends that developers employ strategies to deal with uncertain inputs such as coercion (e.g. to_s, to_i), rejection (guard clause or something more complex to return method), or ignoring invalid inputs. He also talked about his approach to having a zero tolerance for nil in Ruby because it's overused and can be indicative of too many things such as an error, missing data, the default behavior, and an uninitialized variable.

In part 2 of the narrative, perform work, I liked Avdi's comments about how conditionals should be reserved for business logic. For example, consider the following two conditionals:

A) if post.published?  ... end
B) if post ... end

The first line of code most likely represents business logic, while the second line may not. We want to minimize occurrences of instances of the second where conditionals are not representative of business logic. Avdi also touched on writing with the confident styles of chaining and iterative style such as that used in jQuery, where a jQuery selector does nothing when empty.

In part 3 of the narrative, deliver results, Avdi suggested to employ a style to raise a special case or a null object rather than a meaningless nil if there are no results. Finally, while handling failures, Avdi suggested to write the happy path first and have a visual divide in the code between the happy and unhappy paths.

Conclusion

My takeaways from the talks are:

• write code for people first (even yourself), then computers. This seemed to be a recurring recommendation at RailsConf.
• writing code is communicating the business logic. make sure it's clear, componentized, and each method has a single responsibility.
• like photography, sometimes the art is more about what you leave out rather than what you include.

While these takeaways aren't novel, I did like the insight into how both speakers approach development.

JavaScript and APIs at RailsConf 2011

A big trend of RailsConf 2011 has been the presence of JavaScript, whether it be talk of CoffeeScript, node.js, Sproutcore, backbone.js, etc. I attended Yehuda Katz's talk on Building Rails Apps for the Rich Client on Day 4 of the conference.

Part 1: Rails versus Sinatra Smackdown

Yehuda gave a two part talk about developing API-heavy applications. The first bit addressed why to develop in Rails rather than Sinatra if your application is API-heavy and doesn't appear to be utilizing valuable parts of Rails? This is fairly applicable to a couple of Sinatra projects that I've done at End Point — I like Sinatra a lot, but at some point you begin to replicate parts of Rails. Yehuda explained that because it's easy to develop web applciations efficiently using Rails conventions, developers can become forgetful/ignorant of the underlying functionality of Rails that doesn't ship with something like Sinatra, much like how working with an ORM can breed developers who aren't familiar with the underlying database interactions. The checklist for things that Rails manages we might forget about includes:

• ActionDispatch internals
• Session deserialization
• Browser standards mode
• Cookie abstraction and management
• Deserialization of content types like JSON, XML
• Reloading (by comparison Sinatra doesn't reload automagically unless you use a tool like shotgun)
• Handling IP Spoofing
• Routing / Complex Routing
• Browser Caching (ETags, Last-Modified)
• Content negotiation to automatically support different MIMEs
• ActionMailer

This list is is a nice checklist of items you might review when you are considering building in Sinatra to be reminded of things that won't necessarily be trivial to implement.

Part 2: Consistent APIs

The second part of Yehuda's talk covered his experience with building client rich applications with APIs. His general observation is that while Rails follows universal conventions like the ActiveRecord convention that requires foreign keys to be named {table_name}_id, there isn't a convention for APIs. While there has been decent support for building APIs in Rails since 2006, there's been missing documentation on what to generate which has producted APIs with limited consistency. He gave the following pro tips for developing an API:

• Return a JSON object with one or more keys, that is iterable. Rather than returning

  { product_title: "Some Awesome Product." }
  

  return something that has one or more keys that is iterable, such as:

  {
      "products": [
          {
              id: 1,
              title: "Some Awesome Product."
          }
      ]
  }
  

• Avoid nested resources like /users/1/orders. Instead, always make the request do a GET on /orders or a PUT on /orders/1. This will help maintain consistency (and maintainability) with a simple asset reqeusts.
• In general, create a convention and follow conventions. Essentially, apply the principles of Rails here to produce maintainable and consistent code.

Finally, Yehuda presented the bulk_api gem to address the use case where you need to get a set of items, but want to avoid making single requests for multiple items and avoid getting all items and parse through them on the client side. An ecommerce example of this might be applicable in inventory management: Let's say you have ~30 unrelated listed inventory units that need to be updated to an on_hand state. From the admin screen, you may click on checkboxes next to those 30 items and then "Update". Rather than sending a single request to update each item, the bulk_api gem will make a single request to update the inventory unit status for all of the objects. Similarly, you can make a GET request to the bulk api to retrieve a set of objects, such as looking at a set of products on page 2 of a product listing. Yehuda went into more details on authentication and authorization with this gem, so I'd recommend reading the documentation here.

Conclusion

Yehuda concludes that Rails is great for JSON APIs and that we should continue to be building abstractions similar to ActiveRecord abstractions that follow conventions in a sensible way. Hooray for convention over configuration.

Sass at RailsConf 2011

With the inclusion of the Scss gem in Rails 3.1, RailsConf is a nice time to get a refresher on Sass/Scss functionality. Sass defines itself as syntactically awesome stylesheets, or a CSS meta language built to provide more powerful functionality to manipulate website appearances with efficiency and elegance. Note that Sass has two syntaxes and the examples presented in this article use the newer Scss syntax. Around the time of RailsConf two years ago, Sass was a included in Spree, an open-source Ruby on Rails ecommerce framework that End Point supports. At the time, I was skeptical about Sass inclusion in Spree because it wasn't being leveraged to it's full potential and had hopes of taking advantage of Sass, but a few months later it was removed from the core. Since then, I haven't worked with Sass on other projects but hope to do so moving forward after being reminded of it's features and of the fact that it will be included in Rails 3.1 as a default. I attended Chris Eppstein's talk on Sass and explain a few features related to real-life use cases of CSS manipulation.

Variables

While working on a new feature, your client says, "I want this to be the same red that we use all over the site." This is exactly what I experienced while working on Paper Source to build out new Quickview/popup functionality shown in the image below.

Sass variables can be defined and then included in various selectors. Need to change the styling for all those selectors that use this variable? Just change the variable instead of grepping through the code for the color and different variations of the color. Here's an example of what variable definition and use might look like:

$red_error: #33FF00;
.error {
    color: $red_error;
}
.qty_alert {
    color: $red_error;
}

Nesting

Instead of writing code with repeated selectors, take full advantage of nesting in Sass. Below is a good example of a repeated styling selector I use for Paper Source's custom wedding invitation feature:

.editor_box {}
  .editor_box label { font-size: 11px; display: block; padding: 2px 0px; font-style: italic; }
  .editor_box label a { font-size: 9px; color: #82B3CC; }
  .editor_box .fonts { float: left; width: 176px; margin: 0px 5px 5px 0px; }
    .editor_box fonts select { width: 176px; margin: 0px; }

In Sass, this might would look like:

.editor_box {
    label {
        font-size: 11px;
        display: block;
        padding: 2px 0px;
        font-style: italic;
        a { 
            font-size: 9px;
            color: #82B3CC;
        }
    }
    .fonts { 
        float: left;
        width: 176px;
        margin: 0px 5px 5px 0px;
        select {
            width: 176px; margin: 0px;
        }
    }
}

While the nested functionality doesn't produce less lines of code, it does do a better job of following the DRY principle and is more readable in Sass form.

Mixins

Next, while building out a new feature, a client says, "I want you to build a product thumbnail page that has equivalent styling to our other thumbnail pages." See the thumbnails in the image below that share similar styling on multiple product navigation pages through Paper Source's site.

Mixins are reusable sections of CSS that can be included in other selectors to reduce duplicate property and value definitions. A simple example, shown below, includes the product thumbnail mixin in two distinct classes.

@mixin product_thumbnail {
    a { 
        color: $blue;
        text-decoration: none;
        &:hover {
            color: $red;
        }
    }
    img { padding-top: 10px; }
}
.category_products {
    @include product_thumbnail;
}
.special_products {
    @include product_thumbnail;
}

Transformations

Next, while building out another(!) new feature, the client comments, "I want this to be a color close to the other one, but a little different. Can you try what you think looks good?" The image below is a screen snippet with a jQuery-based accordion with different colors representing open and closed sections.

Transformations are calculations that can be applied to values such as color manipulation like saturate, desaturate, lighten, darken, make greyscale. Sass might look like this in the jQuery accordion scenario to show a lightened version of blue on inactive accordion regions:

$blue: #99CCCC;
.active {
    background-color: $blue;
}
.inactive {
    background-color: lighten($blue, 25%);
}

Importing Stylesheets

From a performance perspective, it's ideal to have a single compiled CSS file included on each page. This can be difficult for maintainability as you try to manage an extremely large stylesheet. The CSS directive @import can import additional stylesheets, but these imported files are divided in multiple HTTP requests. Sass's approach allows you to include stylesheets with rules and other Sass functionality, where a single file will be created at compile time. In the Paper Source example, we could do the following to include styles for thumbnails on various pages:

_product_thumbnails.scss
@mixin product_thumbnail {
    a { 
        color: $blue;
        text-decoration: none;
        &:hover { color: $red; }
    }
    img { padding-top: 10px; }
}

category.scss
@import "product_thumbnails"
.category_products {
    @include product_thumbnail;
}

@import "product_thumbnails"
landing_page.scss
.special_products {
    @include product_thumbnail;
}

Check out the Sass documentation to read more about Sass and it's features, including abstractions, calculations, and selector inheritence covered by Chris.

CSS Sprites with Compass

Compass is an additional gem that can be installed on top of Sass. In my opinion, the best feature of Compass mentioned in the talk was automated CSS spriting. CSS sprites is a technique where an aggregate of images is served as one image and CSS is used to show only a portion of the image in or as the DOM element. I've built a few different scripts in Ruby and Perl using ImageMagick that automatically build sprites, and was pleasantly surprised to hear that there is a feature in Compass that handles this. With Compass installed, CSS sprites in Sass might look like the code below, where wrapper elements are automagically compiled into a single sprited image and CSS rules are defined.

@import "wrapper_elements/*.png";
$wrapper-sprite-dimensions: true;
@include all-wrapper-sprites;

Conclusion

Admittedly, the examples shown in this blog article come from a site that runs on Perl-based Interchange, but I used these examples because I can distinctly remember each of these use cases. It might not be quite as easy to include Sass here with Interchange as it will be in Rails 3.1, where Scss/Sass is included as a new default.

Rails 3 at RailsConf 2011

A keynote by DHH kicked off Day 2 of RailsConf, where much of his talk was spent discussing new asset behavior in Rails 3.1. Here's a run down of a few topics I found worth noting:

Asset Pipeline

DHH started by explaining how although a Rails application has lovely organization in terms of Ruby files, the assets (stylesheets, images, and JavaScripts) have become a junk drawer where junk continues to pile in. Once there's more than a handful of files, the broken window theory applies and no one tries to maintain organization of those assets. This gets nasty, like a honey badger.

With Rails 3.1, the asset pipeline addresses the junk drawer. Assets directories (images, stylesheets, and JavaScripts) are now created in the app, lib, and vendor assets directories as they pertain to the main app, plugin dependencies, or introduce new library dependencies like a jquery calendar date select plugin. There's also the introducton of appending to config.assets.paths, which allows you to add new directories that store these assets in arbitrary directories. The new asset pipeline allows you to store these assets in different organization, which encourages JavaScript files to be stored based on their level of abstraction, and the asset pipeline combined with Bundler enables you to track the jquery version with a jquery-rails gem yielding better maintenance.

New Defaults

Rails 3.1 now assumes the default of CoffeeScript and scss (Sass). Jason discussed a Sass talk he attended yesterday at BohConf which includes things like nesting to reduce duplicate code and variables to improve maintainability. I haven't worked with CoffeeScript much, so I'll just link to the CoffeeScript documentation and possibly attend a CoffeeScript talk tomorrow. The argument between setting defaults and setting no defaults was revisited, and defaults won. The new defaults use Bundler to include Sass and coffee-script in the Gemfile:

gem 'sass'
gem 'coffee-script'

And these can simply be commented out of the dependency list if desired. In my case, if I were developing a Rails app tomorrow with a limited budget, I might choose to use Sass since I've worked with it before, but pass on CoffeeScript until I learned more and felt confident working with it.

Scalability and Compiling

Another question that comes up with these new defaults is the scalability and compilability. A new rake task is introduced:

rake assets:precompile

The rake task goes through the load path to precompile application JavaScript and CSS into application-*md5_hash*.js or application-*md5_hash*.css and copy over the images to the application public directory. This new file based method ensures that users will request the correct application file in addition to keeping the older compiled files around. Finally, compression tools are built straight into Rails, uglifier for JavaScript compression and scss for CSS compression. There is no penalty to writing comments or white-space rich code with these compression tools built in.

We need a photo break. A Honey Badger.

The second talk I attended was "SOLID Design Principles Behind the Rails 3 Refactoring" by José Valim, a member of the Rails core team.

Single Responsibility Principle

Jose spent the most time talking about the single responsibility principle, or that a class should have one and only one purpose. José discussed the evolution of the ActionView::Base that was responsible for tracking details, finding templates, compling templates, and rendering to gradually be divided into the following components and responsibilities in Rails 3:

• View Path: holds resolvers
• Resolver: finds template. The resolvers abstraction no longer restricts templates to the filesystem (can be anywhere in the app, web service, or even database) which simplifies testing and therefore improves maintainability.
• Lookup Context: tracks details, lookup context object is passed to the view.
• AV::Renderer: renders templates
• ActionView::Base: renders context

By applying the single responsibility principle to the view rendering functionality in Rails, modularization now allows us to extend or override individual points of the process (such as grabbing a template from a CMS-driven database, or passing a different lookup context object to the view) and ensure maintainability by enabling more testable code.

José talked about the other principles, but some pertain to static languages more so than Ruby as the book was originally written with static languages in mind. These included:

• Open/closed principle: José discussed extending ActiveRecord::Base for one modular bit of application but not another.
• Dependency Inversion Principle
• Liskov Substitution Principle: José's example here in substitution can be applied to the argument between DataMapper versus ActiveRecord and Rails 3.0 took steps to define an API working with an ORM to make it more substitutable.
• Interface Segregation Principle

Since I missed a few points here and there, feel free to check out the conference keynote videos here and I'll add a link to José's talk when it becomes available. Undoubtedly, Rails 3.0 and related Rails 3.0 topics will continue to be a highlight of the conference and I look forward to sharing more!

RailsConf 2011 - Day One

Today was the first official day of RailsConf 2011. As with most technical conferences, this one spent the first day with tutorials and workshops. For those of us without paid access to the tutorial sessions, the BohConf was a nice way to spend our first day of the four-day event.

BohConf is described as the "unconference" of RailsConf. It's a loosely organized collection of presentations, mini-hackathons and barcamp-style meetings. I spent the first half of Monday at the BohConf. Of particular interest to me was Chris Eppstein and Scott Davis' introduction to Sass and Compass. I've dabbled with Sass in the past but only recently learned of Compass.

Sass is a great way to construct your CSS without the tedious duplication that's typical of most modern spreadsheets. Introducing programming features like variables, inheritance and nested blocks, Sass makes it easy to keep your source material concise and logical. Once your source declarations are ready, compile your production spreadsheets with Sass or Compass.

Compass is effectively a framework for easy construction and deployment of spreadsheets using Sass. To hear Scott describe it, "Compass is to Sass as Rails is to Ruby". Together they're a very attractive combination for the Ruby developer who also dabbles in design (and who doesn't these days?). Truth be told, while I'm very impressed with the capabilities of Sass, I worry about the trend to re-introduce logic and presentation. My mom raised me to abstract the presentation layer for ease on graphic designers, and that rule has suited me well to date. Time will tell.

In the afternoon I wandered into a sponsored workshop from VMware. Dave McCrory and Dekel Tankel led a demonstration of their new CloudFoundry service. A nice reward for attending was getting instant approval of your CloudFoundry.com beta registration. Although I felt mildly guilty for it afterwards, I took advantage of this opportunity to get an extra beta account (my personal request had been approved a week earlier).

Dave introduced everyone to the CloudFoundry beta offering and discussed a vague roadmap for the open source project and their own commercial VM product slated for early 2012. They emphasized that the CloudFoundry core project will remain open source, and that interested parties can fork it on github, hack on the code, and deploy their own private "micro-clouds". Dave even hinted that at least one startup has based their PHP PaaS service on CloudFoundry.

Once all the attendees had received their beta accounts, Dekel walked us through the installation and basic command-line usage of the vmc utility. I was able to immediately vmc login with my Beta account credentials and change my password with vmc passwd. I should note that before logging in, I had to choose my target server with vmc target api.cloudfoundry.com. Why is this important? This will allow developers to easily switch between targeted environments. For example, I could install CloudFoundry on my workstation or development server and "target" it as my development or staging environment. Once my tests pass, I can quickly switch targets and push the changes to production.

They had us follow along by recreating a sample Ruby application designed to check if one Twitter account follows another. The examples were simple and easy to follow. Once we had our Gemfile, controller and views in place, we had to bundle package all of the dependencies. Once this was complete, a quick vmc push myappname and we were live!

Unfortunately, most (if not all) of us encountered a gotcha with this example. Because all of our instantiated applications reside behind the same IP address on CloudFoundry's network, we quickly hit Twitter's API quota. I'm not sure if this will be a problem once CloudFoundry officially launches, but it's something to keep in mind. And while it was useful to vmc logs myappname to debug this problem, an astute attendee brought up the fact that there was no way to tail application logs in CloudFoundry. This is a glaring oversight and one I hope they rectify before the Beta is finished.

The workshop continued with an introduction into binding services like Redis, Mongo or MySQL. We added new functionality to our existing application that introduced a Redis backend to store leaderboard information for Twitter activity. Lastly, Dekel demonstrated the ease of scaling our applications with vmc instances myappname 5. This simple command instantiates new copies of the application behind their dynamic load balancer. Coincidentally, they're not currently offering any sort of scalable backend storage, so keep that in mind before you try to launch any production sites on their Beta service. Now you'd never do that, would you? ;-)

The first day of RailsConf 2011 was impressive. I came last year as an exhibitor and am really excited that I get to attend this year for the conference sessions. I can't wait to see all the talks tomorrow!

Ruby, Rails, and Ecommerce

I'm a big fan of Ruby. And I like Rails too. Lately, I've been investigating several Ruby, Rails, and Rails ecommerce framework options (follow-up to discussing general ecommerce options). I've also recently written about developing ecommerce on Sinatra (one, two, and three). Most of End Point's clients are ecommerce clients, so we've seen it all in terms of feature requests (third party integration like QuickBooks, search, PayPal, product features like best sellers, recommended items, related items, checkout features like one-page checkout, guest checkout, backend features like advanced reporting, sales management, inventory management). Our services also include hosting and database consulting for many of our ecommerce clients, so we have a great understanding of what it takes to run an ecommerce solution.

When it comes to ecommerce development, someone who likes coding in Ruby (like me) has a few options:

• Ruby DSL (e.g. Sinatra)
• Pure Rails
• Open Source Ecommerce on Rails: Spree, ROR-Ecommerce, Substruct. End Point admittedly has the most experience with Spree.

Here's a run down of some pros and cons of each option:

Or, here's another way to look at things:

Generalizations I've made after developing in each of the scenarios.

Another thing to consider is what might happen when application changes are required a year after the project was initially implemented. Here's what might happen in each of the scenarios:

• Ruby DSL: You still have to write custom code. You may pick up using the conventions and organization where you left off. You may be able to use open source gems in your application. Ruby probably hasn't changed a ton. The Ruby DSL (e.g. Sinatra) may have changed.
• Rails: You still have to write custom code in Rails. You may be able to use open source gems in your application. Rails may have changed if there was a major release, but chances are your current version of Rails is still supported.
• Open Source Ecommerce on Rails: You may or may not have to write custom code because the community may provide the functionality you need. The platform may have changed significantly, and there's a chance that your current version of the platform is no longer supported.

Any one of the options presented in this article may be suitable for our clients: Sinatra for a small, simple, custom app, Rails for a complex application with custom needs, open source ecommerce on Rails for a client that follows the standard ecommerce mold. As consultants, our goal is to choose the best tool for a client to meed their business needs, long-term goals and budget.

One final note to add is that my generalizations discussed here regarding open source ecommerce on Rails and open source projects in general are highly dependent on the maturity of the framework. As an open source project matures, its stability and flexibility may improve.

ActiveProduct -- Just the Spree Products

ActiveProduct

I wanted to see how difficult it would be to cut out the part of Spree that makes up the product data model and turn it into a self sufficient Rails 3 engine. I followed the tutorial here to get started with a basic engine plugin. Since it sounded good and nobody else seems to be using it, I decided to call this endeavor ActiveProduct.

Which Bits?

The next step was to decide which parts I needed to get. Minimally, I need the models and migrations that support them. If that works out, then I can decide what to do about the controllers and views later.

That said, the lines between what is needed to create a product and the rest of the system are not always so clear cut. You have to cut somewhere, though, so I cut like this:

• Image
• InventoryUnit
• OptionType
• OptionValue
• ProductGroup
• ProductOptionType
• ProductProperty
• Product
• ProductScope
• Property
• Prototype
• Variant

Models

Each of these has a model file in spree/core/app/models, which I just copied over to the app/models directory of my engine.

Migrations

It'd be convenient if I could have just carved the appropriate parts out of the schema.rb file for the migration. But said file does not appear to be in evidence. Building a spree instance, and trying to coerce one out of it just seems too annoying, so I did something else.

I started from the first migration, removed all of the table definitions that didn't interest me and manually applied all of the migrations in the migration file to the remaining definitions. By manually applied, I mean I went through each migration file one at a time and made the specified change to the original set of definitions. There are, of course, all kinds of reasons why this is a terrible idea. For a reasonably small set of tables with a simple set of relations, the trade-off isn't too bad.

Migration Generator

With a single migration in hand, I followed the tutorial at here as a guide to create a generator for it in the engine. With the migration set up as a generator, I went to my sandbox rails app and ran the migration by doing the following:

$ rails g active_product create_migration_file
$ rake db:migrate

Did it Work?

At this point, I had some the tables in the database and the model files in place it was time to see if things worked.

$ rails console
rb(main):001:0> p = Product.new

...and I got a big wall of error messages. So I could not even instantiate the class, much less start using it. Well, I kind of expected that.

Resolving Dependencies

Missing Constants

Following the error messages started with an unresolved dependency on delegate_belongs_to. A little sleuthing lead me into the Spree lib directory were a copy of this plugin lives. Some further knocking around the interwebs lead me to this project which appears to be the canonical version of the plugin. Since I was trying to create a stand alone module, I wanted set this up as an external dependency (which I will defer figuring out how to enforce in the engine until later).

As an aside, (the newer version of) delegate_belongs_to has an issue with and API change in ActiveRecord for Rails 3. A version that will at least load with ActiveProduct can be found here.

The active_product engine currently assumes that delegates_belongs_to is available in the project that it is installed in. I set it up as a normal plugin in the vendor/plugins directory.

Circular Dependencies

With that out of the way, the next error seemed to be about Variants and Scope. In spree/core/lib/scopes there are a couple of files that interact with the Product and Variant classes in a somewhat messy way. In order to make use of the scopes that are defined there, I needed to pull them in. Ultimately, it probably makes sense to include the changes directly into the affected class files. Since I was still experimenting here, I just moved them to approximately the same place in the engine.

It turns out that the dependency relationships between Product, Variant, and the Scopes module are pretty complex. I spent a fair amount of time trying to sort them out manually, but was unable to find any reasonable way. Eventually, I decided to give up and fall back on the auto loader to handle it for me.

The auto-loader in Rails seems to cover a multitude of sins. A well behaved independent module will need to remove all of these interdependencies. There are a couple of significant problems with leaving them be:

• Other potential users of such a module will not necessarily make sure that everything is auto-loaded, so a this module would just be broken. Sinatra comes to mind.
• Interdependencies increase complexity. While complexity is not inherently bad, it can be a source of bugs and errors, so it ought to be avoided when possible.

To start with, I moved the scope.rb file and the scope directory to active_product/lib/auto. And I added the following to the ActiveProduct module definition in active_product/lib/active_product/engine.rb:

module ActiveProduct
  class Engine < Rails::Engine
    config.autoload_paths += %W(#{config.root}/lib/auto)

There are two interesting things about this to note:

• The engine lib directory is not auto-loaded in the same way that app/models, app/controllers, etc are. There apparently is no convention for loading a lib directory. I picked 'lib/auto', but there are not any constraints on what can be added.
• The engine has its own config variable that is loaded and honored as part of the Rails app config.

Now What?

Now when I tried to instantiate the class I found that it called a couple of methods that I'm not quite sure what I am going to do with yet. These are:

make_permalink
search_methods

make_permalink is provided by an interestingly named Railslove module in the spree_core lib and seems harmless enough. For now, I commented the call out of the Product lib.

search_methods is provided by the MetaSearch plugin which is a Spree dependency. Search is neat, but I'll sort it out later. Again, I commented it out and will deal with it if it causes problems.

Where are we now

I can now instantiate a new product object from the console. That seems to somewhat validate the effort to isolate the module. You may be tempted to ask if you could use that product instance for anything; saving a copy to the data store, for instance. Here's a hint: circular dependencies.

The code that I've worked on up till now can be seen here.

A Ruby on Rails Tag Cloud Tutorial with Spree

A tag cloud from a recent End Point blog post.

Tag clouds have become a fairly popular way to present data on the web. One of our Spree clients recently asked End Point to develop a tag cloud reporting user-submitted search terms in his Spree application. The steps described in this article can be applied to a generic Rails application with a few adjustments.

Step 1: Determine Organization

If you are running this as an extension on Spree pre-Rails 3.0 versions, you'll create an extension to house the custom code. If you are running this as part of a Rails 3.0 application or Spree Rails 3.0 versions, you'll want to consider creating a custom gem to house the custom code. In my case, I'm writing a Spree extension for an application running on Spree 0.11, so I create an extension with the command script/generate extension SearchTag.

Step 2: Data Model & Migration

First, the desired data model for the tag cloud data should be defined. Here's what mine will look like in this tutorial:

Next, a model and migration must be created to introduce the class, table and it's fields. In Spree, I run script/generate extension_model SearchTag SearchRecord and update the migration file to contain the following:

class CreateSearchRecords < ActiveRecord::Migration
  def self.up
    create_table :search_records do |t|
      t.string :term
      t.integer :count, :null => false, :default => 0
    end
  end

  def self.down
    drop_table :search_records
  end
end

I also add a filter method to my model to be used later:

class SearchRecord < ActiveRecord::Base
  def self.filter(term)
    term.gsub(/\+/, ' ')
      .gsub(/\s+/, ' ')
      .gsub(/^\s+/, '')
      .gsub(/\s+$/, '')
      .downcase
      .gsub(/[^0-9a-z\s-]/, '')
  end
end

Step 3: Populating the Data

After the migration has been applied, I'll need to update the code to populate the data. I'm going to add an after filter on every user search. In the case of using Spree, I update search_tag_extension.rb to contain the following:

def activate
  Spree::ProductsController.send(:include, Spree::SearchTagCloud::ProductsController)
end

And my custom module contains the following:

module Spree::SearchTagCloud::ProductsController
  def self.included(controller)
    controller.class_eval do
      controller.append_after_filter :record_search, :only => :index
    end
  end

  def record_search
    if params[:keywords]
      term = SearchRecord.filter(params[:keywords])
      return if term == ''
      record = SearchRecord.find_or_initialize_by_term(term)
      record.update_attribute(:count, record.count+1)
    end
  end
end

The module appends an after filter to the products#index action. The after filter method cleans the search term and creates a record or increments the existing record's count. If this is added directly into an existing Rails application, this bit of functionality may be added directly into one or more existing controller methods to record the search term.

Step 4: Reporting the Data

To present the data, I create a controller with script/generate extension_controller SearchTag Admin::SearchTagClouds first. I update config/routes.rb with a new action to reference the new controller:

map.namespace :admin do |admin|
  admin.resources :search_tag_clouds, :only => [:index]
end

And I update my controller to calculate the search tag cloud data, shown below. The index method method retrieves all of the search records, sorts, and grabs the the top x results, where x is some configuration defined by the administrator. The method determines the linear solution for scaling the search_record.count to font sizes ranging from 8 pixels to 25 pixels. This order of terms is randomized (.shuffle) and linear equation applied. This linear shift can be applied to different types of data. For example, if a tag cloud is to show products with a certain tag, the totals per tag must be calculated and scaled linearly.

class Admin::SearchTagCloudsController < Admin::BaseController
  def index
    search_records = SearchRecord.all
      .collect { |r| [r.count, r.term] }
      .sort
      .reverse[0..Spree::SearchTagCloud::Config[:count]]
    max = search_records.empty? ? 1 : search_records.first.first

    # solution is: a*x_factor - y_shift = font size 
    # max font size is 25, min is 8
    x_factor = (Spree::SearchTagCloud::Config[:max] - 
      Spree::SearchTagCloud::Config[:min]) / max.to_f
    y_shift = max.to_f*x_factor - Spree::SearchTagCloud::Config[:max]

    @results = search_records.shuffle.inject([]) do |a, b|
      a.push([b[0].to_f*x_factor - y_shift, b[1]])
      a
    end
  end
end

The data is presented to the user in the following view:

<h3>Tag Cloud:</h3>
<div id="tag_cloud">
<% @results.each do |b| %>
<span style="font-size:<%= b[0] %>px;"><%= b[1] %></span>
<% end -%>
</div>

Step 5: Adding Flexibility

In this project, I added configuration variables for the total number of terms displayed, and maximum and minimum font size using Spree's preference architecture. In a generic Rails application, this may be a nice bit of functionality to include with the preferred configuration architecture.

Example tag cloud from the extension.
Additional modifications can be applied to change the
overall styling or color of individual search terms.

Conclusion

These steps are pretty common for introducing new functionality into an existing application: data migration and model, manipulation on existing controllers, and presentation of results with a new or existing controller and view. Following MVC convention in Rails keeps the code organized and methods simple. In the case of Spree 0.11, this functionality has been packaged into a single extension that is abstracted from the Spree core. The code can be reviewed here, with a few minor differences.

Ruby Ecommerce with Sinatra: Admin and Products

Last week, I wrote about creating a very simple ecommerce application on Ruby with Sinatra. This week, we continue on the yellow brick road of ecommerce development on Ruby with Sinatra.

A yellow brick road.

Part 2: Basic Admin Authentication

After you've got a basic application running which accepts payment for a single product as described in the previous tutorial, the next step is to add admin authorization to allow lookup of completed orders. I found several great resources for this as well as a few Sinatra extensions that may be useful. For the first increment of implementation, I followed the instructions here, which uses Basic::Auth. The resulting code can be viewed here. I also introduce subclassing of Sinatra::Base, which allows us to keep our files a bit more modular and organized.

And if we add an "/admin" method to display orders, we can see our completed orders:

Completed orders.

Part 3: Introducing Products

Now, let's imagine an ecommerce store with different products! Whoa! For this increment, let's limit each order to one product. A migration and model definition is created to introduce products, which contains a name, description, and price. For this increment, product images match the product name and live ~/public/images. The orders table is modified to contain a reference to products.id. The orders model is updated to belong_to :products. Finally, the frontend authorization method is modified to use the order.product.price in the transaction.

# Products ActiveRecord migration
require 'lib/model/product'
class CreateProducts < ActiveRecord::Migration
  def self.up
    create_table :products do |t|
      t.string :name,
        :null => false
      t.decimal :price,
        :null => false
      t.string :description,
        :null => false
    end
  end

  def self.down
    drop_table :products
  end
end

 
# Products model class
class Product < ActiveRecord::Base
  validates_presence_of :name
  validates_presence_of :price
  validates_numericality_of :price
  validates_presence_of :description

  has_many :orders
end

# Order migration update
class CreateOrders < ActiveRecord::Migration
  def self.up
    create_table :orders do |t|
+      t.references :product,
+        :null => false
    end
  end

  def self.down
    drop_table :orders
  end
end

# Order model changes
class Order < ActiveRecord::Base
...
+ validates_presence_of :product_id
+
+  belongs_to :product
end

# in main checkout action
# Authorization amount update
- response = gateway.authorize(1000,
-   credit_card)
+ response = gateway.authorize(order.product.price*100,
+   credit_card)

And let's use Sinatra's simple and powerful routing to build resource management functionality that allows our admin to list, create, update, and delete items, or in this case orders and products. Here's the sinatra code that accomplishes this basic resource management:

# List items
app.get '/admin/:type' do |type|
  require_administrative_privileges
  content_type :json

  begin
    klass = type.camelize.constantize
    objects = klass.all
    status 200
    objects.to_json
  rescue Exception => e
    halt 500, [e.message].to_json 
  end
end

# Delete item
app.delete '/admin/:type/:id' do |type, id|
  require_administrative_privileges
  content_type :json

  begin
    klass = type.camelize.constantize
    instance = klass.find(id)
    if instance.destroy
      status 200
    else
      status 400
      errors = instance.errors.full_messages
      [errors.first].to_json
    end
  rescue Exception => e
    halt 500, [e.message].to_json
  end
end

# Create new item
app.post '/admin/:type/new' do |type|
  require_administrative_privileges
  content_type :json
  input = json_to_hash(request.body.read.to_s)
 
  begin
    klass = type.camelize.constantize
    instance = klass.new(input)
    if instance.save
      status 200
      instance.to_json
    else
      status 400
      errors = instance.errors.full_messages
      [errors.first].to_json
    end
  rescue Exception => e
    halt 500, [e.message].to_json
  end
end

# Edit item
app.post '/admin/:type/:id' do |type, id|
  require_administrative_privileges
  content_type :json
  input = json_to_hash(request.body.read.to_s)
  
  begin
    klass = type.camelize.constantize
    instance = klass.find(id)
    if instance.update_attributes(input)
      status 200
      instance.to_json
    else
      status 400
      errors = instance.errors.full_messages
      [errors.first].to_json
    end
  rescue Exception => e
    halt 500, [e.message].to_json
  end
end

Note that in the code shown above, the request includes the class (product or order in this application), and the id of the item in some cases. The constantize method is used to get the class constant, and ActiveRecord methods are used to retrieve and edit, create, or delete the instance. This powerful routing now allows us to easily manage additional resources with minimal changes to our server-side code.

Next, I use jQuery to call these methods via AJAX, also in such a way that it'll be easy to manage new resources with minimal client side code. That base admin code can be found here. With this jQuery admin base, we now define our empty resource, content for displaying that resource, and content for editing that resource. Examples of this are shown below:

functions.product = {
  edit: function(product) {
    return '<h4>Editing Product: '
      + product.id
      + '</h4>'
      + '<p><label for="name">Name</label>'
      + '<input type="text" name="name" value="'
      + product.name
      + '" /></p>'
      + '<p><label for="price">Price</label>'
      + '<input type="text" name="price" value="'
      + parseFloat(product.price).toFixed(2)
      + '" /></p>'
      + '<p><label for="description">Description</label>'
      + '<textarea name="description">'
      + product.description
      + '</textarea></p>';
  },
  content: function(product) {
    var inner_html = '<h4>Product: '
      + product.id
      + '</h4>'
      + 'Name: '
      + product.name
      + '<br />Price: $'
      + parseFloat(product.price).toFixed(2)
      + '<br />Description: '
      + product.description
      + '<br />';
    return inner_html;
  },
  empty: function() {
    return { name: '',
      price: 0, 
      description: '' };  
  }
};

functions.order = {
  edit: function(order) {
    return '<b>Order: '
      + order.id
      + '</b><br />'
      + '<input name="email" value="'
      + order.email
      + '" />'
      + ' – '
      ...
      //Order editing is limited
  },
  content: function(order) {
    return '<b>Order: '
      + order.id
      + '</b><br />'
      + order.email
      + ' – '
      + order.phone
      + '<br />'
      ...
  },
  empty: function() {
    return { 
      email: '',
      phone: '',
      ...
    };  
  }
};

With a final touch of frontend JavaScript and CSS changes, the following screenshots show the two customer-facing pages from our example store. Like the application described in the previous article, this ecommerce application is still fairly lightweight, but it now allows us to sell several products and manage our resources via the admin panel. Stay tuned for the next increment!

The cupcake images shown in this article are under the Creative Commons license and can be found here, here, and here. The code shown in this article can be found here (branches part2 and part3).

Dissecting a Rails 3 Spree Extension Upgrade

A while back, I wrote about the release of Spree 0.30.* here and here. I didn't describe extension development in depth because I hadn't developed any Rails 3 extensions of substance for End Point's clients. Last month, I worked on an advanced reporting extension for a client running on Spree 0.10.2. I spent some time upgrading this extension to be compatible with Rails 3 because I expect the client to move in the direction of Rails 3 and because I wanted the extension to be available to the community since Spree's reporting is fairly lightweight.

Just a quick rundown on what the extension does: It provides incremental reports such as revenue, units sold, profit (calculated by sales minus cost) in daily, weekly, monthly, quarterly, and yearly increments. It reports Geodata to show revenue, units sold, and profit by [US] states and countries. There are also two special reports that show top products and customers. The extension allows administrators to limit results by order date, "store" (for Spree's multi-site architecture), product, and taxon. Finally, the extension provides the ability to export data in PDF or CSV format using the Ruport gem. One thing to note is that this extensions does not include new models – this is significant only because Rails 3 introduced significant changes to ActiveRecord, which are not described in this article.

	Screenshots of the Spree Advanced Reporting extension.

To deconstruct the upgrade, I examined a git diff of the master and rails3 branch. I've divided the topics into Rails 3 and Spree specific categories.

Rails 3 Specific

SafeBuffers

In my report extension, I utilize Ruport's to_html method, which returns a ruport table object to an HTML table. With the upgrade to Rails 3, ruports to_html was spitting out escaped HTML with the addition of default XSS protection. The change is described here. and required the addition of a new helper method (raw) to yield unescaped HTML:

diff --git a/app/views/admin/reports/top_base.html.erb b/app/views/admin/reports/top_base.html.erb
index 6cc6b70..92f2118 100644
--- a/app/views/admin/reports/top_base.html.erb
+++ b/app/views/admin/reports/top_base.html.erb
@@ -1,4 +1,4 @@
-<%= @report.ruportdata.to_html %>
+<%= raw @report.ruportdata.to_html %>

Common Deprecation Messages

While troubleshooting the upgrade, I came across the following warning:

DEPRECATION WARNING: Using #request_uri is deprecated. Use fullpath instead. (called from ...)

I made several changes to address the deprecation warnings, did a full round of testing, and moved on.

diff --git a/app/views/admin/reports/_advanced_report_criteria.html.erb b/app/views/admin/reports/_advanced_report_criteria.html.erb
index ba69a2e..6d9c3f9 100644
--- a/app/views/admin/reports/_advanced_report_criteria.html.erb
+++ b/app/views/admin/reports/_advanced_report_criteria.html.erb
@@ -1,11 +1,11 @@
 <% @reports.each do |key, value| %>
-  <option <%= request.request_uri == "/admin/reports/#{key}" ? 'selected="selected" ' : '' %>
  value="<%= send("#{key}_admin_reports_url".to_sym) %>">
+  <option <%= request.fullpath == "/admin/reports/#{key}" ? 'selected="selected" ' : '' %>
  value="<%= send("admin_reports_#{key}_url".to_sym) %>">
     <%= t(value[:name].downcase.gsub(" ","_")) %>
   

Integrating Gems

An exciting change in Rails 3 is the advancement of Rails::Engines to allow easier inclusion of mini-applications inside the main application. In an ecommerce platform, it makes sense to break up the system components into Rails Engines. Extensions become gems in Spree and gems can be released through rubygems.org. A gemspec is required in order for my extension to be treated as a gem by my main application, shown below. Componentizing elements of a larger platform into gems may become popular with the advancement of Rails::Engines / Railties.

diff --git a/advanced_reporting.gemspec b/advanced_reporting.gemspec
new file mode 100644
index 0000000..71f00a8
--- /dev/null
+++ b/advanced_reporting.gemspec
@@ -0,0 +1,22 @@
+Gem::Specification.new do |s|
+  s.platform    = Gem::Platform::RUBY
+  s.name        = 'advanced_reporting'
+  s.version     = '2.0.0'
+  s.summary     = 'Advanced Reporting for Spree'
+  s.homepage    = 'http://www.endpoint.com'
+  s.author = "Steph Skardal"
+  s.email = "steph@endpoint.com"
+  s.required_ruby_version = '>= 1.8.7'
+
+  s.files        = Dir['CHANGELOG', 'README.md', 'LICENSE', 'lib/**/*', 'app/**/*']
+  s.require_path = 'lib'
+  s.requirements << 'none'
+
+  s.has_rdoc = true
+
+  s.add_dependency('spree_core', '>= 0.30.1')
+  s.add_dependency('ruport')
+  s.add_dependency('ruport-util') #, :lib => 'ruport/util')
+end

Routing Updates

With the release of Rails 3, there was a major rewrite of the router and integration of rack-mount. The Rails 3 release notes on Action Dispatch provide a good starting point of resources. In the case of my extension, I rewrote the contents of config/routes.rb:

map.namespace :admin do |admin|
  admin.resources :reports, :collection => {
    :sales_total => :get,
    :revenue   => :get,
    :units   => :get,
    :profit   => :get,
    :count   => :get,
    :top_products  => :get,
    :top_customers  => :get,
    :geo_revenue  => :get,
    :geo_units  => :get,
    :geo_profit  => :get,
  }
  map.admin "/admin",
    :controller => 'admin/advanced_report_overview',
    :action => 'index'
end

Rails.application.routes.draw do
  #namespace :admin do
  #  resources :reports, :only => [:index, :show] do
  #    collection do
  #      get :sales_total
  #    end
  #  end
  #end
  match '/admin/reports/revenue' => 'admin/reports#revenue', :via => [:get, :post]
  match '/admin/reports/count' => 'admin/reports#count', :via => [:get, :post]
  match '/admin/reports/units' => 'admin/reports#units', :via => [:get, :post]
  match '/admin/reports/profit' => 'admin/reports#profit', :via => [:get, :post]
  match '/admin/reports/top_customers' => 'admin/reports#top_customers', :via => [:get, :post]
  match '/admin/reports/top_products' => 'admin/reports#top_products', :via => [:get, :post]
  match '/admin/reports/geo_revenue' => 'admin/reports#geo_revenue', :via => [:get, :post]
  match '/admin/reports/geo_units' => 'admin/reports#geo_units', :via => [:get, :post]
  match '/admin/reports/geo_profit' => 'admin/reports#geo_profit', :via => [:get, :post]
  match "/admin" => "admin/advanced_report_overview#index", :as => :admin 
end

Spree Specific

Transition extension to Engine

The biggest transition to Rails 3 based Spree requires extensions to transition to Rails Engines. In Spree 0.11.*, the extension class inherits from Spree::Extension and the path of activation for extensions in Spree 0.11.* starts in initializer.rb where the ExtensionLoader is called to load and activate all extensions. In Spree 0.30.*, extensions inherit from Rails:Engine which is a subclass of Rails::Railtie. Making an extension a Rails::Engine allows it to hook into all parts of the Rails initialization process and interact with the application object. A Rails engine allows you run a mini application inside the main application, which is at the core of what a Spree extension is – a self-contained Rails application that is included in the main ecommerce application to introduce new features or override core behavior.

See the diffs between versions here:

diff --git a/advanced_reporting_extension.rb b/advanced_reporting_extension.rb
deleted file mode 100644
index f75d967..0000000
--- a/advanced_reporting_extension.rb
+++ /dev/null
@@ -1,46 +0,0 @@
-# Uncomment this if you reference any of your controllers in activate
-# require_dependency 'application'
-
-class AdvancedReportingExtension < Spree::Extension
-  version "1.0"
-  description "Advanced Reporting"
-  url "http://www.endpoint.com/"
-
-  def self.require_gems(config)
-    config.gem "ruport"
-    config.gem "ruport-util", :lib => 'ruport/util'
-  end
-
-  def activate
-    Admin::ReportsController.send(:include, AdvancedReporting::ReportsController)
-    Admin::ReportsController::AVAILABLE_REPORTS.merge(AdvancedReporting::ReportsController::ADVANCED_REPORTS)
-
-    Ruport::Formatter::HTML.class_eval do
-      # Override some Ruport functionality
-    end
-  end
-end

diff --git a/lib/advanced_reporting.rb b/lib/advanced_reporting.rb
new file mode 100644
index 0000000..4e6fee6
--- /dev/null
+++ b/lib/advanced_reporting.rb
@@ -0,0 +1,50 @@
+require 'spree_core'
+require 'advanced_reporting_hooks'
+require "ruport"
+require "ruport/util"
+
+module AdvancedReporting
+  class Engine < Rails::Engine
+    config.autoload_paths += %W(#{config.root}/lib)
+
+    def self.activate
+      #Dir.glob(File.join(File.dirname(__FILE__), "../app/**/*_decorator*.rb")) do |c|
+      #  Rails.env.production? ? require(c) : load(c)
+      #end
+
+      Admin::ReportsController.send(:include, Admin::ReportsControllerDecorator)
+      Admin::ReportsController::AVAILABLE_REPORTS.merge(Admin::ReportsControllerDecorator::ADVANCED_REPORTS)
+
+      Ruport::Formatter::HTML.class_eval do
+        # Override some Ruport functionality
+      end
+    end
+
+    config.to_prepare &method(:activate).to_proc
+  end
+end

Required Rake Tasks

Rails Engines in Rails 3.1 will allow migrations and public assets to be accessed from engine subdirectories, but a work-around is required to access migrations and assets in the main application directory in the meantime. There are a few options for accessing Engine migrations and assets; Spree recommends a couple rake tasks to copy assets to the application root, shown here:

diff --git a/lib/tasks/install.rake b/lib/tasks/install.rake
new file mode 100644
index 0000000..c878a04
--- /dev/null
+++ b/lib/tasks/install.rake
@@ -0,0 +1,26 @@
+namespace :advanced_reporting do
+  desc "Copies all migrations and assets (NOTE: This will be obsolete with Rails 3.1)"
+  task :install do
+    Rake::Task['advanced_reporting:install:migrations'].invoke
+    Rake::Task['advanced_reporting:install:assets'].invoke
+  end
+
+  namespace :install do
+    desc "Copies all migrations (NOTE: This will be obsolete with Rails 3.1)"
+    task :migrations do
+      source = File.join(File.dirname(__FILE__), '..', '..', 'db')
+      destination = File.join(Rails.root, 'db')
+      puts "INFO: Mirroring assets from #{source} to #{destination}"
+      Spree::FileUtilz.mirror_files(source, destination)
+    end
+
+    desc "Copies all assets (NOTE: This will be obsolete with Rails 3.1)"
+    task :assets do
+      source = File.join(File.dirname(__FILE__), '..', '..', 'public')
+      destination = File.join(Rails.root, 'public')
+      puts "INFO: Mirroring assets from #{source} to #{destination}"
+      Spree::FileUtilz.mirror_files(source, destination)
+    end
+  end
+end

Relocation of hooks file

A minor change with the extension upgrade is a relocation of the hooks file. Spree hooks allow you to interact with core Spree views, described more in depth here and here.

diff --git a/advanced_reporting_hooks.rb b/advanced_reporting_hooks.rb
deleted file mode 100644
index fcb5ab5..0000000
--- a/advanced_reporting_hooks.rb
+++ /dev/null
@@ -1,43 +0,0 @@
-class AdvancedReportingHooks < Spree::ThemeSupport::HookListener
-  # custom hooks go here
-end

diff --git a/lib/advanced_reporting_hooks.rb b/lib/advanced_reporting_hooks.rb
new file mode 100644
index 0000000..cca155e
--- /dev/null
+++ b/lib/advanced_reporting_hooks.rb
@@ -0,0 +1,3 @@
+class AdvancedReportingHooks < Spree::ThemeSupport::HookListener
+  # custom hooks go here
+end

Adoption of "Decorator" naming convention

A common behavior in Spree extensions is to override or extend core controllers and models. With the upgrade, Spree adopts the "decorator" naming convention:

Dir.glob(File.join(File.dirname(__FILE__), "../app/**/*_decorator*.rb")) do |c|
  Rails.env.production? ? require(c) : load(c)
end

I prefer to extend the controllers and models with module includes, but the decorator convention also works nicely.

Gem Dependency Updates

With the transition to Rails 3, I found that there were changes related to dependency upgrades. Spree 0.11.* uses searchlogic 2.3.5, and Spree 0.30.1 uses searchlogic 3.0.0.*. Searchlogic is the gem that performs the search for orders in my report to pull orders between a certain time frame or tied to a specific store. I didn't go digging around in the searchlogic upgrade changes, but I referenced Spree's core implementation of searchlogic to determine the required updates:

diff --git a/lib/advanced_report.rb b/lib/advanced_report.rb
@@ -13,11 +15,26 @@ class AdvancedReport
     self.params = params
     self.data = {}
     self.ruportdata = {}
+
+    params[:search] ||= {}
+    if params[:search][:created_at_greater_than].blank?
+      params[:search][:created_at_greater_than] =
+        Order.first(:order => :completed_at).completed_at.to_date.beginning_of_day
+    else
+      params[:search][:created_at_greater_than] =
+        Time.zone.parse(params[:search][:created_at_greater_than]).beginning_of_day rescue ""
+    end
+    if params[:search][:created_at_less_than].blank?
+      params[:search][:created_at_less_than] =
+        Order.last(:order => :completed_at).completed_at.to_date.end_of_day
+    else
+      params[:search][:created_at_less_than] =
+        Time.zone.parse(params[:search][:created_at_less_than]).end_of_day rescue ""
+    end
+
+    params[:search][:completed_at_not_null] ||= "1"
+    if params[:search].delete(:completed_at_not_null) == "1"
+      params[:search][:completed_at_not_null] = true
+    end
     search = Order.searchlogic(params[:search])
-    search.checkout_complete = true
     search.state_does_not_equal('canceled')
-
-    self.orders = search.find(:all)
+    self.orders = search.do_search 
 
     self.product_in_taxon = true
     if params[:advanced_reporting]

Rakefile diffs

Finally, there are substantial changes to the Rakefile, which are related to rake tasks and testing framework. These didn't impact my development directly. Perhaps when I get into more significant testing on another extension, I'll dig deeper into the code changes here.

diff --git a/Rakefile b/Rakefile
index f279cc8..f9e6a0e 100644
# lots of stuff

For those interested in learning more about the upgrade process, I recommend the reviewing the Rails 3 Release Notes in addition to reading up on Rails Engines as they are an important part of Spree's core and extension architecture. The advanced reporting extension described in this article is available here.

Ruby on Rails versus CakePHP: A Syntax Comparison Guide

My time is typically split between Interchange and Spree development, but in a recent project for JackThreads, I jumped back into CakePHP code. CakePHP is one of the more popular PHP MVC frameworks and is inspired by Rails. I decided to put together a quick syntax comparison guide between CakePHP and Rails since I occasionally have to look up how to do some Rails-y thing in CakePHP.

app/
  controllers/ models/ views/ helpers/ 
lib/
config/
public
  javascripts/ images/ stylesheets/
vendors/
  plugins/ extensions/

controllers/
models/
views/
  layouts/ elements/ ...
config/
webroot/
tmp/
plugins/
vendors/

rails new my_app # Rails 3 after gem installation
rails my_app # Rails <3

class Zone < ActiveRecord::Base
  validates_presence_of :name
  validates_uniqueness_of :name
end

class User extends AppModel {
  var $name = 'User';
  var $validate = array(
    'email' => array(
      'email-create' => array(
        'rule' => 'email',
        'message' => 'Invalid e-mail.',
        'required' => true,
        'on' => 'create'
      )
    )
  );
}

class Order < ActiveRecord::Base
  belongs_to :user
  has_many :line_items
end

class Invite extends AppModel {
  var $name = 'Invite';
  var $belongsTo = 'User';
  var $hasMany = 'Campaigns';
}

class Address < ActiveRecord::Base
  has_many :billing_checkouts,
    :foreign_key => "bill_address_id",
    :class_name => "Checkout"
end

class Foo extends AppModel {
  var $name = 'Foo';
  var $hasMany = array(
    'SpecialEntity' => array(
      'className' => 'SpecialEntity',
      'foreignKey' => 'entity_id',
      'conditions' =>
  array('Special.entity_class' => 'Foo'),
      'dependent' => true
    ),
  );
}

class FoosController < ActionController::Base
  helper :taxons
  actions :show, :index

  include Spree::Search

  layout 'special'
end

class FooController extends AppController {
  var $name = 'Foo';
  var $helpers = array('Server', 'Cart');
  var $uses = array('SpecialEntity','User');
  var $components = array('Thing1', 'Thing2');
  var $layout = 'standard';
}

class FoosController < ActionController::Base
  before_filter :load_data, :only => :show
end

class FooController extends AppController {
  var $name = 'Foo';

  function beforeFilter() {
    parent::beforeFilter();
    //do stuff
  } 
}

class FoosController < ActionController::Base
  def index
    @special_title = 'This is the Special Title!'
  end
end

class FooController extends AppController {
  var $name = 'Foo';

  function index() {
    $this->set('title',
      'This is the Special Title!');
  }
}

<%= @special_title %>

<?= $special_title ?>

<% @foos.each do |foo| -%>
<%= foo.name %>
<% end -%>

<?php foreach($items as $item): ?>
<?= $item['name']; ?>
<?php endforeach; ?>

<%= render :partial => 'shared/view_name',
  :locals => { :b => "abc" } %>

<?php echo $this->element('account_menu',
  array('page_type' => 'contact')); ?>

<%= javascript_include_tag
  'my_javascript',
  'my_javascript2' %>
<%= stylesheet_link_tag
  'my_style' %>

<?php
  $html->css(array('my_style.css'),
    null, array(), false);
  $javascript->link(array('my_javascript.js'),
    false);
?>

# Rails 3
match '/cart',
  :to => 'orders#edit',
  :via => :get,
  :as => :cart
# Rails <3
map.login '/login',
  :controller => 'user_sessions',
  :action => 'new'

Router::connect('/refer',
  array('controller' => 'invites',
        'action' => 'refer'));
Router::connect('/sales/:sale_id',
  array('controller' => 'sale',
        'action' => 'show'),
  array('sale_id' => '[0-9]+')); 

# Rails 3
namespace :admin do
  resources :foos do
    collection do
      get :revenue
      get :profit
    end
  end
end

# Rails <3
map.namespace :admin do |admin|
  admin.resources :foos, :collection => {
    :revenue            => :get,
    :profit             => :get,
  }
end

Rails.logger.warn "steph!" # Rails 3
logger.warn "steph!" # Rails <3

RAILS_DEFAULT_LOGGER.warn "steph!"

$this->log('steph!', LOG_DEBUG);

If you are looking for guidance on choosing one of these technologies, below are common arguments. In End Point's case, we choose whatever technology makes the most sense for the client. We implemented a nifty solution for JackThreads to avoid a complete rewrite, described here in detail. We also work with existing open source ecommerce platforms such as Interchange and Spree and try to choose the best fit for each client.

Speeding up the Spree demo site

There's a lot that can be done to speed up Spree, and Rails apps in general. Here I'm not going to deal with most of that. Instead I want to show how easy it is to speed up page delivery using standard HTTP server tuning techniques, demonstrated on demo.spreecommerce.com.

First, let's get a baseline performance measure from the excellent webpagetest.org service using their remote Internet Explorer 7 tests:

• First page load time: 2.1 seconds
• Repeat page load time: 1.5 seconds

The repeat load is faster because the browser has images, JavaScript, and CSS cached, but it still has to check back with the server to make sure they haven't changed. Full details are in this initial report.

The demo.spreecommerce.com site is run on a Xen VPS with 512 MB RAM, CentOS 5 i386, Apache 2.2, and Passenger 2.2. There were several things to tune in the Apache httpd.conf configuration:

• mod_deflate was already enabled. Good. That's a big help.
• Enable HTTP keepalive: KeepAlive On and KeepAliveTimeout 3
• Limit Apache children to keep RAM available for Rails: StartServers 5, MinSpareServers 2, MaxSpareServers 5
• Limit Passenger pool size to 2 child processes (down from the default 6), to queue extra requests instead of using slow swap memory: PassengerMaxPoolSize 2
• Enable browser & intermediate proxy caching of static files: ExpiresActive On and ExpiresByType image/jpeg "access plus 2 hours" etc. (see below for full example)
• Disable ETags which aren't necessary once Expires is enabled: FileETag None and Header unset ETag
• Disable unused Apache modules: free up memory by commenting out LoadModule proxy, proxy_http, info, logio, usertrack, speling, userdir, negotiation, vhost_alias, dav_fs, autoindex, most authn_* and authz_* modules
• Disable SSLv2 (for security and PCI compliance, not performance): SSLProtocol all -SSLv2 and SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP

After making these changes, without tuning Rails, Spree, or the database at all, a new webpagetest.org run reports:

• First page load time: 1.2 seconds
• Repeat page load time: 0.4 seconds

That's an easy improvement, a reduction of 0.9 seconds for the initial load and 1.1 seconds for a repeat load! Complete details are in this follow-on report.

The biggest wins came from enabling HTTP keepalive, which allows serving multiple files from a single HTTP connection, and enabling static file caching which eliminates the majority of requests once the images, JavaScript, and CSS are cached in the browser.

Note that many of the resource-limiting changes I made above to Apache and Passenger would be too restrictive if more RAM or CPU were available, as is typical on a dedicated server with 2 GB RAM or more. But when running on a memory-constrained VPS, it's important to put such limits in place or you'll practically undo any other tuning efforts you make.

I wrote about these topics a year ago in a blog post about Interchange ecommerce performance optimization. I've since expanded the list of MIME types I typically enable static asset caching for in Apache. Here's a sample configuration snippet to put in the <VirtualHost> container in httpd.conf:

    ExpiresActive On
    ExpiresByType image/gif   "access plus 2 hours"
    ExpiresByType image/jpeg  "access plus 2 hours"
    ExpiresByType image/png   "access plus 2 hours"
    ExpiresByType image/tiff  "access plus 2 hours"
    ExpiresByType text/css    "access plus 2 hours"
    ExpiresByType image/bmp   "access plus 2 hours"
    ExpiresByType video/x-flv "access plus 2 hours"
    ExpiresByType video/mpeg  "access plus 2 hours"
    ExpiresByType video/quicktime "access plus 2 hours"
    ExpiresByType video/x-ms-asf  "access plus 2 hours"
    ExpiresByType video/x-ms-wm   "access plus 2 hours"
    ExpiresByType video/x-ms-wmv  "access plus 2 hours"
    ExpiresByType video/x-ms-wmx  "access plus 2 hours"
    ExpiresByType video/x-ms-wvx  "access plus 2 hours"
    ExpiresByType video/x-msvideo "access plus 2 hours"
    ExpiresByType application/postscript        "access plus 2 hours"
    ExpiresByType application/msword            "access plus 2 hours"
    ExpiresByType application/x-javascript      "access plus 2 hours"
    ExpiresByType application/x-shockwave-flash "access plus 2 hours"
    ExpiresByType image/vnd.microsoft.icon      "access plus 2 hours"
    ExpiresByType application/vnd.ms-powerpoint "access plus 2 hours"
    ExpiresByType text/x-component              "access plus 2 hours"

Of course you'll still need to tune your Spree application and database, but why not tune the web server to get the best performance you can there?

Spree on Rails 3: Part Two

Yesterday, I discussed my experiences on getting Rails 3 based Spree up and running. I've explained in several blog articles (here and here) that customizing Spree through extensions will produce the most maintainable code – it is not recommended to work directly with source code and make changes to core classes or views. Working through extension development was one of my primary goals after getting Spree up and running.

To create an extension named "foo", I ran rails g spree:extension foo. Similar to pre-Rails 3.0 Spree, a foo directory is created (albeit inside the sandbox/) directory as a Rails Engine. The generator appends the foo directory details to the sandbox/ Gemfile. Without the Gemfile update, the rails project won't include the new foo extension directory (and encompassed functionality). I reviewed the extension directory structure and files and found that foo/lib/foo.rb was similar to the the *_extension.rb file.

require 'spree_core'

module Foo
  class Engine < Rails::Engine

    config.autoload_paths += %W(#{config.root}/lib)

    def self.activate
      # Activation logic goes here.  
      # A good use for this is performing
      # class_eval on classes that are defined
      # outside of the extension 
      # (so that monkey patches are not 
      # lost on subsequent requests in 
      # development mode.)
    end

    config.to_prepare &method(:activate).to_proc
  end
end

class FooExtension < Spree::Extension
  version "1.0"
  description "Describe your extension here"
  url "http://www.endpoint.com/"

  def activate
    # custom application functionality here
  end 
end

I verified that the activate method was called in my extension with the following change:

require 'spree_core'

module Foo
  class Engine < Rails::Engine

    config.autoload_paths += %W(#{config.root}/lib)

    def self.activate
      Spree::BaseController.class_eval do
        logger.warn "inside base controller class eval"
      end
    end

    config.to_prepare &method(:activate).to_proc
  end
end

From here, The Spree Documentation on Extensions provides insight on further extension development. As I began to update an older extension, I ensured that my_extension/lib/my_extension.rb had all the necessary includes in the activate method and I copied over controller and library files to their new locations.

One issue that I came across was that migrations are not run with rake db:migrate and the public assets are not copied to the main project public directory on server restart. The documentation recommends building the migration within the application root (sandbox/), but this is not ideal to maintain modularity of extensions – each extension must include all of its migration files. To work-around this, it was recommended to copy over the install rake tasks from one of the core gems that copies migrations and public assets:

namespace :foo do
  desc "Copies all migrations and assets (NOTE: This will be obsolete with Rails 3.1)"
  task :install do
    Rake::Task['foo:install:migrations'].invoke
    Rake::Task['foo:install:assets'].invoke
  end

  namespace :install do

    desc "Copies all migrations (NOTE: This will be obsolete with Rails 3.1)"
    task :migrations do
      source = File.join(File.dirname(__FILE__), '..', '..', 'db')
      destination = File.join(Rails.root, 'db')
      puts "INFO: Mirroring assets from #{source} to #{destination}"
      Spree::FileUtilz.mirror_files(source, destination)
    end

    desc "Copies all assets (NOTE: This will be obsolete with Rails 3.1)"
    task :assets do
      source = File.join(File.dirname(__FILE__), '..', '..', 'public')
      destination = File.join(Rails.root, 'public')
      puts "INFO: Mirroring assets from #{source} to #{destination}"
      Spree::FileUtilz.mirror_files(source, destination)
    end

  end
end

After creating the extension based migration files and creating the above rake tasks, one would run the following from the application (sandbox/) directory:

steph@machine:/var/www/spree/sandbox$ rake foo:install
(in /var/www/spree/sandbox)
INFO: Mirroring assets from /var/www/spree/sandbox/foo/lib/tasks/../../db to /var/www/spree/sandbox/db
INFO: Mirroring assets from /var/www/spree/sandbox/foo/lib/tasks/../../public to /var/www/spree/sandbox/public

steph@machine:/var/www/spree/sandbox$ rake db:migrate
(in /var/www/spree/sandbox)
# migrations run

Some quick examples of differences in projeect setup and extension generation between Rails 3.* and Rails 2.*:

#clone project
#bundle install
rake sandbox
rails server
rails g spree:extension foo
rails g migration FooThing

#clone project into "sandbox/"
rake db:bootstrap
script/server
script/generate extension Foo
script/generate extension_model Foo thing name:string start:date

Some of my takeaway comments after going through these exercises:

If there's anything I might want to learn about to work with edge Spree, it's Rails Engines. When you run Spree from source and use extensions, the architecture includes several layers of stacked Rails Engines:

Layers of Rails Engines in Spree with extensions.

After some quick googling, I found two helpful articles on Engines in Rails 3 here and here. The Spree API has been inconsistent until now - hopefully the introduction of Rails Engine will force the API to become more consistent which may improve the extension community.

I didn't notice much deviation of controllers, models, or views from previous versions of Spree, except for massive reorganization. Theme support (including Spree hooks) is still present in the core. Authorization in Spree still uses authlogic, but I heard rumors of moving to devise eventually. The spree_dash (admin dashboard) gem still is fairly lightweight and doesn't contain much functionality. Two fairly large code changes I noticed were:

• The checkout state machine has been merged into order and the checkout model will be eliminated in the future.
• The spree_promo gem has a decent amount of new functionality.

Browsing through the spree-user Google Group might reveal that there are still several kinks that need to be worked out on edge Spree. After these issues are worked out and the documentation on edge Spree is more complete, I will be more confident in making a recommendation to develop on Rails 3 based Spree.

Spree on Rails 3: Part One

A couple of weeks ago, I jumped into development on Spree on Rails 3. Spree is an open source Ruby on Rails ecommerce platform. End Point has been involved in Spree since its inception in 2008, and we continue to develop on Spree with a growing number of clients. Spree began to transition to Rails 3 several months ago. The most recent stable version of Spree (0.11.2) runs on Rails 2.*, but the edge code runs on Rails 3. My personal involvement of Rails 3 based Spree began recently; I waited to look at edge Spree until Rails 3 had a bit of momentum and until Rails 3 based Spree had more documentation and stability. My motivation for looking at it now was to determine whether End Point can recommend Rails 3 based Spree to clients and to share insight to my coworkers and other members of the Spree community.

First, I looked at the messy list of gems that have built up on my local machine throughout development of various Rails and Spree projects. I found this simple little script to remove all my old gems:

#!/bin/bash

GEMS=`gem list --no-versions`
for x in $GEMS; do sudo gem uninstall $x --ignore-dependencies -a; done

Then, I ran gem install rails to install Rails 3 and dependencies. The following gems were installed:

abstract (1.0.0)
actionmailer (3.0.1)
actionpack (3.0.1)
activemodel (3.0.1)
activerecord (3.0.1)
activeresource (3.0.1)
activesupport (3.0.1)
arel (1.0.1)
builder (2.1.2)
bundler (1.0.2)
erubis (2.6.6)
i18n (0.4.1)
mail (2.2.7)
mime-types (1.16)
polyglot (0.3.1)
rack (1.2.1)
rack-mount (0.6.13)
rack-test (0.5.6)
rails (3.0.1)
railties (3.0.1)
rake (0.8.7)
thor (0.14.3)
treetop (1.4.8)
tzinfo (0.3.23)

Next, I cloned the Spree edge with the following command from here:

git clone http://github.com/railsdog/spree.git

In most cases, developers will run Spree from the gem and not the source code (see the documentation for more details). In my case, I wanted to review the source code and identify changes. You might notice that the new spree core directory doesn't look much like the old one, which can be explained by the following: the Spree core code has been broken down into 6 separate core gems (api, auth, core, dash, promo, sample) that run as Rails Engines.

After checking out the source code, the first new task to run with edge Spree was bundle install. The bundler gem is intalled by default in Rails 3. It works out of the box in Rails 3, and can work in Rails 2.3 with additional file and configuration changes. Bundler is a dependency management tool. Gemfile and Gemfile.lock in the Spree core specify which gems are required for the application. Several gems were installed with Spree's bundler configuration, including:

Installing webrat (0.7.2.beta.1) 
Installing rspec-rails (2.0.0.beta.19) 
Installing ruby-debug-base (0.10.3) with native extensions 
Installing ruby-debug (0.10.3) 
Installing state_machine (0.9.4) 
Installing stringex (1.1.0) 
Installing will_paginate (3.0.pre2) 
Using spree_core (0.30.0.beta2) from source at /var/www/spree 
Using spree_api (0.30.0.beta2) from source at /var/www/spree 
Using spree_auth (0.30.0.beta2) from source at /var/www/spree 
Using spree_dash (0.30.0.beta2) from source at /var/www/spree 
Using spree_promo (0.30.0.beta2) from source at /var/www/spree
Using spree_sample (0.30.0.beta2) from source at /var/www/spree

The only snag I hit during bundle install was that the nokogiri gem required two dependencies be installed on my machine (libxslt-dev and libxml2-dev).

To create a project and run all the necessary setup, I ran rake sandbox, which completed the tasks listed below. The tasks created a new project, completed the basic gem setup, installed sample data and images, and ran the sample data bootstrap migration. In some cases, Spree sample data will not be used – the latter two steps can be skipped. The sandbox/ application directory contained a directory of folders that one might expect when developing in Rails (app, db, lib, etc.) and sandbox/ itself runs as a Rails Engine.

steph@machine:/var/www/spree$ rake sandbox
(in /var/www/spree)
         run  rails new sandbox -GJT from "."
      append  sandbox/Gemfile
         run  rails g spree:site -f from "./sandbox"
         run  rake spree:install from "./sandbox"
         run  rake spree_sample:install from "./sandbox"
         run  rake db:bootstrap AUTO_ACCEPT=true from "./sandbox"

After setup, I ran rails server, the new command for starting a server in Rails 3.*, and verified my site was up and running.

Hooray - it's up!

There wasn't much to getting a Rails 3 application up and running locally. I removed all my old gems, installed Rails 3, grabbed the repository, allowed bundler to install dependencies and worked through one snag. Then, I ran my Spree specific rake task to setup the project and started the server. Tomorrow, I share my experiences on extension development in Rails 3 based Spree.

Implementing Per Item Discounts in Spree

Discounts in Spree

For a good overview of discounts in Spree, the documentation is a good place to start. The section on Adjustments is particularly apropos.

In general, the way to implement a discount in Spree is to subclass the Credit class and attach or allow for attaching of one or more Calculators to your new discount class. The Adjustment class file has some good information on how this is supposed to work, and the CouponCredit class can be used as a template of how to do such an implementation.

What we Needed

For my purposes, I needed to apply discounts on a per Item basis and not to the entire order.

The issue with using adjustments as-is is that they are applied to the entire order and not to particular line items, so creating per line item discounts using this mechanism is not obviously straight forward. The good news it that there is nothing actually keeping us from using adjustments in this manner. We just need to modify a few assumptions.

Implementation Details

This is going to be a high-level description of what I did with (hopefully) enough hints about what are probably the important parts to point someone who wants to do something similar in the same direction.

Analogous to the Coupon class in Spree, I create a Discount class. It holds the meta-data information about the discount. Specifically, the product that the discount applies to and the business logic for determining under what circumstances to apply the discount and how much to apply.

There is also a DiscountCredit class which subclasses the Credit class. In this class I re-define two methods:

• applicable? returns true when the discount applies to the line_item
• calculate_adjustment calculates the amount of the discount based on the business rules.

I also add a couple of convenience methods:

• line_item returns self.adjustment_source
• discount returns self.line_item.variant.product.discount

The trick (as an astute reader might infer from the convenience methods) is to set the line_item which the discount is getting applied to to the adjustment_source attribute in the discount object.

The adjustment generally expects that you will be setting this to something like an instance of the Discount class, but as long as we ensure that LineItems implement any interface constraints required by Adjustments, we should be okay.

To that end, I monkey patch the LineItem class in my extension to add a method called add_discount. This method creates a new instance of a DiscountCredit object and passes in iteslf as the adjustment_source. I then add this credit object to the adjustments on the order.

I also add a method to iterate through all of the discounts to look for one that might already be applied to this line_item instance. I use this method in the add_discount method to ensure that I don't add more than one credit per line item.

To bring this together, I monkey patch the Order class to add a method that iterates through all of the line items in the order and calls add_discount on each one. I add a after_save callback which calls this method to ensure that discounts are applied to all line items each time the order is updated.

That takes care of the mechanics of applying the discounts. From this point several things will be taken care of by Spree. Any discounts that are not applicable will get removed. The cart totals will get added up properly and discounts will be applied as adjustments.

Other things you might want to do

You may not want Spree to only display applied discounts at checkout as a (potentially) long list of credits tacked on to the end of the order.

For example, I found it useful to create some helpers to peek into the order adjustments and pull out the discount for a particular line item when displaying the cart. I also wanted to consolidate all of the discounts as a total amount under discount, rather than display them independently, so, I modified the views that handled displaying the credits.

In my implementation, I found it more straight-forward to forgo the use of calculators when implementing the business logic. But, they would work just fine as part of the Discount class and the DiscountCredit#calculate_adjustment method can call the calculator#calculate method to determine the amount to discount.

Problems

This approach works because Spree automatically consolidates products/variants into the same line_item in the cart. In my approach, I assigned discounts at the product level, but applied them at the variant level. This worked for me because I didn't have any variants in my data set.

A general solution would probably assign discounts at the product level (it's too annoying to track them on a per-variant basis) and further track enough information to ensure that a discount was properly applied to any valid line_items that contained variants of that product.

Conclusion

All in all, I found that most of the heavy lifting was already done by the Adjustments code. All it really took was looking at the assumptions behind how the credits were working from a slightly different angle to see how I could modify things to allow per line item discounts to be implemented.

Spree Sample Data: Orders and Checkout

A couple of months ago, I wrote about setting up Spree sample data in your Spree project with fixtures to encourage consistent feature development and efficient testing. I discussed how to create sample product data and provided examples of creating products, option types, variants, taxonomies, and adding product images. In this article, I'll review the sample order structure more and give an example of data required for a sample order.

The first step for understanding how to set up Spree order sample data might require you to revisit a simplified data model to examine the elements that relate to a single order. See below for the interaction between the tables orders, checkouts, addresses, users, line items, variants, and products. Note that the data model shown here applies to Spree version 0.11 and there are significant changes with Spree 0.30.

Basic diagram for Spree order data model.

The data model shown above represents the data required to build a single sample order. An order must have a corresponding checkout and user. The checkout must have a billing and shipping address. To be valid, an order must also have line items that have variants and products. Here's an example of a set of fixtures to create this bare minumum sample data:

#orders.yml
order_1:
  id: 1
  user_id: 1
  number: "R00000001"
  state: new
  item_total: 20.00
  created_at: <%= Time.now %>
  completed_at: <%= Time.now %>
  total: 20.00
  adjustment_total: 0.00

#checkouts.yml
checkout_1:
  bill_address: address_1
  ship_address: address_1
  email: 'spree@example.com'
  order_id: 1
  ip_address: 127.0.0.1
  state: complete
  shipping_method: canada_post

#addresses.yml
address_1:
  firstname: Steph
  lastname: Powell
  address1: 12360 West Carolina Drive
  city: Lakewood
  state_id: 889445952
  zipcode: 80228
  country_id: 214
  phone: 000-000-0000

#line_items.yml
li_1:
  order_id: 1
  variant: test_variant
  quantity: 2
  price: 10.00

#variants.yml
test_variant:
  product: test_product
  price: 10.00
  cost_price: 5.00
  count_on_hand: 10
  is_master: true
  sku: 1-master

#products.yml
test_product:
  name: Test Product 1
  description: Lorem ipsum...
  available_on: <%= Time.zone.now.to_s(:db) %>
  count_on_hand: 10
  permalink: test-product

#users.rb
#copy Spree core to create a user with id=1

After adding fixtures for the minimal order data required, you might be interested in adding peripheral data to test custom work or test new feature development. This peripheral data might include:

• shipping methods: A checkout belongs to a shipping method, and has many shipping rates and shipments.
• shipments: An order has many shipments. Shipments are also tied to the shipping method.
• inventory units: An order has many inventory units, corresponding to each item in the order.
• payments: Orders and checkouts have many payments that must cover the cost of an order. Multiple payments can be assigned to each order.
• adjustments: Shipping charges and tax charges are tracked by adjustments, which belong to orders.
• return authorizations: Return authorizations belong to orders and track returns on orders, tied to inventory_units in the order that are returned.

In my experience, I've worked with a few Spree projects where we created fixtures for setting peripheral sample data to test custom shipping and inventory management. Again, note that the data models described in this article are in place in Spree <= 0.11.0. Spree 0.30 will introduce data model changes to be discussed at a later date.

Seeking a Ruby, Rails, Spree developer

Today I realized that we never posted our job announcement on our own blog even though we'd posted it to several job boards. So here it is:

We are looking for a developer who can consult with our clients and develop Ruby web applications. Most of our needs center around Rails, Spree, and SQL, but Sinatra, DataMapper, and NoSQL are lately coming into play too.

End Point is a 15-year-old web consulting company based in New York City, with 20 full-time staff developers working remotely from around the United States. We prefer open source technology and do collaborative development with Git, GNU Screen, IRC, and voice.

Experience with mobile and location-based technologies is a plus.

Please email jobs@endpoint.com to apply.

Learning Spree: 10 Intro Tips

In climbing the learning curve with Spree development here are some observations I've made along the way:

1. Hooks make view changes easier — I was surprised at how fast I could implement certain kinds of changes because Spree's hook system allowed me to inject code without requiring overriding a template or making a more complicated change. Check out Steph's blog entries on hooks here and here, and the Spree documentation on hooks and themes.
2. Core extensions aren't always updated — One of the biggest surprises I found while working with Spree is that some Spree core extensions aren't maintained with each release. My application used the Beanstream payment gateway. Beanstream authorizations (without capture) and voids didn't work out of the box with Spree 0.11.0.
3. Calculators can be hard to understand — I wrote a custom shipping calculator and used calculators with coupons for the project and found that the data model for calculators was a bit difficult to understand initially. It took a bit of time for me to be comfortable using calculators in Spree. Check out the Spree documentation on calculators for more details.
4. Plugins make the data model simpler after learning what they do — I interacted with the plugins resource_controller, state_machine, and will_paginate in Spree. All three simplified the models and controllers interface in Spree and made it easier to identify the core behavior of Spree models and controllers.
5. Cannot revert migrations — Spree disables the ability to revert migrations due to complications with extensions which makes it difficult to undo simple database changes. This is more of a slight annoyance, but it complicated some aspects of development.
6. Coupons are robust, but confusing — Like calculators, the data model for coupons is a bit confusing to learn but it seems as though it's complicated to allow for robust implementations of many kinds of coupons. Spree's documentation on coupons and discounts provides more information on this topic.
7. Solr extension works well — I replaced Spree's core search algorithm in the application to allow for customization of the indexed fields and to improve search performance. I found that the Solr extension for Spree worked out of the box very well. It was also easy to customize the extension to perform indexation on additional fields. The only problem is that the Solr server consumes a large amount of system resources.
8. Products & Variants — Another thing that was a bit strange about Spree is that every product has at least one variant referred to as the master variant that is used for baseline pricing information. Spree's data model was foreign to me as most ecommerce systems I've worked with have had a much different product and variant data model.
9. Routing — One big hurdle I experienced while working with Spree was how Rails routing worked. This probably stemmed from my inexperience with the resource_controller plugin, or from the fact that one of the first times I worked with Rails routing was to create routes for a nested resource. Now that I have learned how routing works and how to use it effectively, I believe it was well worth the initial struggle.
10. Documentation & Community — I found that the documentation for Spree was somewhat helpful at times, but the spree-user Google group was more helpful. For instance, I got a response on Beanstream payment gateway troubleshooting from the Spree extension author fairly quickly after asking on the mailing list.

I believe that Spree is an interesting project with a somewhat unusual approach to providing a shopping cart solution. Spree's approach of trying to implement 90% of a shopping cart system is very different from some other shopping cart systems which overload the code base to support many features. The 90% approach made some things easier and some things harder to do. Things like hooks and extensions makes it far easier to customize than I expected it would be, and it also seems like it helps avoid the build up of spaghetti code which comes from implementing a lot of features. However, allowing for a "90%" solution seems to make some things like calculators a bit harder to understand when getting started with Spree, since the implementation is general and robust to allow for customization.

Ruby on Rails Typo blog upgrade

I needed to migrate a Typo blog (built on Ruby on Rails) from one RHEL 5 x86_64 server to another. To date I've done Ruby on Rails deployments using Apache with FastCGI, mongrel, and Passenger, and I've been looking for an opportunity to try out an nginx + Unicorn deployment to see how it compares. This was that opportunity, and here are the changes that I made to the stack during the migration:

• Apache 2.2.3 to nginx 0.7.64
• Passenger 2.2.1 to Unicorn 1.1.2
• PostgreSQL 8.3.8 to 8.4.4
• Ruby on Rails 2.2.2 to 2.3.8
• Typo 5.3 to 5.5

I used the following packages from End Point's Yum repository for RHEL 5 x86_64:

• nginx-0.7.64-2.ep
• ruby-enterprise-1.8.7-3.ep
• ruby-enterprise-rubygems-1.3.6-3.ep

The rest were standard Red Hat Enterprise Linux packages, including the new RHEL 5.5 postgresql84 packages. The exceptions were the Ruby gems, which were installed locally with the `gem` command as root.

I had to install an older version of one gem dependency manually, sqlite3-ruby, because the current version requires a newer version of sqlite than comes with RHEL 5. The installation commands were roughly:

yum install sqlite-devel.x86_64
gem install sqlite3-ruby -v 1.2.5

gem install unicorn
gem install typo

yum install postgresql84-devel.x86_64
gem install postgres

Then I followed (mostly) the instructions in Upgrading to Typo 5.4, which are still pretty accurate even though outdated by one release. One difference was the need to specify PostgreSQL to override the default of MySQL (even though the default is documented as being sqlite):

typo install /path/to/typo database=postgresql

Then I ran pg_dump on the old Postgres database and imported the data into the new database, and put in place the database.yml configuration file.

The Typo upgrade went pretty smoothly this time. I had to delete the sidebars configuration from the database to stop getting a 500 error for that, and redo the sidebars manually -- which I've had to do with every past Typo upgrade as well. But otherwise it was easy.

I first tested the migrated blog by running unicorn_rails manually in development mode. Then to have Unicorn start at boot time, I wrote this little shell script and put it in ~/bin/start-unicorn.sh:

#!/bin/bash
cd /path/to/app || exit 1
unicorn_rails -E production -D -c config/unicorn.conf.rb

Then added a cron job to run it:

@reboot bin/start-unicorn.sh

That unicorn.conf.rb file contains only:

listen 8080
worker_processes 4

The listen port 8080 is the default, but I may need to change it. Unicorn defaults to only 1 worker process, so I increased it to 4.

I added the following nginx configuration inside the http { ... } block (actually in a separate include file):

upstream app_server {
    server 127.0.0.1:8080 fail_timeout=0;
}

server {
    listen       the.ip.add.ress:80;
    server_name  the.host.name;

    location / { 
        root   /path/to/rails/typo/public/cache;

        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        #proxy_set_header X-Forwarded-Proto https;
        proxy_set_header Host $http_host;
        proxy_redirect off;

        rewrite ^/blog/xml/atom/feed\.xml$ /articles.atom permanent;
        rewrite ^/blog/xml/rss20/feed\.xml$ /articles.rss permanent;

        if (-f $request_filename) {
            break;
        }   

        set $possible_request_filename $request_filename/index.html;
        if (-f $possible_request_filename) {
            rewrite (.*) $1/index.html;
            break;
        }   

        set $possible_request_filename $request_filename.html;
        if (-f $possible_request_filename) {
            rewrite (.*) $1.html;
            break;
        }   

        if (!-f $request_filename) {
            proxy_pass http://app_server;
            break;
        }   
    }   

    # Rails error pages
    error_page 500 502 503 504 /500.html;
    location = /500.html {
        root   /path/to/rails/typo/public;
    }   
}

The configuration was a little complicated to get nginx serving static content directly, including cache files that Typo writes out. I had to add special handling for / which gets cached as /index.html, but can't be called that when passing URIs to Typo, as it doesn't know about any /index.html. And all HTML cache files end in .html, though the URIs don't, so those need special handling too.

But when all is said and done, the blog is now running on the latest version of Typo, on the latest Unicorn, Rails, Ruby Enterprise Edition, PostgreSQL, and nginx, with all static content and fully-cached pages served directly by nginx, and for the most part only dynamic requests being served by Unicorn. I need to tweak the nginx rewrite rules a bit more to get 100% of static content served directly by nginx.

As far as blogging platforms go, I can recommend Typo mainly for Rails enthusiasts who want to write their own plugins, tweak the source, etc. WordPress or Movable Type are so much more widely used that non-programmers are going to have a lot easier time deploying and supporting them. They've had a lot more security vulnerabilities requiring updates, though that may also be a function of popularity and payoff for those exploiting them.

Rails deployment seems to take a lot of memory no matter how you do it. I don't think nginx + Unicorn uses much less RAM than Apache + Passenger, mostly the different between nginx and Apache themselves. But using Unicorn does allow for running the application processes on another server or several servers without needing nginx or Apache on those other servers. It does provide for clean separation between the web server and the application(s), including possibly different SELinux contexts rather than always httpd_sys_script_t as we see with Passenger. Passenger at least switches the child process UID to run with different permissions from the main web server, which is good. Both Passenger and Unicorn are much nicer than FastCGI, which I've always found to be a little buggy, and mongrel, which required specifying a range of ports and load-balancing across all of them in the proxy -- managing multiple port ranges is a pain with multiple apps on the same machine, especially when some need more than others.

I think if you have plenty of RAM, going with Apache + Passenger may still be the easiest Rails web deployment method overall, when mixed with other static content, server-side includes, PHP, and CGIs. But for high-traffic and custom setups, nginx + Unicorn is a nice option.

Spree: Gift Certificates and Coupons

In a recent Spree project, I've been working with Bill to add gift certificate functionality. According to the Spree documentation, gift certificate functionality is trivial to implement using the existing coupon architecture. Here are some of the changes we went through as we tried to use the coupon architecture for gift certificate implementation - we found that it wasn't so simple after all.

Here is a very simplified visualization of the coupon and adjustment data model in Spree. Coupons use polymorphic calculators to compute the applicable discount.

First, Bill and I brainstormed to come up with an initial set of changes required for implementing gift certificates as coupons after we reviewed the data model shown above:

1. Add logic to create a coupon during checkout finalization, which was done with the following:

# coupon object class method
def self.generate_coupon_code
  # some method to generate an unused random coupon code beginning in 'giftcert-'
end

# inside order model during checkout finalization
line_items.select { |li| li.variant.product.is_gift_cert? }.each do |line_item|
  line_item.quantity.times do
    coupon = Coupon.create(:code => Coupon.generate_coupon_code,
                           :description => "Gift Certificate",
                           :usage_limit => 1,
                           :combine => false,
                           :calculator => Calculator::FlatRate.new)
    coupon.calculator.update_attribute(:preferred_amount, line_item.variant.price)
  end
end

2. Add logic to decrease a coupon amount during checkout finalization if used:

   # order model during checkout finalization
   coupon_credits.select{ |cc| cc.adjustment_source.code.include?('giftcert-') }.each do |coupon_credit|
     coupon = coupon_credit.adjustment_source
     amount = coupon.calculator.preferred_amount - item_total
     coupon.calculator.update_attribute(:preferred_amount, amount < 0 ? 0 : amount)
   end
   

3. Add relationship between line item and coupon because we'd want to have a way to associate coupons with line items. The intention here was to limit a gift certificate line item to a quantity of 1 since the gift certificate line item might include personal information like an email in the future.

   LineItem.class_eval do
     has_one :line_item_coupon
     has_one :coupon, :through => :line_item_coupon
   end
   

   class LineItemCoupon < ActiveRecord::Base
     belongs_to :line_item
     belongs_to :coupon
   
     validates_presence_of :line_item_id
     validates_presence_of :coupon_id
   end
   

4. Create the sample data for a gift certificate (coupon) - the implementation offers a master variant for a fixed cost of $25.00. In addition to the code below, Bill created sample data to assign a product property is_gift_cert to the product.

   # products.yml
   gift_certificate:
     name:          Gift Certificate
     description:   Gift Certificate
     available_on:  <%= Time.zone.now.to_s(:db) %>
     permalink:     gift-certificate
     count_on_hand: 100000
   

   # variants.yml
   gift_cert_variant:
     product:       gift_certificate
     sku:           giftcert
     price:         25.00
     is_master:     true
     count_on_hand: 10000
     cost_price:    25.00
     is_extension:  false
   

5. Finally, Bill edited the order mailer view to include gift certificate information

After the above changes were implemented, additional changes were required for our particular Spree application.

1. Adjust the shipping API so it doesn't include gift certificates in the shipping request, because gift certificates aren't shippable. Below is an excerpt of the XML builder code that generates the XML request made to the shipping API:

   # shipping calculator
   -order.line_items.each do |li|
   +order.line_items.select { |li| !li.variant.product.is_gift_cert? }
      x.item {
        x.quantity(li.quantity)
        x.weight(li.variant.weight != 0.0 ? li.variant.weight : Spree::MyShipping::Config[:default_weight])
        x.length(li.variant.depth ? li.variant.depth : Spree::MyShipping::Config[:default_depth])
        x.width(li.variant.width ? li.variant.width : Spree::MyShipping::Config[:default_width])
        x.height(li.variant.height ? li.variant.height : Spree::MyShipping::Config[:default_height])
        x.description(li.variant.product.name)
      }
   

2. Create a new calculator for free shipping applicable to orders with gift certificate line items only, using the is_gift_cert product property:

   # registering the calculator inside Spree site_extension.rb (required for all calculators to be used in Spree)
   [   
     Calculator::GiftCertificateShipping,
   ].each{ |c_model|
     begin
       c_model.register if c_model.table_exists?
     rescue Exception => e
       $stderr.puts "Error registering calculator #{c_model}"
     end 
   }
   

   # shipping method and calculator creation in sample data
   s = ShippingMethod.new(:zone_id => 16, :name => 'Gift Certificate Shipping')
   s.save
   c = Calculator.new
   c.calculable = s
   c.type = 'Calculator::GiftCertificateShipping'
   c.save 
   

   # calculator for free gift cert shipping
   class Calculator::GiftCertificateShipping < Calculator
     ...
     def available?(order)
       order.line_items.inject(0) { |sum, li| sum += li.quantity if !li.variant.product.is_gift_cert?; sum } == 0
     end
   
     def compute(line_items)
       0
     end
   end
   

After Bill implemented these changes, I contemplated the following code more:

coupon_credits.select{ |coupon_credit| coupon_credit.adjustment_source.code =~ /^giftcert-/}.each do |coupon_credit|
  coupon = coupon_credit.adjustment_source
  amount = coupon.calculator.preferred_amount - item_total
  coupon.calculator.update_attribute(:preferred_amount, amount < 0 ? 0 : amount)
end 

I wondered why the coupon amount being decremented by the item_total and not the order total. What about shipping and sales tax? I verified by looking at the the Spree Coupon class that a coupon's amount will only take into account the item total and not shipping or tax, which would present a problem since gift certificates traditionally apply to tax and shipping costs.

In the Spree core, coupons are never applied to shipping or tax costs.

I investigated the following change to separate coupon and gift certificate credit calculation:

def site_calculate_coupon_credit
  return 0 if order.line_items.empty?
  amount = adjustment_source.calculator.compute(order.line_items).abs
  order_total = adjustment_source.code.include?('giftcert-') ? order.item_total + order.charges.total : order.item_total
  amount = order_total if amount > order_total
  -1 * amount
end

After this change, I found that when arriving on the payment page where the gift certificate has covered the entire order including tax and shipping, the payment logic isn't set up handle orders with a total cost of 0. Additional customization on payment implementation, validation and checkout flow would be required to handle orders where gift certificates cover the entire cost. However, rather than implementing these additional customizations, our client was satisfied with the implementation where gift certs don't cover tax and shipping, so I did not pursue this further.

In the future, I'd recommend creating a new model for gift certificate and gift certificate credit management rather than combining the business logic with coupons, because:

1. The coupon implementation in Spree doesn't have a whole lot to it. It uses several custom Spree calculators, has a backend CRUD interface, and credits are applied to orders. Grabbing the coupon implementation and copying and modifying it for gift certificates shouldn't be daunting.
2. It will likely be more elegant to separate coupon logic from gift certificate logic. Coupons and gift certificates share a few business rules, but not all. Gift certificates traditionally apply to tax and shipping and multiple gift certificates can be used on one order (but this part can be configurable). Coupons may have more complex logic to apply to items and do not traditionally get applied to tax and shipping (however, in some cases a free shipping coupon may be needed that covers the cost of shipping only). Additionally, a big difference in business logic is that gift certificates should probably be treated as a payment, where checkout accepts gift certificates as a form of payment, and the backend provides reporting on the gift certificate driven payments. Rather than dirtying-up the the coupon logic with checks for gift certificates versus coupon behavior, it'll be more elegant to separate the logic into classes that address the individual business needs.

Besides "hindsight is 20/20", the takeaway for me here is that you have to understand business rules and requirements for coupon and gift certificate implementation in ecommerce, which can get tricky quickly. We were lucky because the client was satisfied with the resulting behavior of using the coupon architecture for gift certificates. Hopefully, the takeaway for someone not familiar with Spree is that gift certificate implementation might require things like functionality for creating gift certificates after checkout completion, decrementing the gift certificate after it's used, backend reporting to show gift certificates purchase and use and coding for the impact of gift certificate purchase on shipping.

Note that all of the changes described here apply to the latest stable version of Spree (0.11.0). After taking a look at the Spree edge code, I'll mention that there is a bit of an overhaul on coupons (to be called promotions). However, it looks many of the customizations described here would be needed for gift certificate implementation as the edge promotions still apply to item totals only and do not include any core modifications in accepting a credit as a payment.

Learn more about End Point's Ecommerce Development or Ruby on Rails Ecommerce Services.

Spree: Working with Sample Product Data

It's taken me a bit of time to gain a better understanding of working with Spree sample data or fixtures, but now that I am comfortable with it I thought I'd share some details. The first thing you might wonder is why should you even care about sample data? Well, in our project, we had a few motivations for creating sample data:

1. Multiple developers, consistent sample data provides consistency during development. End Point offers SpreeCamps, a hosting solution that combines the open source Spree technology with devcamps to allow multiple development and staging instances of a Spree application. In a recent project, we had a two developers working on different aspects of the custom application in SpreeCamps; creating meaningful sample data allowed each developer to work from the same data starting point.
2. Unit testing. Another important element of our project includes adding unit tests to test our custom functionality. Consistent test sample data gave us the ability to test individual methods and functionality with confidence.
3. Application testing. In addition to unit testing, adding sample data gives the ability to efficiently test the application repeatedly with fresh sample data.

Throughout development, our standard practice is to repeatedly run rake db:bootstrap SKIP_CORE=1 AUTO_ACCEPT=1. Running bootstrap with these arguments will not create Spree's core sample data set, but it will set the core's default data that includes some base zones, zone members, countries, states, and roles, data that is essential for the application to work.

Product Data Model

Product data relationship in Spree

The first data I create is the product data. As you can see from the image above, products data may have relationships with other tables including option_types, option_values, and taxons, or the tables that create has and belongs to many relationships between products and these elements.

The most simple form of sample data might include one test product and its master variant, shown below. If you do not define a master variant for a product, the product page will crash, as a master variant is required to display the product price.

products.yml
test_product:
  id: 1 
  name: Test Product
  description: Lorem ipsum...
  available_on: <%= Time.zone.now.to_s(:db) %>
  count_on_hand: 10
  permalink: test-product

variants.yml
test_variant:
  product: test_product
  price: 10.00
  cost_price: 5.00
  count_on_hand: 10
  is_master: true
  sku: 1-master

Option Types and Option Values

To expand on this, you might be interested in adding option types and values to allow for sizes to be assigned to this variant, shown below. The option type and option value data structure provides a flexible architecture for creating product variants, or multiple varieties of a single product such as different sizes, colors, or combinations of these option values.

option_types.yml
size:
  name: size
  presentation: Size

option_values.yml
small:
  name: Small
  presentation: Small
  option_type: size
large:
  name: Large
  presentation: Large
  option_type: size

product_option_types.yml
test_product_size:
  product: test_product
  option_type: size

variants.yml
# ...  master variant
small_variant:
  product: test_product
  option_values: small
  price: 10.00
  cost_price: 5.00
  count_on_hand: 10
  sku: 1-small
large_variant:
  product: test_product
  option_values: large
  price: 20.00
  cost_price: 10.00
  count_on_hand: 10
  sku: 1-large

Taxonomies and Taxons

Another opportunity for expansion on sample product data is the taxonomy structure, which is very flexible. A root taxonomy can be thought of as a tree trunk with branches; products can be assigned to any number of branches. If we assume you have multiple test products, you might set up the following test data:

taxonomies.yml
category:
  name: Category
brand:
  name: Brand

taxons.yml
category_root:
  id: 1
  name: Category
  taxonomy: category
  permalink: c/
jackets:
  id: 2
  name: Jackets
  taxonomy: category_root
  permalink: c/jackets/
  parent_id: 1
  products: test_product, test_product2, test_product3
pants:
  id: 3
  name: Pants
  taxonomy: category_root
  permalink: c/pants/
  parent_id: 1
  products: test_product4, test_product5, test_product6
brand_root:
  id: 4
  name: Brand
  taxonomy: brand
  permalink: b/
brand_one:
  id: 5
  name: Brand One
  taxonomy: brand
  permalink: b/brand-one/
  parent_id: 4
  products: test_product, test_product3, test_product5
brand_two:
  id: 6
  name: Brand Two
  taxonomy: brand
  permalink: b/brand-two/
  parent_id: 4
  products: test_product2, test_product4, test_product6

I also needed to include the taxons.rb (used in the Spree core sample data) to assign products to taxons correctly.

taxons.rb
Taxon.rebuild!
Taxon.all.each{|t| t.send(:set_permalink); t.save}

Example taxonomies created with Spree sample data

Product Images

My last step in creating Spree sample data is to add product images. I've typically added the image via the Spree backend first, and then copied the images my site extension directory.

# upload Spree images

# create sample image directory
mkdir RAILS_ROOT/vendor/extensions/site/lib/tasks/sample/

# copy the uploaded images to the sample image directory
cp -r RAILS_ROOT/public/assets/products/ RAILS_ROOT/vendor/extensions/site/lib/tasks/sample/

After uploading and copying the images over, I include the image information in assets.yml. The ID for each asset must be equal to the directory containing the multiple image sizes. For example, the directory RAILS_ROOT/vendor/extensions/site/lib/tasks/sample/1/ contains directories original, large, product, small, and mini with images sized respectively.

assets.yml
i1:
  id: 1
  viewable: test_product
  viewable_type: Product
  attachment_content_type: image/jpg
  attachment_file_name: blue_sky.jpg
  attachment_width: 1024
  attachment_height: 683
  type: Image
  position: 1

And finally, I use a modified version of the Spree's core products.rb file to copy over product images during bootstrap:

products.rb
require 'find'
# make sure the product images directory exists
FileUtils.mkdir_p "#{RAILS_ROOT}/public/assets/products/"

# make product images available to the app
target = "#{RAILS_ROOT}/public/assets/products/"
source = "#{RAILS_ROOT}/vendor/extensions/site/lib/tasks/sample/products/"

Find.find(source) do |f|
  # omit hidden directories (SVN, etc.)
  if File.basename(f) =~ /^[.]/
    Find.prune
    next
  end

  src_path = source + f.sub(source, '')
  target_path = target + f.sub(source, '')

  if File.directory?(f)
    FileUtils.mkdir_p target_path
  else
    FileUtils.cp src_path, target_path
  end
end

With my sample data defined in my Spree site extension, I run rake db:bootstrap SKIP_CORE=1 AUTO_ACCEPT=1 to create the above products, variants, and taxonomy structure. I commit my changes to the git repository, and other developers can work with the same set of products, variants, taxonomies including product images. During development, I also add unit tests to test model methods that interact with our sample data. An alternative to setting up Spree sample data described in this article is to dump entire databases and reimport them and manage the sample images manually, but I find that the approach described here forces you to understand the Spree data model better.

Sample product image created with the sample data above.

In addition to setting up sample product data, I've worked through creating sample orders, shipping configuration, and tax configuration. I hope to discuss these adventures in the future.

Learn more about End Point's Ecommerce Development or Ruby on Rails Ecommerce Services.

Spree vs Magento: Feature List Revisited

A little over a month ago, I wrote an article on Spree vs Magento Features. Recently, a client asked me to describe the features mentioned in that article. I thought this was another great opportunity to expand on my response to the client. So, here I am, revisiting ecommerce features in Spree and Magento. The original article can be referenced to compare availability of these features in Spree and Magento.

Features on a Single Product or Group of Product

• Product reviews and/or ratings: functionality to allow customers to review and rate products. See a Backcountry.com product page for an example.
• Product QnA: functionality allow customers to ask and answer questions on products. See a Backcountry.com product page for an example.
• Product SEO (URL, title, meta data control): functionality to allow site administrators to manage product URLs, product page titles, and product meta data.
• Advanced/flexible taxonomy: functionality to build a custom taxonomy / navigation structure for product browsing. For example, build multiple categories and subcategories with complex hierarchy. The taxonomy at Spree's demo includes two categories of brand and category and subcategories in each.
• SEO for taxonomy pages: functionality to allow site administrators to manage taxonomy URLs, taxonomy page titles, and taxonomy meta data.
• Configurable product search: functionality to allow the developers and site administrators to adjust parameters used in search, such as products to show per page, and to show products with no on hand stock.
• Bundled products for discount: functionality to allow site administrators to create bundles or group products together and then apply a discount to the entire bundle. For example, product X, Y and Z purchased together will yield a $10.00 discount.
• Recently viewed products: functionality to show customers products they recently visited. This can be displayed on other product pages or the navigation pages to aid in navigation back to those products if the user would like to revisit the products. See the image below for an example of this functionality in action at Paper Source.
  
• Soft product support/downloads: functionality to sell soft products such as mp3 or pdf files.
• Product comparison: functionality to allow customers to compare multiple products, such as a comparison of price or technical features. See Backcountry.com for an example.
• Upsell: functionality to encourage the customer to purchase a similar product from a higher price point, or to purchase an add on, may or may not include the functionality to allow site administrators to manage the upsell products.
• Cross sell: functionality to encourage the customer to purchase related items, may or may not include the functionality to allow site administrators to manage the cross sell products. See the image below for an example of this functionality at Backcountry.com after adding an item to the cart.
  
• Related items: functionality to display related items on product pages, may or may not include the functionality to allow site administrators to manage the related products.
• RSS feed of products: functionality to produce a RSS feed with product detail releases. steepandcheap.com offers a RSS feed, however, they are unique in that they offer a one deal at a time business model, so the RSS feed contains a stream of products for sale. This may or may not include the functionality to allow site administrators to manage the RSS feed contents.
• Multiple images per product: functionality to allow site administrators to upload multiple products per image. See a Spree demo product for an example.
• Product option selection (variants): functionality to allow site administrators to create and manage variants for products to offer multiple variants per product, such as variants by size and color. See a Spree demo product for an example.
• Wish list: functionality to allow customers to create product wish lists. See Amazon.com for a description of their wish list functionality.
• Send product email to friend: functionality to allow customers to send emails to their friend to visit a specific product.
• Product tagging / search by tagging: functionality to allow site administrators to assign tags to products for navigation or searching. See CityPass's blog for an example of tag use in a content management system; in ecommerce context, the tags would navigate to a set of products instead of a set of blog articles.
• Breadcrumbs: functionality that renders the product navigation hierarchy on navigation and product pages to allow customers to navigate to previous pages visited. See a Spree demo navigation page for an example.

CMS Features

• Blogging functionality: functionality to allow site administrators to create, manage and display blog articles.
• Static page management: functionality to allow site administrators to create, manage, and display static pages such as “About Us”, “Information”.
• Media management: functionality to allow site administrators to create, manage, and display media such as images, video, audio. See the image below for an example of WordPress's dashboard for content management.
  
• Contact us form: functionality to allow customers to submit a request for contact. See End Point's contact page for an example.
• Polls: functionality to allow site administrators to create and manage basic polls, functionality to allow customers to submit answers to basic polls.

Checkout Support

• One page checkout: functionality to allow customers to complete checkout on one page, rather than move forward through checkout through multiple address, payment pages. See Paper Source or Backcountry.com's checkout processes for examples.
• Guest checkout: functionality to checkout without creating a user account. Checkout at the Spree demo without being logged in for an example.
• SSL support: functionality to configure use of SSL during checkout. In Spree's case, the site administrator may turn SSL off during development and on during production.
• Discounts: functionality to allow customers to apply discount coupons to orders for a percentage or dollar amount reduction.
• Gift certificates: functionality to allow customers to purchase and use gift certificates as credit for purchases.
• Saved shopping cart: the functionality to save the products in a customers shopping cart so their shopping cart will be pre-populated on their next visit.
• Saved addresses: functionality to allow customers to create and manage addresses to be selected during checkout for billing or shipping rather than requiring the customer to re-enter their address. See the image below for an example of using saved addresses during Backcountry.com's checkout.
  

Shipping Support

• Real time rate lookup (UPS, USPS, FedEx): the functionality to request rates from UPS, USPS, or FedEx during checkout for more accurate rate pricing rather than using a flat shipping rate.
• Order tracking: functionality to allow the site administrators to enter order tracking information and allow the user to review that tracking information, may or may not include sharing this information in a “Your order has been shipped” email.
• Multiple shipments per order: functionality to allow site administrators to split orders into multiple packages if specific products can not be shipped at the same time or can not be shipped together. See the image below for an example of Spree's backend shipping interface.
  
• Complex rate lookup: functionality to calculate ship rates based on weight or price.
• Free shipping: functionality to offer free shipping.

Payment Support

• Multiple payment gateways: integration of multiple payment gateways such as Authorize.NET, Beanstream, Paypal, SagePay, etc.
• Authorize.Net: integration of the Authorize.Net payment gateway; may or may include the use of profiles (Authorize.Net CIM).
• Authorize and capture versus authorize only: functionality to allow site administrators to configure whether or not credit cards should be authorized only during checkout completion or authorized and captured. If the credit card is authorized only, site administrators may finalize an order by capturing on the backend interface.
• Google Checkout: integration of Google Checkout.
• Paypal Express: integration of Paypal Express.

Admin Features

• Sales reporting: the functionality to display sales statistics, such as profits on sales or year-over-year sales.
• Sales management tools: functionality to allow site administrators to create and manage product sales. For example, the site administrator might create a 50% off sale to cover 25% of the products to begin in a week and end in two weeks.
• Inventory management: functionality to allow the site administrator to manage individual inventory units and their current state (on_hand, shipped, backordered) and order assignment.
• Purchase order management: functionality to allow the site administrator to create and manage purchase orders. See the image below for an example of a potential backend interface for purchase orders in Spree.
  
• Multi-tier pricing for quantity discounts: functionality to allow customers to buy large quantities of products at a discount, the functionality to allow the site administrator to manage the large quantity product discounts.
• Landing page tool: functionality to create custom landing pages that may include targeted content or products, typically used for advertising or marketing.
• Batch import and export of products: functionality to allow the site administrator to import and export products via admin interface or script rather than entering each product individually.
• Multiple sales reports: See "Sales reporting" above.
• Order fulfillment: functionality to allow site administrators to manage fulfillment (inventory selection, shipping) of orders.
• Tax Rate Management: functionality to manage tax rates per zone, where zones are defined by states and/or countries. Note that in Spree, zones can only be defined by a combination of states and countries and tax rates can be tied to one or more zones.

User Account Features

• User addresses: See "Saved addresses" above.
• Feature rich user preferences: integration of various user account tools, such as address management, profile management, order review, etc.
• Order tracking history: functionality to allow a customer to lookup their order history, may or may not include order tracking information.

Site Wide Features

• Extensibility: functionality to extend the ecommerce core with modular components.
• Appearance Theming: functionality to change the appearance of the site.
• Ability to customize appearance at category or browsing level: functionality to create and manage custom and varied appearances for product browsing pages. For example, the categories "Jackets" and "Pants" may have different appearances, motivated by marketing or advertising.
• Localization: the functionality to translate the ecommerce site to a different language. See the image below for a small example of localization in action in Spree.
  
• Multi-store, single admin support: functionality to manage multiple stores from a single administrative location. An example of this might include http://store1.endpoint.com/, and http://store2.endpoint.com/, where both can be managed at http://admin.endpoint.com/.
• Support for multiple currencies: functionality to translate product prices between currencies.
• Web service API: functionality to retrieve data from the ecommerce application for third party use. See Spree's documentation on the Spree API.
• System wide SEO: general site-wide SEO functionality including features such as sitemap, googlebase integration, URL management, page title management.
• Google Analytics: functionality to allow site administrators to create and manage Google Analytics Ids, functionality to track traffic and conversion on the frontend.
• Active community: an active developer community with frequent core and extension contributions.

Most of the features described above are well known to ecommerce developers, but this list might also serve as a good checklist to review with a potential client during the estimate process to make sure expectations of an ecommerce platform are managed, especially with a young platform such as Spree where some features are not yet included in the core.

Learn more about End Point's Ecommerce Development or Ruby on Rails Ecommerce Services.

Upgrading Spree with the help of Git

Lately, I've upgraded a few Spree projects with the recent Spree releases. Spree is a Ruby on Rails ecommerce platform that End Point previously sponsored and continues to support. In all cases, my Spree project was running from the Spree gem (version 0.10.2) and I was upgrading to Spree 0.11.0. I wanted to go through a brief explanation on how I went about upgrading my projects.

First, I made sure my application was running and committed all recent changes to have a clean branch. I follow the development principles outlined here that describe methodology for developing custom functionality on top of the Spree framework core. All of my custom functionality lives in the RAILS_ROOT/vendor/extensions/site/ directory, so that directory probably won't be touched during the upgrade.

steph@The-Laptop:/var/www/ep/myproject$ git status
# On branch master
nothing to commit (working directory clean)

Then, I tried the rake spree:upgrade task with the following results. I haven't upgraded Spree recently, and I vaguely remembered there being an upgrade task.

steph@The-Laptop:/var/www/ep/myproject$ rake spree:upgrade
(in /var/www/ep/myproject)
[find_by_param error] database not available?
This task has been deprecated.  Run 'spree --update' command using the newest gem instead.

OK. The upgrade task has been removed. So, I try spree --update:

Updating to Spree 0.11.0 ...
Finished.

That was easy! I run 'git status' and saw that there were several modified config/ files, and a few new config/ files:

# On branch master
# Changed but not updated:
#   (use "git add ..." to update what will be committed)
#   (use "git checkout -- ..." to discard changes in working directory)
#
#       modified:   config/boot.rb
#       modified:   config/environment.rb
#       modified:   config/environments/production.rb
#       modified:   config/environments/staging.rb
#       modified:   config/environments/test.rb
#       modified:   config/initializers/locales.rb
#       modified:   config/initializers/new_rails_defaults.rb
#       modified:   config/initializers/spree.rb
#
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
#       config/boot.rb~
#       config/environment.rb~
#       config/environments/cucumber.rb
#       config/environments/production.rb~
#       config/environments/staging.rb~
#       config/environments/test.rb~
#       config/initializers/cookie_verification_secret.rb
#       config/initializers/locales.rb~
#       config/initializers/new_rails_defaults.rb~
#       config/initializers/spree.rb~
#       config/initializers/touch.rb
#       config/initializers/workarounds_for_ruby19.rb

Because I had a clean master branch before the upgrade, I can easily examine the code changes for this upgrade from Spree 0.10.2 to Spree 0.11.0:

config/boot.rb

-      load_rails("2.3.5")  # note: spree requires this specific version of rails (change at your own risk)
+      load_rails("2.3.8")  # note: spree requires this specific version of rails (change at your own risk)

config/environment.rb

-  config.gem 'authlogic', :version => '>=2.1.2'
+  config.gem 'authlogic', :version => '2.1.3'
-  config.gem 'will_paginate', :lib => 'will_paginate', :version => '2.3.11'
+  config.gem 'will_paginate', :lib => 'will_paginate', :version => '2.3.14'
-  config.i18n.default_locale = :'en-US'
+  config.i18n.default_locale = :'en'

config/environment/test.rb

-config.gem 'test-unit', :lib => 'test/unit', :version => '~>2.0.5' if RUBY_VERSION.to_f >= 1.9
+config.gem 'test-unit', :lib => 'test/unit', :version => '~>2.0.9' if RUBY_VERSION.to_f >= 1.9

Most of the changes were not surprising, except that the locale changes here are significant because they may require extension locales to be updated. After I reviewed these changes, I installed newer gem dependencies and bootstrapped the data since all my application data was stored in sample data files and restarted the server to test the upgrade. Then, I added the config/ and public/ files in a single git commit. I removed the old temporary configuration files that were left around from the upgrade.

For this particular upgrade, my git log shows changes in the files below. The config/ files were made when I ran the update, and the public/ files were modified when I restarted the server as gem public/ files are copied over during a restart.

commit 96a68e86064aa29f51c5052631f896845c11c266
Author: Steph Powell 
Date:   Mon Jun 28 13:44:50 2010 -0600

    Spree upgrade.

diff --git a/config/boot.rb b/config/boot.rb
diff --git a/config/environment.rb b/config/environment.rb
diff --git a/config/environments/cucumber.rb b/config/environments/cucumber.rb
diff --git a/config/environments/production.rb b/config/environments/production.rb
diff --git a/config/environments/staging.rb b/config/environments/staging.rb
diff --git a/config/environments/test.rb b/config/environments/test.rb
diff --git a/config/initializers/cookie_verification_secret.rb b/config/initializers/cookie_verification_secret.rb
diff --git a/config/initializers/locales.rb b/config/initializers/locales.rb
diff --git a/config/initializers/new_rails_defaults.rb b/config/initializers/new_rails_defaults.rb
diff --git a/config/initializers/spree.rb b/config/initializers/spree.rb
diff --git a/config/initializers/touch.rb b/config/initializers/touch.rb
diff --git a/config/initializers/workarounds_for_ruby19.rb b/config/initializers/workarounds_for_ruby19.rb
diff --git a/public/images/admin/bg/spree_50.png b/public/images/admin/bg/spree_50.png
Binary files a/public/images/admin/bg/spree_50.png and b/public/images/admin/bg/spree_50.png differ
diff --git a/public/images/tile-header.png b/public/images/tile-header.png
Binary files /dev/null and b/public/images/tile-header.png differ
diff --git a/public/images/tile-slider.png b/public/images/tile-slider.png
Binary files /dev/null and b/public/images/tile-slider.png differ
diff --git a/public/javascripts/admin/checkouts/edit.js b/public/javascripts/admin/checkouts/edit.js
diff --git a/public/javascripts/taxonomy.js b/public/javascripts/taxonomy.js
diff --git a/public/stylesheets/admin/admin-tables.css b/public/stylesheets/admin/admin-tables.css
diff --git a/public/stylesheets/admin/admin.css b/public/stylesheets/admin/admin.css
diff --git a/public/stylesheets/screen.css b/public/stylesheets/screen.css

From my experience, the config/ and public/ files are typically modified with a small release. If your project has custom JavaScript, or overrides the Spree core JavaScript, you may have to review the upgrade changes more carefully. Version control goes a long way in highlighting changes and problem areas. Additionally, having custom code abstracted from the Spree core should allow for easier maintenance of your project.

Learn more about End Point's Ecommerce Development or Ruby on Rails Ecommerce Services.

NoSQL at RailsConf 2010: An Ecommerce Example

Even more so than Rails 3, NoSQL was a popular technical topic at RailsConf this year. I haven't had much exposure to NoSQL except for reading a few articles written by Ethan (Quick Thoughts on NoSQL Live Boston Conference, NoSQL Live: The Dynamo Derivatives (Cassandra, Voldemort, Riak), and Cassandra, Thrift, and Fibers in EventMachine), so I attended a few sessions to learn more.

First, it was reinforced several times that if you can read JSON, you should have no problem comprehending NoSQL. So, it shouldn't be too hard to jump into code examples! Next, I found it helpful when one of the speakers presented high-level categorization of NoSQL, whether or not the categories meant much to me at the time:

• Key-Value Stores: Advantages include that this is the simplest possible data model. Disadvantages include that range queries are not straightforward and modeling can get complicated. Examples include Redis, Riak, Voldemort, Tokyo Cabinet, MemcacheDB.
• Document stores: Advantages include that the value associated with a key is a document that exposes a structure that allows some database operations to be performed on it. Examples include CouchDB, MongoDB, Riak, FleetDB.
• Column-based stores: Examples include Cassandra, HBase.
• Graph stores: Advantages include that this allows for deep relationships. Examples include Neo4j, HypergraphDB, InfoGrid.

In one NoSQL talk, Flip Sasser presented an example to demonstrate how an ecommerce application might be migrated to use NoSQL, which was the most efficient (and very familiar) way for me to gain an understanding of NoSQL use in a Rails application. Flip introduced the models and relationships shown here:

In the transition to NoSQL, the transaction model stays as is. As a purchase is created, the Notification.create method is called.

class Purchase < ActiveRecord::Base
  after_create :create_notification

  # model relationships
  # model validations

  def total
    quantity * product.price
  end

  protected
  def create_notification
    notifications.create({
      :action => "purchased #{quantity == 1 ? 'a' : quantity} #{quantity == 1 ? product.name : product.name.pluralize}",
      :description => "Spent a total of #{total}",
      :item => self,
      :user => user
    }
    )
  end
end

Flip moves the product class to Document store because it needs a lot of flexibility to handle the diverse product metadata. The structure of the product class is defined in the product class and nowhere else.

Before

class Product < ActiveRecord::Base
  serialize :info, Hash
end

After

class Product
  include MongoMapper::Document

  key :name, String
  key :image_path, String

  key :info, Hash

  timestamps!
end

The Notification class is moved to a Key-Value store. After a user completes a purchase, the create method is called to store a notification against the user that is to receive the notification.

Before

class Notification < ActiveRecord::Base
  # model relationships
  # model validations
end

After

require 'ostruct'

class Notification < OpenStruct
  class << self
    def create(attributes)
      message = "#{attributes[:user].name} #{attributes[:action]}"
      attributes[:user].follower_ids.each do |follower_id|
        Red.lpush("user:#{follower_id}:notifications", {:message => message, :description => attributes[:description], :timestamp => Time.now}.to_json)
      end
    end
  end
end

The user model remains an ActiveRecord model and uses the devise gem for user authentication, but is modified to retrieve the notifications, now an OpenStruct. The result is that whenever a user's friend makes a purchase, the user is notified of the purchase. In this simple example, a purchase contains one product only.

Before

class User < ActiveRecord::Base
  # user authentication here
  # model relationships

  def notifications
    Notification.where("friend_relationships.friend_id = notifications.user_id OR notifications.user_id = #{id}").
      joins("LEFT JOIN friend_relationships ON friend_relationships.user_id = #{id}")
  end
end

After

class User < ActiveRecord::Base
  # user authentication here
  # model relationships

  def followers
    User.where('users.id IN (friend_relationships.user_id)').
      joins("JOIN friend_relationships ON friend_relationships.friend_id = #{id}")
  end

  def follower_ids
    followers.map(&:id)
  end

  def notifications
    (Red.lrange("user:#{id}:notifications", 0, -1) || []).map{|notification| Notification.new(ActiveSupport::JSON.decode(notification))}
  end
end

The disadvantages to the NoSQL and RDBMS hybrid is that data portability is limited and ActiveRecord plugins can no longer be used. But the general idea is that performance justifies the move to NoSQL for some data. In several sessions I attended, the speakers reiterated that you will likely never be in a situation where you'll only use NoSQL, but that it's another tool available to suit performance-related business needs. I later spoke with a few Spree developers and we concluded that the NoSQL approach may work well in some applications for product and variant data for improved performance with flexibility, but we didn't come to an agreement on where else this approach may be applied.

Learn more about End Point's Ruby on Rails Development or Ruby on Rails Ecommerce Services.

Rails 3 at RailsConf 2010: Code Goodness

At RailsConf 2010, popular technical topics this year are Rails 3 and NoSQL technologies. My first two articles on RailsConf 2010 so far (here and here) have been less technical, so I wanted to cover some technical aspects of Rails 3 and some tasty code goodness in standard ecommerce examples.

Bundler

Bundler, a gem management tool, is a hot topic at the conference, which comes with Rails 3. I went to a talk on Bundler and it was mentioned in several talks, but a quick run through on its use is:

gem install bundler
gem update --system  # update Rubygems to 1.3.6+

Specify your gem requirements in the application root Gemfile directory.

# excerpt from Spree Gemfile in the works
gem 'searchlogic',            '2.3.5'
gem 'will_paginate',          '2.3.11'
gem 'faker',                  '0.3.1'
gem 'paperclip',              '>=2.3.1.1'

bundle install  # installs all required gems
git add Gemfile  # add Gemfile to repository

In Spree, the long-term plan is to break apart ecommerce functional components into gems and implement Bundler to aggregate the necessary ecommerce gems. The short-term plan is to use Bundler for management of all the Spree gem dependencies.

ActiveRecord

ActiveRecord has some changes that affect the query interface. Some ecommerce examples on new querying techniques with the idea of chaining finder methods:

recent_high_value_orders = Order
  .where("total > 1000")
  .where(["created_at >= :start_date", { :start_date => params[:start_date] }])
  .order("created_at DESC")
  .limit(50)

An example with the use of scope:

class Order << ActiveRecord::Base
  scope :high_value_orders where("total > 1000")
    .where(["created_at >= :start_date", { :start_date => Time.now - 5.days )])
    .order("created_at DESC")
end
class SomeController << YourApplication::AdminController
  def index
    orders = Order.high_value_orders.limit(50)
  end
  
  def snapshot
    orders = Order.high_value_orders.limit(10)
  end
  
  def winner
    Order.high_value_orders.first
  end
end

The changes to ActiveRecord provide a more sensible and elegant way to build queries and moves away from the so-called drunkenness on hashes in Rails. ActiveRecord finder methods in Rails 3 include where, having, select, group, order, list, offset, joins, includes, lock, read only, and from. Because the relations are lazily loaded, you have the ability to chain query conditions with no performance effects as the query hasn't been executed yet, and fragment caching is more effective because the query is executed from a view call. Eager loading can be forced by using first, last, and all.

Router Changes

Some new changes are introduced with Rails 3 in routing that move away from hash-itis, clarify flow ownership, and improve conceptual conciseness. A new route in a standard ecommerce site may be:

resources :users do
  member do
    get :index, :show
  end
  resources :addresses
  resources :reviews 
    post :create, :on => :member
  end
end

Another routing change on named routes allows:

get 'login' => 'sessions#new'   # sessions is the controller, new is the action

ActionMailer

Some significant changes were changed to the ActionMailer class after a reexamination of assumptions and the decision to model mailers after a Rails controller instead of a model/controller hybrid. An example of use with ActionMailer now:

class OrderCompleteNotifier < ActionMailer::Base
  default :from => "customerservice@myecommercesite.com"
  
  def order_complete_notification(recipient)
    @recipient = recipient
    mail(:to => recipient.email_address_with_name,
         :subject => "Order information here")
  end
end

And some changes in sending messages, allowing the following:

OrderCompleteNotifier.signup_notification(recipient1).deliver  # sends email
message = OrderCompleteNotifier.signup_notification(recipient2)
message.deliver

RailTies

A few talks about Rails 3 mentioned the use of RailTies, which serves as the interface between the Rails framework and the rest of its components. It accepts configuration from application.rb, sets up initializers in extensions, tells Rails about generators and rake tasks in extensions, gems, plugins.

Rails 3.1

DHH briefly spoke about some Rails 3.1 things he's excited about, including reorganization of the public directory assets and implementing sprite functionality, which I am a big fan of.

Rails 3 Resources

A few recommended Rails 3 learning resources were mentioned throughout the conference, including:

• Rails 3 Screencasts by Gregg Pollack
• The Rails 3 Upgrade Handbook by Jeremy McAnally
• The Rails Dispatch Blog
• The Great Decoupling by Yehuda Katz
• Rails API Documentation

There are tons of resources out there on these topics and more that I found as I was putting this article together. Go look and write code!

Learn more about End Point's Ruby on Rails Development or Ruby on Rails Ecommerce Services.

RailsConf 2010: Spree and The Ecommerce Smackdown, Or Not

Spree has made a good showing at RailsConf 2010 so far this year, new site and logo released this week:

It started off in yesterday's Ecommerce Panel with Sean Schofield, a former End Point employee and technical lead on Spree, Cody Fauser, the CTO of Shopify and technical lead of ActiveMerchant, Nathaniel Talbott, co-founder of Spreedly, and Michael Bryzek, the CTO and founder of Gilt Groupe.

The panel gave a nice overview on a few standard ecommerce questions:

• My client needs a store - what technology should I use? Why shouldn't I just reinvent the wheel? The SaaS reps evangelized their technologies well, explaining that a hosted solution is good as a relatively immediate solution that has minimum cost and risk to a business. A client [of SaaS] need not be concerned with infrastructure or transaction management initially, and a hosted solution comes with freebies like the use of a CDN that improve performance. A hosted solution is a good option to get the product out and make money upfront. Also, both SaaS options offer elegant APIs.
• How do you address client concerns with security? Again, the SaaS guys stressed that there are a few mistakes guaranteed to kill your business and one of those things includes dropping the ball on security, specifically credit card security. The hosted solutions worry about credit card security so the client doesn't have to. One approach to PCI compliance is to securely post credit card information to a 3rd party secure payment request API as the external requests minimizes the risk to a company. Michael (Gilt) discussed The Gilt Groupe's intricate process in place for security and managing encryption keys. Nathaniel (Spreedly) summarized that rather than focus on PCI compliance specifically, it's more important to have the right mindset about financial data security.
• What types of hosting issues should I be concerned about? Next, the SaaS guys led again on this topic by explaining that they worry about the hosting - that a monthly hosted solution cost (Shopify.com starts at $24/month) is less than the cost of paying a developer who knows your technology in an emergency situation when your site goes down on subpar hosting. Michael (Gilt) made a good point by considering that everything is guaranteed to fail at some point - how do you (client) feel about taking the risk of that? do you just trust that the gateway is always up? One interesting thing mentioned by the SaaS guys is that technically, you should not be able to host any solution in the cloud if you touch credit card data, although you may likely be able to "get away with it" - I'm not sure if this was a scare tactic, but it's certainly something to consider. One disadvantage to hosting in the cloud are that you can't do forensic investigation after a problem if the machine has disappeared.

The remaining panel time was spent on user questions that focused on payment and transaction details specifically. There were a few bits of valuable transaction management details covered. Cody (Shopify) has no plans of developing or expanding on alternative payment systems because there isn't a good ROI. The concept of having something like OIDs for credit cards to shop online would likely not receive support from credit card companies, but PayPal [kinda] serves this role currently. Nathaniel (Spreedly) covered interesting details on how user transaction information is tracked: From day one, everything that is stacked on a users account is a transaction model object that mutate the user transaction state over time. The consensus was that a transaction log is the way to track user transaction information - you should never lose track of any dollarz. On the topic of data, Shopify and Spreedly collect and store all data - the first client step is to sell stuff, then later the client can come back to analyze data for business intelligence such as ROI per customer demographic, the average lifespan of a customer, or the computed value of a customer.

Now I take a break for an image, because it's important to have images in blog articles. Here is my view from the plane as I traveled to Baltimore.

After the panel, Spree had a Birds of a Feather session in the evening, which focused more on Spree. Some topics covered:

• What is the current Spree road map to work on Rails 3? Extension development in Rails 3? As Rails 3 stabilizes over time, Spree will begin to transition but no one's actively doing Rails 3 work at this point. I spoke with Brian Quinn, a member of the Spree core team, who mentioned that he's recently spent time on investigating resource controller versus inherited resources or something else. The consensus was that people don't like Resource Controller (one attendee mentioned they used the Spree data model, but ripped out all of the controllers), but that a sensible alternative will need to be implemented at some point. Searchlogic, the search gem used in Spree search functionality, has no plans to upgrade to Rails 3, so Spree will also have to make sensible decisions for search. The Rails 3 generators have a reputation to be good, so this may trickle down to have positive effects on Spree extension development. Rails 3 also encourages more modular development, so the idea with Spree is that things will gradually be broken into more modular pieces, or gems, and the use of Bundler will tie the Spree base components together.
• How's test coverage in Spree? Bad. Contributors appreciated.
• I got scared after I went to the Ecommerce Panel talk - what's PCI compliance and security like in Spree? Spree is PCI compliant with the assumption that the client doesn't store credit cards - there is (was?) actually a Spree preference setting that defaults to not store credit cards in the database, but it will result in unencrypted credit cards being stored in the database if set to true. The Spree core team recently mentioned that this might be removed from the core. Offsite credit card use such as Authorize.Net CIM implementation is included in the Spree core.

The Spree Birds of a Feather session was good: the result of the session was likely a comprehension of the short and long term road map of Spree is as it transitions to Rails 3. This blog post was going to end here, but luckily I sat next to a Shopify employee this morning and learned more about Shopify. My personal opinion of the ecommerce panel was that advantages of the hosted solutions were appropriately represented, but there wasn't much focus on Spree and the disadvantages of SaaS weren't covered much. I learned that one major disadvantage to Shopify is that they don't have user accounts, however, user accounts are in development. Shopify is also [obviously] not a good choice for sites that have customization but there is a large community of applications. One example of a familiar customization is building a site with a one-deal-at-a-time business model (SteepAndCheap.com, JackThreads.com's former business model) - this would be difficult with Shopify. Some highlights of Shopify include it's template engine, based on Liquid, it has a great API, where you can do most things except place an order, and that it scales really well.

Obviously, I drink the Spree kool-aid often, so I learned more from the panel, BOF, and hallway talk on the subject of SaaS, or hosted Rails ecommerce solutions. The BOF session covered details on the Spree to Rails 3 (hot topic!) transition nicely.

Learn more about End Point's Ruby on Rails Development or Ruby on Rails Ecommerce Services.

RailsConf 2010 Rate a Rails Application: Day One, Session One

My first session at RailsConf 2010 was one that I found valuable: 12 hours to Rate a Rails Application presented by Elise Huard. Elise discussed the process of picking up a Rails application and analyzing it, rather than being the developer who develops a Rails application from scratch. This is particularly valuable because I've had to dive into and comprehend Rails projects more in the last few months. I'd imagine this will become more common as Rails matures and legacy code piles up. Elise mentioned that her motivation for application review comes from either acquisition or for project maintenance. Below is the 12-hour timeline covered by Elise:

0:00: Team Overview

First, Elise suggests that speaking to a team will reveal much about the application you are about to jump into. In our case at End Point, we often make up part of or all of the development team. She briefly mentioned some essential personality traits to look for:

• control freak: someone who worries about details and makes other people pay attention to details
• innovator: someone who looks for the next exciting things and doesn't hesitate to jump in
• automator: people who care about process, more sys admin side of things
• visionaries
• methodologizers (ok, i made this word up): someone who has long term planning ability, road mapping insight
• humility: important to be open to understanding code flaws and improve

Of course, there's overlap of personality traits, but the point is that these traits are are reflected in the code base in some way or another. Elise briefly mentioned that having an issue tracker or version control is viewed positively in application review (of course).

2:00: Systemic Review

The next step in a Rails application evaluation is running the app, making sure it works, examining the maintainability, rails version, and license. She also discussed avoiding the NIH syndrome during a review, which I interpreted as reviewing the code and avoiding thinking about reinventing of the wheel and taking the code and functionality as is (not sure I interpreted her intentions correctly) rather than immediately deciding that you would rewrite everything. Additional systemic indications of a good application are applications that use open source gems or plugins that are maintained and used by others, and that the application has passing tests.

3:00: Start Digging Around

The next step in a 12-hour Rails application review should be an initial poke around of the code. Elise likes to look at config/routes.rb because it's an interface application to the user and a good config/routes.rb file will be a representative inventory of the application. Another step in the review is to examine a model diagram, using a tool such as the railroad gem, or via rubymine. Another good overview is to examine how parts of the application are named, as the names should be understandable to someone in the business.

3:30: Metrics, Tools

Elise's next step in application review is using several metrics to examine complexity and elegance of code, which covered several tools that I haven't heard of besides the common and popular (already mentioned a few times at RailsConf) WTF-metric.

An overview of the tools:

• "rake stats": lines of codes, methods, etc. per controllers, helpers, models
• parsetree's ruby_parser: code transformed into an abstract syntax tree
• flog: analysis of code complexity based on the ABC (Assignment Branch Condition) metric
• flay: analysis for similarity between classes
• saikuro: analysis of cyclomatic complexity
• roodi: detection of antipatterns using the visitor pattern
• reek: detection OO code smells
• rails_best_practices: smell for rails antipatterns
• churn: metrics on change of code run on version control history
• rcov: analysis of code coverage
• metric_fu: tool aggregate that includes many of above tools

Elise noted that although metrics are valuable, they don't identify bugs, analyze code performance and don't analyze the human readability of the code.

5:30: Check out the good stuff

Next up in an application review is looking at the good code. She likes to look at the database files: is everything in the migrations? is the database optimized sensibly? Occasionally, Rails developers can become a bit ignorant (sorry, true) to data modeling, so it's important to note the current state of the database. She also looks at the views to analyze style, look for divitis, and identify too much JavaScript or logic.

7:30: Test Code Review

The next step in Elise's review of a Rails application is checking out the test code. As implementation or requirements change, tests should change. The tests should express code responsibility and hide the implementation detail. Tests should reveal expressive code and don't necessarily need to follow the development DRY standard.

9:30: Deployment Methodology Review

Another point of review is to understand the deployment methodology. Automated deployment such as deployment using capistrano, chef, etc. are viewed positively. Similar to software development tests, failures should be expressive. Deployment is also viewed positively if performance tests and/or bottleneck identification is built into deployment practices.

11:00: Brownie Points

In the final hour of application review, Elise looks for brownie-point-worthy coverage such as:

• continuous integration
• documentation and freshness of documentation
• monitoring (nagios, etc.), exception notification, log analyzers
• testing javascript

I found this talk to be informative on how one might approach understanding an existing rails application. As a consultant, we frequently have to pick up a project and just go, Rails or not, so I found the tools and approach presented by Elise insightful, even if I might rearrange some of the tasks if I am going to write code.

The talk might also be helpful in providing details to teach someone where to look in an application for information. For example, a couple of End Point developers are starting to get into Rails, and from this talk I think it's a great recommendation to send someone to config/routes.rb to learn and understand the application routing as a starting point.

Learn more about End Point's Ruby on Rails Development or Ruby on Rails Ecommerce Services.

Spree vs Magento: A Feature List Comparison

Note: This article was written in June of 2010. Since then, there have been several updates to Spree. Check out the current Official Spree Extensions or review a list of all the Spree Extensions.

This week, a client asked me for a list of Spree features both in the core and in available extensions. I decided that this might be a good time to look through Spree and provide a comprehensive look at features included in Spree core and extensions and use Magento as a basis for comparison. I've divided these features into meaningful broader groups that will hopefully ease the pain of comprehending an extremely long list :) Note that the Magento feature list is based on their documentation. Also note that the Spree features listed here are based on recent 0.10.* releases of Spree.

Features on a Single Product or Group of Product

CMS Features

Checkout Support

Shipping Support

Payment Support

Admin Features

User Account Features

System Wide Features

The configurability and complexity of each feature listed above varies. Just because a feature is provided within a platform does not guarantee that it will meet the desired business needs. Magento serves as a more comprehensive ecommerce platform out of the box, but the disadvantage may be that adding custom functionality may require more resources (read: more expensive). Spree serves as a simpler base that may encourage quicker (read: cheaper) customization development simply because it's in Rails and because the dynamic nature of Ruby allows for elegant extensibility in Spree, but a disadvantage to Spree could be that a site with a large amount of customization may not be able to take advantage of community-available extensions because they may not all play nice together.

Rather than focus on the platform features, the success of the development depends on the developer and his/her skillset. Most developers will say that any of the features listed above are doable in Magento, Spree, or Interchange (a Perl-based ecommerce platform that End Point supports) with an unlimited budget, but a developer needs to have an understanding of the platform to design a solution that is easily understood and well organized (to encourage readability and understandability by other developers), develop with standard principles like DRY and MVC-style separation of concerns, and elegantly abstract from the ecommerce core to encourage maintainability. And of course, be able to understand the business needs and priorities to guide a project to success within the given budget. Inevitably, another developer will come along and need to understand the code and inevitably, the business will often use an ecommerce platform longer than planned so maintainability is important.

Please feel free to comment on any errors in the feature list. I'll be happy to correct any mistakes. Now, off to rest before RailsConf!

Learn more about End Point's Ecommerce Development or Ruby on Rails Ecommerce Services.

Spree and Multi-site Architecture for Ecommerce

Running multiple stores from a single ecommerce platform instance seems to be quite popular these days. End Point has worked with several clients in developing a multi-store architecture running from one Interchange installation. In the case of our work with Backcountry.com, the data structure requires that a site_id column be included in product and navigation tables to specify which stores products and categories belong to. Frontend views are generated and "partial views" or "view components" are organized into a per-site directory structure and database calls request products against the current site_id. Below is a simplified view of the data structure used for Backcountry.com's multi-site architecture.

Basic multi-store data architecture

A similar data model was implemented and enhanced for another client, College District. A site_id column exists in multiple tables including the products and navigation tables, and sites can only access data in the current site schema. College District takes one step further in development for appearance management by storing CSS values in the database and enabling CSS stylesheet generation on the fly with the stored CSS settings. This architecture works well for College District because it allows the owners to quickly publish new sites. Below is a simplified view of the data structure used for College District, where the table site_variables contains CSS values (text, background colors, etc.).

Extended multi-store data architecture where site_variables table contains CSS settings. The store frontend can access data in the current store schema only.

In the past year, running a multi-site setup has been a popular topic in the user group for Spree, an open source Ruby on Rails platform. I've been happy to be involved in a couple of client multi-site projects. Here I'll discuss some comments for my Spree multi-site implementation work.

Basic Extension

First, the Spree multi-domain extension developed by the Spree core team serves as a strong starting point. The extension migrations produce the data structure shown below, by creating a new table stores and products_stores, and adding a store_id column to the tracker (Google Analytics data) and orders table.

Code changes in addition to data model changes are:

• A before filter sets the current store by examining the current environment variable "SERVER_NAME".
• Frontend product retrieval is modified to retrieve products in the current store only.
• Frontend navigation is generated based on products assigned to the current store.
• The Tracker (Google Analytics) retrieval method is modified to only retrieve the current store settings.
• Order processing assigns the store_id value to each order.

With this extension, a user can check out across multiple stores running from a single Spree application with the same account, products can be assigned to multiple stores, and a single default store is set to render if no domains match the current SERVER_NAME. The extension does not introduce the advanced schema behavior like College District's data architecture, however, the extension could be customized to do so. The extension suits basic requirements for a multi-store architecture.

Additional Customizations Required

In my project, there were additional changes required. I used the Spree static pages extension, which introduces functionality to present and manage static content pages. I modified this extension to create an additional stores_pages table that introduces a has and belongs to many relationship between stores and pages.

Basic Spree static pages data model.	Expanded Spree static pages data model with multi-store design.

Other custom requirements may include modifying the spree-faq extension to build a has and belongs to many relationship between questions and stores, or similar changes that create relationships between the stores table and other data.

File Structure Organization

The next interesting design choice I faced was how to handle appearance management across multiple sites. As I mentioned before, a method was developed to retrieve and build views based on the current store in Backcountry.com's multi-store solution. With College District, the database stored CSS values and generated new stylesheets on the fly for each site. I chose to implement a simplified version of the College District implementation, where a single stylesheet contains the rules for each site.

In the Spree implementation, the Spree::BaseController class is modified with a before filter to set the store and asset (images, javascript, stylesheet) location:

module Spree::Fantegrate::BaseController
  def self.included(controller)
    controller.class_eval do
      controller.append_before_filter :set_store_and_asset_location
    end
  end
  def set_store_and_asset_location
    @current_store ||= Store.by_domain(request.env['SERVER_NAME']).first
    @current_store ||= Store.default.first
    @asset_location = @current_store.domains.gsub('.mydomain.com', '')
  end
end

The default layout includes the main and site specific stylesheets:

  <%= stylesheet_link_tag "style" %>
  <%= stylesheet_link_tag "#{@asset_location}/site" %>

The site specific stylesheet contains style rules and includes site specific image settings:

body.site_a { background: #101a35 url(/images/site_a/bg.jpg) repeat-x 0 0; }
.site_a h1#logo { float: left; display: inline; width: 376px; height: 131px; position:relative; left: -15px; padding-bottom: 10px; }
.site_a h1#logo a { display: block; height: 131px; background: url(/images/site_a/logo.png); }
.site_a #top-right-info,
.site_a #top-right-info a,
.site_a #top-right-info b { color: #fff; }
...

This implementation acts on the assumption that there will be minimal design differences across stores. This is a simple and effective way to get an initial multi-store architecture in place that allows you to manage multiple site's appearance in a single Spree application.

Advanced Topics

Some advanced topics I've considered with this work are:

• Can we dynamically generate the migrations based on which extensions are installed? For example, a list of tables would be included in the main extension. This list of tables would be iterated through and if the table exists, a dynamic migration is generated to build a has many/belongs to, has one/belongs to, or has and belongs to many relationship between stores and the table.
• SSL setup for our Spree multi-store implementation was accomplished with a wildcard SSL certificate, where each store can be accessed from a different subdomain. Backcountry.com and College District implementation was accomplished with standard SSL certificates because the stores do not share domains. The Spree implementation methods described in this article do not vary for subdomain versus different domain design, but this is certainly something to consider at the start of a multi-site project.
• The multi-domain extension could be customized and enhanced to include a table that contains CSS settings similar to our College District implementation and allows you to generate new stylesheets dynamically to change the look of a site. The obvious advantage of this is that a user with site administrative permissions can change the appearance of the site via backend Spree management without involving development resources.

Learn more about End Point's Ruby on Rails Development or Ruby on Rails Ecommerce Services.