Javascript Blog Archive
UTOSC 2012 talks of interest
It's been two weeks now since the Utah Open Source Conference for 2012. My fellow End Pointers wrote previously about it: Josh Ausborne about the mini Liquid Galaxy we set up there for everyone to play with, and Josh Tolley with a write-up of his talks on database constraints and KML for geographic mapping markup.
There were a lot of interesting talks planned, and I could only attend some of them. I really enjoyed these:
- Rob Taylor on AngularJS
- Brandon Johnson on Red Hat's virtualization with oVirt, Spacewalk, Katello, and Aeolus
- Clint Savage about RPM packaging with Mock & Koji
- Daniel Evans on testing web applications with Capybara, embedded WebKit, and Selenium (which End Pointer Mike Farmer wrote about here back in December)
- Aaron Toponce on breaking full-disk encryption (I missed this talk, but learned about it from Aaron in the hallway track and his slides afterwards)
- Matt Harrison's tutorial Hands-on intermediate Python, covering doctest, function parameters and introspection, closures, function and class decorators, and more.
I gave a talk on GNU Screen vs. tmux, which was fun (and ends with a live demo that predictably fell apart, and audience questions that you can't hear on the recording). Here's the video:
Follow this Direct YouTube link in case the embedded version doesn't work for you. And here are the presentation slides.
I have a bit more to cover from the conference later!
Keeping Your Apps Neat & Tidy With RequireJS
RequireJS is a very handy tool for loading files and modules in JavaScript. A short time ago I used it to add a feature to Whiskey Militia that promoted a new section of the site. By developing the feature as a RequireJS module, I was able to keep all of its JavaScript, HTML and CSS files neatly organized. Another benefit to this approach was the ability to turn the new feature "on" or "off" on the site by editing a single line of code. In this post I'll run through a similar example to demonstrate how you could use RequireJS to improve your next project.
File Structure
The following is the file structure I used for this project:
├── index.html
└── scripts
├── main.js
├── my
│ ├── module.js
│ ├── styles.css
│ └── template.html
├── require-jquery.js
├── requirejs.mustache.js
└── text.js
The dependencies included RequireJS bundled together with jQuery, mustache.js for templates and the RequireJS text plugin to include my HTML template file.
Configuration
RequireJS is included in the page with a script tag and the data-main attribute is used to specify additional files to load. In this case "scripts/main" tells RequireJS to load the main.js file that resides in the scripts directory. Require will load the specified files asynchronously. This is what index.html looks like:
<!DOCTYPE html>
<html>
<head>
<title>RequireJS Example</title>
</head>
<body>
<h1>RequireJS Example</h1>
<!-- This is a special version of jQuery with RequireJS built-in -->
<script data-main="scripts/main" src="scripts/require-jquery.js"></script>
</body>
</html>
I was a little skeptical of this approach working on older versions of Internet Explorer so I tested it quickly with IE6 and confirmed that it did indeed work just fine.
Creating a Module
With this in place, we can create our module. The module definition begins with an array of dependencies:
define([ "require", "jquery", "requirejs.mustache", "text!my/template.html" ],
This module depends on require, jQuery, mustache, and our mustache template. Next is the function declaration where our module's code will live. The arguments specified allow us to map variable names to the dependencies listed earlier:
function(require, $, mustache, html) { ... }
In this case we're mapping the $ to jQuery, mustache to requirejs.mustache and, html to our template file.
Inside the module we're using Require's .toUrl() function to grab a URL for our stylesheet. While it is possible to load CSS files asynchronously just like the other dependencies, there are some issues that arise that are specific to CSS files. For our purposes it will be safer to just add a <link> element to the document like so:
var cssUrl = require.toUrl("./styles.css");
$('head').append($('',
{ rel: "stylesheet", media: "all", type: "text/css", href: cssUrl }));
Next, we define a view with some data for our Mustache template and render it.
var view = {
products: [
{ name: "Apples", price: 1.29, unit: 'lb' },
{ name: "Oranges", price: 1.49, unit: 'lb'},
{ name: "Kiwis", price: 0.33, unit: 'each' }
],
soldByPound: function(){
return (this['unit'] === 'lb') ? true : false;
},
soldByEach: function() {
return (this['unit'] === 'each') ? true : false;
}
}
// render the Mustache template
var output = mustache.render(html, view);
// append to the HTML document
$('body').append(output);
});
The Template
I really like this approach because it allows me to keep my HTML, CSS and JavaScript separate and also lets me write my templates in HTML instead of long, messy JavaScript strings. This is what our template looks like:
<ul class="hot-products">
{{#products}}
<li class="product">
{{name}}: ${{price}} {{#soldByEach}}each{{/soldByEach}}{{#soldByPound}}per lb{{/soldByPound}}
</li>
{{/products}}
</ul>
Including the Module
To include our new module in the page, we simply add it to our main.js file:
require(["jquery", "my/module"], function($, module) {
// jQuery and my/module have been loaded.
$(function() {
});
});
When we view our page, we see that the template was was rendered and appended to the document:
Optimizing Your Code With The r.js Optimizer
One disadvantage of keeping everything separate and using modules in this way is that it adds to the number of HTTP requests on the page. We can combat this by using the the RequireJS Optimizer. The r.js script can be used a part of a build process and runs on both node.js and Rhino. The Optimizer script can minify some or all of your dependencies with UglifyJS or Google's Closure Compiler and will concatenate everything into a single JavaScript file to improve performance. By following the documentation I was able to create a simple build script for my project and build the project with the following command:
node ../../r.js -o app.build.js
This executes the app.build.js script with Node. We can compare the development and built versions of the project with the Network tab in Chrome's excellent Developer Tools.
Development Version:
Optimized with the RequireJS r.js optmizer:
It's great to be able to go from 8 HTTP requests and 360 KB in development mode to 4 HTTP requests and ~118 KB after by running a simple command with Node! I hope this post has been helpful and that you'll check out RequireJS on your next project.
RailsConf 2012: Day One
Today kicks off the official first day of RailsConf 2012, in Austin, Texas (yesterday RailsConf unofficially kicked off with Ignite Rails). This is my fourth RailsConf, and I've found that I get increasingly more out of each one as my time working with Rails has accumulated.
The morning started with a keynote by DHH. DHH talked about progress and keeping an interest in something because it continues to make progress. Initially, the Rails community had a negative reaction to progress like Ruby 1.9, Rails 3, the Asset Pipeline and CoffeeScript, but after the initial learning curve, the new tools and progress was appreciated by the community. DHH talked about how there have been many studies on what prevents people from progressing, and one of those causes is loss aversion, meaning that people hate losing or failing enough that it prevents them from advancing. To me, the point of his talk was to prepare everyone for Rails 4 and how it will challenge and break things. Rails 4 is admittedly not coming out anytime soon, so he didn't touch on any specifics.
Using Backbone.js with Rails
One session talk I attended was Using Backbone.js with Rails by Sarah Mei of Pivotal Labs. Backbone is not an MVC framework in JavaScript, but it is lumped into traditional JavaScript MVC frameworks. It serves as a lightweight infrastructure for organizing code like templates, interactive elements, and events. Because Backbone is a relatively immature framework, like other JavaScript frameworks, there isn't a standard way of writing code, so the implementation of Backbone varies widely.
Sarah talked about the love story between JS and Rails:
She touched on the basics of Backbone from a Rails perspective:
- Models: Backbone and Rails models are very similar. An application will likely have Backbone models that mirror Rails models.
- Templates: Backbone templates are akin to Rails views or a mustache template.
- Views: Views are similar to Rails controllers. They set a template, class name, and define events.
Next, Sarah talked about two methods of implementation using Backbone.js and went through some code examples:
- Greenfield App
- This approach tends to be more API driven and makes sense for an API-driven application that also requires buildout of native mobile apps.
- In this approach, the server side returns [JSON, etc.] data and doesn't render HTML.
- Sarah noted that the amount of code required here can be substantial because the framework is lightweight, so she would recommended considering alternatives (such as ember.js).
- "Backbone as Frosting" Implementation
- In this implementation, there is a mixture of server-side and client-side rendering.
- Sarah noted here that the lightweightedness of Backbone works nicely.
I haven't used Backbone.js in any client projects, but I've looked into it recently out of interest. I like the general idea of improving JavaScript code organization via the frosting approach described by Sarah. A few code examples were shared during the presentation — check out the slides here.
CoffeeScript for the Rubyist
Another talk I attended today was CoffeeScript for the Rubyist by Mark Bates. CoffeeScript is a JS meta language that compiles into JS and can easily integrate with your current JavaScript. It's easy to read, write, and maintain. I took the CoffeeScript lesson over at Code School recently (highly recommended!). Mark went through a brief history of JavaScript and explained that like the evolution to early assembly languages to C to Ruby (with a few missing steps), CoffeeScript's aim is to be an improved tool for writing readable and maintainable code. The syntax of CoffeeScript is a hybrid of Ruby and Python which is simple, uses no semicolons, typically no curly braces, no "function" keyword, relaxed parenthesis, and significant whitespace.
Mark then went into a review of Ruby-like CoffeeScript behavior. Below are some of the important topics in CoffeeScript that he covered, as well as links to the documentation:
- conditionals similarity (inline, postfix)
- objects and hashes are similar (can't screw up last comma)
- ranges
- string interpolation
- heredocs, (""" works)
- functions Ruby 1.9 Lamda syntax is similar to CoffeeScript version
- default arguments
- splats (e.g. first, second, others...)
- loops and comprehensions (for number in numbers)
- combination of loops and comprehensions and conditionals
- classes
- inheritence (e.g. class Manager extends Employee)
- Bound functions (with a fat arrow)
- Existential operator (e.g. if foo?, if console?.log "foo")
In addition to the talks on Backbone.js and CoffeeScript, I'll post the links to slides and video for the other talks I attended when they become available.
What's Next?
It's funny (but not surprising) how many of the talks I attended on Day One weren't actually about Rails or the specifics of Rails. Perhaps I'll try to pick a few more Rails-centric talks in the next couple of days of the conference and report on those. Stay tuned!
Rails 3 remote delete link handlers with Unobtrusive Javascript
I recently encountered a bug in a Rails 3 application that used a remote link_to tag to create a Facebook-style "delete comment" link using unobtrusive javascript. I had never worked with remote delete links like this before, so I figured I’d run through how I debugged the issue.
Here are the relevant parts of the models we're dealing with:
class StoredFile < ActiveRecord::Base has_many :comments, :dependent => :destroy end class Comment < ActiveRecord::Base belongs_to :user belongs_to :stored_file end
Here's the partial that renders a single Comment (from the show.html.erb view for a StoredFile) along with a delete link if the current_user owns that single Comment:
<%= comment.content %> -<%= comment.user.first_name %> <% if comment.user == current_user > <%= link_to 'X', stored_file_comment_path(@stored_file, comment), :remote => true, :method => :delete, :class => 'delete-comment' > <% end ->
Here’s a mockup of the view with 3 comments:
At first, the bug seemed to be that the "X" wasn’t actually a link, and therefore, didn't do anything. Clicking the "X" with Firebug enabled told a different story. There was a link there (hidden by sneaky CSS,) and the Firebug console showed that it was sending the appropriate request to the correct url: /stored_files/78/comments/25
The development.log file on the server corraborated the story and showed a successful delete:
Started DELETE "/stored_files/78/comments/25"
Processing by CommentsController#destroy as JS
Parameters: {"stored_file_id"=>"78", "id"=>"25"}
SQL (0.6ms) DELETE FROM "comments" WHERE "comments"."id" = 25
Completed 200 OK in 156ms
So far, so good. I know that our client code is making the correct request and the Rails app is handling it appropriately. I knew that the existing code "worked," but still didn’t provide UI feedback to inform the user. I needed to write some jQuery to handle the successful (HTTP 200) server response to our unobtrusive javascript call.
Normally, when writing my own handler (e.g. to handle a button click) to initiate an Ajax call with jQuery, I’d use $.ajax or $.post and use its built-in success handler. Something like this:
$.ajax({
type: 'POST',
url: 'my_url',
data: { param_name1: 'param_value1'},
success: function(data){
alert('Successfully did the needful!');
}
});
It turns out that you still define a event handler when handling server responses to unobtrusive javascript, it’s just that the syntax is very different. I needed to bind the ‘ajax:success’ event when it’s fired by any of my comment delete links (where class=”delete-comment”, as specified in my link_to call).
$(document).on('ajax:success', '.delete-comment', function() {
// .parent() is the div containing this "X" delete link
$(this).parent().slideUp();
}
);
(Note that I happen to be using the newer jQuery 1.7+ .on() method and argument list instead of the older, more common .live() style. In this case, they are functionally equivalent, but the methods that .on() replaces are deprecated in jQuery 1.7.)
Now, when the user clicks the "X" to delete one of their own comments, the successful unobtrusive javascript call is detected and the div containing that single comment is neatly hidden with the help of jQuery's slideUp() method.
Much better!
Using Disqus and Ruby on Rails
Recently, I posted about how to import comments from a Ruby on Rails app to Disqus. This is a follow up to that post where I outline the implementation of Disqus in a Ruby on Rails site. Disqus provides what it calls Universal Code which can be added to any site. This universal code is just JavaScript, which asynchronously loads the Disqus thread based on one of two unique identifiers Disqus uses.
Disqus in a development environment
Before we get started, I'd recommend that you have two Disqus "sites"; one for development and one for production. This will allow you to see real content and experiment with how things will really behave once you're in production. Ideally, your development server would be publicly accessible to allow you to fully use the Disqus moderation interface, but it isn't required. Simply register another Disqus site, and make sure that you have your shortname configured by environment. Feel free to use whatever method you prefer for defining these kinds of application preferences. If you're looking for an easy way, considering checking out my article on Working with Constants in Ruby. It might look something like this:
# app/models/article.rb DISQUS_SHORTNAME = Rails.env == "development" ? "dev_shortname".freeze : "production_shortname".freeze
Disqus Identifiers
Each time you load the universal code, you need to specify a few configuration variables so that the correct thread is loaded:
- disqus_shortname: tells Disqus which website account (called a forum on Disqus) this system belongs to.
- disqus_identifier: tells Disqus how to uniquely identify the current page.
- disqus_url: tells Disqus the location of the page for permalinking purposes.
# app/views/disqus/_thread.html.erb
# assumes you've passed in the local variable 'article' into this partial
# from http://docs.disqus.com/developers/universal/
<div id="disqus_thread"></div>
<script type="text/javascript">
var disqus_shortname = '<%= Article::DISQUS_SHORTNAME %>';
var disqus_identifier = '<%= article.id %>';
var disqus_url = '<%= url_for(article, :only_path => false) %>';
/* * * DON'T EDIT BELOW THIS LINE * * */
(function() {
var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
})();
</script>
The above code will populate the div#disqus_thread with the correct content based on your disqus_identifier. By setting up a single partial that will always render your threads, it becomes very easy to adjust this code if needed.
Disqus Identifier Gotcha
We found during our testing a surprising and unexpected behavior in how Disqus associates a thread to a URL. In our application, the landing page was designed to show the newest article as well as the Disqus comments thread. We found that once a new article was posted, the comments from the previous article were still shown! It seems Disqus ignored the unique disqus_identifier we had specified and instead associated the thread with the landing page URL. In our case, a simple routing change allowed us to forward the user to the unique URL for that content and thread. In your case, there may not be such an easy work around, so be certain you include both the disqus_identifier and disqus_url JavaScript configuration variables above to minimize the assumptions Disqus will make. When at all possible, always use unique URLs for displaying Disqus comments.
Comment Counters
Often an index page will want to display a count of how many comments are in a particular thread. Disqus uses the same asynchronous approach to loading comment counts. Comment counts are shown by adding code such as the following where you want to display your count:
# HTML
<a href="http://example.com/article1.html#disqus_thread"
data-disqus-identifier="<%=@article.id%>">
This will be replaced by the comment count
</a>
# Rails helper
<%= link_to "This will be replaced by the comment count",
article_path(@article, :anchor => "disqus_thread"),
:"data-disqus-identifer" => @article.id %>
At first this seemed strange, but it is the exact same pattern used to display the thread. It would likely be best to remove the link text so nothing is shown until the comment count is loaded, but I felt for my example, having some meaning to the test would help understanding. Additionally, you'll need to add the following JavaScript to your page.
# app/view/disqus/_comment_count_javascript.html.erb
# from http://docs.disqus.com/developers/universal/
# add once per page, just above </body>
<script type="text/javascript">
var disqus_shortname = '<%= Article::DISQUS_SHORTNAME %>';
/* * * DON'T EDIT BELOW THIS LINE * * */
(function () {
var s = document.createElement('script'); s.async = true;
s.type = 'text/javascript';
s.src = 'http://' + disqus_shortname + '.disqus.com/count.js';
(document.getElementsByTagName('HEAD')[0] || document.getElementsByTagName('BODY')[0]).appendChild(s);
}());
</script>
Disqus recommends adding it just before the closing </body> tag. You only need to add this code ONCE per page, even if you're planning on showing multiple comment counts on a page. You will need this code on any page with a comment count, so I do recommend putting it in a partial. If you wanted, you could even include it in a layout.
Styling Comment Counts
Disqus provides extensive CSS documentation for its threads, but NONE for its comment counters. In our application, we had some very particular style requirements for these comment counts. I found that in Settings > Appearance, I could add HTML tags around the output of the comments.
This allowed me to style my comments as needed, although these fields are pretty small, so make sure to compress your HTML as much as possible.
Labeling input boxes including passwords
I'm currently working on a new site and one of the design aspects of the site is many of the form fields do not have labels near the input boxes, they utilize labels that are inside the input box and fade away when text is entered. The label is also supposed to reappear if the box is cleared out. Originally I thought this was a pretty easy problem and wrote out some jQuery to do this quickly. The path I went down first was to set the textbox to the value we wanted displayed and then clear it on focus. This worked fine, however I reached a stumbling block when it came to password input boxes. My solution did not work properly because text in a password box is hidden and the label would be hidden as well. Most people would probably understand what went in each box, but I didn't want to risk confusing anyone, so I needed to find a better solution
I did some searching for jQuery and labels for password inputs and turned up several solutions. The first one actually put another text box on top of the password input, but that seemed prone to issue. The solution I decided to ultimately use is called In-Fields Labels, a jQuery plugin by Doug Neiner. In this solution Doug has floating labels that appear over the top of the textbox, and they dim slightly when focus is gained and then disappear completely when typing begins. The plugin does not mess with the value in the input box at all.
It was fairly easy to get up and running. I added the plugin to the page, created some styling for the labels, added label tags with the class of 'overlay' for each input box and called $('label.overlay').inFieldLabels();. This was all that was needed to get us going.
Normal view
Focus in the password box
Typing in the password box
The effect is pretty cool and it provides a good interface for the user as they are reminded up until the time they type in the box what they are supposed to enter.
Nifty In-Button Confirmation
I've been working on a personal email client after work, called Warm Sunrise, that forces myself to keep a manageable inbox. One of the goals of the project was to get to a zero-inbox everyday, so I needed a 'Delete All' button that was easy-to-use without running the risk of accidentally deleting emails. I took a look at JavaScript's confirm, which is jarring, and jQuery's dblClick, which doesn't provide any feedback to the user after the first click, leaving the user to wonder why their emails weren't deleted.
Given these options, I built my own button using Rails 3.1, jQuery, and CoffeeScript, that better fit the goals I set out with. It requires a double click, but gives the user a confirmation in the button itself, without any sort of timeout. You can see a video of it in action here:
Starting with app/views/letters/index.html.erb, I generated the buttons using Rails helpers and Twitter's Bootstrap classes:
<%= link_to 'Write letter', new_letter_path, :class => "btn primary pull-right far-right" %> <%= link_to 'Delete all', '#', :class => "btn pull-right no_danger", :id => "delete_all" %> <%= link_to 'Are you sure?', delete_all_letters_path, :method => :destroy, :class =>"btn pull-right danger confirm", :id => "delete_all", :style => "display:none;" %>
Notice that the 'Delete all' button doesn't actually specify a url and the 'Are you sure?' link's style is set to "display:none"
Here's the relationship I set up in my models:
app/models/letter.rb
belongs to :user
app/models/user.rb
has_many :letters, :dependent => :destroy
I set up config/routes.rb to work with the explicit path I set in:
post 'delete_all_letters' => 'letters#delete_all'
Finally, I finished this lot by adding the delete_all action to my app/controllers/letters_controller.rb:
def delete_all
current_user.letters.delete_all
respond_to do |format|
format.html { redirect_to letters_url, notice: 'Successfully deleted all letters.' }
format.json { head :ok }
end
end
CoffeeScript is a beautiful language that compiles to JavaScript, which I prefer to JavaScript itself. You can read more about it here. Let's take a look at the CoffeeScript that makes this button work:
$('a#delete_all.no_danger').hover( ->
$(this).addClass('danger')
$(this).click( ->
$('a#delete_all.no_danger').hide()
$('a#delete_all.confirm').show()
)
)
$('a#delete_all.no_danger').mouseleave( ->
$(this).removeClass('danger')
)
$('a#delete_all.danger').mouseleave( ->
$(this).hide()
$('a#delete_all.no_danger').show()
)
Since the button's text changes to a confirmation on the first click, makes it better for my purposes than Javascript's dblClick method. Check the video to see what it looks like in action.
Let's take a look at what this compiles to in plain JavaScript, too, since this is the only thing the browser sees:
$('a#delete_all.no_danger').hover(function() {
$(this).addClass('danger');
return $(this).click(function() {
$('a#delete_all.no_danger').hide();
return $('a#delete_all.confirm').show();
});
});
$('a#delete_all.no_danger').mouseleave(function() {
return $(this).removeClass('danger');
});
$('a#delete_all.danger').mouseleave(function() {
$(this).hide();
return $('a#delete_all.no_danger').show();
});Not shown in the video, but I modified index.html.erb to only show the 'Delete all' button when the user has a zero-inbox.
<%= link_to 'Write letter', new_letter_path, :class => "btn primary pull-right far-right" %>
<% if !@letters.empty? %>
<%= link_to 'Delete all', '#', :class => "btn pull-right no_danger", :id => "delete_all" %>
<%= link_to 'Are you sure?', delete_all_letters_path, :method => :destroy, :class =>"btn pull-right danger confirm", :id => "delete_all", :style => "display:none;" %>
<% end %>
Preventing Global Variables in JavaScript
JavaScript's biggest problem is its dependence on global variables
--Douglas Crockford, JavaScript: The Good Parts
Recently I built out support for affiliate management into LocateExpress.com’s Sinatra app using JavaScript and YUI.
I used a working page from the admin, Service Providers, as a starting point to get something up and running for affiliates quickly. By the time I finished, the Affiliates page worked great, but forms on the Service Provider page no longer populated with data.
Identifying a misbehaving global variable
There were no errors in the console, and the forms on the Service Providers page remained broken even after restoring an old copy of service_providers.js. As it turns out, a global variable, edit_map, was defined within service_providers.js, and again in the copied affiliates.js. Credit for spotting the problem goes to Brian Miller.
The fix was as simple as moving edit_map's declaration into the file's YUI sandbox, so that version of edit_map wouldn't be visible to any other pages in the admin.
Preventing global variables
As projects grow and complexity increases, it becomes easier and easier to overlook global variables and thus run into this tough-to-debug problem. Douglas Crockford’s Javascript: The Good Parts covers several workarounds to using global variables.
Rather than declaring variables globally, like this:
var edit_map = { 'business[name]' : 'business_name' };the author recommends declaring them at the beginning of functions whenever possible:
YUI().use("node", "io", "json",
function(Y) {
var edit_map = { 'business[name]' : 'business_name' };
...
});In all other cases, he suggests using Global Abatement, which prevents your global variables from affecting other libraries. For example,
var LocateExpress = {};
LocateExpress.edit_map = { 'business[name]' : 'business_name' };
YUI().use("node", "io", "json",
function(Y) {
...
return LocateExpress.edit_map;
});
I highly recommend JavaScript: The Good Parts to learn about the best JavaScript has to offer and workarounds for its ugly side. The author also wrote a very popular code-checker, JSLint, which could help debug this nasty problem by highlighting implicit global variables.
Rails 3.1: Upgrading a Simple App - Part 1
Here at End Point, I've worked with a few Rails 3 applications in production and a couple of Rails 3.1 apps in development, so I've become familiar with the new features and functionality including the Rails 3.1 Asset Pipeline that I mentioned earlier this year. I thought it was a good time to upgrade our website to Rails 3.1 and share the experience.
To start, here's a quick summary of our website:
- Simple Rails application running on Rails 2.1.2 with no database
- Static pages throughout the site, fully cached
- Rake tasks to generate partials throughout the site to display dynamic blog content
- Site uses a moderate amount of jQuery and jQuery plugins.
- Site is optimized in terms of asset serving (ETags, Expires headers, CSS sprites, etc.)
While I've worked with a few Rails 3 apps, I haven't been involved in the actual upgrade process myself. There are plenty of resources out there with upgrade advice, including a few RailsCasts (one, two, and three). My favorite resource was the rails_upgrade gem, a gem that is now officially supported by Rails to help with the upgrade process. I followed the instructions to install the gem (script/plugin install git://github.com/rails/rails_upgrade.git) to install it as a plugin in our site's application in a fresh git branch (on a camp, of course!).
The rails_upgrade provides a few new rake tasks for checking compatibility, upgrading the routes, creating a Gemfile, and upgrading configuration. For me, the most valuable task was the rake rails:upgrade:check task. Here's what the output looked like for this app:
Deprecated session secret setting
Previously, session secret was set directly on ActionController::Base; it's now config.secret_token.
More information: http://lindsaar.net/2010/4/7/rails_3_session_secret_and_session_store
The culprits:
- config/initializers/session_store.rb
Old router API
The router API has totally changed.
More information: http://yehudakatz.com/2009/12/26/the-rails-3-router-rack-it-up/
The culprits:
- config/routes.rb
New file needed: config/application.rb
You need to add a config/application.rb.
More information: http://omgbloglol.com/post/353978923/the-path-to-rails-3-approaching-the-upgrade
The culprits:
- config/application.rb
Deprecated constant(s)
Constants like RAILS_ENV, RAILS_ROOT, and RAILS_DEFAULT_LOGGER are now deprecated.
More information: http://litanyagainstfear.com/blog/2010/02/03/the-rails-module/
The culprits:
- app/views/layouts/application.rhtml
- ...
Soon-to-be-deprecated ActiveRecord calls
Methods such as find(:all), find(:first), finds with conditions, and the :joins option will soon be deprecated.
More information: http://m.onkey.org/2010/1/22/active-record-query-interface
The culprits:
- app/views/blog_archive/_ruby_on_rails.html.erb
- ...
Deprecated AJAX helper calls
AJAX javascript helpers have been switched to be unobtrusive and use :remote => true instead of having a seperate function to handle remote requests.
More information: http://www.themodestrubyist.com/2010/02/24/rails-3-ujs-and-csrf-meta-tags/
The culprits:
- app/views/blog_archive/_company.html.erb
- ...
Deprecated ActionMailer API
You're using the old ActionMailer API to send e-mails in a controller, model, or observer.
More information: http://lindsaar.net/2010/1/26/new-actionmailer-api-in-rails-3
The culprits:
- app/controllers/contact_controller.rb
Old ActionMailer class API
You're using the old API in a mailer class.
More information: http://lindsaar.net/2010/1/26/new-actionmailer-api-in-rails-3
The culprits:
- app/models/contact_form.rb
As you can see, the upgrade check spits out a list of necessary and recommended upgrades and the corresponding *culprits*. It's also nice that the task provides documentation in the form of a link for each message. Studying the source of the plugin, I found additional examples of upgrade messages: named_scope updates, validate_on_* syntax, test_help path updates, gem bundling configuration, Rails generator API syntax updates, messaging on known broken plugins (e.g. searchlogic, cucumber, nifty-generators), and depracation on ERb helper and AJAX calls.
I went through and applied my updates, according to the checklist. Notable updates were:
Routing updatesBefore
ActionController::Routing::Routes.draw do |map| map.root :controller => 'home', :action => 'index' map.connect 'contact/submit', :controller => 'contact', :action => 'submit' map.connect ':controller/:id' map.connect '*path', :controller => 'redirect' end
After
Endpoint::Application.routes.draw do root :to => 'home#index' match 'contact/submit' => 'contact#submit' match ':controller(/:id)', :action => :index match '*path' => 'redirect#index' endIntroduction of a Gemfile
source 'http://rubygems.org' gem 'rails', '3.1.0' gem 'json' # Gems used only for assets and not required # in production environments by default. group :assets do gem 'sass-rails', " ~> 3.1.0" gem 'coffee-rails', "~> 3.1.0" gem 'uglifier' end gem 'jquery-rails' gem 'fastercsv' gem 'execjs' gem 'therubyracer' gem 'rake', '0.8.7'Renaming rhtml files
Something that didn't come up in the rails upgrade check that is required to have a working app is renaming all rhtml files to html.erb, briefly described here.
Basic Asset ManagementTo get the basic app working, I moved the public/stylesheets and public/javascripts to the new app/assets directories to start. I did not move the images out of the public/ directory because several of the images in the application are referenced by blog articles.
Database-less ApplicationI followed the directions here combined with a bit of troubleshooting to configure a Rails 3.1 app that does not require a database.
ConclusionThe upgrade was a relatively painless process, although it still took a few hours for even the most basic application with only a handful of controllers, routes, and one mailer. My experience suggests that with a more complex application, the upgrade will take at least a few hours, if not much more. This simple app doesn't do much with remote forms and links, so I didn't spend any time upgrading the app to work with the jquery-ujs gem. Also, I obviously didn't mess around with Rails 3.1 ActiveRecord issues since the application is database-less. Both of these items may add significant overhead to the upgrade process.
I spent a significant amount of time working with the new asset pipeline and restructuring the assets, which I plan to describe in Part 2 of the upgrade. Stay tuned!
CSS Fixed, Static Position Toggle
In a recent Rails project, I had to implement a simple but nifty CSS trick. A request came in to give a DOM element fixed positioning, meaning as the user navigates throughout the page, the DOM element stays in one place while the rest of the page updates. This is pretty common behavior for menu bars that show up along one of the borders of a window while a user navigates throughout the site. However, this situation was a bit trickier because the menu bar that needed fixed positioning was already a few hundred pixels down the page below header and navigation content:
I came up with a nifty way of using jQuery to toggle the menu bar CSS between fixed and static positioning. The code uses jQuery's scroll event handler to adjust the CSS position setting of the menu bar as users scroll through the page. If the window scroll position is below it's original top offset, the menu has fixed positioning at the top of the window. If the window scroll position is above it's original top offset, the menu has static positioning. Here's what the code looks like:
var head_offset = jQuery('#fixed_header').offset();
jQuery(window).scroll(function() {
if(jQuery(window).scrollTop() < head_offset.top) {
jQuery('#fixed_header').css({ position: "static"});
} else {
jQuery('#fixed_header').css({ position: "fixed", top: "0px" });
}
});
And perhaps the most effective demonstration of this behavior comes in the form of a video, created with Screencast-O-Matic. I also tried capturing with Jing, which is another handy tool for quick Screenshots and Screencasts. Note that the header content has CSS adjustments for demo purposes only:
jQuery and hidden elements
Out of sight, not out of mind.
While extending some jQuery functionality on a page, I noted that a form element's "change" handler wasn't being invoked, which meant some of the page initialization code would be left out. So I helpfully added it:
$('input[name=foobar]').change(...).change();
What I failed to contemplate was how this would impact another page, which just happened to have some (but not all) of the same form elements, and referenced the same JS code at page load. Specifically, my page's sibling had:
<input name="foobar" type="hidden">
And of course this broke in production.
Well, that's interesting. A change handler on a hidden form input field isn't usually all that useful, so I figured out I really needed:
$('input[name=foobar]').filter(':visible').change(...).change();
It happens that the ".filter()" step is actually more efficient than doing it all in one selector ("input[name=foobar]:visible"), because of some obscurities within jQuery. That little discovery was of value to an End Point co-worker, who realized she could shave a little time off a page load elsewhere, so my minor page malfunction will redeem itself.
Rails Optimization: Digging Deeper
I recently wrote about raw caching performance in Rails and advanced Rails performance techniques. In the latter article, I explained how to use a Rails low-level cache to store lists of things during the index or list request. This technique works well for list pages, but it doesn't necessarily apply to requests to an individual thing, or what is commonly referred to as the "show" action in Rails applications.
In my application, the "show" action loaded at ~200 ms/request with low concurrency, with the use of Rails fragment caching. And with high concurrency, the requests shot up to around 2000 ms/request. This wasn't cutting it! So, I pursued implementing full-page caching with a follow-up AJAX request, outlined by this diagram:
First, the fully-cached is loaded (quickly). Next, an AJAX request is made to retrieve access information. The access information returns a JSON object with information on whether or not there is a user, and if that user has edit access to that thing. If there is no user, the page stays as is. If there is a user, but he does not have edit permissions, the log out button is shown and the username is populated. If there is a user and he has edit permissions, the log out button is shown, the username is populated, and additional buttons requiring edit access are shown.
The Code
To cache the full page, I use the caches_page method, and cache only on requests of HTML format (other formats are not cached):
class ThingsController < ApplicationController
caches_page :show, :if => Proc.new { |c| c.request.format.html? }
...
end
My access level request looks something like this:
def accessibility
respond_to do |format|
format.json do
render :json => {
:logged_in => current_user ? current_user.to_json(:only => [:id, :username]) : false,
:can_edit => current_user ? Thing.find(params[:id]).can_edit?(current_user) : false }
end
end
end
My HTML has some bits of code sprinkled throughout it:
... <a href="#" id="edit_thing" class="requires_editability">Edit</a> ... <a href="#" id="my_account" class="requires_logged_in"><!-- no username yet --></a> ...
My jQuery AJAX request looks something like the code shown below. Note that I remove elements that do not apply to the current request:
$.ajax({
type: 'GET',
cache: false,
url: editability_path, //editability_path is defined in the HTML (a JavaScript variable)
dataType: "JSON",
error: function(xhr){
$('.require_editability,.require_loggged_in').remove();
},
success: function(results) {
if(results.logged_in) {
$('.require_logged_in').show();
$('#my_account').html(results.logged_in.username);
if(results.can_edit) {
$('.require_editability').show();
} else {
$('.require_editability').remove();
}
} else {
$('.require_editability,.require_loggged_in').remove();
}
}
});
And don't forget the sweeper to clear the fully cached page after edits (or other ActiveRecord callbacks):
class ThingSweeper < ActionController::Caching::Sweeper
observe Thing
def after_save(record)
expire_page :controller => :things, :action => :show, :id => record.id
end
end
Additional Notes
There are some additional notes to mention:
- If a user were to hack the AJAX or JavaScript, server-side validation is still being performed when an "edit" action is submitted. In other words, if a hacker somehow enabled an edit button to show up and post an edit, a server-side response would prohibit the update because the hacker does not have appropriate accessibility.
- HTML changes were made to accommodate this caching behavior, which was a bit tricky. HTML has to handle all potential use cases (no user, user & no edit access, user & edit access). jQuery itself can also be used to introduce new elements per use case.
- The access level AJAX request is also hitting more low-level Rails caches: For example, the array of things that a user has edit permissions is cached and the cache is cleared with standard Rails sweepers. With this additional caching component, the access level AJAX request is hitting the database minimally.
- Performance optimization scenarios such as this make an argument against inline editing of resources. If there were a backend admin interface to allow editing of things, full-page caching would be more straight-forward to implement.
Conclusion
With this functionality, fully cached pages are served with an average of less than 5 ms/request, and the AJAX accessibility request appears to be around 20 ms/request (although this is harder to test with simple command line tools). This is an improvement over the 200 ms/request initially implemented. Additionally, requests at a high concurrency don't bog down the system as much.
Company Presentation: jQuery and Rails
Yesterday, I gave a company presentation on jQuery and Rails. The talk covered details on how jQuery and Rails work together to build rich web applications, with a considerable amount of focus on AJAX methods. Check out the slides here:
One piece of knowledge I took away from the talk is how different the Rails 3 approach is for unobtrusive AJAX behavior using helpers like link_to_remote and remote_form_for. Mike Farmer made a recommendation to read the rails.js source here to see how onclick behavior is handled in Rails 3.
JavaScript and APIs at RailsConf 2011
A big trend of RailsConf 2011 has been the presence of JavaScript, whether it be talk of CoffeeScript, node.js, Sproutcore, backbone.js, etc. I attended Yehuda Katz's talk on Building Rails Apps for the Rich Client on Day 4 of the conference.
Part 1: Rails versus Sinatra Smackdown
Yehuda gave a two part talk about developing API-heavy applications. The first bit addressed why to develop in Rails rather than Sinatra if your application is API-heavy and doesn't appear to be utilizing valuable parts of Rails? This is fairly applicable to a couple of Sinatra projects that I've done at End Point — I like Sinatra a lot, but at some point you begin to replicate parts of Rails. Yehuda explained that because it's easy to develop web applciations efficiently using Rails conventions, developers can become forgetful/ignorant of the underlying functionality of Rails that doesn't ship with something like Sinatra, much like how working with an ORM can breed developers who aren't familiar with the underlying database interactions. The checklist for things that Rails manages we might forget about includes:
- ActionDispatch internals
- Session deserialization
- Browser standards mode
- Cookie abstraction and management
- Deserialization of content types like JSON, XML
- Reloading (by comparison Sinatra doesn't reload automagically unless you use a tool like shotgun)
- Handling IP Spoofing
- Routing / Complex Routing
- Browser Caching (ETags, Last-Modified)
- Content negotiation to automatically support different MIMEs
- ActionMailer
This list is is a nice checklist of items you might review when you are considering building in Sinatra to be reminded of things that won't necessarily be trivial to implement.
Part 2: Consistent APIs
The second part of Yehuda's talk covered his experience with building client rich applications with APIs. His general observation is that while Rails follows universal conventions like the ActiveRecord convention that requires foreign keys to be named {table_name}_id, there isn't a convention for APIs. While there has been decent support for building APIs in Rails since 2006, there's been missing documentation on what to generate which has producted APIs with limited consistency. He gave the following pro tips for developing an API:
- Return a JSON object with one or more keys, that is iterable. Rather than returning
{ product_title: "Some Awesome Product." }return something that has one or more keys that is iterable, such as:{ "products": [ { id: 1, title: "Some Awesome Product." } ] } - Avoid nested resources like /users/1/orders. Instead, always make the request do a GET on /orders or a PUT on /orders/1. This will help maintain consistency (and maintainability) with a simple asset reqeusts.
- In general, create a convention and follow conventions. Essentially, apply the principles of Rails here to produce maintainable and consistent code.
Finally, Yehuda presented the bulk_api gem to address the use case where you need to get a set of items, but want to avoid making single requests for multiple items and avoid getting all items and parse through them on the client side. An ecommerce example of this might be applicable in inventory management: Let's say you have ~30 unrelated listed inventory units that need to be updated to an on_hand state. From the admin screen, you may click on checkboxes next to those 30 items and then "Update". Rather than sending a single request to update each item, the bulk_api gem will make a single request to update the inventory unit status for all of the objects. Similarly, you can make a GET request to the bulk api to retrieve a set of objects, such as looking at a set of products on page 2 of a product listing. Yehuda went into more details on authentication and authorization with this gem, so I'd recommend reading the documentation here.
Conclusion
Yehuda concludes that Rails is great for JSON APIs and that we should continue to be building abstractions similar to ActiveRecord abstractions that follow conventions in a sensible way. Hooray for convention over configuration.
Locally served YUI3
For the vast majority of sites serving static JS and CSS files such as those required by YUI, or jQuery, etc. from a CDN makes great sense, improved performance through caching and geography, reduced load, improved uptime, leveraging some large corporations' resources, etc. Unfortunately as soon as you hit an SSL secured page you can't use a CDN's resources securely, a common thing with e-commerce sites and administration panels. In the YUI case that means doing at least a little bit of extra configuration and maintenance of an installed library base (an all too common task in typical server side development that's becoming more common as libraries are maintained for client side usage as well). Toss in "combo loading" and all of a sudden it feels like the software development cycle can't just be discarded cause you are working on the web, maybe this is really what they meant by web 2.0. But I digress...
Since I'm familiar with and working on YUI3, here is how we have it working for non-CDN based serving for an administration panel developed for a newer generation of an Interchange site. Our custom application uses YUI3 (core), modules from the YUI3 Gallery, and a few modules that are pulled in using the YUI 2in3 version compatibility layer. Now down to business...
Each of the libraries is provided by the YUI team through github.com. Since our project has git under it (you are using git, right?) we can set up submodules to help us with tracking new versions as they are released (or really be on the bleeding edge ahead of releases). To start we choose a location to serve the files, this is fairly arbitrary I happened to choose /vendor. Then set up new submodules for each of the libraries in use,
- git submodule add http://github.com/yui/yui3 vendor/yui3
- git submodule add http://github.com/yui/yui3-gallery vendor/yui3-gallery
- git submodule add http://github.com/yui/2in3 vendor/yui-2in3
Follow that with the normal git submodule init, git submodule update cycle to pull down the HEAD version of each library. At this point the library files are available for local serving, and the current version is pegged against the superproject.
With the files locally stored they can be served locally. To do so we need to adjust the "seed" file for the YUI core, the loader, and any CSS files loaded on the page. For instance,
<!-- YUI Seed, Loader, and our primary source --> <script type="text/javascript" src="/combo?vendor/yui3/build/yui/yui.js&vendor/yui3/build/loader/loader.js"></script>(For now ignore the "/combo" portion of this, more on that later.)
With the yui.js and loader.js files loading from a local resource YUI will automatically know where to look for other YUI3 core files. But for non-core files (such as gallery modules or 2in3 loaded modules) we need to provide more information to the loader through the YUI_config global variable.
The above configuration has parameters for both combo and non-combo loading. Lines 4-7 tell the loader where to find core modules explicitly, though the loader should default to these based on the location of the seed file. Starting on line 9 there are definitions for the additional groups of files that we now want locally served. The first group, lines 11-20, are needed to load our local gallery files. Lines 25-44, as the comment above them indicates, are for loading specific CSS files that were not properly handled by the first group, though I've yet to track down why (more robust skin support may fix this issue). Lines 46-62 bring in YUI 2-in-3, note the specific minor version of 2 is pegged in line 47 and 50. With this configuration in place, and properly located files on the filesystem you should be able to use any core, gallery, or 2in3 modules loaded from your local system.
But what about "combo loading." "Combo loading" allows us to reduce the number of roundtrip requests made to pull down the dependencies by concatenating a set of files together (either JS or CSS but not together), when serving from the CDN you get this because Yahoo! has provided a combo loading service. You can see a PHP based loading service provided by the YUI team in the phploader project, https://github.com/yui/phploader/ . Because we are working with Interchange and aren't using PHP I've written a similar script to be used as plain ole' boring CGI, it can be found at github. In the example code this script would be setup to run at the root of our URL space, specifically as "/combo" and the script itself will need to know where to find the local root, "vendor", on the physical file system. (It is important to note that combo loading CSS files takes some special parsing to make images and other referenced files work correctly. Also note that my version does not work with includes in the CSS.)
Finally one of the biggest benefits to this set up is being able to test literally any commit in any of the modules while leaving all else the same. This can allow you to determine when a bug first entered the picture (see git bisect) or test against development branches ahead of releases. To test against the latest and greatest between releases, you simply have to go to one of the submodule paths, fetch any recent updates, then check out the master (or any) branch, merge from the origin branch, and test. If all your tests pass, then you git add the submodule path in the superproject, git commit, and know that your stack is now up to date. Hopefully some day the gallery modules themselves will be git submodules so that maintaining a local gallery will not involve pulling down all modules and an application can peg (or update) a specific module as needed by the application, essentially you'd have your own gallery.
Virtual Page Tracking and Goals with Google Analytics
Sometimes I come across websites that don't use RESTful URLs or have too unique (with an order number) URLs during checkout and I need to implement Goal Tracking in Google Analytics on these user interactions. I've also had to implement Goal Tracking in a non-ecommerce web application where tabbed on-page browsing guides users through a 3-step process. Examples of situations that pose challenges to traditional page tracking in Google Analytics include:
- Throughout Interchange's checkout, URLs are posts to "/process", which makes the user interactions difficult to distinguish.
- Throughout Spree's checkout, URLs are posts to "/order/:id/edit", which are distinct and can be difficult to aggregate.
- In a Sinatra application we developed recently, the single page URL is "/locate.html", but tabbed browsing occurs through three unique steps.
Google Analytics Tagging
To add Goal Tracking by URL, pages must first be tagged as "virtual pages". To implement virtual page tracking in Google, it's as simple as including a new virtual page URL in the _trackPageview action:
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-XXXXXXXX-1']);
- _gaq.push(['_trackPageview']);
+ _gaq.push(['_trackPageview', '/cart/basket']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
In the cases I described above, here are the tags used for each:
| Interchange | Spree | Sinatra App |
On a three page checkout process, the following virtual pages were tagged:
|
On a multi-page checkout process, the following virtual pages were tagged:
|
On a three page process, the following virtual pages were tagged:
|
Goal Definition
After tagging is complete, you'll need to define a Goal in Google Analytics to begin tracking it.
|
First, you'll want to name your goal. Here I've named the goal "Purchase". In this case, the Goal is a URL Destination based on our new tags. The Goal URL is set to "/checkout/complete" for the Spree example described above. In this case, I have it set to a Regular Expression Match, but an Exact Match will work as well. I haven't assigned a Goal Value and I've left the match case insensitive for now. Next, I add the steps involved leading up to the goal URL. This will allow you to visualize a goal funnel and see where users may be abandoning checkout. In this example, each of the steps of the funnel correspond to a checkout step (shopping basket, optional register page, individual checkout steps, receipt/complete page). Note that this screenshot was taken from the new Google Analytics interface. The old Google Analytics interface may appear slightly different, but contains similar content. |
Looking at the Data
With virtual page tracking and goal tracking, you can see how many goals have been completed during a certain time frame:
You can also compare regular segments of traffic: e.g. users that found your site through search versus referrals:
You can compare advanced segments of visitors: e.g. marketing campaign #1 versus marketing campaign #2. This type of comparison may give you more actionable metrics to determine which marketing campaign to put more resources into.
Last but not least, you can visualize the funnel of conversion to see where you might be losing customers. This might also be actionable in that you may be able to A/B test checkout process (or another type of conversion) changes to improve conversion.
Lazy Image Loading in JavaScript with jQuery
This week I had a duh moment while working on a little jQuery-driven interface for a side hobby.
I've occasionally used or attempted to do image preloading in JavaScript, jQuery, and YUI. Preloading images happens after a page is loaded: follow-up image requests are made for images that may be needed, such as hover images, larger sizes of thumbnail images on the page, or images below the fold that do not need to load at page request time. Adobe Fireworks spits out this code for preloading images, which is a bit gross because the JavaScript is typically inline and it doesn't take advantage of common JavaScript libraries. But this is probably acceptable for standalone HTML files that get moved between various locations during design iterations.
<body onload="MM_preloadImages('/images/some_image.png','/images/some_other_image.png')">
I found many examples of preloading images with jQuery that look something like this:
jQuery.preloadImages = function()
{
for(var i = 0; i<arguments.length; i++)
{
jQuery("<img>").attr("src", arguments[i]);
}
}
I implemented this method, but in my code the preloading was happening asynchronously and I needed to find something that would execute some other behavior after the image was loaded. Before I found the solution I wanted, I tried using jQuery's get method and tested jQuery's ready method, but neither was suitable for the desired behavior. I came across jQuery's load event, which binds an event handler to the "load" JavaScript event and can be used on images. So, I came up with the following bit of code to lazily load images:
var img = $('<img>')
.attr('src', some_image_source);
$(element).append(img);
if($(img).height() > 0) {
// do something awesome
} else {
var loader = $('<img>')
.attr('src', 'images/ajax_loader.gif')
.addClass('loader');
$(element).append(loader);
$(img).load(function() {
// do something awesome (the same awesome thing as above)
loader.remove();
});
}
So my bit of code creates a new image element. If the image's height is greater than 0 because it's already been requested, it does some awesome method. If its height is 0, it displays an ajax loader image, then does the same awesome method and removes the ajax loader image. See the screenshot below to get an idea of how this is used.
The image on the left has been loaded and resized to fit its frame. The image to be displayed on the right is loading.
Interestingly enough, the code above works in IE 8, Chrome, and Firefox, but it appears that IE handles image loading a bit differently than the other two browsers — I haven't investigated this further. This lazy-image loading reduces unnecessary requests made to pre-load images that may or may not be accessed by users and the added touch of an ajax loader image communicates to the user that the image is loading. I haven't added a response for image load failure, which might be important, but for now the code makes the assumption that the images exist.
I found a few jQuery plugins for lazy image loading, but I think they might overkill in this situation. One of the jQuery plugins I found is based on YUI's ImageLoader, a utility that similarly delays loading of images.
jQuery and Long-Running Web App Processes: A Case Study
I was recently approached by a client's system administrator with a small but interesting training/development project. The sys-admin, named Rod, had built a simple intranet web application that used a few PHP pages, running on an Amazon EC2 instance, to accept some user input and kick off multiple long-running server side QA deployment processes. He wanted to use Ajax to start the process as well as incrementally display its output, line-by-line, on the same web page. However, waiting for the entire process to finish to display its output was a poor user experience, and he wasn't able to get an Ajax call to return any output incrementally over the lifetime of the request.
Rod asked me to help him get his web app working and train him on what I did to get it working. He admitted that this project was a good excuse for him to learn a bit of jQuery (a good example of keeping your tools sharp) even if it wasn't necessarily the best solution in this case. I have always enjoy training others, so we fired up Skype, got into an IRC channel, and dove right in.
First, I started with the javascript development basics:- Install the Firebug add-on for Firefox
- Use Firebug's Console tab to watch for javascript errors and warnings
- Use Firebug's Net tab to monitor what ajax calls your app is making, and what responses they are getting from the server
- Replace all those hateful debug alert() calls with console.log() calls, especially for ajax work
A special note about console.log(): Some browsers (including any Firefox that doesn't have Firebug installed) do not natively supply a console object. To work around this, we defined the following console.log stub at the very top of our single javascript file:
if (typeof console === 'undefined') {
console = { log: function() {} };
}
Rod's javascript was a mix of old-school javascript that he had remembered from years ago, and new-school jQuery he had recently pulled from various tutorials. His basic design was this: When the user clicked the "Deploy" button, it should kick off two separate Ajax requests; "Request A" would initiate a POST request to deploy.php. Deploy.php made a number of system calls to call slow-running external scripts, and logged their output to a temporary logfile on the server. "Request B" would make a GET request to getoutput.php every 2 seconds (which simply displayed the output of said logfile) and display its output in a scrollable div element on the page.
Hearing Rod describe it to me, I wondered if he might be headed down the wrong path with his design. But, he already had put time into getting the server-side code working and did not want to change direction at this point. Discussing it with him further, it became clear that he did not want to re-write the server-side code and that we could in fact make his current design produce working code with teachable concepts along the way.
To start, Rod told me that his "ajax POST request (Request A) wasn't firing." As the Russian proverb says, "Trust, but verify." So, we opened Firebug's Net tab, clicked the web app's Deploy button (actually its only button - Steve Jobs look out) and saw that the ajax request was in fact firing. However, it was not getting back a successful HTTP 200 status code and as such, was not getting handled by jQuery as Rod expected. Expanding the ajax request in the Net tab let us see exactly what name/value data that was getting POSTed. We spotted a typo in one of his form input names and fixed it. Now Request A was clearly firing, POSTing the correct data to the correct URL, and getting recognized as successful by jQuery. (More on this in a bit.)
Rod's code was making Request A from within a jQuery event handler defined for his form's Deploy button. But, he was making Request B via an HTML onClick attribute within that same HTML tag. He was getting all sorts of strange results with that setup based on which request was returning first, if Request B's function call was correctly returning false to prevent the entire form from getting POSTed to itself, etc. Consolidating logic and control into event handlers that are defined in one place is preferable to peppering a web page with HTML onClick, onChange, etc. javascript calls. So, we refactored his original jQuery event handler and onClick javascript call into the following code snippet:
//global variable for display_output() interval ID
var poll_loop;
$(".deploy_button").click(function() {
$.ajax({
beforeSend: function() {
$('#statusbox').html("Running deployment...");
},
type: "POST",
url: "deploy.php",
data: build_payload(),
success: function() {
console.log('Qa-run OK');
//previously called via an onClick
poll_loop = setInterval(function() {
display_output("#statusbox", 'getoutput.php');
}, 2000);
},
error: function() {
console.log('Qa-run failed.');
}
});
});
That $.ajax(...) call is our jQuery code that initiates the Request A ajax call and defines anonymous functions to call based on the HTTP status code of Request A. If Request A returns an HTTP 200 status code from the server, the anonymous function defined for the 'success:' key will be executed. If any other HTTP code is returned, the anonymous function defined for the (optional) 'error:' key is executed. We refactored the onClick's call to display_output() into the 'success:' function above. Now, it only gets called if Request A is successful, which is the only time we'd want it to execute.
The body of the 'success:' anonymous function calls setInterval() to create an asynchronous (in that it does not block other javascript execution) javascript loop that calls display_output() every 2 seconds. The setInterval() function returns an "interval ID" that is essentially a reference to that interval. We save that interval ID to the 'poll_loop' variable that we intentionally make global (by declaring it with 'var' outside any containing block) so we can cancel the interval later.
Here is the display_output() function that makes Request B and gets called every 2 seconds:
function display_output(elementSelector, sourceUrl) {
$(elementSelector).load(sourceUrl);
var html = $(elementSelector).html();
if (html.search("EODEPLOY") > 0) {
window.clearInterval(poll_loop);
alert('Deployment Finished.');
}
if (html.search("DEPLOY_ERROR") > 0) {
window.clearInterval(poll_loop);
alert('Deployment FAILED.');
}
}
That .load() method is jQuery shorthand for making an ajax GET request and assigning the returned HTML/text into the element object on which it's called. Because the display_output() function is responsible for terminating the interval that calls it, we need to define our end cases. If either "EODEPLOY" (for a successful deployment) or "DEPLOY_ERROR" (for a partially failed deployment) appear as strings within the resulting HTML, we call clearInterval() to stop the infinite loop, and alert the user accordingly. If neither of our end cases are encountered, display_output() will be executed again in 2 seconds.
As it stands, the poll_loop interval will run indefinitely if the server-side code somehow fails to ever return the two strings we're looking for. I left that end case as an exercise up to Rod, but suggested he add a global variable that could be used to measure the number of display_output() calls or the elapsed time since the Deploy button was clicked, and end the loop once an upper limit was hit.
Other suggested features that Rod and I discussed but I've omitted from this article include:- Client-side input validation using javascript regular expressions
- Matching server-side input validation because sometimes the call is coming from inside the house
- Adding a unique identifier that is passed as part of both Request A and Request B to better identify requests and to prevent temp file naming conflicts from multiple concurrent users.
- Packaging display_output()'s "Deployment FAILED" output and providing a button to easily send the output to Rod's team
I'm sure there are a ton of other possible solutions for a project like this. For example, I know that Jon and Sonny developed a more advanced polling solution for another client, www.locateexpress.com, using YUI's AsyncQueue. Without getting to deeply into the server-side design, I'm curious to hear how other people might approach this problem. What do you think?
YUI Extensions and Post Initialization
When using YUI3's provided extension mechanism to enhance (composite, mix in, role, whatever you like to call it, etc.) a Y.Base inherited base class, it is often helpful to have "post initialization" code run after the attributes' values have been set. The following code provides an easy way to hook onto a Y.Base provided attribute change event to run any post initialization code easily.
Using nginx to transparently modify/debug third-party content
In tracking down a recent front-end bug for one of our client sites, I found myself needing to use the browser's JavaScript debugger for stepping through some JavaScript code that lived in a mix of domains; this included a third-party framework as well as locally-hosted code which interfaced with -- and potentially interfered with -- said third-party code. (We'll call said code foo.min.js for the purposes of this article.) The third-party code was a feature that was integrated into the client site using a custom domain name and was hosted and controlled by the third-party service with no ability for us to change directly. The custom domain name was part of a chain of CNAMEs which eventually pointed to the underlying *actual* IP of the third-party service, so their infrastructure obviously relied on getting the Host header correctly in the request to select which among many clients was being served.
It appeared as if there was a conflict between code on our site and that imported by the third party service. As part of the debugging process, I was stepping through the JavaScript in order to determine what if any conflicts there were, as well as their nature (e.g., conflicting library definitions, etc.). Stepping through our code was fine, however the third-party's JS code was (a) unfamiliar, and (b) minified, so this had the effect of putting all of the JavaScript code more-or-less on one line, which made tracing through the code in the debugger much less useful than I had hoped.
My first instinct was to use a JavaScript beautifier to reverse the minification process, but since I had no control over the code being included from the third-party service, this did not seem to be directly feasible. The third-party code was deployed only on our production site and relied on hard-coded domains which would make integrating it into one of our development instances challenging since we had no control over the contents of the returned resources. Since the relevant feature (and subsequent bugs) was only on the production site, making extensive modifications to how things were done and potentially breaking that or other features for users while I was debugging was obviously out as an option.
Enter nginx. I've been doing a lot with nginx lately as far as using it as a reverse proxy cache, so it's been on my mind lately. So I came up with this technique:
- Look up the IP address for the third-party's domain name (used for later purposes).
- Install nginx on localhost, listening to port 80.
- Modify /etc/hosts to point the third-party's domain name to the nginx server's IP (also localhost in this case).
- Configure a new virtual host with the following logical constraints:
- We want to serve specific files (the beautified JavaScript) from our local server.
- We want any other request going through that domain to be passed-through transparently, so neither the browser nor the third-party server treat it differently.
Given these constraints, this is the minimal configuration that I came up with (the interesting parts are located in the server block):
/etc/hosts:
example.domain.com 127.0.0.1
nginx.conf:
worker_processes 1;
events {
worker_connections 10;
}
http {
include mime.types;
default_type application/octet-stream;
server {
server_name example.domain.com;
root /path/to/local_root;
try_files $uri @proxied;
location @proxied {
proxy_set_header Host $http_host;
proxy_pass http://1.2.3.4;
}
}
}
Once I had the above configured/setup, I downloaded/saved the foo.min.js file from the third-party service, ran it through a JS beautifier, and saved it in the local nginx's cache root so it would be served up instead of the actual file from the third-party service. Any other requests for static resources (images, other scripts, etc) would pass-through to the third-party server, so I had my nicely-formatted JavaScript code to step through, the production site worked as normal for anyone else despite potential local changes to the file on my end (i.e., adding JavaScript alert() calls to the file, and no one was the wiser.
A few notes
The try_files directive instructs nginx to first look for a file named after the current URI (foo.min.js in our example) in our local cache, and if this is not found, then fallback to the proxied location block; i.e., relay the request to the original upstream server. We explicitly set the Host header on the proxy request because we want the request to behave normally with respect to name-based hosting, and provide the saved IP address to contact the server in question.
We only needed to preserve/lookup the upstream server's IP address because we're running the nginx server on localhost, so if we used a domain name the lookup would return the same IP defined in /etc/hosts; if the nginx server was running on a different machine, you would be able to just use the domain name as both the server_name and the proxy_pass parameters and set the entry for the host in your local /etc/hosts file to the IP of the nginx server.
A possible extension would be to detect when an upstream request matched a minified URL (via a location ~ \.min\..*\.js$ block) and automatically beautify/cache the content in our local cache. This could be accomplished via the use of an external FastCGI script to retrieve, post-process, and cache the content.
This technique can also be used when dealing with testing changes to a production site on which you are unable or unwilling to make potentially disruptive changes for the purposes of testing static resources. (JavaScript seems the most obvious application here, but this could apply to serving up images or other static content which would be resolvable by the local cache.)
I always need to remind myself to undo changes to /etc/hosts as soon as I'm done testing when using tricks like these. Particularly in something like this which is more-or-less transparent, the behavior would be functionaly identical as long as code/scripts on the third-party site stayed the same, but could easily introduce subtle bugs if the third-party services made changes to their codebase. Since our local copies would mask any remote changes for those non-proxied resources, this could be very confusing if you forget that things are set up this way.
JSON pretty-printer
The other day Sonny and I were troubleshooting some YUI JavaScript code and looking at some fairly complex JSON. It would obviously be a lot easier to read if each nested data structure were indented, and spacing standardized.
I threw together a little Perl program based on the JSON man page:
#!/usr/bin/env perl
use JSON;
my $json = JSON->new;
undef $/;
while (<>) {
print $json->pretty->encode($json->decode($_));
}
It took all of 2 or 3 minutes and I even left out strictures and warnings. Living on the edge!
It turns a mess like this (sample from json.org):
{"glossary":{"title":"example glossary","GlossDiv":{"title":"S","GlossList":
{"GlossEntry":{"ID":"SGML","SortAs":"SGML","GlossTerm":"Standard Generalized Markup Language",
"Acronym":"SGML","Abbrev":"ISO 8879:1986","GlossDef":{"para":
"A meta-markup language,used to create markup languages such as DocBook.",
"GlossSeeAlso":["GML","XML"]},"GlossSee":"markup"}}}}}
into this much more readable specimen:
{
"glossary" : {
"GlossDiv" : {
"GlossList" : {
"GlossEntry" : {
"GlossDef" : {
"para" : "A meta-markup language,used to create markup languages such as DocBook.",
"GlossSeeAlso" : [
"GML",
"XML"
]
},
"GlossTerm" : "Standard Generalized Markup Language",
"ID" : "SGML",
"SortAs" : "SGML",
"Acronym" : "SGML",
"Abbrev" : "ISO 8879:1986",
"GlossSee" : "markup"
}
},
"title" : "S"
},
"title" : "example glossary"
}
}
But today I thought back to that and figured surely something like that must already be at hand if I'd just looked for it. Sure enough, there are many easy options that work conveniently from the shell, similarly to that script:
- json_xs (Perl JSON::XS)
- python -mjson.tool (Python 2.6+)
- prettify_json.rb (Ruby json gem)
And those were just the ones that were likely already on the machine I was using! Hooray for convenience.
jQuery Tips and an Ecommerce Demo
I've recently been jumping back and forth between YUI and jQuery on several different client projects. I prefer working with jQuery, but whenever I work with another framework, I realize what's out there and how I should continue to improve my skills in my preferred framework. I read up on jQuery tips and put together a summary of common tips I need to follow more explained here in an ecommerce demo.
The Setup
|
Before we get started, some notes on the ecommerce demo and performance testing:
|
A screenshot from the demo app.
|
1. The first tip I came across was a recommendation to use a for loop instead of jQuery's each method. I start off with some un-optimized code to test that loops through our products JSON object and renders the images to the page. In this case, the use of the for loop instead of each doesn't give us a significant performance difference.
for(var k=0;k<10;k++) {
$('div.products').html('');
console.time('test ' + k);
for(var j=0;j<10;j++) {
$.each(products, function(index, e) {
var ihtml = $('div.products').html();
ihtml += '<a href="#">'
+ '<img class="product" src="/images/'
+ e.image + '" /></a>';
$('div.products').html(ihtml);
});
}
console.timeEnd('test ' + k);
}
|
for(var k=0;k<10;k++) {
$('div.products').html('');
console.time('test ' + k);
for(var j=0;j<10;j++) {
for(var i=0;i<products.length; i++) {
var ihtml = $('div.products').html();
ihtml += '<a href="#">'
+ '<img class="product" src="/images/'
+ products[i].image + '" /></a>';
$('div.products').html(ihtml);
}
}
console.timeEnd('test ' + k);
}
|
| 10 tests average: 505ms | 10 tests average: 515ms |
Products displayed with a for loop instead of jQuery's each method.
2. The next "I Can't Believe I'm Not Using this jQuery Technique" I found was a recommendation to use data tag. Although I've read about the data tag, I haven't worked with it enough to use it consistently. With this code, we assign product data (name, price) to each product link as it’s added to the DOM. Upon clicking a product, instead of traversing through the products array, we render the "featured_product" contents based on it's data. The data tag is recommended over assigning values to arbitrary HTML tag attributes such as assigning our product <a> and <img> the name and price values to title or alt attributes.
for(var i=0;i<products.length; i++) {
var link = $(document.createElement('a'))
.attr(‘href’, ‘#’)
.html('<img class="product" src="/images/' + products[i].image + '" />')
.data('name', products[i].name)
.data('price', products[i].price);
if(products[i].featured) {
link.addClass('featured');
}
if(products[i].sale) {
link.addClass('sale');
}
$('div.products').append(link);
};
$('div.products a').click(function() {
$('div.featured_product')
.html('<h1>' + $(this).data('name') + '</h1>');
$(this).find('img').clone().appendTo('.featured_product');
});
|
jQuery's data tag is used to populate the right side of the page as a product is clicked.
3. The next tip I came across frequently was a recommendation to cache your selectors, shown in the example below. A selector $('div.products a.featured') is created and used when users like to view "Featured" Items. This gave me a 33% performance gain.
$('a.featured').click(function() {
for(var k=0;k<10; k++) {
console.time('test ' + k);
for(var j=0;j<100;j++) {
$('div.products a').css('background', '#FFF');
$('div.products a.featured').css('background', '#999');
}
console.timeEnd('test ' + k);
}
});
|
var all_products = $('div.products a');
var featured_products = $('div.products a.featured');
$('a.featured').click(function() {
for(var k=0;k<10; k++) {
console.time('test ' + k);
for(var j=0;j<100;j++) {
all_products.css('background', '#FFF');
featured_products.css('background', '#999');
}
console.timeEnd('test ' + k);
}
});
|
| 10 tests average: 31ms | 10 tests average: 21ms |
Multiple products added to test onclick for identifying "featured" products.
4. Another tip I came across was the recommendation on using context in jQuery selectors, described here in the jQuery documentation. In one example, I try updating the $('div.products') selector to set the context as $(‘div.wrapper’), but saw performance worsen here. In another example, I added "this" as a context for populating the featured product, but again saw performance worsen slightly here. In this case, performance gain will depend on your original selector, but it's worth testing.
for(var k=0;k<10;k++) {
$('div.products').html('');
console.time('test ' + k);
for(var j=0;j<10;j++) {
for(var i=0;i<products.length; i++) {
...
$('div.products').append(link);
}
}
console.timeEnd('test ' + k);
}
|
for(var k=0;k<10;k++) {
$('div.products').html('');
console.time('test ' + k);
for(var j=0;j<10;j++) {
for(var i=0;i<products.length; i++) {
...
$('div.products', $('div.wrapper')).append(link);
}
}
console.timeEnd('test ' + k);
}
|
| 10 tests average: 64ms | 10 tests average: 76ms |
5. Another common tip I came across that "I can't believe I don't follow" is to use an id selector instead of a class. I’ve admittedly read this several times, but again it's a practice that I sometimes forget about. In my ecommerce setup, I add a loop to add the products 10x to our DOM, and with a change of selector from 'div.products' to 'div#products', I saw a small performance improvement.
for(var k=0;k<10;k++) {
$('div.products').html('');
console.time('test ' + k);
for(var j=0;j<10;j++) {
for(var i=0;i<products.length; i++) {
...
$('div.products').append(link);
}
}
console.timeEnd('test ' + k);
}
|
for(var k=0;k<10;k++) {
$('div.products').html('');
console.time('test ' + k);
for(var j=0;j<10;j++) {
for(var i=0;i<products.length; i++) {
...
$('div#products').append(link);
}
}
console.timeEnd('test ' + k);
}
|
| 10 tests average: 64ms | 10 tests average: 60ms |
6. I found a recommendation to minimize the DOM minimally. This is a tip that I typically follow, but another one that's easily forgotten. In our ecommerce setup, I create a single point of appending to the div#products selector after generating my product links and data. This gave a ~25% performance gain.
for(var k=0;k<10;k++) {
$('div.products').html('');
console.time('test ' + k);
for(var j=0;j<10;j++) {
for(var i=0;i<products.length; i++) {
...
$('div#products').append(link);
}
}
console.timeEnd('test ' + k);
}
|
for(var k=0;k<10;k++) {
$('div#products').html();
console.time('test ' + k);
var collection = $(document.createElement('div'));
for(var j=0;j<10;j++) {
for(var i=0;i<products.length; i++) {
...
collection.append(link);
}
}
$('div#products').append(collection);
console.timeEnd('test ' + k);
}
|
| 10 tests average: 60ms | 10 tests average: 45ms |
7. Another tip I came across was to use event delegation in jQuery. The idea is that your event has more context to work with than general selectors. You can access the event and manipulate it’s parent. I found that $(e.target).parent() is the same as manipulating $(this) performance-wise. It’s likely that using one or the other is much faster than using a general DOM selector such as $(div.product' + id).
var featured_product = $('div#featured_product');
$('div#products a').click(function(e) {
var product = $(e.target).parent(); // same as $(this)
featured_product
.html('<h1>' + product.data('name') + '</h1>')
.append(product.find('img').clone());
});
|
8. One tip that I've never seen before is to use ".end()" in chaining. Instead of reselecting the $('div.products') region, I use "find()" to apply a css style, then traverse up the chain to find another set of products to apply a css style, and repeat. This change gave a small performance bump, but you might tend to use the cached selectors described in Tip #3 instead of the following.
$('a#special').click(function() {
for(var k=0;k<10;k++) {
console.time('test ' + k);
for(var j=0;j<100; j++) {
$('div#products').find('a').css('background', '#FFF');
$('div#products').find('.featured').css('background', '#999');
$('div#products').find('.sale').css('background', '#999');
}
console.timeEnd('test ' + k);
}
});
|
$('a#special').click(function() {
for(var k=0;k<10;k++) {
console.time('test ' + k);
for(var j=0;j<100; j++) {
$('div#products')
.find('a')
.css('background', '#FFF')
.end()
.find('.featured')
.css('background', '#999')
.end()
.find('sale')
.css('background', '#999');
}
console.timeEnd('test ' + k);
}
});
|
| 10 tests average: 47ms | 10 tests average: 37ms |
Featured and Sale products are highlighted here.
9. I found several examples of writing your own selectors and how easy it is! I wrote two selectors with the following to identify products under $20.00 and products over $1000.00.
$(function() {
$('a#under20').click(function() {
all_products.css('background', '#FFF');
$('div#products a:under20').css('background', '#999');
});
$('a#over1000').click(function() {
all_products.css('background', '#FFF');
$('div#products a:over1000').css('background', '#999');
});
});
$.extend($.expr[':'], {
under20: function(a) {
if($(a).data('price') && $(a).data('price') < 20) {
return true;
}
return false;
},
over1000: function(a) {
if($(a).data('price') && $(a).data('price') > 1000) {
return true;
}
return false;
}
});
|
Products priced over $1000 are highlighted with a custom selector.
10. I also found several examples of how to write your own chain methods. In this example, I create two chain methods to set the product background to white or grey and update my under20 & over1000 methods to use this new chain method. The nice thing about creating your own chain methods is that these methods can be easily modified in the future with minimal code changes because it follows the DRY principle. I'm not sure if it's intended, but the use of my custom chain method did not work with the end() chain method described in Tip #8.
$.fn.unhighlight_product = function() {
return $(this).css('background', '#FFF');
}
$.fn.highlght_product = function() {
return $(this).css('background', '#999');
}
$(function() {
$('a#under20').click(function() {
all_products.unhighlight_product();
$('div#products a:under20').highlight_product();
});
$('a#over1000').click(function() {
all_products.unhighlight_product();
$('div#products a:over1000').highlight_product();
});
});
|
Products priced under $20 are highlighted with a custom selector and custom chain method.
Again, the product images for this demo app are from DryIcons.com and the final application code can be found here. The application was also deployed on Heroku to verify that the code works in IE [8], Chrome, and Firefox.
In Case You are Interested
During development of the demo, I found a significant performance differences between Chrome and Firefox. See the following tests:
- Tip #1: 6200ms (FF) vs 515ms (Chrome)
- Tip #3: 106ms (FF) vs 21ms (Chrome)
- Tip #4: 258ms (FF) vs 60ms (Chrome)
- Tip #6: 169ms (FF) vs 45ms (Chrome)
- Tip #8: 154ms (FF) vs 37ms (Chrome)
Or in visual form:
New Year Bug Bites
Happy New Year! And what would a new year be without a new year bug bite? This year we had one where figuring out the species wasn't easy.
On January 2nd one of our ecommerce clients reported that starting with the new year a number of customers weren't able to complete their web orders because of credit card security code failures. Looking in the Interchange server error logs we indeed found a significant spike in the number of CVV2 code verification failures (Payflow Pro gateway error code "114") starting January 1st.
We hadn't made any programming or configuration changes on the system in the recent days. We double-checked to make sure: nope, no code changes. So it had to be a New Year's bug and presumably something with the Payflow Pro gateway or banks further upstream. We checked error logs for other customers to see if they were being similarly impacted, but they weren't. Our client contacted PayPal (the vendor for Payflow Pro) and they reported there were no problems with their system. The failures must indeed be card failures or a problem with the website according to them. We further checked our code looking for what we could possibly have done that might be the cause, double-checking our Git repository (which showed no recent changes) and reexamining our checkout code for possible year-based logic flaws.
Our client's top-notch customer service group got on the phone with a customer who'd gotten a security code failure and got PayPal tech support on another line. The customer service rep tried to place the customer's order on the website using the customer's credit card info and once again got the CVV2 error. She then did the credit card transaction using the swipe machine in the office, and lo and behold the order went through! What was going on??!
It turned out that despite the Payflow Pro gateway returning CVV2 verification errors what was really happening was that the year of the credit card was coming into the Payflow Pro gateway as "2012"—not as "2011" as entered into the checkout form. We knew all along that it was possible that the 114 error code responses were possibly misleading because payment gateway error codes are notorious this way. (Payment gateways blame the banks, saying they can only pass along what the banks give them. Some banks' credit card validations don't actually even care about the years being correct, but just that they not be in the past; but I digress...)
Previously we'd reviewed the checkout pages and the dropdown menus to verify that the dropdown menus weren't off, but nevertheless it very much sounded like this rather stupid problem could very well be the culprit. So we checked and checked again. What we found is that sometimes on the checkout form the year dropdown menu was mangled such that the values associated with the displayed years were YYYY+1.
The oddly intermittent behavior of the problem, the process of elimination and the all around hair pulling this loss of business was causing made somebody in the marketing group at our client realize that they are in fact still running an Omniture Test & Target A/B test on the checkout pages that they thought had been discontinued. To quote David Christensen (thanks, David!): "The Omniture system works by replacing select content for randomly chosen users in an effort to track user behavior/response to proposed site changes. Alternate site content is created and dynamically replaced for these users as they use the site, such as the specific content on the checkout page in this instance."
We verified that the Omniture A/B test's JavaScript replacement code was alternately mangling and not mangling the year dropdown on the checkout form as mentioned. Our client took out the A/B test and the "security code errors" dropped back to a normal low level.
This was a difficult and expensive problem—not only was there business lost because of the problem, but there were a lot of resources put into troubleshooting it. We've come away from this episode with some lessons learned and with plenty of food for thought. I'll leave it to commentators to opine away on this, including the End Point folks who scratched this itch: David Christensen, Jeff Boes, Mark Johnson, and Jon Jensen.
jQuery code for making a block level element clickable while maintaining left/middle/right clicking
While working on a recent redesign for a client we needed the ability to click on a div and have it function as a link to a product page. The initial implementation used the jQuery plugin BigTarget.js. The plugin searches within the div for a link and when the div is clicked changes the location to the link that is found. This plugin worked fine and was fairly easy to setup, however there was one drawback that we found once it was released in the wild. Most people expect to be able to right click, middle click, shift click, and control click to get the context-sensitive menu, open in a new window, or open in a new tab.
Enter Superlink.js, a jQuery plugin that uses a cool trick of moving the location of a link to the location of the mouse within the block level element (such as a div, li, tr, td, etc.). With this implementation the mouse is actually over a link so that the various ways of clicking function as expected. Initially I started moving the clickable area within a table, as shown in the example, but then quickly realized there was no reason this shouldn't work within a div or li. One other neat thing with this plugin is that the events attached to the block will continue to function, they are passed to the event handlers for the link being moved within the block.
Utah Open Source Conference 2010 part 1
It's been about a little over a month since the 2010 Utah Open Source Conference, and I decided to take a few minutes to review talks I enjoyed and link to my own talk slides.
Magento: Mac Newbold of Code Greene spoke on the Magento ecommerce framework for PHP. I've somewhat familiar with Magento, but a few things stood out:
- He finds the Magento Enterprise edition kind of problematic because Varien won't support you if you have any unsupported extensions. Some of his customers had problems with Varien support and went back to the community edition.
- Magento is now up to around 30 MB of PHP files!
- As I've heard elsewhere, serious customization has a steep learning curve.
- The Magento data model is an EAV (Entity-Attribute-Value) model. To get 50 columns of output requires 50+ joins between 8 tables (one EAV table for each value datatype).
- There are 120 tables total in default install -- many core features don't use the EAV tables for performance reasons.
- Another observation I've heard in pretty much every conversation about Magento: It is very resource intensive. Shared hosting is not recommended. Virtual servers should have a minimum of 1/2 to 1 GB RAM. Fast disk & database help most. APC cache recommended with at least 128 MB.
- A lot of front-end things are highly adjustable from simple back-end admin options.
- Saved credit cards are stored in the database, and the key is on the server. I didn't get a chance to ask for more details about this. I hope it's only the public part of a public/secret keypair!
It was a good overview for someone wanting to go beyond marketing feature lists.
Node.js: Shane Hansen of Backcountry.com spoke on Node, comparing it to Tornado and Twisted in Python. He calls JavaScript "Lisp in C's clothing", and says its culture of asynchronous, callback-driven code patterns makes Node a natural fit.
Performance and parallel processing are clearly strong incentives to look into Node. The echo server does 20K requests/sec. There are 2000+ Node projects on GitHub and 500+ packages in npm (Node Package Manager), including database drivers, web frameworks, parsers, testing frameworks, payment gateway integrations, and web analytics.
A few packages worth looking into further:
- express - web microframework like Sinatra
- Socket-IO - Web Sockets now; falls back to other things if no Web Sockets available
- hummingbird - web analytics, used by Gilt.com
- bespin - "cloud JavaScript editor"
- yui3 - build HTML via DOM, eventbus, etc.
- connect - like Ruby's Rack
I haven't played with Node at all yet, and this got me much more interested.
Metasploit: Jason Wood spoke on Metasploit, a penetration testing (or just plain penetrating!) tool. It was originally in Perl, and now is in Ruby. It comes with 590 exploits and has a text-based interactive control console.
Metasploit uses several external apps: nmap, Maltego (proprietary reconnaissance tool), Nessus (no longer open source, but GPL version and OpenVAS fork still available), Nexpose, Ratproxy, Karma.
The reconnaissance modules include DNS enumeration, and an email address collector that uses the big search engines.
It can backdoor PuTTY, PDFs, audio, and more.
This is clearly something you've got to experiment with to appreciate. Jason posted his Metasploit talk slides which have more detail.
So Many Choices: Web App Deployment with Perl, Python, and Ruby: This was my talk, and it was a lot of fun to prepare for, as I got to take time to see some new happenings I'd missed in these three languages communities' web server and framework space over the past several years.
The slides give pointers to a lot of interesting projects and topics to check out.
My summary was this. We have an embarrassment of riches in the open source web application world. Perl, Python, and Ruby all have very nice modern frameworks for developing web applications. They also have several equivalent solid options for deploying web applications. If you haven't tried the following, check them out:
That's about half of my notes on talks, but all I have time for now. I'll cover more in a later post.
Cross Browser Development: A Few CSS and JS Issues
Coding cross browser friendly JavaScript and CSS got you down? In a recent project, Ron, David, and I worked through some painful cross browser issues. Ron noted that he even banged his head against the wall over a couple of them :) Three of these issues come up frequently in my other projects full of CSS and JS development, so I wanted to share.
Variable Declaration in JS
In several cases, I noticed that excluding variable declaration ("var") resulted in broken JavaScript-based functionality in IE only. I typically include variable declaration when I'm writing JavaScript. In our project, we were working with legacy code and conflicting variable names may have be introduced, resulting in broken functionality. Examples of before and after:
| Bad | Better |
|---|---|
var display_cart_popup = function() {
popup_id = '#addNewCartbox';
left = (parseInt($(window).width()) - 772) / 2;
...
};
|
var display_cart_popup = function() {
var popup_id = '#addNewCartbox';
var left = (parseInt($(window).width()) - 772) / 2;
...
};
|
... address_display = ''; country = $(type+'_country').value; address = $(type+'_address').value; address2 = $(type+'_address2').value; city = $(type+'_city').value; state = $(type+'_state').value; zip = $(type+'_zip').value; ... |
... var address_display = ''; var country = $(type+'_country').value; var address = $(type+'_address').value; var address2 = $(type+'_address2').value; var city = $(type+'_city').value; var state = $(type+'_state').value; var zip = $(type+'_zip').value; ... |
I researched this to gain more insight, but I didn't find much except a reiteration that when you create variables without the "var" declaration, they become global variables which may have resulted in conflicts. However, all the "learning JavaScript" documentation I browsed through includes variable declaration and there's no reason to leave it out for these lexically scoped variables.
Trailing Commas in JSON objects
According to JSON specifications, trailing commas are not permitted (e.g obj = { "1" : 2, }). From my experience, JSON objects with trailing commas might work in Firefox and WebKit browsers, but it dies silently in IE. Some recent examples:
| Bad | Better |
|---|---|
|
//JSON response from an ajax call
{
"response_message" : '<?= $response_message ?>',
"subtotal" : <?= $subtotal ?>,
"shipping_cost" : <?= $shipping ?>,
"carttotal" : <?= $carttotal ?>,
<?php if($add_taxes) { ?>
"taxes" : <?= $taxes ?>
<?php } ?>
}
|
//JSON response from an ajax call
{
"response_message" : '<?= $response_message ?>',
"subtotal" : <?= $subtotal ?>,
"shipping_cost" : <?= $shipping ?>,
<?php if($add_taxes) { ?>
"taxes" : <?= $taxes ?>,
<?php } ?>
"carttotal" : <?= $carttotal ?>
}
|
|
//Page load JSON object defined
var fonts = {
[loop list=`$Scratch->{fonts}`]
'[loop-param name]' : {
'bold' : "[loop-param bold]",
'italic' : "[loop-param italic]"
},[/loop]
};
|
//Page load JSON object defined
var fonts = {
[loop list=`$Scratch->{fonts}`]
'[loop-param name]' : {
'bold' : "[loop-param bold]",
'italic' : "[loop-param italic]"
},[/loop]
'dummy' : {}
};
|
Additional solutions to avoid the trailing comma include using join (Perl, Ruby) or implode (PHP), conditionally excluding the comma on the last element of the array, or using library methods to serialize data to JSON.
Floating Elements in IE
Often times, you'll get a design like the one shown below. There will be a static width and repeating components to span the entire width. You may programmatically determine how many repeating elements will be displayed, but using CSS floating elements yields the cleanest code.
Example of a given design with repeating elements to span a static width.
You start working in Chrome or Firefox and apply the following CSS rules:
CSS rules for repeating floating elements.
When you think you're finished, you load the page in IE and see the following. Bummer!
Floating elements wrap incorrectly in IE.
This is a pretty common scenario. In IE, if the combined widths of consecutive floating elements is greater than or equal to 100% of the available width, the latter floating element will jump down based on the IE float model. Instead of using floating elements, you might consider using tables or CSS position rules, but my preference is to use tables only for elements that need vertical align settings and to stay away from absolute positioning completely. And I try to stay away from absolute positioning in general.
The simplest and minimalist change I've found to work can be described in a few steps. Let's say your floating elements are <div>'s inside a <div> with an id of "products":
<div id="products"> <div class="product">product 1</div> <div class="product">product 2</div> <div class="product" class="last">product 3</div> <div class="product">product 4</div> <div class="product">product 5</div> <div class="product" class="last">product 6</div> </div>
And let's assume we have the following CSS:
<style>
div#products { width: 960px; }
div.product { float: left; width: 310px; margin-right: 15px; height: 100px; }
div.last { margin-right: 0px; }
</style>
Complete these steps:
- First, add another div to wrap around the #products div, with an id of "outer_products"
- Next, update the 'div#products' width to be greater than 960 pixels by several pixels.
- Next, add a style rule for 'div#outer_products' to have a width of "960px" and overflow equal to "hidden".
Yielding:
<div id="outer_products">
<div id="products">
<div class="product">product 1</div>
<div class="product">product 2</div>
<div class="product" class="last">product 3</div>
<div class="product">product 4</div>
<div class="product">product 5</div>
<div class="product" class="last">product 6</div>
</div>
</div>
And:
<style>
div#outer_products { width: 960px; overflow: hidden; }
div#products { width: 980px; }
div.product { float: left; width: 310px; margin-right: 15px; height: 100px; }
div.last { margin-right: 0px; }
</style>
The solution is essentially creating a "display window" (outer_products), where overflow is hidden, but the contents are allowed to span a greater width in the inside <div> (products).
The white border outlines the outer_products "display window".
Some other issues that I see less frequently include the double-margin IE6 bug, chaining CSS in IE, and using '#' vs. 'javascript:void(0);'.
Simple audio playback with Yahoo Mediaplayer
Recently I had need to show a list of MP3 files with a click-to-play interface.
I came upon a very simple self-contained audio player:
<script type="text/javascript" src="http://mediaplayer.yahoo.com/js"></script>
The code to set up my links for playing was dirt-simple:
<script type="text/javascript">
var player = document.getElementById('player');
function add_to_player() {
var link = this;
player.src.replace(/audioUrl=.*/,'audioUrl=' + link.src);
return false;
}
var links = document.getElementsByTagName('A');
for (var i = 0; i < links.length; i++) {
if (links[i].src.match(/\.mp3$/)) {
links.onclick = add_to_player;
}
}
</script>
You could use various ways to identify the links to be player-ized, but I chose to just associate the links with a class, "mp3":
<a class="mp3" href="/path/to/file.mp3">Audio File 1</a>
Obviously, if jQuery is in use for your page, you can reduce the code to an even smaller snippet.
Debugging jQuery
A recent reskin project for a client requires that we convert some of their old Prototype code to jQuery as well as create new jQuery code. I have not done much with converting Prototype to jQuery and I felt like my debugging tools for JavaScript were under par. So this morning I set out to find what was available for jQuery and I found this article on the subject.
I've used Firebug for some time now, but was unaware of some of the supporting plugins that would certainly help with debugging JavaScript. Some of the plugins and methods found in the article that I found immediately helpful were:
- FireFinder: Makes it quite easy to verify that the selector values in your code are correct and that the proper elements are returned. I was able to immediately pinpoint problems with my selectors and this brought to light why certain events were not firing.
- Firebug Console: Using the console.log function allowed me to check values without littering my code with alert statements.
- FireQuery: At a glance this plugin for Firebug shows which elements have event handlers bound to them.
- Firebug Breakpoints: Setting breakpoints and watch statements in your code makes it easier to see what is happening in the JavaScript code as it is executed instead of trying to figure out what happened after the code has run its course.
Thanks to the author of the article, Elijah Manor, for the in-depth information on debugging jQuery code.
jQuery Auto-Complete in Interchange
"When all you have is a hammer, everything looks like a nail."
Recently, I've taken some intermediate steps in using jQuery for web work, in conjunction with Interchange and non-Interchange pages. (I'd done some beginner stuff, but now I'm starting to see nails, nails, and more nails.)
Here's how easy it was to add an auto-complete field to an IC admin page.
In this particular application, a <select> box would have been rather unwieldy, as there were 400+ values that could be displayed.
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"
type="text/javascript"></script>
<script type="text/javascript" src="http://dev.jquery.com/view/trunk/plugins/autocomplete/lib/jquery.bgiframe.min.js"></script>
<script type="text/javascript" src="http://dev.jquery.com/view/trunk/plugins/autocomplete/lib/jquery.dimensions.js"></script>
<script type="text/javascript" src="http://dev.jquery.com/view/trunk/plugins/autocomplete/jquery.autocomplete.js"></script>
That's the requisite header stuff. Then you set up the internal list of autocomplete terms:
<script type="text/javascript">
$('document').ready(function(){
var auto_list = "[perl]...[/perl]".split(" ");
$('input[name="auto_field"]').autocomplete(auto_list);
});
</script>
The [perl] block just needs to emit a space-delimited list of the autocomplete terms. For instance,
[perl table="foo"]
return join(' ', map { $_->[0] }
@{ $Db{foo}->query('SELECT keycol FROM foo ORDER BY 1') });
[/perl]
jQuery UI Sortable Tips
I was recently tasked with developing a sorting tool to allow Paper Source to manage the sort order in which their categories are displayed. They had been updating a sort column in a database column but wanted a more visual aspect to do so. Due to the well-received feature developed by Steph, it was decided that they wanted to adapt their upsell interface to manage the categories. See here for the post using jQuery UI Drag Drop.
The only backend requirements were that the same sort column was used to drive the order. The front end required the ability to drag and drop positions within the same container. The upsell feature provided a great starting point to begin the development. After a quick review I determined that the jQuery UI Sortable function would be more favorable to use for the application.
Visual feedback was used to display the sorting in action with:
// on page load
$('tr.the_items td').sortable({
opacity: 0.7,
helper: 'clone',
});
// end on page load
Secondly I reiterate "jQuery UI Event Funtionality = Cool"
I only needed to use one function for this application to do the arrange the sorting values once the thumbnail had been dropped. This code calls a function which loops through all hidden input variables on the page and updates the sorting order.
// on page load
$('tr.the_items td').sortable({
stop: function(event, ui) { do_drop(this); },
});
// end on page load
Validating the sorting fields was a little different from the previously developed feature in that the number of available items could change depending on the category. The number of items could easily be 3 or 30. Therefore I needed a quick way to check the ever changing number. I decided to use a nested loop using the each function.
$('input.new_sku').each(
function( intIndex, obj ) {
$('input.new_sku').each(
function( secIndex, secObj ) {
if( (intIndex != secIndex) && ($(obj).val() == $(secObj).val()) ) {
error = true;
}
});
}
);
The rest of the feature uses some of the same logic previously documented here.
All in all I learned that the jQuery UI is very versatile and a pleasure to work with. I hope to be using more of its features in the near future.
jQuery UI Drag Drop Tips and an Ecommerce Example
This week, I implemented functionality for Paper Source to allow them to manage the upsell products, or product recommendations. They wanted a better way to visualize, organize, and select the three upsell products for every product. The backend requirements of this functionality were relatively simple. A new table was created to manage the product upsells.
The frontend requirements were more complex: They wanted to be able to drag and drop products into the desired upsell position (1, 2, or 3). I was allowed a bit of leeway on the interactivity level of the functionality, but the main requirement was to have drag and drop functionality working to provide a more efficient way to manage upsells. A mockup similar to the image shown below was provided at the onset of the project.
The mockup provided did not demonstrate the "interactiveness" of the drag and drop functionality. Items below the current upsells were ordered by cross sell revenue, or the revenue of each related item purchased with the current item.
Since I was familiar with jQuery, I knew that the jQuery UI included drag and drop functionality. I also had heard of several other jQuery drag and drop plugins, but since the jQuery UI is well supported, I was hopeful that the UI would have the functionality that I envisioned needing. Throughout the project, I learned a few valuable tips to consider with drag and drop implementation. To begin development, I downloaded the latest jQuery and UI Core in addition to the draggable and droppable UI components.
Visual Feedback = Helpful
The first thing I learned from working on the drag and drop functionality, was that visual feedback is very helpful in interactive design and that the jQuery UI has functionality built in to provide visual feedback. The first bit of visual feedback I included was to use a "clone" helper with semi-opaque styling to provide visual feedback that the object was being dragged. This was accomplished using the following code:
// on page load
$('div.common_item').draggable({
opacity: 0.7,
helper: 'clone'
});
// end on page load
And is shown here as the Lake Peace 1.25" Circle Stickers product is dragged:
The second bit of visual feedback I included was adding a class to the droppable item when a draggable item hovered over it. I added the "hoveringover" class to the droppable item which was defined in by the stylesheet to have a different colored background. This was accomplished using the following code:
// on page load
$('tr.upsells td').droppable({
hoverClass: 'hoveringover'
});
// end on page load
And is shown here as the Shimmer Silver A7 Envelope product hovers above the Quilt on Night with Curry A2 Stationers in upsell position #2:
jQuery UI Event Functionality = Useful
The second tip I learned from working on the drag and drop functionality was that the jQuery drag and drop UI includes valuable event functionality to manage events during the drag and drop process.
By adding the code shown below, at the initiation of dragging, I set a hidden input variable to track which element was being dragged. This value was later used to populate the product upsell form.
// on page load
$('div.common_item').draggable({
start: function(event, ui) { $('input#is_dragging').val($(this).attr('id')); }
});
// end on page load
By adding the code shown below, at the conclusion of dragging, I cleared the hidden input variable that indicated which item was being dragged.
// on page load
$('div.common_item').draggable({
stop: function(event, ui) { $('input#is_dragging').val(''); }
});
// end on page load
A final event response was added to be called when an item is dropped on a droppable item. The function do_drop is called at this drop time. The do_drop function replaces the html of the current upsells if the dropped sku is different than the current upsell sku, updates the hidden form element, adds visual feedback by adding a class to show that the item had been replaced, and displays the "Save" and "Revert" options to save to database or revert the upsell items.
// on page load
$('tr.upsells td').droppable({
drop: function(event, ui) { do_drop(this); }
});
// end on page load
var do_drop = function(obj) {
var current_sku = $('input#is_dragging').val();
if(current_sku != $(obj).find('img').attr('class')) {
//show "Save" and "Revert" options
show_drag_form();
//update hidden form element
$('input#' + $(obj).attr('id').replace('td_', '')).val(current_sku);
//replace html and add visual feedback by adding a class to show that the item was replaced
$(obj).html($('div#' + current_sku).html()).addClass('replaced');
}
};
Shown below, the Curry Dots A9 Printable Party Invitations have been replaced with the Olive Natsuki Gel Roller and the background color change signifies the item has been modified.
jQuery UI Documentation and Examples = Awesome
I found the jQuery UI documentation and examples to be very helpful. Another jQuery UI draggable component that was used was to force draggable items to be contained to a region on the page. I contained the elements to the entire parent table using the following code.
$('div.common_item').draggable({
containment: 'table#drag_table'
});
The Envelope Liners product is shown below to be confined to the table that contained potential and current upsell products. I could not drag the Envelope Liners any further to the right.
Because the functionality was a backend admin tool, the client requested that the functionality not be over-engineered to work across browsers. I did, however, verify that the drag and drop functionality worked in Firefox, Internet Explorer 7 and 8, Chrome, and Safari with a small amount of styling tweaking.
The final drag-drop JavaScript initiation is similar to the following code:
$(function() {
$('div.common_item').draggable({
opacity: 0.7,
helper: 'clone',
start: function(event, ui) { $('input#is_dragging').val($(this).attr('id')); },
stop: function(event, ui) { $('input#is_dragging').val(''); },
containment: 'table#drag_table'
});
$('tr.upsells td').droppable({
hoverClass: 'hoveringover',
drop: function(event, ui) { do_drop(this); }
});
})
Shown below is an example of the product upsell in action for the Chrysanthemum Letterpress Thank You Notes.
New End Point Site Launched: Rails, jQuery, Flot, Blogger
This week we launched a new website for End Point. Not only did the site get a facelift, but the backend content management system was entirely redesigned.
Goodbye Old Site:
Hello New Site:
Our old site was a Rails app with a Postgres database running on Apache and Passenger. It used a custom CMS to manage dynamic content for the bio, articles, and service pages. The old site was essentially JavaScript-less, with the exception of Google Analytics.
Although the new site is still a Rails application, it no longer uses the Postgres database. As developers, we found that it is more efficient to use Git as our "CMS" rather than developing and maintaining a custom CMS to meet our ever-changing needs. We also trimmed down the content significantly, which further justified the design; the entire site and content is now comprised of Rails views and partial views. Also included in the new site is cross browser functioning jQuery and flot. Some of the interesting implementation challenges are discussed below.
jQuery Flot Integration
The first interesting JavaScript component I worked on was using flot to improve interactivity to the site and to decrease the excessive text that End Pointers are known for [for example, this article]. Flot is a jQuery data plotting tool that contains functionality for plot zooming, data interactivity, and various configuration display settings (see more flot examples). I've used flot before in several experiments but had yet to use it on a live site. For the implementation, we chose to plot our consultant locations over a map of the US to present our locations in an interactive and fun to use way. The tedious part of this implementation was actually creating the datapoints to align with cities. Check out the images below for examples.
Flot has built in functionality for on hover events. When a point on the plot is hovered over, correlating employees are highlighted using jQuery and their information is presented to the right of the map.
When a bio picture is hovered over, the correlating location is highlighted using jQuery and flot data point highlighting.
We also implemented a timeline using flot to map End Point's history. Check out the images below.
When a point on the plot is hovered over, the history details are revealed in the section below.
The triangle image CSS position is adjusted when a point on the plot is activated.
Dynamic Rails Partial Generation
One component of the old site that was generated dynamically sans-CMS was Blogger article integration into the site. A cron job ran daily to import new Blogger article title, link, and content snippets into the Postgres database. We opted for removing dependency on a database with the new site, so we investigated creative ways to include the dynamic Blogger content. We developed a rake task that is run via cron job to dynamically generate partial Rails views containing Blogger content. Below is an example and explanation of how the Blogger RSS feed is retrieved and a partial is generated:
Open URI and REXML are used to retrieve and parse the XML feed.
require 'open-uri' require 'rexml/document' ...
The feed is retrieved and a REXML object created from the feed in the rake task:
data = open('http://blog.endpoint.com/feeds/posts/default?alt=rss&max-results=10', 'User-Agent' => 'Ruby-Wget').read
doc = REXML::Document.new(data)
The REXML object is iterated through. An array containing the Blogger links and titles is created.
results = []
doc.root.each_element('//item') do |item|
author = item.elements['author'].text.match(/\(.+/).to_s.gsub(/\.|\(|\)/,'')
results << '<a href="' + item.elements['link'].text + '">' + item.elements['title'].text + '</a>'
end
Finally, a Rails dynamic partial is written containing the contents of the results array:
File.open(#{RAILS_ROOT}/app/views/blog/_index.rhtml", 'w') { |f| f.write(results.inject('') { |s, v| s = s + '<p>' + v + '</p>'}) }
A similar process was applied for bio and tag dynamic partials. The partials are included on pages such as the End Point service pages, End Point bio pages, and End Point home page.
jQuery Carousel Functionality
Another interesting JavaScript component I worked on for the new site was the carousel functionality for the home page and client page. Carousels are a common "web 2.0" JavaScript component where visible items slide one direction out of view and new items slide into view from the other direction. I initially planned on implementing a simple carousel with a jQuery plugin, such as jCarousel. Other JavaScript frameworks also include carousel functionality such as the YUI Carousel Control or the Prototype UI. I went along planning to implement the existing jQuery carousel functionality, but then was asked, "Can you make it a circular carousel where the left and right buttons are always clickable?" In many of the existing carousel plugins and widgets, the carousel is not circular, so this request required custom jQuery. After much cross-browser debugging, I implemented the following (shown in images for a better explanation):
Step 1: The page loads with visible bios surrounded by empty divs with preset width. The visibility of the bios is determined by CSS use of the overflow, position, and left attributes.
Step 2: Upon right carousel button click, new bios populate the right div via jQuery.
Step 3: To produce the carousel or slider effect, the left div uses jQuery animation functionality and shrinks to a width of 0px.
Step 4: Upon completion of the animation, the empty left div is removed, and a new empty div is created to the right of the new visible bios.
Step 5: Finally, the left div's contents are emptied and the carousel is in its default position ready for action!
Another request for functionality came from Jon. He asked that we create and use "web 2.0" URLs to load specific content on page load for the dynamic content throughout our site, such as http://www.endpoint.com/clients#citypass, http://www.endpoint.com/clients#backcountry.
Upon page load, JavaScript is used to detect if a relative link exists:
if(document.location.href.match('#.+')) {
var id = document.location.href.match('#.*').toString().replace('#', '');
}
The id retrieved from the code snippet above is used to populate the dynamic page content. Then, JavaScript is used during dynamic page functionality, such as carousel navigation, to update the relative link:
document.location.href = document.location.href.split('#')[0] + '#' + anchor;
Twitter Integration
Another change in the new site was importing existing functionality previously written in Python to update End Point's Twitter feed automagically. The rake task uses the Twitter4R gem to update the Twitter feed and is run via cron job every 30 minutes. See the explanation below:
The public twitter feed is retrieved using Open URI and REXML.
data = open('http://twitter.com/statuses/user_timeline/endpoint.xml', 'User-Agent' => 'Ruby-Wget').read
doc = REXML::Document.new(data)
An array containing all the titles of all tweets is created.
doc.each_element('statuses/status/text') do |item|
twitter << item.text.gsub(/ http:\/\/j\.mp.*/, '')
end
The blogger RSS feed is retrieved and parsed. An array of hashes is created to track the un-tweeted blog articles.
data = open('http://blog.endpoint.com/feeds/posts/default?alt=rss&max-results=10000', 'User-Agent' => 'Ruby-Wget').read
doc = REXML::Document.new(data)
found_recent = false
doc.root.each_element('//item') do |item|
found_recent = true if twitter.include?(item.elements['title'].text)
blog << { 'title' => item.elements['title'].text, 'link' => item.elements['link'].text } if !found_recent
end
Using the j.mp api, a short url is generated. A Twitter message is created from the short URL.
data = open('http://api.j.mp/shorten?version=2.0.1&longUrl=' + blog.last['link'] + '&login=**&apiKey=*****&format=xml')
...
twitter_msg = blog.last['title'] + ' ' + short_url
The twitter4r gem is used to login and update the Twitter status message.
client = Twitter::Client.new(:login => **, :password => *****)
begin
status = client.status(:post, twitter_msg)
rescue
end
Google Event Tracking
Finally, since we implemented dynamic content throughout the site, we decided to use Google Event Tracking to track user interactivity. We followed the standard Google Analytics event tracking to add events for events such as the slider carousel user involvement, the team page bio and history hover user involvement:
//pageTracker._trackEvent(category, action, optional_label, optional_value);
pageTracker._trackEvent('Team Page Interaction', 'Map Hover', bio);
We are happy with the new site and we hope that it presents our skillz including Interchange Development, Hosting Expertise and Support, and Database Wizardry!
Learn more about End Point's rails development.
Client Side Twitter Integration
I recently was assigned a project that required an interesting solution, Crisis Consultation Services. The site is essentially composed of five static pages and two dynamic components.
The first integration point required PayPal payment processing. Crisis Consultation Services links to PayPal where payment processing is completed through PayPal. Upon payment completion, the user is bounced back to a static receipt page. This integration was quite simple as PayPal provides the exact form that must be included in the static HTML.
The second integration point required a unique solution. The service offered by the static brochure site is dependent on the availability and schedule of the company employees, so the service availability remains entirely dynamic. The obvious solution was to include dynamic functionality where the employees would update their availability. Some thoughts that crossed our minds of how to update the availability were:
- Could we build an app for the employees to update the availability given the budget constraints?
- Could the employees use ftp or ssh to upload a single file containing details on their availability?
- Are there other dynamic tools that we could use to track the availability of the consultant such as SMS or Twitter?
Initially, we investigated using Google App Engine with a Python app that retrieved the availability information from an existing tool. To keep the budget down and try to stick with a purely static site on the server, we decided to investigate using Twitter for integration. I reviewed the Twitter API and found some code snippets for integrating Twitter via JavaScript. Below are snippets and explanations of the resulting code.
First, a script that retrieves the Twitter feed is appended to the document body. In this case, the endpoint Twitter account is pinged to get the most recent comment (count=1), and the resulting callback 'twitterAfter' is made after the JSON feed has been retrieved.
var url = 'http://twitter.com/statuses/user_timeline/endpoint.json?callback=twitterAfter&count=1';
var script = document.createElement('script');
script.setAttribute('src', url);
document.body.appendChild(script);
Next, the callback 'twitterAfter' function is called. The callback function includes logic to determine if the consultant is available based on the most recent twitter message. If the datetime is in the future, the consultant is not available and will be available at that future datetime. If the datetime is in the past, the consultant is available and has been available since that datetime.
var twitterAfter = function(obj) {
var now = new Date();
var available = new Date(obj[0].text.replace(/-/g, '/'));
if (now >= available) {
alert('Consultant is available!');
// do other whizbang stuff here
}
return;
};
In another example of a more complex callback, the availability of the consultant is calculated.
var twitterAfter_advanced = function(obj) {
var now = new Date();
var available = new Date(obj[0].text.replace(/-/g, '/'));
mins_available = parseInt((available.getTime() - now.getTime())/60000);
if (mins_available < 1) {
alert('Consultant is available!');
// do other whizbang stuff here
} else {
alert('Consultant is not available. The consultant will be available in ' + mins_available + ' minute(s).');
// do other whizbang stuff here
}
return;
};
Here is an example Twitter feed to be used with this client side code:
2009-09-13 9:00 - 6:00pm Sept 12th from web 2009-09-12 8:30 - 7:00pm Sept 11th from web 2009-09-10 22:00 - 5:00pm Sept 10th from web
The above example Twitter feed would yield the following availability:
Sept 10th 5pm - Sept 10th 10pm: Not Available Sept 10th 10pm - Sept 11th 7pm: Available Sept 11th 7pm - Sept 12th 8:30am: Not Available Sept 12th 8:30am - Sept 12th 6pm: Available Sept 12th 6pm - Sept 13th 9am: Not Available Sept 13th 9am - now: Available
In both the basic and advanced callback methods above, content on the page is updated to inform users of service or consultant availability. In the application of the advanced callback method, the user is notified when the consultant will be available.
The client side Twitter integration solution fit our budget and server constraints - the functionality lives entirely on the client side, so we weren't concerned about server installation, setup, or requirements. Additionally, Twitter is such a popular app that there are many convenient ways to tweet availability from a mobile environment.
JavaScript fun with IE 8
I ran into, and found solutions for, two major gotchas targeting IE 8 with a jQuery-based (and rather JavaScript-heavy) web application.
First is to specify the 'IE 8 Standard' rendering mode by adding the following meta tag:
<meta equiv="X-UA-Compatible" content="IE=8">
The default rendering mode is rather glitchy and tends to produce all sorts of garbage from 'clean' HTML and JavaScript. The result renders slightly different sizes, reports incorrect values from common jQuery calls, etc.
The default rendering also caused various layout issues (CSS handling looked more like IE 6 than IE 7). Also, minor errors (an extra '' tag on one panel) caused the entire panel to not render.
Another issue is the browser is overly lazy about invalidating the cache for AJAX pulled content, especially (X)HTML. This means that though you think you're pulling current data, in reality it keeps feeding you the same old data. This also means that if you use the same exact URL for HTML & JSON data, you must add a parameter to avoid running into cache collisions. IE 8 only seemed to honor 'Cache-control: no-cache' in the header to cause it to behave properly.
On the other side, I've got a big thumbs up for jQuery. I was able to produce a skinned fairly 'heavy' client-side application that works equally well (and looks almost the same) on Firefox, Chrome, Safar, and now IE 8.
