Interchange Blog Archive
Paper Source: Case Study with Google Maps API
Basic Google map with location markers.
Recently, I've been working with the Google Maps API on Paper Source. Paper Source is one of our Interchange clients who has over 40 physical stores throughout the US. On their website, they had previously been managing static HTML pages for these 40 physical stores to share store information, location, and hours. They wanted to move in the direction of something more dynamic with interactive maps. After doing a bit of research on search options out there, I decided to go with the Google Maps API. This article discusses basic implementation of map rendering, search functionality, as well as interesting edge case behavior.
Basic Map Implementation
In it's most simple form, the markup required for adding a basic map with markers is the shown below. Read more at Google Maps Documentation.
HTML
<div id="map"></div>
CSS
#map {
height: 500px;
width: 500px;
}
JavaScript
//mapOptions defined here
var mapOptions = {
center: new google.maps.LatLng(40, -98),
zoom: 3,
mapTypeId: google.maps.MapTypeId.ROADMAP
};
//map is the HTML DOM element ID where it will be rendered
var map = new google.maps.Map(document.getElementById("map"), mapOptions);
//all locations is a JSON object representing locations,
//where each location has a latitude and longitude
$.each(all_locations, function(i, loc) {
var marker = new google.maps.Marker({
map: map,
position: new google.maps.LatLng(loc.latitude, loc.longitude)
});
})
Building Search Functionality
Search interface. Search results are listed on the left, and map with markers is shown on the right.
Next up, I needed to build out search functionality. Google has its own geocoder to allow address searches. Here is the basic markup for running a search:
var geocoder = new google.maps.Geocoder();
//search is a variable representing the user search, such as a zip code, city name, or state name
geocoder.geocode({ 'address' : search }, function(results, status) {
var search_center = results[0].geometry.bounds.getCenter();
var mapOptions = {
center: search_center,
zoom: 10,
mapTypeId: google.maps.MapTypeId.ROADMAP
};
var map = new google.maps.Map(document.getElementById("map"), mapOptions);
$.each(all_locations, function(i, loc) {
var marker = new google.maps.Marker({
map: map,
position: new google.maps.LatLng(loc.latitude, loc.longitude)
});
})
}
In the above code, the search term is passed into the Geocoder object and a map with all locations marked is rendered. To determine which markers are in the visible map boundaries, the following map.getBounds().contains() method would be leveraged:
var visible_locations = [];
$.each(all_locations, function(i, loc) {
if(map.getBounds().contains(new google.maps.LatLng(loc.latitude, loc.longitude))) {
visible_locations.push(loc);
}
});
//render visible locations to the left of the map
One final step here is to add a listener to the map, so that visible locations are updated when the user zooms in and out. This is accomplished with the following listener:
google.maps.event.addListener(map, 'zoom_changed', function() {
//call method to rerender visible locations
});
Handling Zero Results
What happens if your Geocoder object can't find the address? A simple conditional can be used:
geocoder.geocode({ 'address' : search }, function(results, status) {
if(status == "ZERO_RESULTS") {
//notify customer that no results have been found
} else {
//got results, render location
}
}
Calculate and Sort by Distance
The next layer of logic I needed to add was the ability to determine the distance between the search address and sort the results by distance. To calculate distance, I did some research and settled on the following code:
var R = 6371;
$.each(all_locations, function(i, loc) {
var loc_position = new google.maps.LatLng(loc.latitude, loc.longitude);
var dLat = locations.rad(loc.latitude - search_center.lat());
var dLong = locations.rad(loc.longitude - search_center.lng());
//calculate spherical distance between search position and location
var a = Math.sin(dLat/2) * Math.sin(dLat/2) +
Math.cos(locations.rad(search_center.lat())) *
Math.cos(locations.rad(search_center.lat())) *
Math.sin(dLong/2) * Math.sin(dLong/2);
var c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1-a));
var d = R * c;
loc.distance = d;
//convert distance to miles
loc.readable_distance =
(google.maps.geometry.spherical.computeDistanceBetween(search_center, loc_position) *
0.000621371).toFixed(2);
});
To sort the locations by distance, I leverage jQuery sort:
var sort_by_distance = function(obj) {
return obj.sort(function(a, b) {
if(a.distance > b.distance) {
return 1;
} else {
return -1;
}
})
};
var sorted_locations = sort_by_distance(all_locations);
Adjust Map Boundaries to Include Specific Markers
Another interesting use case I needed to handle was forcing the map to zoom out to include stores within 100 miles if there was nothing in the initial map boundaries, e.g.:
The search for "27103" doesn't return any nearby stores, so the map is extended to include stores within 100 miles.
To accomplish this functionality, I added a bit of code to extend the map boundaries:
geocoder.geocode({ 'address' : search }, function(results, status) {
var search_center = results[0].geometry.bounds.getCenter();
var mapOptions = {
center: search_center,
zoom: 10,
mapTypeId: google.maps.MapTypeId.ROADMAP
};
var map = new google.maps.Map(document.getElementById("map"), mapOptions);
var current_bounds = results[0].geometry.bounds;
$.each(all_locations, function(i, loc) {
var loc_position = new google.maps.LatLng(loc.latitude, loc.longitude);
var dLat = locations.rad(loc.latitude - search_center.lat());
var dLong = locations.rad(loc.longitude - search_center.lng());
//calculate spherical distance between search position and location
var a = Math.sin(dLat/2) * Math.sin(dLat/2) +
Math.cos(locations.rad(search_center.lat())) *
Math.cos(locations.rad(search_center.lat())) *
Math.sin(dLong/2) * Math.sin(dLong/2);
var c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1-a));
var d = R * c;
loc.distance = d;
//convert distance to miles
loc.readable_distance =
(google.maps.geometry.spherical.computeDistanceBetween(search_center, loc_position) *
0.000621371).toFixed(2);
var marker = new google.maps.Marker({
map: map,
position: new google.maps.LatLng(loc.latitude, loc.longitude)
});
if(loc.readable_distance < 100) {
current_bounds.extend(loc_position);
}
});
//Google map method to fit map boundaries to desired boundaries
map.fitBounds(current_bounds);
}
Disable Scroll and Zoom on Mobile-Sized Devices
One final behavior needed was to disable map zooming and scrolling on mobile devices, to improve the usability on mobile/touch interfaces. Here's how this was accomplished:
var options_listener = google.maps.event.addListener(map, "idle", function() {
if($(window).width() < 656) {
map.setOptions({
draggable: false,
zoomControl: false,
scrollwheel: false,
disableDoubleClickZoom: true,
streetViewControl: false
});
}
google.maps.event.removeListener(options_listener);
});
Conclusion
With all this code, the final location search functionality at Paper Source includes:
- Basic United States map rendering to display all physical store locations.
- Search by location which shows stores within 100 miles, and allows users to zoom in and out to adjust their search. Search lists results sorted by distance.
- "Saved" or "Quick" searches by states, which displays all physical stores by state.
- Adjustment of mobile display map options.
Crossed siting; or How to Debug iOS Flash issues with Chrome
This situation had all the elements of a programming war story: unfamiliar code, an absent author, a failure that only happens in production, and a platform inaccessible to the person in charge of fixing this: namely, me.
Some time ago, an engineer wrote some Javascript code to replace a Flash element on a page with an HTML5 snippet, for browsers that don't support Flash (looking at you, iOS). For various reasons, said code didn't make it to production. Fast forward many months, and that engineer has left for another position, so I'm asked to test it, and get it into production.
Of course, it works fine. My only test platform is an iPod, but it looks great here. Roll it out, and ker-thunk: it doesn't work. Of course, debugging Javascript on an iPod is less than optimal, so I enlisted others with Apple devices and found that it mostly failed, but maybe worked a few times, depending on [SOMETHING].
To make matters a bit worse, the Apache configurations for the test and production environments differed, just enough to raise my suspicions and convince me that was worth investigating. Once I went down that path, it was tough to jar myself loose from that suspicion.
I tried disabling Flash in Firefox to trigger the substitution, but that didn't seem to have the desired effect (as the replacement didn't happen, which was a different error than the replacement failing). I tried a browser emulation site (which shall remain nameless for this post, as I don't think they are bad at what they do, but they don't emulate iOS browsers in this capacity).
Eventually we disabled flash in Chrome (by visiting the chrome://plugins page). That unveiled the hidden error:
XMLHttpRequest cannot load http://www.somewhere.com/ajax/newstuff.html. Origin http://somewhere.com is not allowed by Access-Control-Allow-Origin.
There's the crux of it: the browser was sitting on an address which appeared different than that of the AJAX target.
The site involved is an Interchange site, and page constructed a URL using the [area] tag, which makes a fully-qualified URL from a fragment like "ajax/newstuff". That URL was being seen by the iOS browsers as a cross-site scripting attempt, as it didn't precisely match where the browser found the page. The error was not visible in the Safari browser on my iPod, and browsers I had access to which could have displayed the error, weren't suffering it.
I replaced the [area] tag with a plain relative URL and the problem disappeared.
TL;DR:
$.ajax({
- url: "[area href=|ajax/newstuff|]",
+ url: "ajax/newstuff.html",
type: 'html',
The original code caused a cross-site scripting failure.
Paper Source: The Road to nginx Full Page Caching in Interchange
Background & Motivation
During the recent holiday season, it became apparent that some efforts were needed to improve performance for Paper Source to minimize down-time and server sluggishness. Paper Source runs on Interchange and sells paper and stationery products, craft products, personalized invitations, and some great gifts! They also have over 40 physical stores which in addition to selling products, offer on-site workshops.
Over the holiday season, the website experienced a couple of instances where server load spiked causing extreme sluggishness for customers. Various parts of the site leverage Interchange's timed-build tag, which creates static caches of parts of a page (equivalent to Rails' and Django's fragment caching). However, in all cases, Interchange is still being hit for the page request and often the pages perform repeated logic and database hits that opens an opportunity for optimization.
The Plan
The long-term plan for Paper Source is to move towards full page nginx caching, which will yield speedily served pages that do not require Interchange to be touched. However, there are several code and configuration hurdles that we have to get over first, described below.
Step 1: Identify Commonly Visited Pages
First, it's important to recognize which pages are visited the most frequently and to tackle optimization on those pages first, essentially profiling the site to determine where we will gain the most from performance optimization. In the case of Paper Source, popular pages include:
- Thumbnail page, or the template where multiple products are shown in list format
- Product detail page, or the template that serves the basic product page
- Swatching page, or the template that serves a special product page with special product options
- Personalization detail page, or a template that serves the special product page for personalizeable products (e.g. wedding invitatations, birth announcements, etc.)
Step 2: Remove Dynamic User Elements on pages of interest
The next step in the process is to remove dynamic elements on the page, by having cookies or AJAX render these dynamic elements. Below are a couple of examples of these dynamic elements on two primary page templates.
The thumbnail page contains two dynamic elements: the mini-cart template, which shows how many items are in the user's cart, and the log in information, which shows "my account" and "log out" links if the user is logged in, and shows a "log in" link if the user is not logged in.
In addition to the mini-cart and logged in elements, the product page contains additional dynamic elements which signify if a user has added an item to their cart, and presentation of the user's previously viewed items.
In the examples above, the following changes were applied to replace these dynamic elements:
- Mini-cart Component: This section utilizes cookies with the jQuery.cookie plugin. A cookie stored in the browser identifies the number of cart items and the cart subtotal. After the DOM loads, the mini-cart is rendered and displayed if the user has a non-empty cart. These cookies are manipulated whenever the cart contents are modified.
- Login Component: This section also reads a browser-stored cookie. If the cookie indicates the user is logged in, the navigation elements are updated to reflect that.
- Added to cart Component: On the product detail page, this code was invasively modified to allow items to be added to cart via AJAX. The AJAX call results in an update of the cart cookies specified above. The feedback of the AJAX call is presented to the user to indicate that the item has been successfully added.
- Previously Viewed Component: Finally, the product page also contains previously viewed items. This is generated via a cookie that stores recent skus visited by the user, and each sku has an associated cookie that includes information such as the image source, link, description, and price. Because a maximum of three previously viewed items is shown, cookies here of older previously viewed items are deleted to minimize cookie build-up.
Step 3: Implement fully timed-build caching pages
During this incremental process to reach the end goal of full nginx caching, the next step is to implement fully timed-build pages, or use Interchange's caching mechanism to fully cache these pages and reduce repetitive database hits and backend logic. In this step, the entire page is wrapped in a timed-build tag, which results in writing and serving a static cached file for that page. While this step is not a necessity, it does allow for us to deploy and test our changes in preparation for nginx caching. In adddition to giving us an opportunity to work out kinks, this step also gives us an added bump in performance because several of these page templates have no caching at all.
Step 4: Reproduce redirect logic outside of Interchange
Next up, we plan to move logic that handles page redirects outside of Interchange to nginx. At the moment, Interchange is responsible for handling 301 redirects on old product and navigation pages. This will need to be moved to nginx redirects to minimize the hits on Interchange here.
Step 5: Implement nginx architecture on camps
Another non-trivial step in this process will be to implement nginx architecture on DevCamps (or camps). DevCamps is an open source tool developed by End Point for developing on multiple instances of copies of the production server. Camps are heavily used for Paper Source because several End Point and internal Paper Source employees simultaneously work on different projects on their development instances or camps. Nginx caching will need to be set up to also work with the camp system in place.
Step 6: Turn nginx caching on!
Finally, we can turn on nginx caching for specific pages of interest. Nginx will then serve these fully cached pages and will avoid Interchange entirely. Cookies and AJAX will still be used to render the dynamic elements on the fully cached pages. While we'd ideally like to cache every page on the site except for the cart, checkout and my account pages, it makes more sense to find the bottlenecks and tackle them incrementally.
Where are we now?
At the moment, I've made progress on steps 1-3 for several subsets of pages, including the thumbnail and product detail pages. I plan to continue these steps for additional bottleneck pages. I have worked out out a couple of minor kinks throughout the recent progress, but things have been progressing well. Richard plans to make progress on the nginx related tasks in preparation for reaching the end goal.
Lazy AJAX
Don't do this, at least not without a good reason. It's not the way to design AJAX interfaces from scratch, but it serves well in a pinch, where you have an existing CGI-based page and you don't want to spend a lot of time rewriting it.
I was in a hurry, and the page involved was a seldom-used administration page. I was attempting to convert it into an AJAX-enabled setup, wherein the page would stand still, but various parts of it could be updated with form controls, each of which would fire off an AJAX request, and use the data returned to update the page.
However, one part of it just wasn't amenable to this approach, or at least not quick-and-dirty. This part had a relatively large amount of inline interpolated (Interchange) data (if you don't know what Interchange is, you can substitute "PHP" in that last sentence and you'll be close enough.) I wanted to run the page back through the server-side processing, but only cared about (and would discard all but) one element of the page.
My lazy-programmer's approach was to submit the page itself as an AJAX request:
$.ajax({
url: '/@_MV_PAGE_@',
data: {
'order_date': order_date,
'shipmode' : shipmode
},
method: 'GET',
async: true,
success: function(data, status){
$('table#attraction_booklet_order').replaceWith(
$(data).find('#attraction_booklet_order').get(0)
);
$('table#attraction_booklet_order').show();
}
});
In this excerpt, "MV_PAGE" is a server-side macro that evaluates to the current page's path. The element I care about is a rather complex HTML table containing all sorts of interpolated data. So I'm basically reloading the page, or at least that restricted piece of it. The tricky bit, unfamiliar to jQuery newcomers, lets you parse out something from the returned document much as you would from your current document.
Again, don't do this without a reason. When I have more time, I'll revisit this and improve it, but for now it's good enough for the current needs.
Slash URL
There's always more to learn in this job. Today I learned that Apache web server is smarter than me.
A typical SEO-friendly solution to Interchange pre-defined searches (item categories, manufacturer lists, etc.) is to put together a URL that includes the search parameter, but looks like a hierarchical URL:
/accessories/Mens-Briefs.html
/manufacturer/Hanes.html
Through the magic of actionmaps, we can serve up a search results page that looks for products which match on the "accessories" or "manufacturer" field. The problem comes when a less-savvy person adds a field value that includes a slash:
accessories: "Socks/Hosiery"
or
manufacturer: "Disney/Pixar"
Within my actionmap Perl code, I wanted to redirect some URLs to the canonical actionmap page (because we were trying to short-circuit a crazy Web spider, but that's beside the point). So I ended up (after several wild goose chases) with:
my $new_path = '/accessories/' .
Vend::Tags->filter({body => (join '%2f' => (grep { /\D/ } @path)),
op => 'urlencode', }) .
'.html';
By this I mean: I put together my path out of my selected elements, joined them with a URL-encoded slash character (%2f), and then further URL-encoded the result. This was counter-intuitive, but as you can see at the first link in this article, it's necessary because Apache is smarter than you. Well, than me anyway.
Insidious List Context
Recently, I fell into a deep pit. Not literally, but a deep pit of Perl debugging. As a result, I'm here to warn you and yours about "Insidious List Context(TM)".
(Note: this is a fairly elementary discussion, for people early in their Perl wizardry training.)
Perl has two contexts for evaluating expressions: list and scalar. (All who know this stuff cold can skip down a ways.) "Scalar" context is what non-Perl languages just call "normal reality", but Perl likes to do things ... differently ... so we have more than one context.
In scalar context, a scalar is a scalar is a scalar, but a list becomes a scalar that represents the number of items in the list. Thus,
@x = (1, 1, 1); # @x is a list of three 1s # vs. $x = (1, 1, 1); # $x is "3", the list size
In list context, a list of things is still a list of things. That's pretty simple, but when you are expecting a scalar and you get a list, your world can get pretty confused.
Okay, now the know-it-alls have rejoined us. I had a Perl hashref being initialized with code something like this:
my $hr = {
KEY1 => $value1,
KEY2 => $value2,
KEY_TROUBLE => (defined($foo) ? mysub($foo) : 1),
KEY3 => $value3,
};
So here is the issue: if mysub() returns a list, then the hashref will get extra data. Remember, Perl n00bs, "=>" is not a magical operator, it's just a "fat comma". So a construction like this:
1 => (2, 3, 4)is really the same as:
1, 2, 3, 4
Here's a complete example to illustrate just what size and shape hole I fell into:
use strict;
use Data::Dumper;
my($value1,$value2,$value3,$foo) = qw(value1 value2 value3 foo);
my $hr = {
KEY1 => $value1,
KEY2 => $value2,
KEY_TROUBLE => (defined($foo) ? mysub($foo) : 1),
KEY3 => $value3,
};
print Data::Dumper->Dumper($hr);
sub mysub {
return qw(junk extrajunk);
}
This outputs:
$VAR1 = 'Data::Dumper';
$VAR2 = {
'extrajunk' => 'KEY3',
'KEY2' => 'value2',
'KEY1' => 'value1',
'value3' => undef,
'KEY_TROUBLE' => 'junk'
};
Now, the actual subroutine involved in my little adventure was even more insidious: it returned a list context because it was evaluating a regular expression, in a list context. Its actual source:
sub is_yes {
return( defined($_[0]) && ($_[0] =~ /^[yYtT1]/));
}
So watch those expression-evaluation contexts; they can turn fairly harmless expressions into code-busters.
Hidden inefficiencies in Interchange searching
A very common, somewhat primitive approach to Interchange searching uses an approach like this:
The search profile contains something along the lines of --
mv_search_type=db
mv_search_file=products
mv_column_op=rm
mv_numeric=0
mv_search_field=category
[search-region]
[item-list]
[item-field description]
[/item-list]
[/search-region]
In other words, we search the products table for rows whose column "category" matches an expression (with a single query), and we list all the matches (description only). However, this can be inefficient depending on your database implementation: the item-field tag issues a query every time it's encountered, which you can see if you "tail" your database log. If your item-list contains many different columns from the search result, you'll end up issuing many such queries:
[item-list]
[item-field description], [item-field weight], [item-field color],
[item-field size], [item field ...]
...
resulting in:
SELECT description FROM products WHERE sku='ABC123' SELECT weight FROM products WHERE sku='ABC123' SELECT color FROM products WHERE sku='ABC123' SELECT size FROM products WHERE sku='ABC123' ...
(Now, some databases are smart enough to cache query results, but some aren't, so avoiding this extra work is probably worth your trouble even on a "smart" database, in case your Interchange application gets moved to a "dumb" database sometime in the future.)
Fortunately, it's easy to correct:
mv_return_fields=*
and then
...
[item-param description]
...
in place of "item-field".
Interchange "on-the-fly" items
Interchange has a handy feature (which, in my almost-seven-years of involvement, I'd not seen or suspected) allowing you to create an item "on-the-fly", without requiring any updates to your products table. Here's a recipe for making this work.
First, you need to tell Interchange that you're going to make use of this feature (in catalog.cfg).
OnFly onfly
Simple, no? The "OnFly" directive names a subroutine that is called to pre-process the custom item before it's added to the cart. The default "onfly" routine can be found in the system tag "onfly": code/SystemTag/onfly.coretag in the standard Interchange installation. (If you need more that what it provides, that's beyond the scope of my post, so good luck, bon voyage, and please write back to let us know what you learned!)
Then, you need to submit some special form parameters to set up the cart:
- mv_order_item: the item number identifying this line
- mv_order_fly: a structured string with | (vertical bar) delimiters. Each sub-field specifies something about the custom item, thus:
description=My custom item|price=12.34
Now, in my particular case, I was encapsulating an XML feed of products from another site (a parts supplier) so that the client (a retail seller) could offer replacement parts, but not have to incorporate thousands of additional lines in the "products" table. So after drilling down to the appropriate model and showing the available parts, each item got the following bit of JavaScript (AJAX) code associated with its add-to-cart button:
var $row = $(this).parents('tr');
$.ajax({
url: '/cgi-bin/mycat/process',
data: {
mv_todo: 'refresh',
mv_order_quantity: 1,
mv_order_item: $row.find('td.item_number').html(),
mv_order_fly: 'description='
+ $row.find('td.description').html().replace('|','')
+ '|'
+ 'price='
+ $row.find('td.price').html().replace('$','').replace(',','')
},
method: 'POST',
success: function(data, status) {
$('#msg_div').html('Added to cart.')
}
});
And that's all it took. With Interchange, you don't even need a special "landing page" for your AJAX submission; Interchange handles all the cart-updating out of sight.
I still need to add some post-processing to handle errors, and update the current page so I can see the new cart line count, but the basics are done.
World of Powersports Client Report
World of Powersports is a family of websites that runs on Interchange. Carl Bailey describes how a few years after working on their initial website, World of Powersports came to End Point to develop a new website called Dealer Orders which has been very successful. This has allowed End Point the opportunity to work on several other related websites for the client.
Since then, we have worked on several other sites including:
All of the websites pull from a single database that is fed by various APIs from parts vendors such as Honda, Suzuki, and Polaris. This updates the inventory counts and other related information for all of the sites. It also interacts with online sites such as eBay, Google Base, and Amazon for checking part availability and pricing.
Implementing the interactions between these different entities has provided End Point with much of the challenge of these sites but continues to provide the client and customers with great value.
Interchange Caching Implementation Under Fire
Richard and David presented a recent case study on an e-commerce hosting client.
Several Interchange catalogs drive their individually branded storefronts, on top of a standard single-server LAMP stack boosted by an SSD drive.
Last year the sites came under an intense Distributed Denial of Service attack which lasted nearly two weeks. End Point responded immediately and soon engaged third-party DDoS mitigation firms. This experience later prompted an Interchange caching implementation.
Cache population and expiration is difficult for any dynamic web application using sessions, and doubly so for e-commerce sites. Every shopping cart needs a session, but delaying session creation until the first POST submission enables efficient caching for most of the sitemap. Other Interchange caching improvements made it back into the upstream code.
Web service integration in PHP, jQuery, Perl and Interchange
Jeff Boes presented on one of his latest projects.
CityPass.com decided on a project to convert their checkout from being served by Interchange to have the interface served by PHP, but continue to interact with Interchange for the checkout process through a web service.
The original site was entirely served by Interchange, the client then took on a project to convert the frontend to PHP while leveraging Interchange for frontend logic such as pricing and shipping as well as for backend administration for order fulfillment.
Technologies used in the frontend rewrite:
- PHP
- jQuery for jStorage, back-button support and checkout business logic
- AJAX web services for prices, discounts, click-tracking
The Interchange handler is conduit.am that handles the processing of the URL. From this ActionMap the URLs are decoded and passed to a Perl module, Data.pm, which handles processing the input and returning the results.
An order is just a JSON object so testing of the web service is easy. We have a known hash, we post to the proper URL and compare the results and verify they are the same. New test cases are also easy, we can capture any order (JSON) to a log file and add it as a test case.
Multi-store Architecture for Ecommerce
Something that never seems to go out of style in ecommerce development is the request for multi-site or multi-store architecture running on a given platform. Usually there is interest in this type of setup to encourage build-out and branding of unique stores that have shared functionality.
A few of Backcountry.com's stores driven by a multi-store architecture, developed with End Point support.
End Point has developed several multi-store architectures on open source ecommerce platforms, including Backcountry.com (Interchange/Perl), College District (Interchange/Perl), and Fantegrate (Spree/Rails). Here's an outline of several approaches and the advantages and disadvantages for each method.
Option #1: Copy of Code Base and Database for Every Site
This option requires multiple copies of the ecommerce platform code base, and multiple database instances connected to each code base. The stores could even be installed on different servers. This solution isn't a true multi-store architecture, but it's certainly the first stop for a quick and dirty approach to deploy multiple stores.
The advantages to this method are:
- Special template logic doesn't have to be written per site – the templates would simply follow the ecommerce platform's template pattern.
- Relative to Option #3 described below, no custom database development is required.
- Custom business logic may be more easily applied to a set of the stores, without affecting the other stores.
The disadvantages to this method are:
- Maintenance of the applications can be time consuming, as changes must be applied to all instances.
- Custom changes must be applied to all multi-store instances.
- Users and administrator accounts are not shared across multiple stores.
Option #2: Single Code Base, Single DatabaseIn this method, there is one copy of the source code that interacts with one database. The single database would be modified to contain a store specific id per product, order, and peripheral tables. The code base would also have to be modified to be able to limit the visible products described here. In this method, the individual store may be identified by the domain or subdomain. With this method, there may also be code customization that allows for custom templates per store. The advantages to this method are:
The disadvantages to this method are:
|
A second option in multi-store architecture may use a data model with store specific entries in various tables, described here. |
Option #3: Single Code Base, Single Database with Schemas or Views Per Store
In this method, there is one copy of the source code that interacts with a database that has views specific to that store, or a schema specific to that store. In this case, the code base would not necessarily need customization since the data model it accesses should follow the conventions of the ecommerce platform. However, moderate database customization is required in this method. With this method, there may also be code customization that allows for custom templates per store.
The advantages to this method are:
- Relative to Option #1, maintenance for one code base is relatively simple.
- Relative to option #2, code base changes are minimal.
- User accounts may or may not be shared across stores.
- Relative to option #2, there may be a potential performance gain by removing time spent limiting data to the current store instance.
The disadvantage to this method is:
- Customization and development is required for database configuration and management of multi-store database schemas.
A tangential variation on the methods above are two different codebases and functionality attached to one back-end web service and backing database, such as the architecture we implemented for Locate Express. And a similar tangential variation I've investigated before is one that might use a Sinatra driven front-end and a Rails backed admin, such as RailsAdmin used in Piggybak.
College District has a collection of stores driven by a multi-store architecture, developed with End Point support.
Conclusion
In most cases for our clients, there is cost-benefit analysis that drives the decision between the three options described above. Option #1 might be an acceptable solution for someone interested in building out two or three stores, but the latter two options would be more suitable for someone interested in spinning up many additional instances quickly with lower long term maintenance costs.
Interchange loops using DBI Slice
One day I was reading through the documentation on search.cpan.org for the DBI module and ran across an attribute that you can use with selectall_arrayref() that creates the proper data structure to be used with Interchange's object.mv_results loop attribute. The attribute is called Slice which causes selectall_arrayref() to return an array of hashrefs instead of an array of arrays. To use this you have to be working in global Perl modules as Safe.pm will not let you use the selectall_arrayref() method.
An example of what you could use this for is an easy way to generate a list of items in the same category. Inside the module, you would do like this:
my $results = $dbh->selectall_arrayref(
q{
SELECT
sku,
description,
price,
thumb,
category,
prod_group
FROM
products
WHERE
category = ?},
{ Slice => {} },
$category
);
$::Tag->tmpn("product_list", $results);
In the actual HTML page, you would do this:
<table cellpadding=0 cellspacing=2 border=1>
<tr>
<th>Image</th>
<th>Description</th>
<th>Product Group</th>
<th>Category</th>
<th>Price</th>
</tr>
[loop object.mv_results=`$Scratch->{product_list}` prefix=plist]
[list]
<tr>
<td><a href="/cgi-bin/vlink/[plist-param sku].html"><img src="[plist-param thumb]"></a></td>
<td>[plist-param description]</td>
<td>[plist-param prod_group]</td>
<td>[plist-param category]</td>
<td>[plist-param price]</td>
</tr>
[/list]
[/loop]
</table>
We normally use this when writing ActionMaps and using some template as our setting for mv_nextpage.
Some great press for College District
College District has been getting some positive press lately, the most recent being a Forbes article which talks about the success they have been seeing in the last few years.
College District is a company that sells collegiate merchandise to fans. They got their start focusing on the LSU Tigers at TigerDistrict.com and have branched out to teams such as the Oregon Ducks and Alabama Roll Tide.
We've been working with Jared Loftus @ College District for more then four and a half years. College District is running on a heavily modified Interchange system with some cool Postgres tricks. The system can support a nearly unlimited number of sites, running on 2 catalogs (1 for the admin, 1 for the front end) and 1 database. The key to the system is different schemas, fronted by views, that hide and expose records based on the database user that is connected. The great thing about this system is that Jared can choose to launch a new store within a day and be ready for sales, something he has taken advantage of in the past when a team is on fire and he sees an opportunity he can't pass up.
We are currently preparing for a re-launch of the College District site that will focus on crowd-sourced designs. Artists and fans will submit their designs, have them voted on, some will be chosen to be sold and the folks that have their designs chosen will get paid for their efforts. The goal here is to grow a community that guides what College District and the individual school sites ultimately sell.
With College District's quick growth we've also been helping them improve their order fulfillment process. This includes streamlining how orders are picked, packed and shipped. The introduction of bar code scanners will help with the accuracy and speed of the process.
We get a kick out of seeing our clients succeed, especially those that come to us with a clear vision and a good attitude, and then put the hard work in to make it happen. It's an exciting year ahead for College District and we'll be right there supporting them on the journey.
Interchange Search Caching with "Permanent More"
Most sites that use Interchange take advantage of Interchange's "more lists". These are built-in tools that support an Interchange "search" (either the search/scan action, or result of direct SQL via [query]) to make it very easy to paginate results. Under the hood, the more list is a drill-in to a cached "search object", so each page brings back a slice from the cache of the original search. There are extensive ways to modify the look and behavior of more lists and, with a bit of effort, they can be configured to meet design requirements.
Where more lists tend to fall short, however, is with respect to SEO. There are two primary SEO deficiencies that get business stakeholders' attention:
- There is little control over the construction of the URLs for more lists. They leverage the scan actionmap and contain a hash key for the search object and numeric data to identify the slice and page location. They possess no intrinsic value in identifying the content they reference.
- The search cache by default is ephemeral and session-specific. This means all those results beyond page 1 the search engine has cataloged will result in dead links for search users who try to land directly on the more-listed pages.
It is the latter issue that I wish to address because there is--and has been for some time now--a simple mechanism called "permanent more" to remedy the default behavior.
You can leverage "permanent more" by adding the boolean mv_more_permanent, or the shorthand pm, to your search conditions. E.g.:
Link:
<a href="[area search="
co=1
sf=category
se=Foo
op=rm
more=1
ml=5
pm=1
"]">All Foos</a>
Loop:
[loop search="
co=1
sf=category
se=Foo
op=rm
more=1
ml=5
pm=1
"]
...loop body with [more-list]...
[/loop]
Query:
[query
list=1
more=1
ml=10
pm=1
sql="SELECT * FROM products WHERE category LIKE '%Foo%'"
]
...same as loop but with 10 matches/page...
[/query]
If the initial search is defined with the "permanent more" setting, it will produce the following adjustments:
- The hash key used to store and identify the search cache is deterministic based on the search conditions. Many searches for Interchange are category driven. Thus, all end users who wish to browse a category end up clicking identical links, which create duplicate search caches, belonging uniquely to them. With permanent more, they all share the same cache, with the same identifier. As long as the search conditions don't change, neither does the cache identifier. Even as the cache is refreshed with new executions of the search, the object remains in the same location. Thus, the results a search engine produced this morning reference links still valid now, tomorrow, or next week, provided they reference the same search conditions.
- The cached search object has no session affinity. Any link referencing the cache with the correct hash key has access to the content.
Taken together, "permanent more" removes (for the most part, addressed later) dead links from more lists cataloged by search engines. There are, however, other benefits to "permanent more" beyond those intended as described above:
- As stated in passing, standard Interchange search caching produces duplicate search objects for common search conditions. For a busy site, these caches can have an impact on storage. Typically, maintenance is implemented to clean up cache files for all such files whose age exceeds by some amount the session duration (standard is 48 hours). With permanent more, duplicate caches are eliminated. A cache location is reused by all users with the same search requirements, keeping data-storage requirements for caches to the minimum necessary. As searches change, ophaned caches can still easily be cleaned up as they will immediately start to age with no more access to them necessary for storage.
- For the same reason that "permanent more" resolves search-engine links, it also resolves content management for individual sites using a reverse proxy for caching. Because most (and certainly the easiest) caching keys are based off of URL, the deterministic nature of the hash keys for "permanent more" allows assurance that the cached content in the proxy accurately reflects the search content over time, and that all users will hit the cached resource and not generate new, unique links with varying hash keys.
One shortcoming of "permanent more" to be aware of is the impact of changing data underneath the search. Even if search conditions do not change, the count and order of matching record sets may. So, e.g., enough products may be removed from a given category to cause the last page of a more list to become empty, which would cause any specific link into that page to become dead. More minor, but still a possibility, is the introduction or removal of products so that a particularly searched-for term has been "bumped" to another page within the search cache since the last time the search engine crawled the more lists. For searches backed by particularly volatile data, "permanent more" may not be sufficient to address search-engine or caching demands.
Finally, "permanent more" should be avoided for any search features that may cache data sensitive to an individual user. This is unlikely to happen as, under most circumstances, the configuration of the search itself will change based on the unique characteristics of the user executing the search (e.g., a username included in a query to review order history). However, it is still possible that context-sensitive information could be stored in the search object and, if so, all other users with access to the more lists would have access to that information.
Global Variables in Interchange Jobs
Those familiar with writing global code in Interchange are certainly familiar with the number of duplicate references of certain global variables in different namespaces. For example, the Values reference is found in both the main namespace ($::Values) as well as in Vend::Interpolate ($Values usually from within usertags). One can also access the Values reference through the Session reference, which itself can be found in main ($::Session), Vend ($Vend::Session), and Vend::Interpolate ($Session usually from within usertags) namespaces with, e.g., $::Session->{values}. Most times, as long as context allows, any of those access points are interchangeable, and there's a good mix you see from developers using all of them.
In recent work for a client, I had developed an actionmap that incorporated access to the session for some of its coding--certainly not an uncommon occurrence. When I work in global space, I tend to use the main namespace references since they are available in all contexts within Interchange (or so I thought). The actionmap was constructed, tested, and put into production, where it worked as expected.
After a short period of operation, the client came to us and noted that in their actual operating procedure, the actionmap must process many more data points than we had it operate on in testing, causing it to take much more time. Thus, for their usual workload, they found the process was timing out and Interchange housekeeping reaping the process.
After a brief discussion, we decided the expedient course of action was to convert the work from a browser-initiated actionmap into an Interchange job. The code was easily exposed as a usertag as well, so in very short order we had the same functionality available as a job, where the job was now triggered by the browser access previously running the actionmap.
The change resolved the immediate problem, so now all work was completing, but the client brought a new issue to our attention. The reporting from the job was not as it was supposed to be. None of the code had been modified in the changeover, and the code when run as an actionmap produced the proper reporting.
The problem tracked down eventually to that session access. When the code was run in the context of the job, the Session reference was not copied into the main (or, as it turns out, Vend::Interpolate) namespace. Without the assumed session values in place, it was causing the report to produce invalid output.
To demonstrate, I constructed a simple usertag to dump the reference addresses of the 5 mentioned global variables:
UserTag ic-globals Routine <<EOR
sub {
return <<EOP;
. \$Session: $Session
\$Vend::Session: $Vend::Session
\$::Session: $::Session
\$Values: $Values
\$::Values: $::Values
EOP
}
EOR
I then created both a test page and an IC job that only called [ic-globals]. Running them both demonstrates the problem quite clearly.
From test page:
$Session: HASH(0xb0e1898)
$Vend::Session: HASH(0xb0e1898)
$::Session: HASH(0xb0e1898)
$Values: HASH(0xb0e1dd8)
$::Values: HASH(0xb0e1dd8)
Output from job:
$Session:
$Vend::Session: HASH(0xb221fa0)
$::Session:
$Values: HASH(0x926ddd8)
$::Values: HASH(0x926ddd8)
Interchange jobs provide yet a new context where you must consider your global variable usage. In particular, if you find code executed in the context of a job produces inconsistencies with the same code in other contexts, review your global variable usage and confirm those variables are what you assume they are.
SQL errors in Interchange
Interchange has a little feature whereby errors in a [query] tag are reported back to the session just like form validation errors. That is, given the intentional syntax error here:
[query ... sql="select 1 from foo where 1="]
Interchange will paste the error from your database in
$Session->{errors}{'table foo'}
That's great, but it comes with a price: now you have a potential for a page with SQL in it, which site security services like McAfee will flag as "SQL injection failures". Sometimes you just don't want your SQL failures plastered all over for the world to see.
Simple solution:
DatabaseDefault LOG_SESSION_ERROR 0
in your Interchange configuration file, possibly constrained so it only affects production (because you'd love to see your SQL errors when you are testing, right?).
Spree Performance Benchmarking
I see a lot of questions regarding Spree performance in the spree-user group, but they are rarely answered with metrics. I put together a quick script using the generic benchmark tool ab to review some data. Obviously, the answer to how well a site performs and scales is highly dependent on the host and the consumption of the web application, so the data here needs to be taken with a grain of salt. Another thing to note is that only two of the following use cases are running on Rails 3.0 — many of our current Spree clients are on Spree 0.11.2 or older. I also included one non-Spree Rails ecommerce application, in addition to a few non-Rails applications for comparison. All of the tests were run from my home network, so in theory there shouldn't be bias on performance tests for sites running on End Point servers.
| ab -n 100 | |||||
| -c 2 homepage | -c 20 homepage | -c 2 product page |
-c 20 product page |
||
|
Client #1 Spree: 0.11.2 Hosting: 4 cores, 512 GB RAM DB: MySQL # Products: <100 |
7.49 | 24.75 | 6.49 | 19.87 | Requests per second |
| 266.889 | 808.041 | 307.997 | 1006.552 | Time per request (ms) | |
|
Client #2 Spree 0.11.2 Hosting: Engineyard, medium instance DB: MySQL # Products: 100s |
5.32 | 20.28 | 5.36 | 18.03 | Requests per second |
| 375.713 | 986.309 | 373.289 | 1109.524 | Time per request (ms) | |
|
Client #3 Spree: 0.9.0 Hosting: 4 cores, 1 GB RAM DB: PostgreSQL # Products: <100 |
4.91 | 25.39 | 1.98 | 6.54 | Requests per second |
| 407.135 | 787.782 | 1011.875 | 3060.062 | Time per request (ms) | |
|
(Former) Client #4 Spree: 0.11.2 Hosting: Unknown DB: PostgreSQL # Products: >5000 |
20.69 | 8.84 | 10.15 | 19.28 | Requests per second |
| 96.673 | 2262.105 | 196.996 | 1037.146 | Time per request (ms) | |
|
Client #5 Spree: 0.11.2 Hosting: EngineYard, small instance DB: MySQL # Products: 1 |
12.28 | 16.23 | N/A | N/A | Requests per second |
| 162.909 | 1231.945 | N/A | N/A | Time per request (ms) | |
|
Client #6 Spree: 0.40 Hosting: 4 cores, 1 GB RAM DB: MySQL # Products: 50-100 |
3.61 | 8.93 | 2.96 | 3.06 | Requests per second |
| 553.569 | 2240.657 | 675.306 | 6539.433 | Time per request (ms) | |
|
SpreeDemo Spree: Edge Hosting: Heroku, 2 dynos DB: Unknown # Products: 100s |
8.17 | 12.79 | 4.7 | 5.48 | Requests per second |
| 244.831 | 1563.642 | 425.27 | 3652.927 | Time per request (ms) | |
|
Client #7 *custom Rails ecommerce app Hosting: 1.0 GB RAM DB: MySQL # Products: 1000s |
5.43 | 29.8 | 4.45 | 23.14 | Requests per second |
| 368.409 | 671.082 | 448.962 | 864.24 | Time per request (ms) | |
|
Interchange Demo Hosting: 4 cores, 2 GB RAM DB: MySQL # Products: >500 |
7.41 | 55.27 | 7.5 | 13.93 | Requests per second |
| 269.942 | 361.875 | 266.492 | 1435.51 | Time per request (ms) | |
|
Client #8 *PHP site, serves fully cached pages with nginx with no app server or db hits Hosting: 4 cores, 4 GB RAM |
10.81 | 30.54 | 6.05 | 9.87 | Requests per second |
| 184.994 | 654.858 | 330.727 | 2027.092 | Time per request (ms) | |
|
Magento Demo Hosting: Unknown DB: Unknown # Products: 100s |
4.26 | 44.85 | 2.68 | 36.29 | Requests per second |
| 469.831 | 445.931 | 745.472 | 551.11 | Time per request (ms) | |
Here's the same data in graphical form:
Requests per Second
Time Per Request (ms)
We expect to see high performance on some of the sites with significant performance optimization. On smaller VPS, we expect to see the the server choke with higher concurrency.
Product Personalization for Ecommerce on Interchange with Scene7
One of the more challenging yet rewarding projects Richard and I have worked on over the past year has been an ecommerce product personalization project with Paper Source. I haven't blogged about it much, but wanted to write about the technical challenges of the project in addition to shamelessly self-promote (a bit).
Personalize this and many other products at Paper Source.
Paper Source runs on Interchange and relies heavily on JavaScript and jQuery on the customer-facing side of the site. The "personalization" project allows you to personalize Paper Source products like wedding invitations, holiday cards, stationery, and business cards and displays the dynamic product images with personalized user data on the fly using Adobe's Scene7. The image requests are made to an external location, so our application does not need to run Java to render these dynamic personalized product images.
Technical Challenge #1: Complex Data Model
To say the data model is complex is a bit of an understatement. Here's a "blurry" vision of the data model for tables driving this project. The tables from this project have begun to exceed the number of Interchange core tables.
A snapshot of the data model driving the personalization project functionality.
To give you an idea of what business needs the data model attempts to meet, here are just a few snapshots and corresponding explanations:
 |
At the highest level, there are individual products that can be personalized. |
 |
Each product may or may not have different color options, or what we refer to as colorways. The card shown here has several options: gravel, moss, peacock, night, chocolate, and black. Clicking on each colorway here will update the image on the fly. |
 |
In addition to colorways, each product will have corresponding paper types and print methods. For example, each product may be printed on white or cream paper and each product may have a "digital printing" option or a letterpress option. Colorways shown above apply differently to digital printing and letterpress options. For example, letterpress colorways are typically a subset of digital printing colorways. |
|
Each card has a set of input fields with corresponding fonts, sizes, and ink colors. The input fields can be input fields or text boxes. All cards have their own specific set of data to control the input fields – one card may have 4 sections with 1 text field in each section while another card may have 6 sections with 1 text field in some sections and 2 text fields in other sections. In most cases, inks are limited between card colorways. For example, black ink is only offered on the black colorway card, and blue ink is only offered on the blue colorway card. |
 |
Each card also has a set of related items assigned to it. When users toggle between card colorways, the related item thumbnails update to match the detail option. This allows users to see an entire suite of matching cards: a pink wedding invite, RSVP, thank you, and stationery or a blue business card, matching letterhead stationery, and writing paper shown here. |
|
In addition to offering the parent product, envelopes are often tied to the products. In most cases, there are default envelope colors tied to products. For example, if a user selected a blue colorway product, the blue envelope would show as the default on the envelopes page. |
 |
In addition to managing personalization of the parent products, the functionality also meets the business needs to offer customization of return address printing on envelopes tied to products. For example, here is a personalized return address printing tied to my wedding invitation. |
Technical Challenge #2: Third Party Integration with Limited Documentation
There are always complexities that come up when implementing third-party service in a web application. In the case of this project, there is a fairly complex structure for image requests made to Scene7. In the case of dynamic invitations, cards, and stationery, examples of image requests include:
|
https://a248.e.akamai.net/f/248/9086/10h/origin-d7.scene7.com/is/image/?layer=0& anchor=-50,-50 &size=2000,2000&layer=1&src=is{PaperSource/A7_env_back_closed_sfwhite}& anchor=2900,-395 &rotate=-90&op_usm=1,1,8,0&resMode=sharp&qlt=95,1&pos=100,50&size=1800,1800& layer=3&src=fxg {PaperSource/W136-122208301?&imageres=150}&anchor=0,0&op_usm=1,1,1,0&pos=500,315& size=1732,1732 &effect=0&resMode=sharp&fmt=jpg |
|
https://a248.e.akamai.net/f/248/9086/10h/origin-d7.scene7.com/is/image/?layer=0& anchor=-50, 50&size=2000,2000&layer=1&src=is{PaperSource/4bar_env_back_closed_fig}&anchor=0,0& pos=115,375 &size=1800,1800&layer=2&src=fxg{PaperSource/4barV_white_background_key}&anchor=0,0& rotate=-90& pos=250,1757&size=1733,1733&layer=3& src=fxg{PaperSource/ST57-2011579203301?&$color_fig=true&$color_black=false&$ color_chartreuse =false&$color_espresso=false&$color_moss=false&$color_peacock=false &$ink_0=780032 &$ink_2=780032&$ink_1=780032&imageres=150}&anchor=0,0 &op_usm=2,1,1, 0&pos=255,513&size=1721,1721&resMode=sharp&effect=0&fmt=jpg |
|
https://a248.e.akamai.net/f/248/9086/10h/origin-d7.scene7.com/is/image/?layer=0&anchor=-50,- 50&size=2000,2000&layer=1&src=is{PaperSource/4bar_env_back_closed_night}&anchor=0,0&pos =115,375&&size=1800,1800&layer=2&src=fxg{PaperSource/4bar_white_sm} &anchor=0,0&rotate=-90&pos=250,1757&size=1733,1733&layer=3&src=fxg{PaperSource/W139-201203301?} &anchor=0,0&op_usm=2,1,1,0&pos=255,513&size=1721,1721&resMode=sharp&effect=0&fmt=jpg |
Each argument is significant to the dynamic image; background envelope color, card colorway, ink color, card positioning, envelope positioning, image quality, image format, and paper color are just a few of the factors controlled by the image arguments. And part of the challenge was dealing with the lack of documentation to build the logic to render the dynamic images.
Conclusion
As I mentioned above, this has been a challenging and rewarding project. Paper Source has sold personalizable products for a couple of years now. They continue to move their old personalized products to use this new functionality including many stationery products moved yesterday. Below are several examples of Paper Source products that I created with the new personalized functionality.
 |
 |
 |
 |
 |
 |
Ecommerce Solutions: What are the Options?
Lately, I've been evaluating ecommerce options for use on a side hobby/business. I'm obviously a developer, so in theory I could use one of End Point's supported ecommerce frameworks or just write my own framework. But, my bottom line is that I don't need the feature set offered by some of the ecommerce options out there and I don't necessarily have the resources to develop a custom solution.
In addition to personal interest, End Pointers constantly encounter potential clients who aim to get a better understanding of the cost of using open source, our preferred ecommerce solution. I put together two infographics on ecommerce options, ongoing cost, feature sets, and the ability to customize. Before anyone *flips out* about the infographics, note that they represent my broad generalizations regarding the ongoing cost, feature sets and ability to customize. I'm intimately familiar with some of these options and less familiar with a couple of them.
Feature Set versus Ongoing Cost of Ecommerce Solutions
Ability to Customize versus Ongoing Cost of Ecommerce Solutions
Some notes on on the ecommerce solutions shown in the infographics:
- Online payment service (Paypal): An online payment collection service like PayPal offers a minimal "ecommerce" feature set and might be suitable for someone looking to simply collect money. It also provides almost no ability to customize. The ongoing cost of PayPal is lower than many of the other options, where a percentage of each sale goes to PayPal.
- Online catalog service (Etsy, eBay): An online catalog service such as Etsy or eBay offers very basic ecommerce listing features, little to no ability to customize, but has relatively low ongoing cost. For example, Etsy charges $0.20 to list a single item for four months and takes 3.5% of the sales fee.
- Hosted ecommerce (shopify, Big Cartel, Big Commerce, Yahoo Merchant): A hosted ecommerce solution offers the ability to customize the appearance, typically in the form of a custom template language and has a basic ecommerce feature set. Ongoing costs would include the cost of the service, domain registration cost, and payment gateway (e.g. Authorize.NET) fees. Additional cost may apply if development services are required for building a custom template. Shopify's basic solution costs $29/mo. with a max of 100 skus. Shopify offers several other higher-priced options which include more features and have less limitations. Most other hosted ecommerce solutions are similarly priced.
- Open source ecommerce (Interchange, Spree, Magento, Zen Cart, osCommerce, prestaShop): An open-source ecommerce solution tends to have generic ecommerce features, provides the opportunity for a large amount of customization, but is typically more expensive than hosted ecommerce solutions and online catalog services. The software itself is free, but ongoing costs of hosting (server, domain registration, SSL certificate), development (any piece of customization that does not fit into the generic mold), and payment gateway fees apply. One positive about open source ecommerce is that additional plugins or add-ons are produced by members of the community. If the business needs are satisfied by the community-available extensions, cost from additional development or customization may be eliminated or reduced. I'd also probably group in existing open source plugins or modules for open source CMS solutions like WordPress and Drupal into this category: the generic solution is free, but additional resources may be required for customization.
- Enterprise ecommerce (ATG, Magento Enterprise): From my experience, I've observed that enterprise ecommerce solutions tend to offer a large feature set and a similar ability to customize as open source frameworks. The ongoing cost is high: for example, Magento Enterprise starts at $12,990/year. In addition to the licensing cost, hosting, development, and payment gateway fees may apply.
- Custom ecommerce (homegrown): The cost of writing your own ecommerce framework depends on the functionality requirements. The ability to customize is unlimited since the entire solution is custom. The feature set is likely to be proportional to resources spent on the project. Ongoing costs here include hosting, development and payment gateway fees.
Which one do you choose?
It has been my experience that most of End Point's ecommerce clients need some type of customization: a custom appearance, discount functionality, shipping integration, payment gateway integration, social media integration, and other third-party integration. Choosing an option that allows easy customization tends to benefit our customers in the long run. We're pretty biased at End Point towards open source ecommerce solutions, but my opinion recently is that with the advancement of web frameworks and web framework tools (e.g. gems in ruby), development of custom solutions can be done efficiently and may be a better option for a site that is outside the realm of standard ecommerce. For businesses not able to pay development or consulting costs, hosted ecommerce solutions are affordable and provide the essentials needed as the business grows.
Using nginx to transparently modify/debug third-party content
In tracking down a recent front-end bug for one of our client sites, I found myself needing to use the browser's JavaScript debugger for stepping through some JavaScript code that lived in a mix of domains; this included a third-party framework as well as locally-hosted code which interfaced with -- and potentially interfered with -- said third-party code. (We'll call said code foo.min.js for the purposes of this article.) The third-party code was a feature that was integrated into the client site using a custom domain name and was hosted and controlled by the third-party service with no ability for us to change directly. The custom domain name was part of a chain of CNAMEs which eventually pointed to the underlying *actual* IP of the third-party service, so their infrastructure obviously relied on getting the Host header correctly in the request to select which among many clients was being served.
It appeared as if there was a conflict between code on our site and that imported by the third party service. As part of the debugging process, I was stepping through the JavaScript in order to determine what if any conflicts there were, as well as their nature (e.g., conflicting library definitions, etc.). Stepping through our code was fine, however the third-party's JS code was (a) unfamiliar, and (b) minified, so this had the effect of putting all of the JavaScript code more-or-less on one line, which made tracing through the code in the debugger much less useful than I had hoped.
My first instinct was to use a JavaScript beautifier to reverse the minification process, but since I had no control over the code being included from the third-party service, this did not seem to be directly feasible. The third-party code was deployed only on our production site and relied on hard-coded domains which would make integrating it into one of our development instances challenging since we had no control over the contents of the returned resources. Since the relevant feature (and subsequent bugs) was only on the production site, making extensive modifications to how things were done and potentially breaking that or other features for users while I was debugging was obviously out as an option.
Enter nginx. I've been doing a lot with nginx lately as far as using it as a reverse proxy cache, so it's been on my mind lately. So I came up with this technique:
- Look up the IP address for the third-party's domain name (used for later purposes).
- Install nginx on localhost, listening to port 80.
- Modify /etc/hosts to point the third-party's domain name to the nginx server's IP (also localhost in this case).
- Configure a new virtual host with the following logical constraints:
- We want to serve specific files (the beautified JavaScript) from our local server.
- We want any other request going through that domain to be passed-through transparently, so neither the browser nor the third-party server treat it differently.
Given these constraints, this is the minimal configuration that I came up with (the interesting parts are located in the server block):
/etc/hosts:
example.domain.com 127.0.0.1
nginx.conf:
worker_processes 1;
events {
worker_connections 10;
}
http {
include mime.types;
default_type application/octet-stream;
server {
server_name example.domain.com;
root /path/to/local_root;
try_files $uri @proxied;
location @proxied {
proxy_set_header Host $http_host;
proxy_pass http://1.2.3.4;
}
}
}
Once I had the above configured/setup, I downloaded/saved the foo.min.js file from the third-party service, ran it through a JS beautifier, and saved it in the local nginx's cache root so it would be served up instead of the actual file from the third-party service. Any other requests for static resources (images, other scripts, etc) would pass-through to the third-party server, so I had my nicely-formatted JavaScript code to step through, the production site worked as normal for anyone else despite potential local changes to the file on my end (i.e., adding JavaScript alert() calls to the file, and no one was the wiser.
A few notes
The try_files directive instructs nginx to first look for a file named after the current URI (foo.min.js in our example) in our local cache, and if this is not found, then fallback to the proxied location block; i.e., relay the request to the original upstream server. We explicitly set the Host header on the proxy request because we want the request to behave normally with respect to name-based hosting, and provide the saved IP address to contact the server in question.
We only needed to preserve/lookup the upstream server's IP address because we're running the nginx server on localhost, so if we used a domain name the lookup would return the same IP defined in /etc/hosts; if the nginx server was running on a different machine, you would be able to just use the domain name as both the server_name and the proxy_pass parameters and set the entry for the host in your local /etc/hosts file to the IP of the nginx server.
A possible extension would be to detect when an upstream request matched a minified URL (via a location ~ \.min\..*\.js$ block) and automatically beautify/cache the content in our local cache. This could be accomplished via the use of an external FastCGI script to retrieve, post-process, and cache the content.
This technique can also be used when dealing with testing changes to a production site on which you are unable or unwilling to make potentially disruptive changes for the purposes of testing static resources. (JavaScript seems the most obvious application here, but this could apply to serving up images or other static content which would be resolvable by the local cache.)
I always need to remind myself to undo changes to /etc/hosts as soon as I'm done testing when using tricks like these. Particularly in something like this which is more-or-less transparent, the behavior would be functionaly identical as long as code/scripts on the third-party site stayed the same, but could easily introduce subtle bugs if the third-party services made changes to their codebase. Since our local copies would mask any remote changes for those non-proxied resources, this could be very confusing if you forget that things are set up this way.
New Year Bug Bites
Happy New Year! And what would a new year be without a new year bug bite? This year we had one where figuring out the species wasn't easy.
On January 2nd one of our ecommerce clients reported that starting with the new year a number of customers weren't able to complete their web orders because of credit card security code failures. Looking in the Interchange server error logs we indeed found a significant spike in the number of CVV2 code verification failures (Payflow Pro gateway error code "114") starting January 1st.
We hadn't made any programming or configuration changes on the system in the recent days. We double-checked to make sure: nope, no code changes. So it had to be a New Year's bug and presumably something with the Payflow Pro gateway or banks further upstream. We checked error logs for other customers to see if they were being similarly impacted, but they weren't. Our client contacted PayPal (the vendor for Payflow Pro) and they reported there were no problems with their system. The failures must indeed be card failures or a problem with the website according to them. We further checked our code looking for what we could possibly have done that might be the cause, double-checking our Git repository (which showed no recent changes) and reexamining our checkout code for possible year-based logic flaws.
Our client's top-notch customer service group got on the phone with a customer who'd gotten a security code failure and got PayPal tech support on another line. The customer service rep tried to place the customer's order on the website using the customer's credit card info and once again got the CVV2 error. She then did the credit card transaction using the swipe machine in the office, and lo and behold the order went through! What was going on??!
It turned out that despite the Payflow Pro gateway returning CVV2 verification errors what was really happening was that the year of the credit card was coming into the Payflow Pro gateway as "2012"—not as "2011" as entered into the checkout form. We knew all along that it was possible that the 114 error code responses were possibly misleading because payment gateway error codes are notorious this way. (Payment gateways blame the banks, saying they can only pass along what the banks give them. Some banks' credit card validations don't actually even care about the years being correct, but just that they not be in the past; but I digress...)
Previously we'd reviewed the checkout pages and the dropdown menus to verify that the dropdown menus weren't off, but nevertheless it very much sounded like this rather stupid problem could very well be the culprit. So we checked and checked again. What we found is that sometimes on the checkout form the year dropdown menu was mangled such that the values associated with the displayed years were YYYY+1.
The oddly intermittent behavior of the problem, the process of elimination and the all around hair pulling this loss of business was causing made somebody in the marketing group at our client realize that they are in fact still running an Omniture Test & Target A/B test on the checkout pages that they thought had been discontinued. To quote David Christensen (thanks, David!): "The Omniture system works by replacing select content for randomly chosen users in an effort to track user behavior/response to proposed site changes. Alternate site content is created and dynamically replaced for these users as they use the site, such as the specific content on the checkout page in this instance."
We verified that the Omniture A/B test's JavaScript replacement code was alternately mangling and not mangling the year dropdown on the checkout form as mentioned. Our client took out the A/B test and the "security code errors" dropped back to a normal low level.
This was a difficult and expensive problem—not only was there business lost because of the problem, but there were a lot of resources put into troubleshooting it. We've come away from this episode with some lessons learned and with plenty of food for thought. I'll leave it to commentators to opine away on this, including the End Point folks who scratched this itch: David Christensen, Jeff Boes, Mark Johnson, and Jon Jensen.
Version Control Visualization and End Point in Open Source
Over the weekend, I discovered an open source tool for version control visualization, Gource. I decided to put together a few videos to showcase End Point's involvement in several open source projects.
Here's a quick legend to help understand the videos below:
| The branches and nodes correlate to directories and files, respectively. In the case of the image to the left, the repository has a main directory with several files and three directories. One of the child directories has one file and the other two have multiple files. |
| A big dot represents a person, and a flash connecting the person and a file signifies a commit. |
| White + blue dots represent current End Point employees. |
| White + grey dots represent former End Point employees. |
| White dots represent other people, out there! |
The Videos
Interchange from endpoint on Vimeo.
Bucardo from endpoint on Vimeo.
One of the articles that references Gource suggests that the videos can be used to visualize and analyze the community involvement of a project (open source or not). One might also be able to qualitatively analyze the stability of project file architecture from a video, but this won't reveal anything definitive about the code stability since external factors can influence file structure. For example, since I am intimately familiar with the progress of Spree, I can identify when Spree transitioned to Rails 3 in the video, which required reorganization of the Spree core functionality (read more about this here and here).
In the case of this article, I wanted to highlight End Point's involvement in a few open source projects where we've had various levels of involvement. We've contributed to Interchange since 2000. We've been involved in Spree less lately, but had more presence in early 2009. In the smaller projects Bucardo and pgsi, End Point employees have worked on a team to be the primary contributors to the projects in addition to a few external contributors. Open source is important to End Point, and it's great to see our presence demonstrated in these cute videos.
Character encoding in perl: decode_utf8() vs decode('utf8')
When doing some recent encoding-based work in Perl, I found myself in a situation which seemed fairly unexplainable. I had a function which used some data which was encoded as UTF-8, ran Encode::decode_utf8() on said data to convert to Perl's internal character format, then converted the "wide" characters to the numeric entity using HTML::Entities::encode_entities_numeric(). Logging/printing of the data on input confirmed that the data was properly formatted UTF-8, as did running `iconv -f utf8 -t utf8 output.log >/dev/null` for the purposes of review.
However when I ended up processing the data, it was as if I had not run the decode function at all. In this case, the character in question was € (unicode code point U+20AC). The expected behavior from encode_entities_numeric() would be to turn any of the hi-bit characters in the perl string (i.e. all Unicode code points > 0x80) into the corresponding numeric entity (€ - € in this case). However instead of that specific character's numeric entity appearing in the output, the entities which appeared were: € i.e., the raw UTF-8 encoded value for €, with each octet being treated as an independent character instead of part of the whole encoded value.
What was particularly confusing was that extracting the relevant parts from the script in question resulted in the expected answer, so it was clearly not an issue of HTML::Entities not being able to deal with Unicode characters, as this code snippet demonstrates:
$ perl -MHTML::Entities+encode_entities_numeric -MEncode -e '$c=qq{\xE2\x82\xAC}; print encode_entities_numeric(decode_utf8($c))'
--> €
In the actual non-extracted version of the code, I was scratching my head. This was exhibiting the signs of doubly-encoded data, however I couldn't see how that could be the case. There were no PerlIO layers (e.g., :utf8 or :encoding) at play, the data I was outputting to a log file for verification purposes was being written via a brand new filehandle from a bare open(); I verified in multiple ways that the raw octets being passed in to the function were not doubly-encoded (printing the raw character points, counting lengths of the runs of octets and verifying that these matched the length of the UTF-8 encoded value for the represented characters, etc). The more things I tried the more puzzled I got. Finally, I changed the Encode::decode_utf8() call to a Encode::decode('utf8') one, providing the encoding explicitly. At this point, the processing pipeline started working as expected, and hi-bit characters were being output as their full numeric entities.
Since the documentation for decode_utf8 indicated that it should be identical to decode('utf8'), I resorted to the code to find out why it worked with the version that specified the encoding explicitly. I found that decode_utf8() does one additional thing that the regular decode('utf8') does not, and that is that before processing via the regular decode() function, decode_utf8 first checks the UTF-8 flag of the data that is being passed in, and if it is set it returns the data without further decoding*. My best guess is that this is to prevent errors if someone attempts to decode UTF-8 data in a string which is already in Perl's internal format, so in most cases this will provide a caller-friendly interface that will DWYM in many expected cases.
Armed with this knowledge, I verified that for some reason, the data that was being passed into the function had the UTF-8 flag set, so using the explicit decode('utf8') in lieu of decode_utf8() fixed the issue for me. (Tracing down the reason for the UTF-8 flag being set on this data was out of scope for this exercise, but is the true fix.) And just to verify that this was in fact the cause of the issue at hand, here's our example, modified slightly (we use the utf8::upgrade function to turn the UTF-8 flag on in the data and treat as actual encoded characters instead of raw octets):
$ perl -l -MHTML::Entities+encode_entities_numeric -MEncode -Mutf8 -e '$c=qq{\xE2\x82\xAC}; utf8::upgrade($c); print encode_entities_numeric(decode_utf8($c))'
--> €
* The UTF-8 flag is more-or-less an implementation detail of how Perl is able to deal with legacy 8-bit binary data in no particular encoding (i.e., raw octets, which it treats as latin-1) as well as the full range of Unicode data, and deal with both efficiently and in a backwards-compatible manner.
SearchToolbar and dropped Interchange sessions
A new update to Interchange's robots.cfg can be found here. This update adds "SearchToolbar" to the NotRobotUA directive which is used to exclude certain user agent strings when determining whether an incoming request is from a search engine robot or not. The SearchToolbar addon for IE and FireFox is being used more widely and we have received reports that users of this addon are unable to add items to their cart, checkout, etc. You may remember a similiar issue with the Ask.com toolbar that we discussed in this post. If you are using Interchange you should download the latest robots.cfg and restart Interchange.
Keep the Aisles Clean at Checkout
It's no mystery in ecommerce that checkout processing must flow smoothly for an effective store. Providing products or services in high demand doesn't mean much if they cannot be purchased, or the purchase process is so burdensome that would-be customers give up in frustration.
Unfortunately, checkout also tends to include the most volatile elements of a web store. It virtually always involves database writes, which can be hindered by locking. It often involves real-time network access to 3rd-party providers, with payment transactions being at the top of the list. It can involve complex inventory assessments, where high concurrency can make what's normally routine highly unpredictable. Meanwhile, your customers wait, while the app sifts through complexity and waits on responses from various services. If they wait too long, you might lose sales; even worse, you might lose customers.
Even armed with the above knowledge, it's all too easy to fall into the trap of expediency. A particular action is so logically suited to be included as part of the checkout routine, and a superficial evaluation makes it seem like such a low-risk operation. That action can be tucked in there just after we've passed all the hurdles and are assured the order won't be rejected--why, it'll be so simple, and all the data we need for the action are readily at hand.
Just such expediency was at the heart of a checkout problem that had been plaguing an Interchange client of ours for months. The client would receive regular complaints that checkouts were timing out or taking so long that the customer was reloading and trying again. Many times, these customers would come to find that their orders had been placed, but that the time to complete them was exceeding the web server's timeout (or their patience). In far less common instances, but still occurring regularly, log and transaction evidence existed that showed an order attempt produced a valid payment transaction, but there was no hint of the order in their database or even in the application's system logs.
In the latter case of behavior, I had seen this before for other clients. If an action within order routing takes long enough, the Interchange server handling the request will be hammered by housekeeping. The telltale sign is the lack of log evidence for the attempt since order routes are logged at the end of the route's run; when that's interrupted, then no logging occurs.
I added considerably more explicit real-time logging and picked off some of the low-hanging fruit--code practices that had often been implicated before as the culprit in these circumstances. After collecting enough data for problematic order attempts, I was able to isolate the volatility to mail-list maintenance. The client utilizes a 3rd-party provider for managing their various mail lists, and that provider's API was contacted during order routing with all the data the provider needed for managing said lists. The data transfer for the API was very simple, and in most cases would process in sub-second time. Unfortunately, it turned out that, in enough cases, the calls to the API would take 10s to even 100s of seconds to process.
The placement of maintaining mail lists within order routing was merely convenience. The success or failure of adding to the mail lists was insignificant compared to the success or failure of the order itself. Once identified, the API calls were moved into a post-order processing routine, which was specifically built to anticipate the demonstrated volatility. As a result, complaints from customers on long or timed-out checkouts have dwindled to near zero, and the mail-list maintenance is more reliable since the background process is designed to catch excessively long process calls and retry until we receive an affirmative response from the list maintainers.
When deciding what belongs within checkout processing, ideally limit that activity to only those actions absolutely imperative to the success of the order. For each piece of functionality, ask yourself (or your client): is the outcome of this action worth adding to the wait a customer experiences placing an order? Should the outcome of this action affect whether the order attempt is successful? If the answer to those questions is "no", account for that action outside of checkout. It may be more work to do so, but keeping the checkout aisles clean, without obstruction, should be paramount.
SEO friendly redirects in Interchange
In the past, I've had a few Interchange clients that would like the ability to be able to have their site do a SEO friendly 301 redirect to a new page for different reasons. It could be because either a product had gone out of stock and wasn't going to return or they completely reworked their url structures to be more SEO friendly and wanted the link juice to transfer to the new URLs. The normal way to handle this kind of request is to set up a bunch of Apache rewrite rules.
There were a few issues with going that route. The main issue is that to add or remove rules would mean that we would have to restart or reload Apache every time a change was made. The clients don't normally have the access to do this so it meant they would have to contact me to do it. Another issue was that they also don't have the access to modify the Apache virtual host file to add and remove rules so again, they would have to contact me to do it. To avoid the editing issue, we could have put the rules in a .htaccess file and allow them to modify it that way, but this can present its own challenges because some text editors and FTP clients don't handle hidden files very well. The other issue is that even though overall basic rewrite rules are pretty easy to copy, paste and reuse, they still can have nasty side effects if not done properly and can also be difficult to troubleshoot so I devised a way to allow them to be able to manage their 301 redirects using a simple database table and Interchange's Autoload directive.
The database table is a very simple table with two fields. I called them old_url and new_url with the primary key being old_url. The Autoload directive accepts a list of subroutines as its arguments so this requires us to create two different GlobalSubs. One to actually do the redirect and one to check the database and see if we need to redirect. The redirect sub is really straight forward and looks like this:
sub redirect {
my ($url, $status) = @_;
$status ||= 302;
$Vend::StatusLine = qq|Status: $status moved\nLocation: $url\n|;
$::Pragma->{download} = 1;
my $body = '';
::response($body);
$Vend::Sent = 1;
return 1;
}
The code for the sub that checks to see if we need to redirect looks like this:
sub redirect_old_links {
my $db = Vend::Data::database_exists_ref('page_redirects');
my $dbh = $db->dbh();
my $current_url = $::Tag->env({ arg => "REQUEST_URI" });
my $normal_server = $::Variable->{NORMAL_SERVER};
if ( ! exists $::Scratch->{redirects} ) {
my $sth = $dbh->prepare(q{select * from page_redirects});
my $rc = $sth->execute();
while ( my ($old,$new) = $sth->fetchrow_array() ) {
$::Scratch->{redirects}{"$old"} = $new;
}
$sth->finish();
}
if ( exists $::Scratch->{redirects} ) {
if ( exists $::Scratch->{redirects}{"$current_url"} ) {
my $path = $normal_server.$::Scratch->{redirects}{"$current_url"};
my $Sub = Vend::Subs->new;
$Sub->redirect($path, '301');
return;
} else {
return;
}
}
}
We normally create these as two different files and put them into our own directory structure under the Interchange directory called custom/GlobalSub and then add this, include custom/GlobalSub/*.sub, to the interchange.cfg file to make sure they get loaded when Interchange restarts. After those files are loaded, you'll need to tell the catalog that you want it to Autoload this subroutine and to do that you use the Autoload directive in your catalog.cfg file like this:
Autoload redirect_old_links
After modifying your catalog.cfg file, you will need to reload your catalog to ensure to change takes effect. Once these things are in place, you should just be able to add data into the page_redirects table and start a new session and it will redirect you properly. When I was working on the system, I just created an entry that redirected /cgi-bin/vlink/redirect_test.html to /cgi-bin/vlink/index.html so I could ensure that it was redirecting me properly.
Providing Database Handle for Interchange Testing
I've recently begun using the test driven development approach to my projects using Perl's Test::More module. Most of my projects lately have been with Interchange which has some hurdles to get around as far as test driven development is concerned. Primarily this is because Interchange runs as a daemon and provides some readily available utilites like the database handle. This method is not available to our tests, so they need to be made available as discussed below.
I develop Usertags, GlobalSubs and ActionMaps where applicable as it helps keep the separation of business logic and views clear. I generally organize these to call a function within a Perl module so they can be tested properly. Most of these tags involve some sort of connection with the database to present information to the user in which I uses the Interchange ::database_exists_ref method.
When it comes to testing I want to ensure that the test script invokes the same method. Otherwise, your script will not be testing the code as its used in production.
Let's say you are building a Perl module that looks something like this:
package YourMagic;
use strict;
sub do_something {
my ($opt) = @_;
# some code
my $dbh = ::database_exists_ref($opt->{table})->dbh
or return undef;
# ... more code
return $output;
}
1;
The ::database_exists_ref() method will not be available for a test script and needs to be defined. It should return an object to the dbh method in the test script as it does within Interchange. There is no need to test the method itself, as it is not part of the "what" that is being developed. The following code needs to be added to the test script so it can handle the correct type of database reference returned by Interchange.
use lib '/home/user/interchange/custom/lib';
use Test::More tests => 2;
use DBI;
# Here are the methods to provide proper reference to our database handle
################################
sub ::database_exists_ref {
my $table = shift;
return undef if !$table;
# return an object with a dbh method
return bless({}, __PACKAGE__);
}
sub dbh {
# define a dbh method
my $db = DBI->connect('dsn, 'user', 'pass');
return $db;
}
##################################
use YourMagic;
is(
YourMagic::do_something(),
undef,
'do_something() returns undef when called with no arguments',
);
is(
YourMagic::do_something(\%opt),
undef,
'do_something() returns ...',
);
It is also worthwhile to note that you'll need to use the ::database_exists_ref method to look up some information from the existing table that is valuable to test against. Now the do_something() method will call ::database_exists_ref() when invoked.
This approach allows us to use, reuse, and add new tests without worrying about mock data during the intial development. You can be sure that the existing test scripts will function properly against the latest data that is available.
I will cover some other topics regarding Interchange Test Driven Development in future posts. For more information regarding Unit Testing in general see this post by Ethan.
jQuery Auto-Complete in Interchange
"When all you have is a hammer, everything looks like a nail."
Recently, I've taken some intermediate steps in using jQuery for web work, in conjunction with Interchange and non-Interchange pages. (I'd done some beginner stuff, but now I'm starting to see nails, nails, and more nails.)
Here's how easy it was to add an auto-complete field to an IC admin page.
In this particular application, a <select> box would have been rather unwieldy, as there were 400+ values that could be displayed.
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"
type="text/javascript"></script>
<script type="text/javascript" src="http://dev.jquery.com/view/trunk/plugins/autocomplete/lib/jquery.bgiframe.min.js"></script>
<script type="text/javascript" src="http://dev.jquery.com/view/trunk/plugins/autocomplete/lib/jquery.dimensions.js"></script>
<script type="text/javascript" src="http://dev.jquery.com/view/trunk/plugins/autocomplete/jquery.autocomplete.js"></script>
That's the requisite header stuff. Then you set up the internal list of autocomplete terms:
<script type="text/javascript">
$('document').ready(function(){
var auto_list = "[perl]...[/perl]".split(" ");
$('input[name="auto_field"]').autocomplete(auto_list);
});
</script>
The [perl] block just needs to emit a space-delimited list of the autocomplete terms. For instance,
[perl table="foo"]
return join(' ', map { $_->[0] }
@{ $Db{foo}->query('SELECT keycol FROM foo ORDER BY 1') });
[/perl]
Guidelines for Interchange site migrations
I'm involved at End Point often with Interchange site migrations. These migrations can be due to a new client coming to us and needing hosting or migrating from one server to another within our own infrastructure.
There are many different ways to do a migration, in the end though we need to hit on certain points to make sure that the migration goes smoothly. Below you will find steps which you can adapt for your specific migration.
During the start of the migration it might be a good time to introduce git for source control. You can do this by creating the repository and cloning it to /home/account/live, setting up .gitignore files for logs, counter files, gdbm files. Then commit the changes back to the repo and you've now introduced source control without much effort, improving the ability to make changes to the site in the future. This is also helpful to document the changes you make to the code base along the way during the migration in case you need to merge changes from the current production site before completing the migration.
- Export all of the gdbm databases to their text file equivalents on the production server
- Take a backup from production of the database, catalog, interchange server, htdocs
- Setup an account
- Create the database and user
- Restore the database, catalog, interchange server and htdocs
- Update the paths in interchange/bin for each script to point at the new location
- Grep the restored code for hard coded paths and update those paths to the new locations. Better yet move these paths out to a catalog_local.cfg where environment specific information can go.
- Grep the restored code for hard coded urls and use the [area] tag to generate the urls
- Update the urls in products/variable.txt to point at the test domain
- Update the sql settings in products/variable.txt to point at the new database using the new user
- Remove the gdbm databases so they will be recreated on startup from the source text files
- Install a local Perl if it's not already installed (./configure -des will compile and install Perl locally)
- Install Bundle::InterchangeKitchenSink
- Install the DBD module for MySQL or PostgreSQL
- Review the code base looking for use statements in custom code and Require module settings in interchange.cfg. Install the Perl modules found into the local Perl.
- Setup a non ssl and ssl virtual host using a temporary domain. Configure the temporary domain to use the SSL certificate from the production domain.
- Firewall or password protect the virtual host so it is not accessible to the public
- Generate a vlink using interchange/bin/compile and copy it into the cgi-bin directory and name it properly
- Startup the new Interchange
- Review error messages and resolve until Interchange will start properly
- Test the site thoroughly, resolving issues as they appear. Make sure that checkout, charging credit cards, sending of emails, using the admin, etc all function.
- Migrate any cron jobs running on the current production site, such as session expiration scripts
- Setup logrotation for the new logs that will be created
- Verify that you have access to make DNS changes
- Set the TTL for the domain to a low value such as 5 minutes
- Modify the new production site to respond to the production url, test by updating your hosts file to manually set the IP address of the domain
- Shutdown the new Interchange
- Restore a copy of the original backup for Interchange, the catalog and htdocs to /tmp on the production server
- Shutdown the production Interchange, put up a maintenance note on the production site.
- Take a backup of the production database and restore on the new server
- Diff the Interchange, catalog and htdocs directory between /tmp and the current production locations, making note of the files that have changed since we took the original copy.
- Copy the files that have changed, making sure to merge with any changes we have made on the new production site. Making sure to copy over all .counter and .autonumber files to the new production site.
- Start Interchange on the new production server
- Test the site thoroughly on the new production server, using the production url. Make sure that checkout with charging the credit card functions properly.
- Resolve any remaining issues found during the testing
- Setup the Interchange daemon to start at boot for this site in /etc/rc.d/rc.local or in cron using @reboot
- Update DNS to point at the new production IP address
- Update the TTL of the domain to a longer value
- Open the site to the public by opening the firewall or removing the password protection
- Keep an eye on the error logs for any issues that might crop up
This will hopefully give you a solid guide for performing an Interchange site migration from one server to another and some of the things to watch out for that might cause issues during the migrations.
