Interchange loops using DBI Slice

One day I was reading through the documentation on search.cpan.org for the DBI module and ran across an attribute that you can use with selectall_arrayref() that creates the proper data structure to be used with Interchange's object.mv_results loop attribute. The attribute is called Slice which causes selectall_arrayref() to return an array of hashrefs instead of an array of arrays. To use this you have to be working in global Perl modules as Safe.pm will not let you use the selectall_arrayref() method.

An example of what you could use this for is an easy way to generate a list of items in the same category. Inside the module, you would do like this:

my $results = $dbh->selectall_arrayref(
  q{
    SELECT
      sku,
      description,
      price,
      thumb,
      category, 
      prod_group
    FROM
      products
    WHERE
      category = ?},
  { Slice => {} }, 
  $category
);
$::Tag->tmpn("product_list", $results);

In the actual HTML page, you would do this:

<table cellpadding=0 cellspacing=2 border=1>
<tr>
  <th>Image</th>
  <th>Description</th>
  <th>Product Group</th>
  <th>Category</th>
  <th>Price</th>
</tr>
[loop object.mv_results=`$Scratch->{product_list}` prefix=plist]
[list]
<tr>
  <td><a href="/cgi-bin/vlink/[plist-param sku].html"><img src="[plist-param thumb]"></a></td>
  <td>[plist-param description]</td>
  <td>[plist-param prod_group]</td>
  <td>[plist-param category]</td>
  <td>[plist-param price]</td>
</tr>
[/list]
[/loop]
</table>

We normally use this when writing ActionMaps and using some template as our setting for mv_nextpage.

Some great press for College District

College District has been getting some positive press lately, the most recent being a Forbes article which talks about the success they have been seeing in the last few years.

College District is a company that sells collegiate merchandise to fans. They got their start focusing on the LSU Tigers at TigerDistrict.com and have branched out to teams such as the Oregon Ducks and Alabama Roll Tide.

We've been working with Jared Loftus @ College District for more then four and a half years. College District is running on a heavily modified Interchange system with some cool Postgres tricks. The system can support a nearly unlimited number of sites, running on 2 catalogs (1 for the admin, 1 for the front end) and 1 database. The key to the system is different schemas, fronted by views, that hide and expose records based on the database user that is connected. The great thing about this system is that Jared can choose to launch a new store within a day and be ready for sales, something he has taken advantage of in the past when a team is on fire and he sees an opportunity he can't pass up.

We are currently preparing for a re-launch of the College District site that will focus on crowd-sourced designs. Artists and fans will submit their designs, have them voted on, some will be chosen to be sold and the folks that have their designs chosen will get paid for their efforts. The goal here is to grow a community that guides what College District and the individual school sites ultimately sell.

With College District's quick growth we've also been helping them improve their order fulfillment process. This includes streamlining how orders are picked, packed and shipped. The introduction of bar code scanners will help with the accuracy and speed of the process.

We get a kick out of seeing our clients succeed, especially those that come to us with a clear vision and a good attitude, and then put the hard work in to make it happen. It's an exciting year ahead for College District and we'll be right there supporting them on the journey.

Interchange Search Caching with "Permanent More"

Most sites that use Interchange take advantage of Interchange's "more lists". These are built-in tools that support an Interchange "search" (either the search/scan action, or result of direct SQL via [query]) to make it very easy to paginate results. Under the hood, the more list is a drill-in to a cached "search object", so each page brings back a slice from the cache of the original search. There are extensive ways to modify the look and behavior of more lists and, with a bit of effort, they can be configured to meet design requirements.

Where more lists tend to fall short, however, is with respect to SEO. There are two primary SEO deficiencies that get business stakeholders' attention:

• There is little control over the construction of the URLs for more lists. They leverage the scan actionmap and contain a hash key for the search object and numeric data to identify the slice and page location. They possess no intrinsic value in identifying the content they reference.
• The search cache by default is ephemeral and session-specific. This means all those results beyond page 1 the search engine has cataloged will result in dead links for search users who try to land directly on the more-listed pages.

It is the latter issue that I wish to address because there is--and has been for some time now--a simple mechanism called "permanent more" to remedy the default behavior.

You can leverage "permanent more" by adding the boolean mv_more_permanent, or the shorthand pm, to your search conditions. E.g.:

Link:

    <a href="[area search="
        co=1
        sf=category
        se=Foo
        op=rm
        more=1
        ml=5
        pm=1
    "]">All Foos</a>

Loop:

    [loop search="
        co=1
        sf=category
        se=Foo
        op=rm
        more=1
        ml=5
        pm=1
    "]
    ...loop body with [more-list]...
    [/loop]

Query:

    [query
        list=1
        more=1
        ml=10
        pm=1
        sql="SELECT * FROM products WHERE category LIKE '%Foo%'"
    ]
    ...same as loop but with 10 matches/page...
    [/query]

If the initial search is defined with the "permanent more" setting, it will produce the following adjustments:

• The hash key used to store and identify the search cache is deterministic based on the search conditions. Many searches for Interchange are category driven. Thus, all end users who wish to browse a category end up clicking identical links, which create duplicate search caches, belonging uniquely to them. With permanent more, they all share the same cache, with the same identifier. As long as the search conditions don't change, neither does the cache identifier. Even as the cache is refreshed with new executions of the search, the object remains in the same location. Thus, the results a search engine produced this morning reference links still valid now, tomorrow, or next week, provided they reference the same search conditions.
• The cached search object has no session affinity. Any link referencing the cache with the correct hash key has access to the content.

Taken together, "permanent more" removes (for the most part, addressed later) dead links from more lists cataloged by search engines. There are, however, other benefits to "permanent more" beyond those intended as described above:

• As stated in passing, standard Interchange search caching produces duplicate search objects for common search conditions. For a busy site, these caches can have an impact on storage. Typically, maintenance is implemented to clean up cache files for all such files whose age exceeds by some amount the session duration (standard is 48 hours). With permanent more, duplicate caches are eliminated. A cache location is reused by all users with the same search requirements, keeping data-storage requirements for caches to the minimum necessary. As searches change, ophaned caches can still easily be cleaned up as they will immediately start to age with no more access to them necessary for storage.
• For the same reason that "permanent more" resolves search-engine links, it also resolves content management for individual sites using a reverse proxy for caching. Because most (and certainly the easiest) caching keys are based off of URL, the deterministic nature of the hash keys for "permanent more" allows assurance that the cached content in the proxy accurately reflects the search content over time, and that all users will hit the cached resource and not generate new, unique links with varying hash keys.

One shortcoming of "permanent more" to be aware of is the impact of changing data underneath the search. Even if search conditions do not change, the count and order of matching record sets may. So, e.g., enough products may be removed from a given category to cause the last page of a more list to become empty, which would cause any specific link into that page to become dead. More minor, but still a possibility, is the introduction or removal of products so that a particularly searched-for term has been "bumped" to another page within the search cache since the last time the search engine crawled the more lists. For searches backed by particularly volatile data, "permanent more" may not be sufficient to address search-engine or caching demands.

Finally, "permanent more" should be avoided for any search features that may cache data sensitive to an individual user. This is unlikely to happen as, under most circumstances, the configuration of the search itself will change based on the unique characteristics of the user executing the search (e.g., a username included in a query to review order history). However, it is still possible that context-sensitive information could be stored in the search object and, if so, all other users with access to the more lists would have access to that information.

Global Variables in Interchange Jobs

Those familiar with writing global code in Interchange are certainly familiar with the number of duplicate references of certain global variables in different namespaces. For example, the Values reference is found in both the main namespace ($::Values) as well as in Vend::Interpolate ($Values usually from within usertags). One can also access the Values reference through the Session reference, which itself can be found in main ($::Session), Vend ($Vend::Session), and Vend::Interpolate ($Session usually from within usertags) namespaces with, e.g., $::Session->{values}. Most times, as long as context allows, any of those access points are interchangeable, and there's a good mix you see from developers using all of them.

In recent work for a client, I had developed an actionmap that incorporated access to the session for some of its coding--certainly not an uncommon occurrence. When I work in global space, I tend to use the main namespace references since they are available in all contexts within Interchange (or so I thought). The actionmap was constructed, tested, and put into production, where it worked as expected.

After a short period of operation, the client came to us and noted that in their actual operating procedure, the actionmap must process many more data points than we had it operate on in testing, causing it to take much more time. Thus, for their usual workload, they found the process was timing out and Interchange housekeeping reaping the process.

After a brief discussion, we decided the expedient course of action was to convert the work from a browser-initiated actionmap into an Interchange job. The code was easily exposed as a usertag as well, so in very short order we had the same functionality available as a job, where the job was now triggered by the browser access previously running the actionmap.

The change resolved the immediate problem, so now all work was completing, but the client brought a new issue to our attention. The reporting from the job was not as it was supposed to be. None of the code had been modified in the changeover, and the code when run as an actionmap produced the proper reporting.

The problem tracked down eventually to that session access. When the code was run in the context of the job, the Session reference was not copied into the main (or, as it turns out, Vend::Interpolate) namespace. Without the assumed session values in place, it was causing the report to produce invalid output.

To demonstrate, I constructed a simple usertag to dump the reference addresses of the 5 mentioned global variables:

UserTag  ic-globals  Routine <<EOR
sub {
    return <<EOP;
.     \$Session: $Session
\$Vend::Session: $Vend::Session
    \$::Session: $::Session

       \$Values: $Values
     \$::Values: $::Values
EOP
}
EOR

I then created both a test page and an IC job that only called [ic-globals]. Running them both demonstrates the problem quite clearly.

From test page:

      $Session: HASH(0xb0e1898)
$Vend::Session: HASH(0xb0e1898)
    $::Session: HASH(0xb0e1898)

       $Values: HASH(0xb0e1dd8)
     $::Values: HASH(0xb0e1dd8)

Output from job:

      $Session:
$Vend::Session: HASH(0xb221fa0)
    $::Session:

       $Values: HASH(0x926ddd8)
     $::Values: HASH(0x926ddd8)

Interchange jobs provide yet a new context where you must consider your global variable usage. In particular, if you find code executed in the context of a job produces inconsistencies with the same code in other contexts, review your global variable usage and confirm those variables are what you assume they are.

SQL errors in Interchange

Interchange has a little feature whereby errors in a [query] tag are reported back to the session just like form validation errors. That is, given the intentional syntax error here:

  [query ... sql="select 1 from foo where 1="]

Interchange will paste the error from your database in

  $Session->{errors}{'table foo'}

That's great, but it comes with a price: now you have a potential for a page with SQL in it, which site security services like McAfee will flag as "SQL injection failures". Sometimes you just don't want your SQL failures plastered all over for the world to see.

Simple solution:

  DatabaseDefault LOG_SESSION_ERROR 0

in your Interchange configuration file, possibly constrained so it only affects production (because you'd love to see your SQL errors when you are testing, right?).

Spree Performance Benchmarking

I see a lot of questions regarding Spree performance in the spree-user group, but they are rarely answered with metrics. I put together a quick script using the generic benchmark tool ab to review some data. Obviously, the answer to how well a site performs and scales is highly dependent on the host and the consumption of the web application, so the data here needs to be taken with a grain of salt. Another thing to note is that only two of the following use cases are running on Rails 3.0 — many of our current Spree clients are on Spree 0.11.2 or older. I also included one non-Spree Rails ecommerce application, in addition to a few non-Rails applications for comparison. All of the tests were run from my home network, so in theory there shouldn't be bias on performance tests for sites running on End Point servers.

	ab -n 100
	-c 2 homepage	-c 20 homepage	-c 2 product page	-c 20 product page
Client #1 Spree: 0.11.2 Hosting: 4 cores, 512 GB RAM DB: MySQL # Products: <100	7.49	24.75	6.49	19.87	Requests per second
	266.889	808.041	307.997	1006.552	Time per request (ms)
Client #2 Spree 0.11.2 Hosting: Engineyard, medium instance DB: MySQL # Products: 100s	5.32	20.28	5.36	18.03	Requests per second
	375.713	986.309	373.289	1109.524	Time per request (ms)
Client #3 Spree: 0.9.0 Hosting: 4 cores, 1 GB RAM DB: PostgreSQL # Products: <100	4.91	25.39	1.98	6.54	Requests per second
	407.135	787.782	1011.875	3060.062	Time per request (ms)
(Former) Client #4 Spree: 0.11.2 Hosting: Unknown DB: PostgreSQL # Products: >5000	20.69	8.84	10.15	19.28	Requests per second
	96.673	2262.105	196.996	1037.146	Time per request (ms)
Client #5 Spree: 0.11.2 Hosting: EngineYard, small instance DB: MySQL # Products: 1	12.28	16.23	N/A	N/A	Requests per second
	162.909	1231.945	N/A	N/A	Time per request (ms)
Client #6 Spree: 0.40 Hosting: 4 cores, 1 GB RAM DB: MySQL # Products: 50-100	3.61	8.93	2.96	3.06	Requests per second
	553.569	2240.657	675.306	6539.433	Time per request (ms)
SpreeDemo Spree: Edge Hosting: Heroku, 2 dynos DB: Unknown # Products: 100s	8.17	12.79	4.7	5.48	Requests per second
	244.831	1563.642	425.27	3652.927	Time per request (ms)
Client #7 *custom Rails ecommerce app Hosting: 1.0 GB RAM DB: MySQL # Products: 1000s	5.43	29.8	4.45	23.14	Requests per second
	368.409	671.082	448.962	864.24	Time per request (ms)
Interchange Demo Hosting: 4 cores, 2 GB RAM DB: MySQL # Products: >500	7.41	55.27	7.5	13.93	Requests per second
	269.942	361.875	266.492	1435.51	Time per request (ms)
Client #8 *PHP site, serves fully cached pages with nginx with no app server or db hits Hosting: 4 cores, 4 GB RAM	10.81	30.54	6.05	9.87	Requests per second
	184.994	654.858	330.727	2027.092	Time per request (ms)
Magento Demo Hosting: Unknown DB: Unknown # Products: 100s	4.26	44.85	2.68	36.29	Requests per second
	469.831	445.931	745.472	551.11	Time per request (ms)

Here's the same data in graphical form:

Requests per Second

Time Per Request (ms)

We expect to see high performance on some of the sites with significant performance optimization. On smaller VPS, we expect to see the the server choke with higher concurrency.

Product Personalization for Ecommerce on Interchange with Scene7

One of the more challenging yet rewarding projects Richard and I have worked on over the past year has been an ecommerce product personalization project with Paper Source. I haven't blogged about it much, but wanted to write about the technical challenges of the project in addition to shamelessly self-promote (a bit).

Personalize this and many other products at Paper Source.

Paper Source runs on Interchange and relies heavily on JavaScript and jQuery on the customer-facing side of the site. The "personalization" project allows you to personalize Paper Source products like wedding invitations, holiday cards, stationery, and business cards and displays the dynamic product images with personalized user data on the fly using Adobe's Scene7. The image requests are made to an external location, so our application does not need to run Java to render these dynamic personalized product images.

Technical Challenge #1: Complex Data Model

To say the data model is complex is a bit of an understatement. Here's a "blurry" vision of the data model for tables driving this project. The tables from this project have begun to exceed the number of Interchange core tables.

A snapshot of the data model driving the personalization project functionality.

To give you an idea of what business needs the data model attempts to meet, here are just a few snapshots and corresponding explanations:

	At the highest level, there are individual products that can be personalized.
	Each product may or may not have different color options, or what we refer to as colorways. The card shown here has several options: gravel, moss, peacock, night, chocolate, and black. Clicking on each colorway here will update the image on the fly.
	In addition to colorways, each product will have corresponding paper types and print methods. For example, each product may be printed on white or cream paper and each product may have a "digital printing" option or a letterpress option. Colorways shown above apply differently to digital printing and letterpress options. For example, letterpress colorways are typically a subset of digital printing colorways.
	Each card has a set of input fields with corresponding fonts, sizes, and ink colors. The input fields can be input fields or text boxes. All cards have their own specific set of data to control the input fields – one card may have 4 sections with 1 text field in each section while another card may have 6 sections with 1 text field in some sections and 2 text fields in other sections. In most cases, inks are limited between card colorways. For example, black ink is only offered on the black colorway card, and blue ink is only offered on the blue colorway card.
	Each card also has a set of related items assigned to it. When users toggle between card colorways, the related item thumbnails update to match the detail option. This allows users to see an entire suite of matching cards: a pink wedding invite, RSVP, thank you, and stationery or a blue business card, matching letterhead stationery, and writing paper shown here.
	In addition to offering the parent product, envelopes are often tied to the products. In most cases, there are default envelope colors tied to products. For example, if a user selected a blue colorway product, the blue envelope would show as the default on the envelopes page.
	In addition to managing personalization of the parent products, the functionality also meets the business needs to offer customization of return address printing on envelopes tied to products. For example, here is a personalized return address printing tied to my wedding invitation.

Technical Challenge #2: Third Party Integration with Limited Documentation

There are always complexities that come up when implementing third-party service in a web application. In the case of this project, there is a fairly complex structure for image requests made to Scene7. In the case of dynamic invitations, cards, and stationery, examples of image requests include:

	https://a248.e.akamai.net/f/248/9086/10h/origin-d7.scene7.com/is/image/?layer=0& anchor=-50,-50 &size=2000,2000&layer=1&src=is{PaperSource/A7_env_back_closed_sfwhite}& anchor=2900,-395 &rotate=-90&op_usm=1,1,8,0&resMode=sharp&qlt=95,1&pos=100,50&size=1800,1800& layer=3&src=fxg {PaperSource/W136-122208301?&imageres=150}&anchor=0,0&op_usm=1,1,1,0&pos=500,315& size=1732,1732 &effect=0&resMode=sharp&fmt=jpg
	https://a248.e.akamai.net/f/248/9086/10h/origin-d7.scene7.com/is/image/?layer=0& anchor=-50, 50&size=2000,2000&layer=1&src=is{PaperSource/4bar_env_back_closed_fig}&anchor=0,0& pos=115,375 &size=1800,1800&layer=2&src=fxg{PaperSource/4barV_white_background_key}&anchor=0,0& rotate=-90& pos=250,1757&size=1733,1733&layer=3& src=fxg{PaperSource/ST57-2011579203301?&$color_fig=true&$color_black=false&$ color_chartreuse =false&$color_espresso=false&$color_moss=false&$color_peacock=false &$ink_0=780032 &$ink_2=780032&$ink_1=780032&imageres=150}&anchor=0,0 &op_usm=2,1,1, 0&pos=255,513&size=1721,1721&resMode=sharp&effect=0&fmt=jpg
	https://a248.e.akamai.net/f/248/9086/10h/origin-d7.scene7.com/is/image/?layer=0&anchor=-50,- 50&size=2000,2000&layer=1&src=is{PaperSource/4bar_env_back_closed_night}&anchor=0,0&pos =115,375&&size=1800,1800&layer=2&src=fxg{PaperSource/4bar_white_sm} &anchor=0,0&rotate=-90&pos=250,1757&size=1733,1733&layer=3&src=fxg{PaperSource/W139-201203301?} &anchor=0,0&op_usm=2,1,1,0&pos=255,513&size=1721,1721&resMode=sharp&effect=0&fmt=jpg

Each argument is significant to the dynamic image; background envelope color, card colorway, ink color, card positioning, envelope positioning, image quality, image format, and paper color are just a few of the factors controlled by the image arguments. And part of the challenge was dealing with the lack of documentation to build the logic to render the dynamic images.

Conclusion

As I mentioned above, this has been a challenging and rewarding project. Paper Source has sold personalizable products for a couple of years now. They continue to move their old personalized products to use this new functionality including many stationery products moved yesterday. Below are several examples of Paper Source products that I created with the new personalized functionality.

Ecommerce Solutions: What are the Options?

Lately, I've been evaluating ecommerce options for use on a side hobby/business. I'm obviously a developer, so in theory I could use one of End Point's supported ecommerce frameworks or just write my own framework. But, my bottom line is that I don't need the feature set offered by some of the ecommerce options out there and I don't necessarily have the resources to develop a custom solution.

In addition to personal interest, End Pointers constantly encounter potential clients who aim to get a better understanding of the cost of using open source, our preferred ecommerce solution. I put together two infographics on ecommerce options, ongoing cost, feature sets, and the ability to customize. Before anyone *flips out* about the infographics, note that they represent my broad generalizations regarding the ongoing cost, feature sets and ability to customize. I'm intimately familiar with some of these options and less familiar with a couple of them.

Feature Set versus Ongoing Cost of Ecommerce Solutions

Ability to Customize versus Ongoing Cost of Ecommerce Solutions

Some notes on on the ecommerce solutions shown in the infographics:

• Online payment service (Paypal): An online payment collection service like PayPal offers a minimal "ecommerce" feature set and might be suitable for someone looking to simply collect money. It also provides almost no ability to customize. The ongoing cost of PayPal is lower than many of the other options, where a percentage of each sale goes to PayPal.
• Online catalog service (Etsy, eBay): An online catalog service such as Etsy or eBay offers very basic ecommerce listing features, little to no ability to customize, but has relatively low ongoing cost. For example, Etsy charges $0.20 to list a single item for four months and takes 3.5% of the sales fee.
• Hosted ecommerce (shopify, Big Cartel, Big Commerce, Yahoo Merchant): A hosted ecommerce solution offers the ability to customize the appearance, typically in the form of a custom template language and has a basic ecommerce feature set. Ongoing costs would include the cost of the service, domain registration cost, and payment gateway (e.g. Authorize.NET) fees. Additional cost may apply if development services are required for building a custom template. Shopify's basic solution costs $29/mo. with a max of 100 skus. Shopify offers several other higher-priced options which include more features and have less limitations. Most other hosted ecommerce solutions are similarly priced.
• Open source ecommerce (Interchange, Spree, Magento, Zen Cart, osCommerce, prestaShop): An open-source ecommerce solution tends to have generic ecommerce features, provides the opportunity for a large amount of customization, but is typically more expensive than hosted ecommerce solutions and online catalog services. The software itself is free, but ongoing costs of hosting (server, domain registration, SSL certificate), development (any piece of customization that does not fit into the generic mold), and payment gateway fees apply. One positive about open source ecommerce is that additional plugins or add-ons are produced by members of the community. If the business needs are satisfied by the community-available extensions, cost from additional development or customization may be eliminated or reduced. I'd also probably group in existing open source plugins or modules for open source CMS solutions like WordPress and Drupal into this category: the generic solution is free, but additional resources may be required for customization.
• Enterprise ecommerce (ATG, Magento Enterprise): From my experience, I've observed that enterprise ecommerce solutions tend to offer a large feature set and a similar ability to customize as open source frameworks. The ongoing cost is high: for example, Magento Enterprise starts at $12,990/year. In addition to the licensing cost, hosting, development, and payment gateway fees may apply.
• Custom ecommerce (homegrown): The cost of writing your own ecommerce framework depends on the functionality requirements. The ability to customize is unlimited since the entire solution is custom. The feature set is likely to be proportional to resources spent on the project. Ongoing costs here include hosting, development and payment gateway fees.

Which one do you choose?

It has been my experience that most of End Point's ecommerce clients need some type of customization: a custom appearance, discount functionality, shipping integration, payment gateway integration, social media integration, and other third-party integration. Choosing an option that allows easy customization tends to benefit our customers in the long run. We're pretty biased at End Point towards open source ecommerce solutions, but my opinion recently is that with the advancement of web frameworks and web framework tools (e.g. gems in ruby), development of custom solutions can be done efficiently and may be a better option for a site that is outside the realm of standard ecommerce. For businesses not able to pay development or consulting costs, hosted ecommerce solutions are affordable and provide the essentials needed as the business grows.

Using nginx to transparently modify/debug third-party content

In tracking down a recent front-end bug for one of our client sites, I found myself needing to use the browser's JavaScript debugger for stepping through some JavaScript code that lived in a mix of domains; this included a third-party framework as well as locally-hosted code which interfaced with -- and potentially interfered with -- said third-party code. (We'll call said code foo.min.js for the purposes of this article.) The third-party code was a feature that was integrated into the client site using a custom domain name and was hosted and controlled by the third-party service with no ability for us to change directly. The custom domain name was part of a chain of CNAMEs which eventually pointed to the underlying *actual* IP of the third-party service, so their infrastructure obviously relied on getting the Host header correctly in the request to select which among many clients was being served.

It appeared as if there was a conflict between code on our site and that imported by the third party service. As part of the debugging process, I was stepping through the JavaScript in order to determine what if any conflicts there were, as well as their nature (e.g., conflicting library definitions, etc.). Stepping through our code was fine, however the third-party's JS code was (a) unfamiliar, and (b) minified, so this had the effect of putting all of the JavaScript code more-or-less on one line, which made tracing through the code in the debugger much less useful than I had hoped.

My first instinct was to use a JavaScript beautifier to reverse the minification process, but since I had no control over the code being included from the third-party service, this did not seem to be directly feasible. The third-party code was deployed only on our production site and relied on hard-coded domains which would make integrating it into one of our development instances challenging since we had no control over the contents of the returned resources. Since the relevant feature (and subsequent bugs) was only on the production site, making extensive modifications to how things were done and potentially breaking that or other features for users while I was debugging was obviously out as an option.

Enter nginx. I've been doing a lot with nginx lately as far as using it as a reverse proxy cache, so it's been on my mind lately. So I came up with this technique:

1. Look up the IP address for the third-party's domain name (used for later purposes).
2. Install nginx on localhost, listening to port 80.
3. Modify /etc/hosts to point the third-party's domain name to the nginx server's IP (also localhost in this case).
4. Configure a new virtual host with the following logical constraints:
   • We want to serve specific files (the beautified JavaScript) from our local server.
   • We want any other request going through that domain to be passed-through transparently, so neither the browser nor the third-party server treat it differently.

Given these constraints, this is the minimal configuration that I came up with (the interesting parts are located in the server block):

/etc/hosts:

example.domain.com 127.0.0.1

nginx.conf:

worker_processes 1;

events {
    worker_connections 10;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    
    server {
        server_name example.domain.com;
        root /path/to/local_root;

        try_files $uri @proxied;

        location @proxied {
            proxy_set_header Host $http_host;
            proxy_pass http://1.2.3.4;
        }
    }
}

Once I had the above configured/setup, I downloaded/saved the foo.min.js file from the third-party service, ran it through a JS beautifier, and saved it in the local nginx's cache root so it would be served up instead of the actual file from the third-party service. Any other requests for static resources (images, other scripts, etc) would pass-through to the third-party server, so I had my nicely-formatted JavaScript code to step through, the production site worked as normal for anyone else despite potential local changes to the file on my end (i.e., adding JavaScript alert() calls to the file, and no one was the wiser.

A few notes

The try_files directive instructs nginx to first look for a file named after the current URI (foo.min.js in our example) in our local cache, and if this is not found, then fallback to the proxied location block; i.e., relay the request to the original upstream server. We explicitly set the Host header on the proxy request because we want the request to behave normally with respect to name-based hosting, and provide the saved IP address to contact the server in question.

We only needed to preserve/lookup the upstream server's IP address because we're running the nginx server on localhost, so if we used a domain name the lookup would return the same IP defined in /etc/hosts; if the nginx server was running on a different machine, you would be able to just use the domain name as both the server_name and the proxy_pass parameters and set the entry for the host in your local /etc/hosts file to the IP of the nginx server.

A possible extension would be to detect when an upstream request matched a minified URL (via a location ~ \.min\..*\.js$ block) and automatically beautify/cache the content in our local cache. This could be accomplished via the use of an external FastCGI script to retrieve, post-process, and cache the content.

This technique can also be used when dealing with testing changes to a production site on which you are unable or unwilling to make potentially disruptive changes for the purposes of testing static resources. (JavaScript seems the most obvious application here, but this could apply to serving up images or other static content which would be resolvable by the local cache.)

I always need to remind myself to undo changes to /etc/hosts as soon as I'm done testing when using tricks like these. Particularly in something like this which is more-or-less transparent, the behavior would be functionaly identical as long as code/scripts on the third-party site stayed the same, but could easily introduce subtle bugs if the third-party services made changes to their codebase. Since our local copies would mask any remote changes for those non-proxied resources, this could be very confusing if you forget that things are set up this way.

New Year Bug Bites

Happy New Year! And what would a new year be without a new year bug bite? This year we had one where figuring out the species wasn't easy.

On January 2nd one of our ecommerce clients reported that starting with the new year a number of customers weren't able to complete their web orders because of credit card security code failures. Looking in the Interchange server error logs we indeed found a significant spike in the number of CVV2 code verification failures (Payflow Pro gateway error code "114") starting January 1st.

We hadn't made any programming or configuration changes on the system in the recent days. We double-checked to make sure: nope, no code changes. So it had to be a New Year's bug and presumably something with the Payflow Pro gateway or banks further upstream. We checked error logs for other customers to see if they were being similarly impacted, but they weren't. Our client contacted PayPal (the vendor for Payflow Pro) and they reported there were no problems with their system. The failures must indeed be card failures or a problem with the website according to them. We further checked our code looking for what we could possibly have done that might be the cause, double-checking our Git repository (which showed no recent changes) and reexamining our checkout code for possible year-based logic flaws.

Our client's top-notch customer service group got on the phone with a customer who'd gotten a security code failure and got PayPal tech support on another line. The customer service rep tried to place the customer's order on the website using the customer's credit card info and once again got the CVV2 error. She then did the credit card transaction using the swipe machine in the office, and lo and behold the order went through! What was going on??!

It turned out that despite the Payflow Pro gateway returning CVV2 verification errors what was really happening was that the year of the credit card was coming into the Payflow Pro gateway as "2012"—not as "2011" as entered into the checkout form. We knew all along that it was possible that the 114 error code responses were possibly misleading because payment gateway error codes are notorious this way. (Payment gateways blame the banks, saying they can only pass along what the banks give them. Some banks' credit card validations don't actually even care about the years being correct, but just that they not be in the past; but I digress...)

Previously we'd reviewed the checkout pages and the dropdown menus to verify that the dropdown menus weren't off, but nevertheless it very much sounded like this rather stupid problem could very well be the culprit. So we checked and checked again. What we found is that sometimes on the checkout form the year dropdown menu was mangled such that the values associated with the displayed years were YYYY+1.

The oddly intermittent behavior of the problem, the process of elimination and the all around hair pulling this loss of business was causing made somebody in the marketing group at our client realize that they are in fact still running an Omniture Test & Target A/B test on the checkout pages that they thought had been discontinued. To quote David Christensen (thanks, David!): "The Omniture system works by replacing select content for randomly chosen users in an effort to track user behavior/response to proposed site changes. Alternate site content is created and dynamically replaced for these users as they use the site, such as the specific content on the checkout page in this instance."

We verified that the Omniture A/B test's JavaScript replacement code was alternately mangling and not mangling the year dropdown on the checkout form as mentioned. Our client took out the A/B test and the "security code errors" dropped back to a normal low level.

This was a difficult and expensive problem—not only was there business lost because of the problem, but there were a lot of resources put into troubleshooting it. We've come away from this episode with some lessons learned and with plenty of food for thought. I'll leave it to commentators to opine away on this, including the End Point folks who scratched this itch: David Christensen, Jeff Boes, Mark Johnson, and Jon Jensen.

Version Control Visualization and End Point in Open Source

Over the weekend, I discovered an open source tool for version control visualization, Gource. I decided to put together a few videos to showcase End Point's involvement in several open source projects.

Here's a quick legend to help understand the videos below:

	The branches and nodes correlate to directories and files, respectively. In the case of the image to the left, the repository has a main directory with several files and three directories. One of the child directories has one file and the other two have multiple files.
	A big dot represents a person, and a flash connecting the person and a file signifies a commit.
	White + blue dots represent current End Point employees.
	White + grey dots represent former End Point employees.
	White dots represent other people, out there!

The Videos

Interchange from endpoint on Vimeo.

pgsi from endpoint on Vimeo.

Spree from endpoint on Vimeo.

Bucardo from endpoint on Vimeo.

One of the articles that references Gource suggests that the videos can be used to visualize and analyze the community involvement of a project (open source or not). One might also be able to qualitatively analyze the stability of project file architecture from a video, but this won't reveal anything definitive about the code stability since external factors can influence file structure. For example, since I am intimately familiar with the progress of Spree, I can identify when Spree transitioned to Rails 3 in the video, which required reorganization of the Spree core functionality (read more about this here and here).

In the case of this article, I wanted to highlight End Point's involvement in a few open source projects where we've had various levels of involvement. We've contributed to Interchange since 2000. We've been involved in Spree less lately, but had more presence in early 2009. In the smaller projects Bucardo and pgsi, End Point employees have worked on a team to be the primary contributors to the projects in addition to a few external contributors. Open source is important to End Point, and it's great to see our presence demonstrated in these cute videos.

Character encoding in perl: decode_utf8() vs decode('utf8')

When doing some recent encoding-based work in Perl, I found myself in a situation which seemed fairly unexplainable. I had a function which used some data which was encoded as UTF-8, ran Encode::decode_utf8() on said data to convert to Perl's internal character format, then converted the "wide" characters to the numeric entity using HTML::Entities::encode_entities_numeric(). Logging/printing of the data on input confirmed that the data was properly formatted UTF-8, as did running `iconv -f utf8 -t utf8 output.log >/dev/null` for the purposes of review.

However when I ended up processing the data, it was as if I had not run the decode function at all. In this case, the character in question was € (unicode code point U+20AC). The expected behavior from encode_entities_numeric() would be to turn any of the hi-bit characters in the perl string (i.e. all Unicode code points > 0x80) into the corresponding numeric entity (€ - € in this case). However instead of that specific character's numeric entity appearing in the output, the entities which appeared were: € i.e., the raw UTF-8 encoded value for €, with each octet being treated as an independent character instead of part of the whole encoded value.

What was particularly confusing was that extracting the relevant parts from the script in question resulted in the expected answer, so it was clearly not an issue of HTML::Entities not being able to deal with Unicode characters, as this code snippet demonstrates:

$ perl -MHTML::Entities+encode_entities_numeric -MEncode -e '$c=qq{\xE2\x82\xAC}; print encode_entities_numeric(decode_utf8($c))'
--> €

In the actual non-extracted version of the code, I was scratching my head. This was exhibiting the signs of doubly-encoded data, however I couldn't see how that could be the case. There were no PerlIO layers (e.g., :utf8 or :encoding) at play, the data I was outputting to a log file for verification purposes was being written via a brand new filehandle from a bare open(); I verified in multiple ways that the raw octets being passed in to the function were not doubly-encoded (printing the raw character points, counting lengths of the runs of octets and verifying that these matched the length of the UTF-8 encoded value for the represented characters, etc). The more things I tried the more puzzled I got. Finally, I changed the Encode::decode_utf8() call to a Encode::decode('utf8') one, providing the encoding explicitly. At this point, the processing pipeline started working as expected, and hi-bit characters were being output as their full numeric entities.

Since the documentation for decode_utf8 indicated that it should be identical to decode('utf8'), I resorted to the code to find out why it worked with the version that specified the encoding explicitly. I found that decode_utf8() does one additional thing that the regular decode('utf8') does not, and that is that before processing via the regular decode() function, decode_utf8 first checks the UTF-8 flag of the data that is being passed in, and if it is set it returns the data without further decoding*. My best guess is that this is to prevent errors if someone attempts to decode UTF-8 data in a string which is already in Perl's internal format, so in most cases this will provide a caller-friendly interface that will DWYM in many expected cases.

Armed with this knowledge, I verified that for some reason, the data that was being passed into the function had the UTF-8 flag set, so using the explicit decode('utf8') in lieu of decode_utf8() fixed the issue for me. (Tracing down the reason for the UTF-8 flag being set on this data was out of scope for this exercise, but is the true fix.) And just to verify that this was in fact the cause of the issue at hand, here's our example, modified slightly (we use the utf8::upgrade function to turn the UTF-8 flag on in the data and treat as actual encoded characters instead of raw octets):

$ perl -l -MHTML::Entities+encode_entities_numeric -MEncode -Mutf8 -e '$c=qq{\xE2\x82\xAC}; utf8::upgrade($c); print encode_entities_numeric(decode_utf8($c))'
--> €

* The UTF-8 flag is more-or-less an implementation detail of how Perl is able to deal with legacy 8-bit binary data in no particular encoding (i.e., raw octets, which it treats as latin-1) as well as the full range of Unicode data, and deal with both efficiently and in a backwards-compatible manner.

SearchToolbar and dropped Interchange sessions

A new update to Interchange's robots.cfg can be found here. This update adds "SearchToolbar" to the NotRobotUA directive which is used to exclude certain user agent strings when determining whether an incoming request is from a search engine robot or not. The SearchToolbar addon for IE and FireFox is being used more widely and we have received reports that users of this addon are unable to add items to their cart, checkout, etc. You may remember a similiar issue with the Ask.com toolbar that we discussed in this post. If you are using Interchange you should download the latest robots.cfg and restart Interchange.

Keep the Aisles Clean at Checkout

It's no mystery in ecommerce that checkout processing must flow smoothly for an effective store. Providing products or services in high demand doesn't mean much if they cannot be purchased, or the purchase process is so burdensome that would-be customers give up in frustration.

Unfortunately, checkout also tends to include the most volatile elements of a web store. It virtually always involves database writes, which can be hindered by locking. It often involves real-time network access to 3rd-party providers, with payment transactions being at the top of the list. It can involve complex inventory assessments, where high concurrency can make what's normally routine highly unpredictable. Meanwhile, your customers wait, while the app sifts through complexity and waits on responses from various services. If they wait too long, you might lose sales; even worse, you might lose customers.

Even armed with the above knowledge, it's all too easy to fall into the trap of expediency. A particular action is so logically suited to be included as part of the checkout routine, and a superficial evaluation makes it seem like such a low-risk operation. That action can be tucked in there just after we've passed all the hurdles and are assured the order won't be rejected--why, it'll be so simple, and all the data we need for the action are readily at hand.

Just such expediency was at the heart of a checkout problem that had been plaguing an Interchange client of ours for months. The client would receive regular complaints that checkouts were timing out or taking so long that the customer was reloading and trying again. Many times, these customers would come to find that their orders had been placed, but that the time to complete them was exceeding the web server's timeout (or their patience). In far less common instances, but still occurring regularly, log and transaction evidence existed that showed an order attempt produced a valid payment transaction, but there was no hint of the order in their database or even in the application's system logs.

In the latter case of behavior, I had seen this before for other clients. If an action within order routing takes long enough, the Interchange server handling the request will be hammered by housekeeping. The telltale sign is the lack of log evidence for the attempt since order routes are logged at the end of the route's run; when that's interrupted, then no logging occurs.

I added considerably more explicit real-time logging and picked off some of the low-hanging fruit--code practices that had often been implicated before as the culprit in these circumstances. After collecting enough data for problematic order attempts, I was able to isolate the volatility to mail-list maintenance. The client utilizes a 3rd-party provider for managing their various mail lists, and that provider's API was contacted during order routing with all the data the provider needed for managing said lists. The data transfer for the API was very simple, and in most cases would process in sub-second time. Unfortunately, it turned out that, in enough cases, the calls to the API would take 10s to even 100s of seconds to process.

The placement of maintaining mail lists within order routing was merely convenience. The success or failure of adding to the mail lists was insignificant compared to the success or failure of the order itself. Once identified, the API calls were moved into a post-order processing routine, which was specifically built to anticipate the demonstrated volatility. As a result, complaints from customers on long or timed-out checkouts have dwindled to near zero, and the mail-list maintenance is more reliable since the background process is designed to catch excessively long process calls and retry until we receive an affirmative response from the list maintainers.

When deciding what belongs within checkout processing, ideally limit that activity to only those actions absolutely imperative to the success of the order. For each piece of functionality, ask yourself (or your client): is the outcome of this action worth adding to the wait a customer experiences placing an order? Should the outcome of this action affect whether the order attempt is successful? If the answer to those questions is "no", account for that action outside of checkout. It may be more work to do so, but keeping the checkout aisles clean, without obstruction, should be paramount.

SEO friendly redirects in Interchange

In the past, I've had a few Interchange clients that would like the ability to be able to have their site do a SEO friendly 301 redirect to a new page for different reasons. It could be because either a product had gone out of stock and wasn't going to return or they completely reworked their url structures to be more SEO friendly and wanted the link juice to transfer to the new URLs. The normal way to handle this kind of request is to set up a bunch of Apache rewrite rules.

There were a few issues with going that route. The main issue is that to add or remove rules would mean that we would have to restart or reload Apache every time a change was made. The clients don't normally have the access to do this so it meant they would have to contact me to do it. Another issue was that they also don't have the access to modify the Apache virtual host file to add and remove rules so again, they would have to contact me to do it. To avoid the editing issue, we could have put the rules in a .htaccess file and allow them to modify it that way, but this can present its own challenges because some text editors and FTP clients don't handle hidden files very well. The other issue is that even though overall basic rewrite rules are pretty easy to copy, paste and reuse, they still can have nasty side effects if not done properly and can also be difficult to troubleshoot so I devised a way to allow them to be able to manage their 301 redirects using a simple database table and Interchange's Autoload directive.

The database table is a very simple table with two fields. I called them old_url and new_url with the primary key being old_url. The Autoload directive accepts a list of subroutines as its arguments so this requires us to create two different GlobalSubs. One to actually do the redirect and one to check the database and see if we need to redirect. The redirect sub is really straight forward and looks like this:

sub redirect {
   my ($url, $status) = @_;
   $status ||= 302;
   $Vend::StatusLine = qq|Status: $status moved\nLocation: $url\n|;
   $::Pragma->{download} = 1;
   my $body = '';
   ::response($body);
   $Vend::Sent = 1;
   return 1;
}

The code for the sub that checks to see if we need to redirect looks like this:

sub redirect_old_links {
   my $db = Vend::Data::database_exists_ref('page_redirects');
   my $dbh = $db->dbh();
   my $current_url = $::Tag->env({ arg => "REQUEST_URI" });
   my $normal_server = $::Variable->{NORMAL_SERVER};
   if ( ! exists $::Scratch->{redirects} ) {
       my $sth = $dbh->prepare(q{select * from page_redirects});
       my $rc  = $sth->execute();
       while ( my ($old,$new) = $sth->fetchrow_array() ) {
           $::Scratch->{redirects}{"$old"} = $new;
       }
       $sth->finish();
   }
   if ( exists $::Scratch->{redirects}  ) {
       if ( exists $::Scratch->{redirects}{"$current_url"} ) {
           my $path = $normal_server.$::Scratch->{redirects}{"$current_url"};
           my $Sub = Vend::Subs->new;
           $Sub->redirect($path, '301');
           return;
       } else {
          return;
       }
   }
}

We normally create these as two different files and put them into our own directory structure under the Interchange directory called custom/GlobalSub and then add this, include custom/GlobalSub/*.sub, to the interchange.cfg file to make sure they get loaded when Interchange restarts. After those files are loaded, you'll need to tell the catalog that you want it to Autoload this subroutine and to do that you use the Autoload directive in your catalog.cfg file like this:

Autoload redirect_old_links

After modifying your catalog.cfg file, you will need to reload your catalog to ensure to change takes effect. Once these things are in place, you should just be able to add data into the page_redirects table and start a new session and it will redirect you properly. When I was working on the system, I just created an entry that redirected /cgi-bin/vlink/redirect_test.html to /cgi-bin/vlink/index.html so I could ensure that it was redirecting me properly.

Providing Database Handle for Interchange Testing

I've recently begun using the test driven development approach to my projects using Perl's Test::More module. Most of my projects lately have been with Interchange which has some hurdles to get around as far as test driven development is concerned. Primarily this is because Interchange runs as a daemon and provides some readily available utilites like the database handle. This method is not available to our tests, so they need to be made available as discussed below.

I develop Usertags, GlobalSubs and ActionMaps where applicable as it helps keep the separation of business logic and views clear. I generally organize these to call a function within a Perl module so they can be tested properly. Most of these tags involve some sort of connection with the database to present information to the user in which I uses the Interchange ::database_exists_ref method.

When it comes to testing I want to ensure that the test script invokes the same method. Otherwise, your script will not be testing the code as its used in production.

Let's say you are building a Perl module that looks something like this:

package YourMagic;
use strict;

sub do_something {
    my ($opt) = @_;

    # some code

    my $dbh = ::database_exists_ref($opt->{table})->dbh
        or return undef;

    # ... more code
    return $output;
}

1;

The ::database_exists_ref() method will not be available for a test script and needs to be defined. It should return an object to the dbh method in the test script as it does within Interchange. There is no need to test the method itself, as it is not part of the "what" that is being developed. The following code needs to be added to the test script so it can handle the correct type of database reference returned by Interchange.

use lib '/home/user/interchange/custom/lib';
use Test::More tests => 2;
use DBI;

# Here are the methods to provide proper reference to our database handle
################################
sub ::database_exists_ref {
    my $table = shift;
    return undef if !$table;

    # return an object with a dbh method
    return bless({}, __PACKAGE__);
}

sub dbh {
    # define a dbh method
    my $db = DBI->connect('dsn, 'user', 'pass');

    return $db;
}
##################################

use YourMagic;

is(
    YourMagic::do_something(),
    undef,
    'do_something() returns undef when called with no arguments',
);

is(
    YourMagic::do_something(\%opt),
    undef,
    'do_something() returns ...',
);

It is also worthwhile to note that you'll need to use the ::database_exists_ref method to look up some information from the existing table that is valuable to test against. Now the do_something() method will call ::database_exists_ref() when invoked.

This approach allows us to use, reuse, and add new tests without worrying about mock data during the intial development. You can be sure that the existing test scripts will function properly against the latest data that is available.

I will cover some other topics regarding Interchange Test Driven Development in future posts. For more information regarding Unit Testing in general see this post by Ethan.

jQuery Auto-Complete in Interchange

"When all you have is a hammer, everything looks like a nail."

Recently, I've taken some intermediate steps in using jQuery for web work, in conjunction with Interchange and non-Interchange pages. (I'd done some beginner stuff, but now I'm starting to see nails, nails, and more nails.)

Here's how easy it was to add an auto-complete field to an IC admin page. In this particular application, a <select> box would have been rather unwieldy, as there were 400+ values that could be displayed.

<script src="//ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js" type="text/javascript"></script>
<script type="text/javascript" src="http://dev.jquery.com/view/trunk/plugins/autocomplete/lib/jquery.bgiframe.min.js"></script>
<script type="text/javascript" src="http://dev.jquery.com/view/trunk/plugins/autocomplete/lib/jquery.dimensions.js"></script>
<script type="text/javascript" src="http://dev.jquery.com/view/trunk/plugins/autocomplete/jquery.autocomplete.js"></script>


That's the requisite header stuff. Then you set up the internal list of autocomplete terms:

<script type="text/javascript">
$('document').ready(function(){
var auto_list = "[perl]...[/perl]".split(" ");
$('input[name="auto_field"]').autocomplete(auto_list);
});
</script>


The [perl] block just needs to emit a space-delimited list of the autocomplete terms. For instance,

[perl table="foo"]
return join(' ', map { $_->[0] }
@{ $Db{foo}->query('SELECT keycol FROM foo ORDER BY 1') });
[/perl]

Guidelines for Interchange site migrations

I'm involved at End Point often with Interchange site migrations. These migrations can be due to a new client coming to us and needing hosting or migrating from one server to another within our own infrastructure.

There are many different ways to do a migration, in the end though we need to hit on certain points to make sure that the migration goes smoothly. Below you will find steps which you can adapt for your specific migration.

During the start of the migration it might be a good time to introduce git for source control. You can do this by creating the repository and cloning it to /home/account/live, setting up .gitignore files for logs, counter files, gdbm files. Then commit the changes back to the repo and you've now introduced source control without much effort, improving the ability to make changes to the site in the future. This is also helpful to document the changes you make to the code base along the way during the migration in case you need to merge changes from the current production site before completing the migration.

• Export all of the gdbm databases to their text file equivalents on the production server
• Take a backup from production of the database, catalog, interchange server, htdocs
• Setup an account
• Create the database and user
• Restore the database, catalog, interchange server and htdocs
• Update the paths in interchange/bin for each script to point at the new location
• Grep the restored code for hard coded paths and update those paths to the new locations. Better yet move these paths out to a catalog_local.cfg where environment specific information can go.
• Grep the restored code for hard coded urls and use the [area] tag to generate the urls
• Update the urls in products/variable.txt to point at the test domain
• Update the sql settings in products/variable.txt to point at the new database using the new user
• Remove the gdbm databases so they will be recreated on startup from the source text files
• Install a local Perl if it's not already installed (./configure -des will compile and install Perl locally)
• Install Bundle::InterchangeKitchenSink
• Install the DBD module for MySQL or PostgreSQL
• Review the code base looking for use statements in custom code and Require module settings in interchange.cfg. Install the Perl modules found into the local Perl.
• Setup a non ssl and ssl virtual host using a temporary domain. Configure the temporary domain to use the SSL certificate from the production domain.
• Firewall or password protect the virtual host so it is not accessible to the public
• Generate a vlink using interchange/bin/compile and copy it into the cgi-bin directory and name it properly
• Startup the new Interchange
• Review error messages and resolve until Interchange will start properly
• Test the site thoroughly, resolving issues as they appear. Make sure that checkout, charging credit cards, sending of emails, using the admin, etc all function.
• Migrate any cron jobs running on the current production site, such as session expiration scripts
• Setup logrotation for the new logs that will be created
• Verify that you have access to make DNS changes
• Set the TTL for the domain to a low value such as 5 minutes
• Modify the new production site to respond to the production url, test by updating your hosts file to manually set the IP address of the domain
• Shutdown the new Interchange
• Restore a copy of the original backup for Interchange, the catalog and htdocs to /tmp on the production server
• Shutdown the production Interchange, put up a maintenance note on the production site.
• Take a backup of the production database and restore on the new server
• Diff the Interchange, catalog and htdocs directory between /tmp and the current production locations, making note of the files that have changed since we took the original copy.
• Copy the files that have changed, making sure to merge with any changes we have made on the new production site. Making sure to copy over all .counter and .autonumber files to the new production site.
• Start Interchange on the new production server
• Test the site thoroughly on the new production server, using the production url. Make sure that checkout with charging the credit card functions properly.
• Resolve any remaining issues found during the testing
• Setup the Interchange daemon to start at boot for this site in /etc/rc.d/rc.local or in cron using @reboot
• Update DNS to point at the new production IP address
• Update the TTL of the domain to a longer value
• Open the site to the public by opening the firewall or removing the password protection
• Keep an eye on the error logs for any issues that might crop up

This will hopefully give you a solid guide for performing an Interchange site migration from one server to another and some of the things to watch out for that might cause issues during the migrations.

Using charge tag in Interchange's profiles, and trickiness with logic and tag interpolation order

One of the standard ways to charging in older versions of the Interchange demo was to do the charging from a profile using the &charge command. New versions of the demo store do the charging from log_transaction once the order profiles have finished, so it is not an issue there. I've come across quite a few catalogs where the &charge command is replaced with the [charge] tag wrapped in if-then-else blocks in an order profile. It had been so long since I had used &charge so I needed to lookup how options are passed to it, which may be why people tend to use the tag version instead of the &charge command. The problem here is that Interchange tags interpolate before any of the profile specifications execute, so if you have a [charge] tag in an order profile, it executes before any of the other checks, such as validation of fields.

Here's a stripped down example of where a profile will have tags executed before the other profile checks:

lname=required Last name required
fname=required First name required
&fatal=yes
&credit_card=standard keep

[charge route="[var MV_PAYMENT_MODE]" amount="[scratch some_total_calculation]"]

&final=yes

In this situation even if lname, fname or the credit card number are invalid, charge will execute before all of those checks occur, calling your payment gateway with invalid parameters. This could even cause a weird state where a credit card was charged, but the order not placed because the last name check fails for example, after the charge is successful.

The way around this is either to move the credit card charging out of the order profile into log_transaction or use the &charge command like so:

&charge=[var MV_PAYMENT_MODE] amount=[scratch some_total_calculation]

Another situation where you should be careful is using if-then-else blocks, if you need to do a profile checks that are dependent upon the results of other calls in the profile then you will need to create a custom order check to do that processing, otherwise sections of your if-then-else may execute that are not intended to.

Tip: Find all non-UTF-8 files

Here's an easy way to find all non-UTF-8 files for later perusal:

find . -type f | xargs -I {} bash -c "iconv -f utf-8 -t utf-16 {} &>/dev/null || echo {}" > utf8_fail

I've needed this before for converting projects over into UTF-8; obviously certain files are going to be binary and will show up in this list, so manual vetting will need to be done before converting all your images over into UTF-8.

Safari 4 Top Sites feature skews analytics

Safari version 4 has a new "Top Sites" feature that shows thumbnail images of the sites the user most frequently visits (or, until enough history is collected, just generally popular sites).

Martin Sutherland describes this feature in details and shows how to detect these requests, which set the X-Purpose HTTP header to "preview".

The reason this matters is that Safari uses its normal browsing engine to fetch not just the HTML, but all embedded JavaScript and images, and runs in-page client JavaScript code. And these preview thumbnails are refreshed fairly frequently -- possibly several times per day per user.

Thus every preview request looks just like a regular user visit, and this skews analytics which see a much higher than average number of views from Safari 4 users, with lower time-on-site averages and higher bounce rates since no subsequent visits are registered (at least as part of the preview function).

The solution is to simply not output any analytics code when the X-Purpose header is set to "preview". In Interchange this is easily done if you have an include file for your analytics code, by wrapping the file with an [if] block such as this:

[tmp x_purpose][env HTTP_X_PURPOSE][/tmp]
[if scratch x_purpose eq 'preview']
<!-- skip analytics for browser previews -->
[else]
(normal Google Analytics, Omniture SiteCatalyst, or other analytics code)
[/else]
[/if]

In Ruby on Rails you'd check request.env["HTTP_X_PURPOSE"].

In PHP you'd check $_SERVER["HTTP_X_PURPOSE"].

In Django you'd check request.META["HTTP_X_PURPOSE"] or the equivalent request.META.get("HTTP_X_PURPOSE") (from the HttpRequest class).

And so on.

I confirmed the analytics tracking code was omitted by waiting for Safari to make its preview request and inspecting the response with the Fiddler proxy, on Windows. The same can be done for Safari on Mac OS X with a suitable Mac OS X HTTP proxy.

DevCamps on different systems, including Plesk, CPanel and ISPConfig

In the last few months I've been active setting up DevCamps for several of our newer clients. DevCamps is an open source development environment system, that once setup, allows for easily starting up and tearing down a development environment for a specific site/code base.

I've done many camps setups, and you tend to run into surprises from system to system, but what was most interesting and challenging about these latest installs was that they were to be done on systems running Plesk, CPanel, and ISPConfig. Some things that are different between a normal deployment and one on the above mentioned platforms are:

• On the Plesk system there was a secured Linux called 'Atomic Secured Linux' which includes the grsecurity module. One restriction of this module is (TPE) Trusted Path Execution which required the camp bin scripts to be owned by root and the bin directory could not be writable by other groups, otherwise they would fail to run.
• Permissions are a mixed bag, where typically we set all of the files to be owned by the site owner, in Plesk there are special groups such as psacln that the files need to be owned by.
• On the CPanel system we needed to move the admin images for Interchange to a different directory since CPanel includes Interchange and has aliases for /interchange/ and /interchange-5/ to point at a central location which we would not be using.
• On ISPConfig and Plesk the home directories of the sites are in different places, which required deploying the code in such places as /var/www/clients/client/user/domain.com or /var/www/vhosts/domain.com.

In the end we were able to get DevCamps to run properly on these various platforms both in development and production. If you are starting a new project or working on an existing project and could use a strong development environment, consider DevCamps.

XZ compression

XZ is a new free compression file format that is starting to be more widely used. The LZMA2 compression method it uses first became popular in the 7-Zip archive program, with an analogous Unix command-line version called 7z.

We used XZ for the first time in the Interchange project in the Interchange 5.7.3 packages. Compared to gzip and bzip2, the file sizes were as follows:

interchange-5.7.3.tar.gz   2.4M
interchange-5.7.3.tar.bz2  2.1M
interchange-5.7.3.tar.xz   1.7M

Getting that tighter compression comes at the cost of its runtime being about 4 times slower than bzip2, but a bonus is that it decompresses about 3 times faster than bzip2. The combination of significantly smaller file sizes and faster decompression made it a clear win for distributing software packages, leading to it being the format used for packages in Fedora 12.

It's also easy to use on Ubuntu 9.10, via the standard xz-utils package. When you install that with apt-get, aptitude, etc., you'll get a scary warning about it replacing lzma, a core package, but this is safe to do because xz-utils provides compatible replacement binaries /usr/bin/lzma and friends (lzcat, lzless, etc.). There is also built-in support in GNU tar with the new --xz aka -J options.

Dropped sessions when Ask.com Toolbar is installed

We've been dealing with an issue on a client's site where customers were reporting that they could not login and when they added items to their cart the cart would come up empty. This information pointed towards a problem with the customer's session being dropped, but we were unable to determine the common line across these customer's environments and came up empty handed. This was a case of being unable to reproduce a problem which made it nearly impossible to fix.

This morning on the Interchange users list there was a post from Racke discussing a similiar issue. His customer had the Ask.com toolbar installed and Interchange's robot matching code was mistakenly matching the Ask.com toolbar as a search spider. The user agent of the browser with Ask.com installed appeared as so:

"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; msn OptimizedIE8;ENUS; AskTB5.6)"

A quick look at the current robots.cfg that Steven Graham linked showed that 'AskTB' had been added to the NotRobotUA directive which instructs Interchange to not consider AskTB a search spider, thus allowing proper use of sessions on the site.

Updating the robots.cfg on our client's site allowed users with Ask.com to browse, login and checkout as expected. Those with Interchange sites who see reports of similiar issues should consider a false positive spider match a possibility and update their robots.cfg.

Performance optimization of icdevgroup.org

Some years ago Davor Ocelić redesigned icdevgroup.org, Interchange's home on the web. Since then, most of the attention paid to it has been on content such as news, documentation, release information, and so on. We haven't looked much at implementation or optimization details. Recently I decided to do just that.

Interchange optimizations

There is currently no separate logged-in user area of icdevgroup.org, so Interchange is primarily used here as a templating system and database interface. The automatic read/write of a server-side user session is thus unneeded overhead, as is periodic culling of the old sessions. So I turned off permanent sessions by making all visitors appear to be search bots. Adding to interchange.cfg:

RobotUA *

That would not work for most Interchange sites, which need a server-side session for storing mv_click action code, scratch variables, logged-in state, shopping cart, etc. But for a read-only content site, it works well.

By default, Interchange writes user page requests to a special tracking log as part of its UserTrack facility. It also outputs an X-Track HTTP response header with some information about the visit which can be used by a (to my knowledge) long defunct analytics package. Since we don't need either of those features, we can save a tiny bit of overhead. Adding to catalog.cfg:

UserTrack No

Very few Interchange sites have any need for UserTrack anymore, so this is commonly a safe optimization to make.

HTTP optimizations

Today I ran the excellent webpagetest.org test, and this was the icdevgroup.org test result. Even though icdevgroup.org is a fairly simple site without much bloat, two obvious areas for improvement stood out.

First, gzip/deflate compression of textual content should be enabled. That cuts down on bandwidth used and page delivery time by a significant amount, and with modern CPUs adds no appreciable extra CPU load on either the client or the server.

We're hosting icdevgroup.org on Debian GNU/Linux with Apache 2.2, which has a reasonable default configuration of mod_deflate that does this, so it's easy to enable:

a2enmod deflate

That sets up symbolic links in /etc/apache2/mods-enabled for deflate.load and deflate.conf to enable mod_deflate. (Use a2dismod to remove them if needed.)

I added two content types for CSS & JavaScript to the default in deflate.conf:

AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css application/x-javascript

That used to be riskier when very old browsers such as Netscape 3 and 4 claimed to support compressed CSS & JavaScript but actually didn't. But those browsers are long gone.

The next easy optimization is to enable proxy and browser caching of static content: images, CSS, and JavaScript files. By doing this we eliminate all HTTP requests for these files; the browser won't even check with the server to see if it has the current version of these files once it has loaded them into its cache, making subsequent use of those files blazingly fast.

There is, of course, a tradeoff to this. Once the browser has the file cached, you can't make it fetch a newer version unless you change the filename. So we'll set a cache lifetime of only one hour. That's long enough to easily cover most users' browsing sessions at a site like this, but short enough that if we need to publish a new version of one of these files, it will still propagate fairly quickly.

So I added to the Apache configuration file for this virtual host:

ExpiresActive On
ExpiresByType image/gif  "access plus 1 hour"
ExpiresByType image/jpeg "access plus 1 hour"
ExpiresByType image/png  "access plus 1 hour"
ExpiresByType text/css   "access plus 1 hour"
ExpiresByType application/x-javascript "access plus 1 hour"
FileETag None
Header unset ETag

This adds the HTTP response header "Cache-Control: max-age=3600" for those static files. I also have Apache remove the ETag header which is not needed given this caching and the Last-modified header.

There are cases where the above configuration would be too broad, for example, if you have:

• images that differ with the same filename, such as CAPTCHAs
• static files that vary based on logged-in state
• dynamically-generated CSS or JavaScript files with the same name

If the website is completely static, including the HTML, or identical for all users at the same time even though dynamically generated, we could also enable caching the HTML pages themselves. But in the case of icdevgroup.org, that would probably cause trouble with the Gitweb repository browser, live documentation searches, etc.

After those changes, we can see the results of a new webpagetest.org run and see that we reduced the bytes transferred, and the delivery time. It's especially dramatic to see how much faster subsequent page views of the Hall of Fame are, since it has many screenshot thumbnail images.

Optimizing a simple non-commerce site such as icdevgroup.org is easy and even fun. With caution and practicing on a non-production system, complex ecommerce sites can be optimized using the same techniques, with even more dramatic benefits.

Interchange news

Tomorrow we'll be having an Interchange community meeting on IRC. All Interchange users and any other interested parties are invited to participate.

Also, just recently, End Point's own David Christensen joined the Interchange Development Group and became a core committer. Congratulations, David, and keep up the good work!

LinuxTag 2009 day 1

Today was the first day of LinuxTag 2009. Representing the Interchange project are Stefan Hornburg of LinuXia Systems in Germany, Davor Ocelić of Spinlock Solutions in Croatia, and I, Jon Jensen, of End Point in the U.S.

Tuesday afternoon we set up the booth (here still underway):

That was a fairly quiet affair since many exhibitors showed up later that afternoon or early Wednesday morning. But it was nice to get it all done early. The setup involves the network and power wiring behind the scenes, hanging the signs, unloading the marketing materials, and getting all the equipment tested (and then put away again for the night). At night back in our apartment we made some updates to the slide presentation to include many more examples of some of the busy and interesting sites we have current data on that appear in the Interchange Hall of Fame.

We're sharing a booth with the YaCy distributed search project, and have had a few good discussions with their people.

Booth traffic was probably about the same this year as it was the first day last year -- a little slow. We talked with several people who were interested in hosted e-commerce solutions such as Interchange is.

In two cases, very interested visitors were not at all clear about how the open source partnership between project, individuals, and businesses works, and we were able to explain it. (Hopefully clearly enough that it still made sense after we were done talking!)

Specifically, one visitor represents a hosting company that wanted to pay to include Interchange in their hosting offerings. We are of course happy to take his money but have no set price to offer because, as he later succinctly put it, "Sie leben von Ihren Dienstleistungen." That is, "You live from your [custom] services." Exactly.

We explained the service model, his ability to download the software for free not just to evaluate, but to permanently use and resell as a hosted service to others. At the end it was clear it would've been simpler for him to hear it costs something like €500 or €5000 per year to be part of our hosting partner program. Yet that wouldn't have answered the question of what support we provide, how his programmers can contribute back to the Interchange community as they customize the software, etc. So the elaboration is necessary. And we explained that each of us three Interchange developers represents our three different consulting companies, and that's who you actually do business with, not "Interchange" per se.

The other visitor who didn't have a background with open source software wanted something similar, a fixed price to be allowed to deploy the software and customize it for his own consulting customers. A similar discussion was had. The bottom line is, the software's free, but work you hire us to do specifically for you is not. That's not too complicated once you get used to it!

In between talking with visitors, we talked about some work Stefan's been doing on the WellWell catalog, the need for new experimental Interchange branches which is much easier now with our new Git repository, and some custom work that's been done before that needs to be genericized and committed to mainline Interchange. We also dropped in on the Freenode #interchange IRC channel and worked a bit with Gert van der Spoel, René Hertell, David Christensen, and others.

Otherwise, I had time to attend one talk, Die Mathematik hinter RAID, by Michael Schwartzkopff. He worked through the math to show the probability of various kinds of failures when using RAID 1 and RAID 5, discussed RAID 6 and Sun's ZFS RAID-Z2. It was quite interesting and a good reminder that as hard disk capacity grows, what once seemed like incredibly small chances of failure (a one-bit read error, or a failure of a disk's mechanism) become both more likely and more catastrophic when they do occur.

So we're off to a good start, with three more days to go.

Using the new-style Google Analytics pageTracker functions in Interchange

For a while now there have been two different ways to setup the JavaScript calls to report traffic back to Google Analytics. The older method uses functions names that mention "urchin," while the newer method uses a function named "pageTracker". This post describes an approach for using the new method at a standard Interchange store.

You can see an example of the new method of reporting a page view here. Nothing Interchange-related is required for normal page tracking, but you may want to use a variable for the Google Account Number, of which more below.

If you have your Google Analytics account setup to treat the website as an E-commerce site, then you can also add the order tracking tags to your receipt page, so that it sends order data over to Google Analytics at the time of conversion. The order tracking tags can be viewed here. This gist shows the typical Interchange tags you might want to use to transmit the order specifics. Of course you might need to change the field used for the category for the products since not everyone uses the prod_group field from the products table to hold this information.

As you can see, both normal and the order-conversion scripts need to be modified to contain the individual Google Analytics account number for the website. I tend to set up an Interchange variable such as GOOGLE_ANALYTICS_ID in the variable.txt file or catalog.cfg.

Learn more about End Point's analytics expertise.

In Interchange, You Might Need to [try] [goto]. What's the [catch]?

Interchange provides tags that allow error trapping and handling within ITL--[try] and [catch]--that can be thought of as analogous to perl's eval {} followed by if ($@) {}. However, as I discovered the hard way, the analogy is not perfect.

I set up a block of ITL within [try] that had two major actions, with the 2nd depending on the success of the first. In particular, these two actions were a credit card authorization, followed by a capture of that auth as long as (a) the authorization succeeded, and (b) the merchant's internal rules for analyzing order content compared to AVS results "passed". (b) was necessary as a fraud-protection measure, tightening up the impact of AVS results based on the historic tendency of certain products to be targeted by crooks. In the event that the auth succeeded, but the tests from (b) failed, it is very important that the capture never be attempted because, to the gateway, the auth is entirely valid and the catpure attempt would succeed.

The code that assesses whether AVS passes is done in its own [calc]. From within the code, if the assessment does not pass, the code issues a die(), which in fact does trigger [try] to log the error that becomes accessible in [catch] via the $ERROR$ token, and thus does trigger [catch] to execute its body contents. In that way, the [try] did trap the error, and the error was handled in [catch], but of course that's not the end of the story or this post wouldn't exist.

After the code had been in production for some time, David Christensen brought to my attention that he noticed in development a test order attempt, where the order attempt failed, but both the auth and the capture succeeded. I was highly dubious of this claim and went over in great detail just what he had done. We narrowed down the condition that produced the problem to (b) above: a successful auth, but abort the order attempt anyway because a high-value product in the cart was coupled with a questionable AVS. When I went to the logs, I could see the result spelled out, but the result made no sense to my understanding:

### starting credit card processing ###
Real-time full auth succeeded. ID=***
0
Real-time full capture succeeded. ID=***
Error detected for order xxx in the credit card charge.

The 0 indicated the [calc] had failed (died), yet the capture later in the [try] was still executing. The only conclusion was that, unlike eval {}, when [try] trapped an error, it just kept right on processing the continuing ITL. [try] always went forward and processed all its ITL to the end, and whatever happened to be the last die() called within that batch of ITL would be the thing that [catch] caught and displayed.

To resolve the problem, I introduced [goto] into the block, which stops the instance of interpolate_html() running at the point of encounter and returns. Continuing to use the die() call to populate the error code from [try], immediately after the [calc] test block I called [goto] conditionally on whether the [calc] block, in fact, died. The [goto] call then terminated the instance of interpolate_html() that [try] had invoked on its body, which had the effect of stopping ITL execution at the point of the die().

This approach to emulating eval {}/if ($@) {} has the significant flaw of developers needing to know ahead of time exactly where in the [try] block such failures are expected. If such is unknowable, it leaves developers in the unenviable position of having to follow each tag call with a conditional [goto] that has to know when the previous tag "failed" (i.e., triggered a die() somewhere).

Subverting Subversion for Fun and Profit

One of our clients recently discovered a bug in a little-used but vital portion of the admin functionality of their site. (Stay with me here...) After traditional debugging techniques failed on the code in question, it was time to look to the VCS for identifying the regression.

We fortunately had the code for their site in version control, which is obviously a big win. Unfortunately (for me, at least), the repository was stored in Subversion, which means that my bag o' tricks was significantly diminished compared to my favorite VCS, git. After attempting to use 'svn log/svn diff -c' to help identify the culprit based on what I *thought* the issue might be, I realized that svn was just not up to the task.

Enter git-svn. Using git svn clone file://path/to/repository/trunk, I was able to acquire a git-ized version of the application's repository. For this client, we use DevCamps exclusively, so the entire application stack is stored in the local directory and run locally, including apache instance and postgres cluster. These pieces are necessarily unversioned, and are ignored in the repository setup. I was able to stop all camp services in the old camp directory (svn-based), rsync over all unversioned files to the new git repository (excluding the .svn metadata), replace the svn-based camp with the new git-svn based one, and fire up the camp services again. Started up immediately and worked like a charm. I now had git installed and working in what had previously only been svn-capable before.

Now that I had a git installation, I was able to pull one of my favorite tools from my toolbox when fighting regressions: git-bisect. In my previous svn contortions, I had located a previous revision several hundred commits back which did not exhibit the regression, so I was able to start the bisect with the following command: git bisect start bad good. In this case, bad was master and good was the revision I had found previously. Using git svn find-rev rnumber, I found the SHA1 commit for the good ref as git saw it.

From this point, I was able to quickly identify the commit which introduced the regression. In reviewing the diff, there was nothing that I would have expected to cause the issue at hand; the code did not touch any of the affected area of the admin. But git had never lied to me before. I compared the code currently in master with that introduced in the implicated commit and saw that most of it was still in place. I began selectively commenting out pieces of the code the commit introduced, and was able to enable/disable the bug with increasingly fine granularity. Finally, I was able to identify the single line which when removed caused the issue to evaporate. This was a line in an innocuous template which had a simple variable interpolation (inside an HTML comment, nonetheless); however, this line (which was in a file which was included with every document, added in the implicated commit) revealed a bug in the parser of the app-server which was causing the symptoms in the unrelated admin area.

It's certain that I would never have been able to find the source of this issue without git-bisect, as manual bisection with svn would have been too tedious to even consider. I am able to happily interact with the rest of the development team with git being my secret weapon; git svn dcommit enables me to push my commits upstream, and git svn fetch/git svn rebase enable me to pull in the upstream changes. I'll never need to tell my subversive secret (except, you know, on the company blog), and my own happiness and productivity has increased. Profit!!11 all around.

Interchange jobs caveat

I'd used Interchange's jobs feature to handle sending out email expirations and re-invites for a client. However I found out the hard way that scratch variables persisted between individual sub-jobs in the job set. I'd tested each of the two sub-jobs in isolation and had had no issues.

This bit me because I'd assumed each job component was run in isolation and variables were initialized with sensible (aka empty) content. In my case it fortunately only affected the reporting of each piece of the job system, but definitely could have affected larger pieces of the system.

The lessons? 1) Always explicitly initialize your variables; you don't know the ultimate context they'll be run in. 2) Individual component testing is no substitute for testing a system as a whole; you can reveal bugs that would otherwise slip through.

Standardized image locations for external linkage

Here's an interesting thought: http://www.boingboing.net/2008/09/01/publishers-should-al.html

Nutshell summary: publishers should put cover images of books into a standard, predictable location (like http://www.acmebooks.com/covers/{ISBN}.jpg).

This could be extended for almost any e-commerce site where the product image might be useful for reviews, links, etc.

At very least, with Interchange action maps, a site could capture external references to such image requests for further study. (E.g., internally you might reference a product image as [image src="images/products/current{SKU}"], but externally as "/products/{SKU}.jpg"; the actionmap wouldn't be used for the site, but only for other sites linking to your images.)