Development

In 4K Labs, (our own version of google “20% time”) we’ve built a web-based, Drupal-powered version of those “magnetic poetry” kits you may have seen on someone’s refrigerator. Drupal Poetry is a web version of “magnetic poetry,” complete with a touch-and-drag interface for tablets/phones and includes several Drupal-centric word packs. After you’ve crafted your masterpiece, log in to Twitter, and share you poem with other Drupalers and the world!

Drupal is a great platform, but it can’t do everything. As your site grows, you’ll likely encounter use cases that Drupal can’t or shouldn’t do. Some examples include interacting with third-party APIs while preserving good page loads, or performing repeated actions (polling, etc) that result in site updates. Fortunately, separating these kinds of problems from Drupal and moving them to specialized “sub-stacks” within or near your Drupal stack is easy to do.

I had a really great time attending Drupal Camp Asheville and giving presentations there, a my “Debugging Techniques for Drupal and LAMP” (I have presented versions of that before, most recently at DrupalCamp Colorado) and a new presentation, “Simple DevOps Using Jenkins.”

Here at Four Kitchens we make BIG websites — a lot of them. In the past, keeping up with the development environments for all of these sites added a lot of overhead, which ultimately means more time managing code than writing it. So like good developers we asked ourselves: “how can we make this better, lower maintenance, and reusable?” What we came up with isn’t necessarily anything new, but has completely changed how our internal development process works.

One really useful HTML element that doesn’t seem to get much love is optgroup. It allows you to group the items of a select list in a way that may be more meaningful (and more readable) to your users than a long, uninterrupted list. The option group labels themselves can’t be selected, so there’s no need for back end logic to filter them out of your form data.

Drupal’s Form API provides option groups, but it isn’t immediately obvious how to use them. A quick check of the select type and #options attribute in the Form API reference doesn’t provide any clues about option groups, so it’s necessary to dive into Drupal’s source code.

Drupal and Pressflow have too much magic in them, and not the good kind. On the recent Facebook webcast introducing HipHop PHP, their PHP-to-C++ converter, they broke down PHP language features into two categories: magic and mundane. The distinction is how well each capability of PHP, a dynamic language, translates to a static language like C++. “Mundane” features translate well to C++ and get a big performance boost in HipHop PHP. “Magic” features are either unsupported, like eval(), or run about as fast as today’s PHP+APC, like call_user_func_array().

Pagination is one of the hardest problems for web applications supporting access-control lists (ACLs). Drupal and Pressflow support ACLs through the node access system.

Problems with traditional pagination

• Because pagination uses row offsets into the results, browsing listings where newly published items get added to the beginning of the results creates “page drift.” Page drift is where a user already browsing through paginated results sees, for example, items E, D, and C on page one, waits awhile, clicks to the next page, and sees items C, B, and A. Going back to page one again shows F (newly published), E, and D. Item C “drifted” to page two while the user was reading page one. If new items are published frequently enough, pagination can become unusable due to this drifting effect.
• Even if content and ordering are fully indexed, jumping n rows into the results remains inefficient; it scales linearly with depth into pagination.
• Paginating sets where the content and ordering are not fully indexed is even worse, often to the point of being unusable.
• The design is optimized around visiting arbitrary page offsets, which does not reflect user needs. Users only need to make relative jumps in pagination of up to 10 pages (or so) in either direction or to start from the end of the results. (If users are navigating results by hopping to arbitrary pages to drill down to what they need, there are other flaws in the system.)

Anyone experienced with high-performance, scalable PHP development is familiar with APC and memcached. But used alone, they each have serious limitations:

APC

• Advantages
  • Low latency
  • No need to serialize/unserialize items
  • Scales perfectly with more web servers
• Disadvantages
  • No enforced consistency across multiple web servers
  • Cache is not shared; each web server must generate each item

memcached

• Advantages
  • Consistent across multiple web servers
  • Cache is shared across all web servers; items only need to be generated once
• Disadvantages
  • High latency
  • Requires serializing/unserializing items
  • Easily shards data across multiple web servers, but is still a big, shared cache

Combining the two

Traditionally, application developers simply think about consistency needs. If consistency is unnecessary (or the scope of the application is one web server), APC is great. Otherwise, memcached is the choice. There is, however, a third, hybrid option: use memcached as a coordination system for invalidation with APC as the main item cache. This functions as a loose L1/L2 cache structure. To borrow terminology from multimaster replication systems, memcached stores “tombstone” records.

The Vancouver Planetarium. Photo by Qole Pejorian.

chx and I gathered last week in Vancouver’s West End for a two-person performance sprint during the final code slush days, allowing us to finish several key improvements to Drupal’s database layer. Right afterward, many more people joined us for another sprint to port key modules to Drupal 7. People worked in-person, voluntarily over IRC, and involuntarily over IRC (lost passport).

I can say — without reservation — that our work was successful. We kicked off the weekend with Drupal 6 versions of Coder and Views. (Though there had been a touch of prior work on the Views port to Drupal 7’s new database layer.)

For modern applications, the word “schema” has become synonymous with the tables, columns, constraints, indexes, and foreign keys in a relational database management system. A typical relational schema affects physical concerns (like record layout on disk) and logical concerns (like the cascading deletion of records in related tables).

Schemas have gotten a bad name because current RDBMS tools give them these rotten attributes: