After winning control of the New York State Senate for the first time in almost 50 years, the Democratic party turned to Advomatic to quickly deploy a new website powered by Drupal. There were many challenges to be overcome, and their requirements pushed the envelope in some areas such as permissions and work flow. In the end, this site stands as a testament to what can be accomplished by a focused and experienced development team using the power and flexibility offered by Drupal and its community.
Multi-Site Structure
Previously, each Senator controlled their own little section of the web with a proprietary ASP-driven site, with little continuity between them. The Senate wanted to allow each senator to continue to control their individual sub-sites, but wanted content to be more easily shared between the sites, and for desired articles from individual senators to occasionally be featured on the main section of the site.
This obviously required a multi-site solution, and this was the first issue to be tackled, since its implementation would inform the rest of the development.
We looked at several proposed solutions and modules that help with deploying such a site, and in the end decided to create a custom method (similar to what we deployed for Air America Radio), because of the unique requirements. This consisted of creating a ‘Senator’ content type, to which every other content could reference, which would be used for grouping. For instance, Senator Malcom A. Smith’s section groups all of his content, with a unique header, a specialized menu navigation, blog, etc.
The design (created by our very own Amanda Luker and Jack Haas) called for two basic themes, dependent on party affiliation. We simply added a drop-down field with a ‘Blue’ and ‘Red’ option, originally with the intention of switching the global $custom_theme as needed. In the end, we decided to simply print that in the body class, as it was easier to simply use a CSS switch rather than an entirely unique theme. We used Cufón for image replacement throughout the site.
Menu Navigation
Menu navigation was far more challenging to implement. They wanted each senator to have its own menu for primary navigation and have the menu automatically pre-populated, but to also allow each senator to override the menu as desired. We quickly dismissed the standard menu as an option, as we did not want to open up the can of worms of allowing full access to the system menu by a disgruntled office editor. Even if we modified the access to only allow access to a specific menu, that wouldn’t solve the need to automatically create a suitable menu from a template, which would serve 95% of the senators.
In the end, we opted to use a Link field for the menu, which would be printed by overriding $vars[‘primary_links’] in phptemplate_preprocess_page. Then we simply checked the path to see if we were on a page referencing a senator, in which case we would do the override. We also implemented a custom token for this, so that we could fill it with such useful default paths as [senator]/insights and [senator]/gallery.
As there were many other cases for which we needed to know if a specific page load referenced a senator, we put all the arg sniffing in a static variable during hook_init, so that we could easily discover that: a call to nyss_senator() would return either FALSE or the referenced senator’s node.
Data Migration
After creating the basic content types, with required senator node reference fields, it was time to populate our site. The original vendor had provided us with a backup disk with about 12,000 pieces of content. We wrote a quick migration script to map the original data with our new content types. In the past, we would simply do this over the command line. This time, I wanted to dive into Drupal 6’s batch operations, which offers a slick interface, so I put the scripts into a special one-time only admin callback so I could watch a druplified progress bar while I ate popcorn.
The original data being in a Microsoft Access database presented several problems for migration. Firstly, the original dump was corrupted, as the original migration returned about three quarters of the nodes with odd characters. After investigating why apostrophes were being turned into ’ (among other problems), we learned the data had been corrupted when the original vendor backed up the database. Normally, PHP’s iconv would have been able to convert from Latin-1 character encoding to the required UTF-8. However, the corruption was in the data itself, which meant we had to get a new backup. A few hours later, the new backup was delivered, which was properly encoded.
There were other problems with the original data: there was no unique field on the rows, images and pdfs were stored inline (which meant we had to modify those on the fly to match our new consolidated /files folder, and also turn them into suitable file objects for storage in a filefield). Then there was the problem of editing the 300 page State Constitution as a single node.
These issues were easily surmountable, and soon we had a working site. Interestingly, the original proprietary CMS didn’t have anything approaching Drupal’s check_plain, which meant that in the past, some innovative senatorial editors had gotten away with such fancies as displaying a page’s title as a JS marquee…
Editor Permissions
With 12,000 nodes filling in the sections for 60+ senators, not all of whom play nice together, we next had to tackle the roles and permissions for the site. This was not a straight-forward matter: they all had to be able to edit content from their respective senators, but not that from other senators. Additionally, if a senator had multiple editors, they would need to be able to edit each other’s work, and we had to account for the possibility of some editors being shared amongst senators.
We made use of hook_node_access_records and hook_node_grants to handle the grunt work for this. We created a grant realm for the possible field_senator’s nid on a node, and mapped this to the office editors assigned to a senator. This in turn was assigned with a user_reference field selecting from a view of users with the Office Editor role. Finally, we altered the menu callback for node/[nid]/edit to check the permissions, to keep folks from being mislead by an edit tab to nowhere.
Flag FTW
We made heavy use of the Flag module throughout the site. This included not only promoting content to the front carousel (which was created using jQuery Cycle) and to other blocks on the front page, but also to a similar carousel and tabs on each senator’s page, galleries, and other places throughout the site.
Of course, this presented its own set of problems for our unique permissions structure, as a senator should not be able to flag content for another senator. To circumvent the problems, we altered the flag links so they would not appear if the editor did not have permission to edit a specific node.
Bread and Butter
Views, of course, filled the site with our content. After creating and finalizing the View blocks and pages, we exported them all for better performance. There were no real problems with this stage of things, except for a few view queries that were not possible to create with the standard UI. In those few cases, we made use of hook_views_query_alter. Additionally, because many of the views required an argument of a referenced senator, and we had a custom token for our senator’s path that didn’t necessarily match up to the title, we had to write our own views argument handler that we stuck in the argument PHP validation block when required.
As a site of any complexity would need, we needed to implement several custom blocks. We have refined a system over several sites that we continued here, that allows for the flexibility allowed with Drupal’s custom block system, while extending it to allow things to be easily controlled through code, rather than embedding such things as a block’s per-page appearance on the administration form. Basically, when creating a new custom block, we define its visibility as 2 (so the pages are controlled with a PHP callback), and set that to return a custom callback with the $delta. Then if we decide in the future to move one or more blocks to another page, we can change the settings in the callback, rather than submitting each block configuration page individually. To further ease configuration, and take advantage of Drupal’s just in time file loading, we also create theme functions automatically when a new custom $delta is added, loading the new theme_custom_block_$delta.tpl.php file as needed.
There are a lot of little things I would love to tell you about, and there was much more that went into creating this site than I can get into without turning this case study into a novel-length exposition. I haven’t even discussed Committees, Legislation, or their Open Data project.
Advomatic’s development team was superb! I was assisted by Jonathan DeLaigle, Amanda Luker, Jack Haas, and Marco Carbone. Overseeing the project were Fred Gooltz and Dylan Clear. Craig Leinoff also jumped in feet first from the Senate’s CIO office to help develop the site and get it ready to launch.
I am excited to see more of our government extolling the virtues of Open Source, and am proud to have helped to deliver a robust Drupal site to the New York State Senate.
Aaron Winborn is a developer with Advomatic. Besides development, Advomatic also offers a wide range of other Drupal services, including maintenance and clustered hosting. In addition to helping roll out sites such as the New York State Senate, Air America, and Mozilla, Aaron also contributes heavily to the Drupal community, including such modules as Embedded Media Field and Views SlideShow. He has written Drupal Multimedia, published by Packt Publishing, and is mentoring a Google Summer of Code project to help roll out the upcoming Media module.