Archive for the 'development' category

Return to blog homepage

Templating is a real hot button in the web development community. The only other topic I can think of more prone to sanctimony and religous flame wars is quality control and testing.

The high horses that people usually get on are that all too familiar TLA MVC (Model/View/Controller) architecture and "separation of presentation and business logic".

The poor pedestrians upon which they look down are those who have written "spaghetti code" - templates where presentation logic, markup, business logic, database access configuration and whatever else you might imagine are mixed up in the same file.

Well, I've got some news for you: you're all wrong.

Get off your high horse

What is spaghetti code anyway?

Spaghetti code is a really old term dating back to the days of goto statements and the transition from unstructured to structured programming languages.

There's not really one true definition and it's meaning has evolved over time but the common thread is that it refers to code where things are mixed together that shouldn't be, the result being difficult to understand if you didn't write the application (and sometimes even if you did).

In the world of web development, the term most often refers to the inclusion of both business and presentation logic in the same file as your HTML (and javascript and css or whatever else you have on your page).

A typical example written in PHP would look like this:

Why all templating sucks

This is where folks will start to get technical about why their code isn't spaghetti code.

Oh no! Not me because I use (Django|Rails|Haskell|Node.js|Symfony|CakePHP|Tornado|Plone) and the superior templating language (Django Template|ERB|Mustache|PHP|TAL|Smarty) doesn't allow business logic in templates and enforces a separation of concerns

You, the reader, December 2011

The problem is that in every single one of these circumstances, the templating engine has 2 primary responsibilities:

  • The fragmentification and inclusion of common interface assets (to avoid duplication and maintaining the same code in multiple files) - eg. headers, footers, sidebars and the like
  • The output of dynamic content

The output of dynamic content requires logic (if/else, loops being about the bare minimum).

The creators of templating languages have, to varying degress, availed themselves of a false dichotomy, namely that there exists "presentation" logic and "business" logic, the former of which belongs in templates and the latter of which has absolutely no place in templates and must be banished immediately.

I say false dichotomy because this is an arbitrary separation - a choice made by the person who created the templating engine and is completely subjective.

Even Mustache, which boasts to be "logic-less templates" is guilty of having some logic in their templates. After all, if I have a construct that performs logic, even if that logic doesn't look like logic, isn't it still logic? To take the demo on their home page:

See that last line that says "#empty"? In the demo you modify some JSON and render the template and it suggests "Try setting empty to true". Low and behold IF you set empty to TRUE, THEN the paragraph tag indicating that the list is empty is rendered. However IF you set empty to FALSE, THEN the paragraph tag indicating that the list is empty is not rendered.

So basically I call bullshit on Mustache being "logic-less". It may have the most ruthless attitude to excluding "business" logic (whatever the hell that is) from the templates but there's logic in there, to be sure, even if they're trying really really hard to make it look like there isn't.

It's all spaghetti code

So I'd like to proffer that all templating languages produce spaghetti code.

Surely not, because that Mustache template was heaps easier to read than the PHP example you gave at the beginning of this article

You, the reader, 2 minutes later, December 2011

Well let me hit you with this: although there are varying degrees of entanglement and obfuscation, and certainly one person might write more readable, maintainable spaghetti code than another person, that doesn't change the fact that it's still spaghetti.

I would like to formalise a new and more ruthless definition of spaghetti code:

As soon as I'm looking at more than one programming or markup language in the same file, I'm looking at spaghetti code.

Iain Dooley, December 2011

This issue was been popularly addressed in JavaScript for quite some time. The first time I recall seeing the concept of banishing things like onclick attributes from markup was when I first read about the Behaviour.js library and the whole thing just seemed incredibly sensible to me. Why is it that we still accept the equivalent in our server side templating engines?

The holy grail: Template Animation

So long as I'm updating the definition of spaghetti code, I might as well coin a new term while I'm at it:

Template animation: the practice of using static HTML files as a resource from within your application to be manipulated via the DOM to generate dynamic output.

Iain Dooley, December 2011

Instead of splitting up the task of templating in the following, arbitrary fashion:

Traditional templating separation

We can create a non-arbitrary, completely objective separation where the view/template layer truly contains no logic what so ever:

A proposed, new way of separating concerns

When I first had this idea, it was because David Grinton and I had actually created a little PHP script for doing Django style template inheritance in XML.

It was a very short mental walk to realise that, with a system in place that was responsible solely for fragmentification of documents and re-use of common assets capable of returning head-to-foot complete HTML files, all the backend programmer has to do is load the file and manipulate it using the DOM to output the required response.

Whilst the impetus for this was the fact that we'd created a tool for fragmentification of XML documents, it really doesn't matter what the frontend developer uses.

Even Dreamweaver is capable of allowing a frontend developer to re-use common view components and it has no impact what-so-ever on the application layer so long as the server side developer is presented with a complete document to work with.

I created a little proof of concept which you can see in this class file just done using raw DOM functions in PHP.

Since DOM manipulation totally sucks I went looking for a jQuery port to PHP and found phpQuery and queryTemplates by the same author (neither of which appear to be being actively maintained), as well as HTML::Zoom for Perl on CPAN.

What are the advantages?

Dramatically simpler resourcing and training (in-house or outsourced teams)

The single biggest advantage here is simplicity of resourcing and separation of skills. To illustrate this let's look at a counter example.

Recently, Luke Wroblewski posted his notes from the "Design for Continuous Deployment" presentation by Randy Hunt from Etsy. From the article:

The company has 140 people working on the product (design & engineering) and they are all pushing production code between 50-60 times a day.

In the section entitled "How do you do it?" the two key points are:

  • Everyone at Etsy has a development environment on their machine. All the designers have local environments to do design work on GUIs.
  • Quick start guides and packages are created for designers to bring them up to speed on how the process works.

This is emblematic of the problem in traditional templating engines/development frameworks, and it's fine if you have a design and engineering department, a profitable business and time for designers to get "up to speed on how the process works" by reading guides and packages (as well as the time to develop those resources in the first place) but ultimately this is not a solution for smaller, scrappier teams with less time and money on their hands.

The fact that Etsy even had to discuss this as a feat of engineering achievement shows just how big of a problem it is, and changing the way we approach templating solves this problem completely.

By defining an application first as a series of static, clickable HTML mockups and then passing these off to developers who "animate" them, you obviate the need for designers to be exposed to things like local development environments or logic of any kind.

You also create non-blocking workflows between designers and developers. Nothing a developer can do will ever break what the designer is working on. The designer will never have to wait for a piece of backend code to be written in order to design a new feature or refine an existing one.

Certain strategies can be used such as not changing ids and classes so that these "hooks" remain constant and you break the application less when changing it, but regardless of this, a breakage due to a change in the markup is solely a coder's problem. It's something that automated testing or even basic user acceptance testing will isolate quickly and which one person (ie. the developer) can fix on their own without the need for costly communication between the people working on the front and the backend.

I'll stake my reputation (whatever that's worth) on the fact that any "brittleness" introduced on the developer's side of the equation as a result of having to target specific pieces of markup can a) be reduced to an extent by not changing classes and ids too frequently and b) will be far less costly than the traditional headfuck involved with designers having to have a local copy of the entire application running in order to iterate on your application's interfaces.

Even as a single person on a project you'll find it easier to iterate and easier to find someone else to work on it if you want to at some point.

Dramatic reduction of technical debt on any project and any platform

Technical debt manifests itself most frequently in the inability to iterate quickly and keep your product current. This reduces your ability to compete and thusly your effectiveness as a business.

This approach to templating basically eradicates the biggest source of technical debt: having your interface tied inextricably to your codebase.

It is a simple fact of programming on the web that HTML(4/5/X whatever), CSS and JavaScript are present in every technology stack.

By prototyping your entire application in static HTML, CSS and JavaScript, you isolate the most valuable part of your application (the feature set and user experience) from the means with which it is implemented (or "animated").

The beauty of this approach is really that it doesn't matter at all what you use on the backend: or even that you are consistent, or that you start with the same approach you continue on with later, or that everyone uses the same coding standards or whatever.

If you're not happy with the code that's driving your application, just pick up and go somewhere else. I'm not saying this is free mind you, but it would be a hell of a lot easier to take a complete, static HTML prototype that was written with a PHP backend using Template Animation and reimplement it whatever-the-hell-language-is-trendy-these-days (or what language is easiest to find talented developers to work on). In other words: you can stick your language wars, I don't give a shit what it's written in so long as it works like this in the browser.

Not only does Template Animation release you from the long term bonds of your technical choices in backend technology, but it shortens the lifecycle of iterations in your product and makes it quicker, cheaper and easier to change the user experience because you can iterate in your static mockup without having to "re-templatify" - that is re-add a bunch of template tags/logic or re-fragment your templates.

Even things like modernising and progressively enhancing your "Web 1.0 application" to be some wonder of modern engineering, single page application with hash bangs out the whazoo would be far simpler when your HTML templates are completely independent of the codebase used to animate them.

Okay smarty pants, but some logic is always required in templates LOL!!!1

In considering the implications of building an application using this method I came up with a few hypothetical scenarios to test the waters and I've narrowed the type of "logic" that you still need in templates to 2 things:

  1. Conditionally printing some value to the screen (eg. a form error or different menu items depending on whether the user is logged in or not)
  2. Conditionally including some structural element (eg. a sidebar when logged in or a shopping cart if you've added items to it)

The solutions, however, are trivial and require no logic to be present in your templates.

In the first case, you simply include everything in the markup that might need to be there, and the programmer removes whatever is not necessary:

In the second case, the designer/HTML/template guy simply implements two versions of the template and the developer loads whichever is appropriate:

In that second example of course, we've got a bunch of duplicate markup, but that's not our problem. That is the job of whatever system you're using to fragmentify your templates and re-use common assets. Hell, designers don't even have to use the same tools as each other in order for this to work, let alone having to use the same tools as the developers.

What about haml/markdown/{insert markup abstraction technology}

The longer I work as a programmer the less I care for abstractions. This goes, too, for ActiveRecord as an ORM strategy (although that's a subject for a different post) - Joel Spolsky summed up the Law of Leaky Abstractions years ago and I think that's still relevant today.

But if you really want to use some abstraction on the front end that means you have to type less markup or write less JavaScript, be my guest! Just make sure that by the time I'm working with the templates and adding in my logic to them, they're head-to-foot complete HTML documents with no crap in them.

Just the same as our Fragmentify script runs in PHP and allows the HTML guy to iterate on fragmented templates, so too could a simple HAML or Markdown parser allow an HTML guy to iterate with a minimal locally installed environment. Whatever. I don't care. It's all HTML, CSS and JavaScript in the end, the only thing that Template Animation advocates is that the technological barrier between the frontend and the backend is never crossed - that our templates are truly logic-less and that I never have to try and find a freelance designer that is also familiar with Django.

If you've read this far, please don't hesitate to follow Working Software on Twitter, become a fan of Working Software on Facebook and subscribe to this blog via RSS. You can also enter your email address below and I'll send you an email when we update the blog.

Filed under: development | View Comments

Just a quick note to let everyone know about a new website deployment that we're very proud to have been a part of deploying:

http://www.energyforopportunity.org/

Energy for Opportunity is an organisation that focuses on skills and training for building sustainable power generation capacity in remote 3rd world communities.

Working Software supplied Django development services to get the website up and running, and helped with deployment and choosing a web host.

Congratulations to all the team at EFO - we're really impressed by the work they're doing and wish them all the best for the future.

Filed under: development | View Comments

I just watched another fantastic TED talk video that Will Franco (a.k.a FlyWheel) over at Jive Systems linked me to a couple of weeks ago.

The video focuses primarily on changing from the "carrot and stick" model of management and incentives and into a "ROWE" (Results Only Work Environment) model for creative, 21st century tasks.

He frames the problem as being one of tasks that involve creativity vs. those that only require "mechanical" skills using a series of experiments done with the Candle Problem.

More specifically, that when 2 different versions of the same problem - one requiring the participant to overcome the problem of "functional fixedness" creatively and the other not - traditional extrinsic motivators either do not work or do work respectively.

About halfway through the talk, however, Pink states that software development is one of the things that can be pretty easily outsourced, and that's where I would disagree.

This reminds me of the famous section of the Mythical Man Month by Frederick P. Brooks, Jr.:

Men and months are interchangeable commodities only when a task can be partitioned among many workers with no communication between them. This is true of reaping wheat or picking cotton; it is not even approximately true of systems programming

This ties in closely with what Dan Pink is saying, but in quite a subtle way. What both Brooks and Pink are describing is a "purely mechanical" task vs. one which requires thought, creativity and communication.

The challenge in building software is that it takes a software engineer to know the difference.

Whilst it's simple for anyone to change the Candle Problem from one that requires creative output to one that doesn't, the process of "removing creativity" from the requirements of developing software is a software engineering task, and one that requires immense amounts of creativity.

In fact, the task of partitioning tasks on a project, particularly for a very early stage prototype, is often so much work that whoever might be doing the work would have been able to do the task faster themselves (and been more motivated to do it a better job of it!)

So think next time you are, or someone you know is, wanting to outsource software development: who's doing the creative tasks?

Filed under: development | View Comments

PickDrop is available for download free of charge. If you use and enjoy it, please feel free to donate via PayPal.

A couple of weeks ago, I created a command line bourne shell utility that let you "pick" files up from multiple locations on your computer then "drop" them all in one place. I uploaded it to GitHub (My First Project (TM)) and posted it on Hacker News (where it sailed off the "newest" page and into oblivion) and also on reddit, where some awesome shell scripting gurus contributed improvements to my crappy first version.

It was fun, and more than anything I now really understand what GitHub means when they say "social coding".

That day I was already using PickDrop quite a bit and finding it very handy. I thought it would be great if there were a Pick and Drop item in the Finder context menu on OSX. Since OSX is, after all, a Unix distribution, and since the original command line utility was written in the bourne shell it should be portable.

I did a bit of hunting around and found that I could create an Automator script pretty easily and call it a "Service" (in OSX 10.6 Snow Leopard) and have it just execute the scripts for me. After a couple of hours poking around with Automator I had PickDrop running in the Finder context menu and it was actually kind of cool.

So cool, in fact, that I thought "this might be a neat thing to put on the Mac App store". I did a quick search for some information about uploading to the app store but couldn't really find any detailed information. It looked as though I had to actually sign up with the Apple developer program (and pay them 120 semolians) before I found out any real information. I wasn't even sure I could make this really work as an installer yet so I decided to get a working installer before I shelled out the signup fee.

About 12 hours later, after praying to the Voodoo High Priestess that is PackageMaker and getting some friends to test the installer (seriously, PackageMaker is a dark art, I wouldn't wish that hell on anyone) I had a package that I thought I could upload to the Mac App store, so I went and joined up.

The signup process was relatively hassle free and the support team was responsive when I appealed their decision to require me to get a note from my doctor to verify my identity (because I had used my initials as they appear on my credit card in one place, and my full name in another). The process took about 48 hours to complete and then I had access to all the documentation and information about uploading.

One thing I notice on the "Submitting to the Mac App store page is":

Using the PackageMaker application to archive your application is not supported.

No wonder! It's a pretty lousy piece of software, that's for sure.

Of more concern, though, is that it states that an app should only install files to the following locations:

~/Library/Application Support/<app-identifier>
~/Library/<app-identifier>
~/Library/Caches/<app-identifier>

Since there's no way to make an item appear in the Finder context menu other than to install it to ~/Library/Services/ that means I can't distribute PickDrop via the Mac App store.

The first thing on the page regarding "Submitting to the Mac App Store" is a notice that says:

Important: This is a preliminary document. Although it has been reviewed for technical accuracy, it is not final...

So I figured maybe they just hadn't thought of the fact that someone might want to sell an Automator workflow as a service to add an item to the Finder context menu. It is, after all, a pretty dinky little "app" and possibly an edge case they had not considered.

I emailed developer support who replied within only a few hours (one thing is for sure: the developer support is responsive!) with the following:

Thank you for contacting Apple Developer Support regarding the Mac Developer Program. Developers should be testing and developing their apps in line with the Mac Developer Program License Agreement and the App Store Review Guidelines.

Fair enough. At this point I give up on the idea of distributing my package in the Mac App store and figure I'll just stick it online with a PayPal donate button instead and see what happens.

Something doesn't feel right ...

As dinky as this application is, this whole process really got me thinking about the Apple Developer program. The fact that they're expecting me to conform to their guidelines and use only their tools and submit applications in a very particular way makes me pretty uncomfortable.

Generally, I don't set out to build an application, then decide what to build. I generally come up with some idea or scratch my own itch and then decide that it's something I could monetise or that I'd like to share with others.

In his article "App is Crap (why Apple is bad for your health)" I think Mark Suster really hits the nail on the head when he says that the Apple App store is "... a channel disguised in business clothing."

I can't help but be reminded of Steve Ballmer frantically screaming "DEVELOPERS! DEVELOPERS! DEVELOPERS!" and the countless thousands of windows only application boxes you see in bargain bins at your local Post Office.

It's not that one can categorically say that development and distribution using Apple's tools and channels is a Bad Thing(TM). Certainly many people are making a fine living from developing for the App Store(s), but to me the browser still feels like a much better place to do business.

Filed under: development | View Comments

Subscribe

Subscribe via RSS

Building software in the real world - the Working Software blog

We write about our experiences, ideas and interests in business, software and the business of software. We also sometimes write about our own products (in order to promote them).