Idea Machine

Getting started with modular JavaScript

Thu, 12 Apr 2012 19:07:00 +0200

If you are developing rich internet application, and moving business logic to the client-side by truckloads, there is nothing better than using some form of module system, a loader that can take care of dependencies and asynchronous loading of your modules, and a system to package everything up for production use as a minified single JavaScript file. This is a very quick introduction to RequireJS and AMD modules which makes all of above come true.

In this tutorial, you will find just one way to do things. There are many more, but as soon as you figure out this one way of doing things, you will most certainly discover the others with ease. Therefore, we won’t dwell on all the nuances of AMD modules, and instead focus on how I generally do things so you can get started quickly and painlessly.

NodeJS (we’ll need it later)

Before you start, you will need to install NodeJS in your computer. Installation is well covered on the homepage, so I won’t go into details, but keep in mind that, if you use Linux, you probably do not want to install the version from your distribution’s package repository unless you are running Arch Linux and use AUR to do it. Grab the source and compile it (trust me, it’s not hard). For Linux users, here’s a copy-paste series of commands that will install NodeJS 0.6.15 into your user’s home directory (note it needs to be all one line, so I had to break it up with ):

cd ~ 
wget http://nodejs.org/dist/v0.6.15/node-v0.6.15.tar.gz
tar xvf node-v0.6.15.tar.gz
mkdir -p ~/local
cd node-v0.6.15
./configure --prefix=$HOME/local
make install
echo 'export PATH=$PATH:$HOME/local/bin' >> ~/.bashrc
echo 'export NODE_PATH=$HOME/local' >> ~/.bashrc
source ~/.bashrc

Now you can type node -v and confirm NodeJS is working. Also make sure NPM is installed along with NodeJS by typing npm -v.

Setting things up

Apart from NodeJS, you will also need to create a project directory. Let’s call our project test (yes, very creative, I know).

mkdir test
cd test
mkdir js
mkdir css
mkdir img

There’s our project tree.

Now we need a few more things. First, an index file that looks like this:

<!doctype html>
<html>
  <head>
    <title>RIA test</title>
    <link rel="stylesheet" href="css/main.css">
    <script src="js/require.js" data-main="js/boot"></script>
  </head>
  <body>
    <!-- move along, nothing to see here! -->
  </body>
</html>

This file will remain absolutely intact throughout the development. It does mention a few files and paths that we currently don’t have, so let’s take care of those.

Create an empty CSS file:

touch css/main.css

Download require.js and jquery.js, and create empty boot.js:

cd js
wget http://requirejs.org/docs/release/1.0.7/comments/require.js
wget http://code.jquery.com/jquery-1.7.2.js
touch boot.js
cd ..

And… that’s it. That’s the skeleton setup for our RIA frontend. If you haven’t bothered to create these, you can grab a zip file with the contents we have so far.

Now, a word about what these files will do.

require.js is our module loader with dependency resolution. It has an API that conforms to the AMD spcification which we will make use of to build our modules.

jquery.js… ok, seriously, does anyone need an explanation about what jQuery is and what it is used for?

boot.js is our main module, which kicks off the whole application (more on that later).

main.css is a CSS file that we will use to collect all our CSS modules (yes, you read right, CSS modules).

Ok, let’s get the ball rolling.

Baby step

If you load the index.html now, you’ll see that nothing actually happens, and our page is blank. That’s because our boot module isn’t doing anything yet. So let’s add something to it.

Before we do this, you should be aware that boot module is special, not just because it’s the last module that gets loaded, but also because… Um, what’s that? Yes, I said ‘last’. It’s last because any dependencies that it lists will get loaded first. And dependencies of those dependencies before that… And so on. Got it? Ok, where was I…

boot.js uses a syntax that is a wee bit different than other modules. I’ll get around to that, but first, let me show you the code:

require(['jquery'], function($) {
  $(document).ready(function() {
    alert('ready!');
  });
});

If the code is correct, you should see no JavaScript errors, and you should see an alert box that says ready!. What that means is, jquery was successfully loaded, and the ready even handler was fired correctly as well. So everything works. (Get the zip file of this step.)

The whole module is basically a require() call with two arguments. The first argument is a list of dependencies. You list dependencies without the .js extension, and relative to wherever you boot script is. So if your boot script is in js/ directory (as it should be per this tutorial), then a dependency that looks like lib/somemodule.js will be loaded from js/lib/somemodule.js. Although paths are relative, true relative paths cannot be used. So, you cannot say something like ../lib/somemodule.js or ./lib/somemodule.js. There are workarounds for this, but those workarounds are neither elegant, nor strictly required (no pun intended) to have a good experience developing with AMD.

Let’s build our first AMD modules

First order of business is templating. We want to make a very simple template library, and use it as a module.

Create a directory called lib in the js directory. That’s where we will keep all our helper libraries:

mkdir js/lib
touch js/lib/template.js

Let’s edit the template.js now:

define([], function() {

});

First thing first, that’s the module skeleton. It looks very much like boot.js, except that it uses define() call instead of require() call. And that’s basically your AMD module… more or less.

The empty array is our dependency list, which is empty because we have (and need) no dependencies. The function is called a factory function, and its job is to return the finished module.

The module is currently empty, because factory function doesn’t do anything. Since it has no code in it, it’s not very useful (not useful at all, that is), so let’s work on that:

define([], function() {
  var template = {};
  return template;
});

Ok, that’s better. What we’ve done is, we’ve created an empty object called template, and returned it. Now our template module is actually a real module, albeit it still does nothing.

The template object actually represents the eternal interface of our module. Anything that we put in that object will be exposed (because the object is returned from the function). This is where we define the API for the module.

The template module obviously needs a rendering function. So let’s expose such a function through the template object:

  /**
   * Render template
   * 
   * @param {String} t Template
   * @param {Object} d Data
   * @return {String} Rendered template
   */
  template.render = function(t, d) {
    // For each key in `d`, subsitute key in template with value
    for (key in d) {
      // We'll use '$key' format for placeholders in our template
      t = t.replace(new RegExp('\\$' + key, 'g'), d[key]);
    }

    // Return 'rendered' template
    return t;
  };

This is rudimentary template ‘engine’ that simply takes an object with keys, and replaces $+key placeholders in a string with values corresponding to those keys. Let’s test this out. Edit the boot module to look like this:

require(['jquery', 'lib/template'], function($, template) {
  $(document).ready(function() { 
    var tHithere = "Hi there. It's now $time";
    $('body').append(template.render(tHithere, {time: new Date()}));
  });
});

If you reload the page now, it should print out “Hi there, it’s now NOW”, where NOW is your current date and time were you are (in my case it’s “Hi there, it’s now Thu Apr 12 2012 17:26:35 GMT+0200 (CEST)”). (zip file of current progress)

So, let me show you something interesting now. Open your development console and try rendering a template. For example:

> template.render('My name is $name', {name: 'Branko'});

What you get is a ReferenceError. Why? Because modules are not global. Whatever you defined in your module is not accessible globally, and even when the module is referenced from another module, it’s only accessible from within that module. This makes it possible to have complete freedom when defining stuff inside your modules.

Privacy

So, module exposed methods are accessible from modules that reference (load) the module. How about if we wanted to keep some things private? Let’s do that now. The line where we build the regular expression is a bit ugly, so let’s move that part out into a private function:

define([], function() {
  var template = {};

  // Build regular expression
  function keyRe(key) {
    return new RegExp('\\$' + key, 'g');
  }

  /**
   * Render template
   * 
   * @param {String} t Template
   * @param {Object} d Data
   * @return {String} Rendered template
   */
  template.render = function(t, d) {
    // For each key in `d`, subsitute key in template with value
    for (key in d) {
      // We'll use '$key' format for placeholders in our template
      t = t.replace(keyRe(key), d[key]);
    }

    // Return 'rendered' template
    return t;
  };

  return template;
});

The new function keyRe is not exposed through a property on the template object, so it’s therefore not accessible to other modules. You can try calling keyRe() from the boot module, or even try template.keyRe, but both attempts would fail. (zip file of current progress)

Loading text templates

This is all fine and dandy, but if we wanted to edit HTML templates, it would be cumbersome to use JavaScript strings, right? So, we obviously need a way to load templates from text files.

Before we get to that, let’s first define a template that we’ll use. Create a directory called templates in your js directory:

mkdir js/templates
touch js/templates/main.tpl

Now edit main.tpl to look like this:

<h1>RIA test</h1>

<p>Hi, it's now $time, and you're looking at 
foxbunny's <em>AMD tutorial</em>.</p>

That’s done. But how do we load it? RequireJS sports a plugin system, which allows for various load plugins to be used. One such plugin is the aptly named ‘text’ plugin, which can be used to load plain-text files from a server. Download that plugin and put it in js directory:

cd js
wget http://requirejs.org/docs/release/1.0.7/comments/text.js
cd ..

Let’s modify our boot module to use this plugin and fetch main.tpl:

require(
  [
    'jquery', 
    'lib/template', 
    'text!templates/main.tpl'
  ], function(
    $, 
    template, 
    tMain
  ) {

  $(document).ready(function() { 
    $('body').append(template.render(tMain, {time: new Date()}));
  });

});

As you can see, I’ve changed the layout a bit so I can keep better track of dependencies. The way we used the text plugin is we just prepended text! to the path of the template file we wanted to load. Note that we must also include the extension in the template’s path. (zip file of current progress)

If you are using Chrome, the text plugin may trigger a security error “NETWORK_ERR: XMLHttpRequest Exception 101” because Chrome does not allow AJAX requests to be made to local file system. If you want to get around this, you must use a server to server your project.

Adding some style

Let’s add some stylesheets to spice things up a bit. We’ll divide the CSS properties into two modules, one dealing with layout, and one dealing with typography. In reality, you might take a completely different approach, and divide stylesheets by functionality, or by widget. The basic principle of writing modular stylesheets is the same.

Add two files, one called layout.css, and one called type.css to your css directory:

touch css/layout.css
touch css/type.css

Now edit layout.css to look like this:

* {
  margin: 0;
  padding: 0;
}

body {
  margin: 30px;
  width: 300px;
  border-radius: 10px;
  background: #996;
  padding: 1em;
  box-shadow: 0 0 10px rgba(0, 0, 0, 0.7);
  border: 1px solid #fff;
}

html {
  background: #588;
}

Contents of the type.css might look like this:

html, body {
  font-family: Helvetica, Arial, "Liberation Sans", sans-serif;
  font-size: 14px;
}

h1 {
  font-size: 2em;
  margin-bottom: 1em;
  color: #ee9;
}

To load these modules, just use the @import rules (note that if you add any CSS properties to main.css, the @import rules must be at the top of the file, or they won’t work). Edit the main.css file to look like this:

@import url('./layout.css');
@import url('./type.css');

The rendered page now looks more acceptable, depending on where you come from. (your zipball with stylesheets)

Final steps

Now that you’ve learned how to use RequireJS (I hope you have), and also a bit about organizing your code, let’s take a look at our payload.

That’s 10 requests and 348kb of payload right there. A bit on the heavy side. But no matter how heavy, the number of requests is always the main reason your site loads so slow (or maybe it doesn’t, but if it did, it would be the number of items, and then the size of the items).

We can cut this down to 4 (one for index.html, one for require.js loader, one of all other JavaScript files, and one for all CSS). Technically, you can even remove require.js from the equation by including it with other JS files, but I’ll leave that as an exercise to you.

The reason we installed NodeJS before is because of this. RequireJS project provides a nice optimization tool called r.js which will help us package our project into a set of 4 files we mentioned above.

First thing first, let’s install r.js:

npm install requirejs -g

Note the -g flag which tells NPM (Node package manager) to install the requirejs package globally. With that done, we now have access to r.js command. Before you can try this new toy, you need to write a project description. Just grab this one and put it in a file called app.build in the test directory:

({ appDir: “.”, baseUrl: “js”, dir: “../build”, uglify: { top_level: false, accii_only: false, beautify: false }, optimizeCss: “standard”, inlineText: true, findNestedDependencies: true, modules: [ { name: “boot”, path: “js/boot” } ] })

You can read more about what each option does by looking at the ‘full version’ of the build file in r.js’s github repository. In our case, we optimize CSS, inline any plain-text/HTML templates into modules that require them (so that they won’t be loaded separately after the page is loaded), and specify a few options for uglifyJS (JavaScript minifier) to avoid headaches with non-ASCII characters, etc.

The most important options above are appDir which points to a directory relative to the build file, and dir, which specifies where the optimized project will be output. Note that the output directory will contain everything that your source directory does.

Now that we’ve got our build profile, we can proceed to actually build it. (zip file of the current progress)

To build it simply run this command:

r.js -o app.build

The built project will be dumped into a directory called build just outside the test directory.

Loading the built project now gives us this in our network tab:

Yes, that’s right. 4 requests, and exactly 114kb of payload, all your dependencies and templates neatly packed into a single JS file, and all your CSS modules neatly packed into a single CSS file. How cool is that!

Conclusion

You’ve hopefully learned how to write AMD modules, and use them with RequireJS. You’ve possibly learned a thing or two about writing modular code in general. And you’ve learned a thing or two about how to spare your users from that 5-minute load time. But if you’re not used to rich internet applications in general, you might be missing a point here. So, load up that test page again, right-click on it, and do “View source”. Here’s what it looks like here:

<!doctype html>
<html>
  <head>
    <title>RIA test</title>
    <link rel="stylesheet" href="css/main.css">
    <script src="js/require.js" data-main="js/boot"></script>
  </head>
  <body>
    <!-- move along, nothing to see here! -->
  </body>
</html>

Yes, <body> is completely empty. Even though the page does contain things, and the developer tools show you all the elements on the page, the “View source” function is empty because it is showing you the page as it was originally loaded from the server. We’ve actually built the whole page using JavaScript.

Yeah, I knew you knew. These things aren’t exactly news. I hope, though, that RequireJS has made your work much easier, and much more organized.

The source code of this project can be downloaded from its GitHub repository. Feel free to use it as you see fit, without any restrictions. You can also use it as a template for your future projects. I use this template in production at Monwara LLC, so it should work.

The New Resistance: Aftermath of the Serbian Revolution

Sat, 21 Jan 2012 16:03:00 +0100

On 5 October 2000, almost twelve years ago, Serbia has won its freedom. Its people have ousted Slobodan Milošević, and ended an almost ten-year-long rule of the Serbian Socialist Party (SPS). And then what? For the next twelve years, have the Serbs seen any improvement?

The key to revolution’s success was a youth organization called Otpor. Today, Otpor’s founders are traveling across the globe to teach the tactics they used to other nations. A typical session might include wisdom like this:

“Do a small thing and if it is successful, you have the confidence to do another one and another one,” Popovic said. “You recruit people, train them, and keep them constantly active. You hit, proclaim victory — and get the hell out. If it is successful, people will come to you. Participating in small successes, you build self-confidence. Nonviolent struggle changes the way people think of themselves.” — “Revolution U”, Foreign Policy, 16 February 2011

Have people really changed the way they think of themselves, though? Twelve years after the 5 October revolution in Serbia, I’ve talked to many young people that participated directly or indirectly in the events, and observed the people around me. There is definitely a permanent change that resulted from the past events. But I also feel those changes are not exactly what Popović and others thought or hoped they’d be.

Economic situation in Serbia is not at all good. This country is still transitioning towards free market capitalism, and western-style parliamentary democracy still hasn’t rooted itself in the Serbian collective consciousness. Some of the today’s students are again involved in protests, still looking for a definitive change towards a better society. There are voices that claim 1990s were vastly better.

During the 1990s, Serbia was under economic blockade imposed by the western countries. Trade with most other countries was virtually non-existent. Domestic production (mainly foods) and smuggling formed the basis of day-to-day survival. Inflation was unimaginably high, and paychecks usually invalidated in under 12 hours. Those crisis, however, had a sense of reality to it that people knew what to do with. They were desperate, but not apathetic. They fought and prevailed in the end.

During the 78-day NATO military campaign over Serbia in 1998, I felt most strongly the high spirit of survivalism in Serbia. Although, as a foreigner, I have had problems myself, I still adored this energy, and things used to be fun during those times. Most people did not fear. The Early 1990’s economic crisis seemed like the worst they survived, and that nothing could ever break them. Unfortunately, this was the last climax of the Serbian spirit before it was permanently smashed after the 5 October revolution in 2000.

Today’s crisis is different. People are drowning in debt, struggling at the verge of existence, constantly pressed to make ends meet. Whether one will earn enough to pay the electricity bills that are several month overdue has become a common concern. Even worse, poverty brought about a tremendous storm of corruption, and young mothers in some Serbian cities have a realistic concern of having to come up with bribes in excess of their monthly income simply to prevent their child from being murdered by corrupted doctors. (Of course, you won’t see these in news, but I actually know people who are concerned about this.) The everyday stress is slowly but surely breaking people into fatalist, apathetic wrecks.

Unlike in 1990s, though, the lack of immediate and obvious danger suspends them between an illusion that they might make it, and the actual poverty that surrounds them. They are locked in passivity and insensitivity, hoping subconsciously for a hero that would salvage them from the bad situation they are in, instead of believing in their own power as Otpor’s former leaders would like to think (or have us think at least).

The danger of the new situation is that the minimum level of survival is still guaranteed (or at least most people think so). This provides a tiny safe spot in the sea of poverty, that people can cling on to. People usually fight if they believe there is no place to go, but as long as they have something, they will stick to it, and not try to move for the fear of losing this foothold.

Another phenomenon I’ve observed talking to people is that there are people who actually believe it’s much better now. The case is usually that things are somewhat better than having to scrape for life, but only for them alone. For example, graphic designers are now sought for, unlike before the revolution, and most graphic designers can get decent jobs as a result. (“Decent”, is of course, subject to debate.) Talking to these people, you quickly realize they have very little understanding of the current situation, and that they, essentially, live in a surreal world that heavily draws on the ideas of the early 2000s. These people are, I feel, as spiritually wrecked as the poor, and they invent this illusion of a better Serbia in order to cover it up. For them, if something is wrong, then it is the direct result of the evils Slobodan Milošević did over a decade ago, and everything else is “fine and dandy” until proven otherwise. This distortion creates a much bigger danger to the society at large, as such undisillusionable people are more likely to have a reactionary attitude, in case people decide to oppose the new powers in Serbia.

While the former Otpor leaders, now working as CANVAS, are selling their ideology to countries around the globe that are only too eager to jump on the revolutionary bandwagon, people have to be aware of the consequences of such revolutions. The art of war is not the same as the art of post-war. What remains after a non-violent revolution is not much better than what remains after a war: a sea of spiritually wrecked people that are even weaker and even more susceptible to manipulation than before. Or perhaps, financiers of CANVAS actually look forward to sailing on those seas?

Apple laptop parody featuring Plum D series, an imaginary...

Fri, 07 Oct 2011 01:18:49 +0200

Apple laptop parody featuring Plum D series, an imaginary Serbian-made laptop with pretty cool specs.

After watching the Trigun series and the movie edition, I...

Tue, 04 Oct 2011 11:18:51 +0200

After watching the Trigun series and the movie edition, I couldn’t resist recreating the sunglasses worn by the protagonist, Vash the Stampede. Used Blender as usual. Before you ask, yes, the internal renderer is still missing the caustics.

Node Web Development

Tue, 13 Sep 2011 19:23:00 +0200

Quit scratching your head already. Of course you’re doing it, scratching your head and mumbling to yourself, “What’s a browser language doing on the server?” In truth, JavaScript has a long and largely unknown history outside the browser. JavaScript is a programming language, just like any other language, and the better question to ask is “Why should JavaScript remain trapped inside browsers?”

With these words, begins your jorney though web development using server-side JavaScript with Node.js. Right from the start, David Herron’s Node Web Development (2011, Packt) gives hints as to who this book would be most useful to: experienced software engineers, getting ready to explore the Node.js platform for the very first time. Frontend engineers that are looking to get into backend development, would probably find this book very useful.

If you are the tartget audience, though, you will find this book to be one of the best resources on Node.js available.

Before I begin my review, I would like to note that the review was requested by Packt. Although it has been requested, you can consider this to be my completely open and honest review of the book. In other words, it really is a good book.

Node.js isn’t just any new thing that comes to Web. You could say that Node.js is as important to server-side as something like client-side Python (substitute ‘Python’ with your favorite language) would be to browsers. It also proves (hopefully once and for all) that JavaScript is a powerful language and does away with the myth that it’s only good for light scripting in browsers. Finally, it creates an opportunity for talented front-end engineers to also try their JavaScript skills on the server-side.

Node Web Development is therefore a book that reveals the true face of JavaScript, in addition to being a good introduction to Node.js platform, and thus makes an important contribution to modern web development as a whole.

With Node Web Development, it’s like walking into a restaurant for a burger, and getting salad, drinks, and free WiFi with it, for free. The order of topics in the book is a bit non-standard, but very much in line with how work is done in reality. That’s probably one of the things experienced users will find very useful. If you’re already a web developer, you will appreciate having things that interest you the most covered early on. As for newbies, it will lay down a good foundation, which is also a good thing. All in all, you will learn, I believe, a proper way to do things, from someone that’s been doing it for a very long time. ;)

If you are an experienced JavaScript developer, you will appreciate the fact that an exotic thing like JavaScript modules is covered first thing. Modules are probably one of the most important things frontend developers should learn coming to Node.

NPM, Node Package Manager, is covered in great detail early on. That is excellent, considering it has become a de facto standard in the Node developer toolbox.

The book also covers some of the libraries that are typically used for web development on Node. These include Connect, Express, and Mongoose (MongoDB ORM).

All in all, you will learn 80% of Node-related ‘stuff’ that needs knowing if you want to do web development on Node.

One thing I found missing was unit testing. Node.js has a few very good unit testing libraries that are definitely worth covering. I guess covering the full choice would be outside the scope of most books (except maybe a book dedicated to unit testing in Node), but I think at least expresso should have been mentioned.

As someone who has been using Node for almost two months now, I definitely found the book useful. If you are just starting, I think this book is a must.

Herman's Thought Soup

Mon, 12 Sep 2011 17:26:36 +0200

Herman's Thought Soup:

People who follow me on Twitter know about it, but I thought I’d leave a short note here as well. So anyway, there’s this thing, a blog about anything and everything Herd-Hound-related. There will be quite a few posts about the technical stuff that goes on under the hood, but also the business stuff. Stay tuned. ;)

We have a new family member, the lovely Labrador Retriever we...

Wed, 07 Sep 2011 19:16:19 +0200

We have a new family member, the lovely Labrador Retriever we call Hatchin. :)

Packt launch sixth annual Open Source Awards

Wed, 07 Sep 2011 17:29:00 +0200

The 2011 Open Source Awards was launched today by Packt, inviting people to visit www.PacktPub.com and submit nominations for their favorite Open Source project. Now in its sixth year, the Awards continue in its aim of encouraging, supporting, recognizing and rewarding all Open Source projects.

The 2010 Open Source Award Winners included the Open Source Content Management System (CMS) Award winner CMS Made Simple, Open Source JavaScript Libraries Award winner jQuery and Pimcore the winner of the Most Promising Open Source Project Award.

The 2011 Awards will feature a prize fund of $24,000 with several new categories introduced and the vote of the public becoming more influential. This year all CMS projects will compete in an even tighter contest in the Open Source CMS Award category with the now defunct Hall of Fame CMS finalists re-entered into the CMS category. Projects such as Drupal and Joomla! will face off with CMS Made Simple and MODx for the first time since 2008.

While the Most Promising Open Source Project and the Open Source JavaScript Libraries categories will be back for a second year, Packt is introducing new categories for Open Source Business Applications, Open Source Multimedia Software and Open Source Mobile Toolkit and Libraries. These new categories will ensure that the Open Source Awards remain committed to providing the platform to recognise excellence within the community while supporting Open Source projects both new and old.

“We’ve managed to continue to provide new levels of accessibility for Open Source projects, while encouraging a more competitive nature in the contest by increasing the public votes influence. Additionally, we thought it would be a great idea to reward more projects thus we’ve introduced sub-category awards across a number of the categories during the voting stage. We expect the Awards this year to be bigger and better.” said Julian Copes, organizer of this year’s Awards. Packt has opened up nominations for people to submit their favorite Open Source projects for each category at www.PacktPub.com/open-source-awards-home . The top five in each category will go through to the final, which begins mid-September. For more information on the categories, read Packt’s recent announcement: www.packtpub.com/blog/2011-open-source-awards-announcement

About Packt

Packt is a modern, unique publishing company with a focus on producing cutting-edge books for communities of developers, administrators, and newbies alike.

Packt’s books and publications share the experiences of fellow IT professionals in adapting and customizing today’s systems, applications, and frameworks. Their solutions-based books give readers the knowledge and power to customize the software and technologies they’re using to get the job done.

For more information, please visit www.PacktPub.com

Show Me Your Email!

Wed, 07 Sep 2011 10:41:59 +0200

If you sign up for a social networking account, you can understand that the terms of service contain stuff like “you give us permission to publicly display your information for the purpose of providing you the service”. How would you feel if your email provider had the same in their TOS?

I was thinking about email today. I use email for everything and anything. My social networking goes on in my inbox. Work goes without saying. Personal stuff, too. I don’t use Facebook, nor Google+, and I only use Twitter to publish stuff from my blogs.

Now, I’ve always considered email the most private service on Internet. If there was one service on which my privacy would be absolutely protected it would be email, so I thought. But I guess it doesn’t hold true anymore. Here’s an excerpt from the TOS from an email provider I’m currently using (replaced company name with ‘ACME’):

“We respect your right to ownership of content created or stored by you. You own the content created or stored by you. Unless specifically permitted by you, your use of the Services does not grant ACME the license to use, reproduce, adapt, modify, publish or distribute the content created by you or stored in your user account for ACME’s commercial, marketing or any similar purpose. But you grant ACME permission to access, copy, distribute, store, transmit, reformat, publicly display and publicly perform the content of your user account solely as required for the purpose of providing the Services to you.”

So, the ACME service respect my ownership of the content, but they do get to ‘publicly display’… my email… I’m sure they didn’t mean it to come out that way, but it did, didn’t it? I don’t know if it makes sense at all. If I post on Facebook, I can understand that it is meant to be publicly displayed to a fairly large chunk of the public. And I understand that I need to give Facebook permission to do so. But what about email? It is decidedly not meant to be public. In fact, I use it specifically so that it doesn’t go public. Otherwise, I’d just tell everyone to watch my blog and post personal messages here, right? So I definitely don’t understand why “publicly display” has to appear in email (and office) SaaS provider’s TOS.

IANAL, but I checked the Y! Mail’s TOS and no such clause appears there, so it’s generally speaking (from IANAL perspective) ‘safer’ to use for email than the service I’m using right now. You only give Yahoo! permission to publicly display stuff that you upload to ‘publicly accessible areas’ which makes a lot of sense. Google has something similar in their TOS, but they aren’t as clear as Yahoo!. The FAQ says this: “We do require that you give us a licence to the content you post so that we can host it and, if you ask us to, make it available to others.” So, with my permission, they can post it publicly, but not without it.

As far as services accessing our email’s content, I believe Google does that to facilitate context-sensitive ads in Gmail. There’s also a service called GMX Mail that outright reads its users’ email (supposedly to facilitate their security). Such practice is not only undesirable, but should be alarming to most webmail users (I stopped using Gmail a while ago, and was kicked out of GMX because they believed that one of my messages, which they obviously read, violated their TOS.)

It would be very difficult to check each and every TOS out there to find a reliable email provider that would really care about the privacy of my email. However, it would be good to keep in mind that email isn’t as sacred as it used to be (if it ever were). Apart for self-hosting email, there is no telling what might happen to your messages. You might see them leaked publicly one day, and find yourself in a situation where you cannot even sue the company that leaked them because you gave them the permission to begin with…

New Node.js Book from Packt

Fri, 26 Aug 2011 16:13:00 +0200

Node.js is the new cool thing on the web, and everybody seems to be talking about it these days. And when everyone starts talking about something you start seeing books on the subject. Some time ago, I’ve caught David’s anouncement of the book he was writing, and was wondering about when it will be available, when Packt contacted me asking for a review of David’s Node web development, which has just been released (August 2011).

Now, you could say I’m a bit biased when it comes to Packt. They’ve published a great book on Python 3 by my friend and mentor, Dusty Phillips, and I also consider them a friend of open-source, which automatically promotes them to the friend-of-mine status. David himself is also a veteran developer (currently at Yahoo!), so I must say I have really high hopes for David’s book. We’ll see if it lives up to the standards that my previous contact with Packt books (and books from other publishers like Apress and O’Reilly) have set for me.

For now, I can only tell you that it’s a book for people starting with Node.js, and that it’s 172 pages. Cover’s also nice, as usual. (That plant represents a single-threaded JavaScript VM?) Expect a more thorough review in a week or two, but for now, here’s the free chapter to give you some idea of what the book is like.

Bullet-proofing Setters and Getters

Wed, 17 Aug 2011 21:27:00 +0200

JavaScript has two very handy methods that you can use on an object: __defineSetter__(), and __defineGetter__(). They allow you to write accessor functions for your objects. Here, I’ll show you how to write them so they are as tamper-free as possible.

I needed a bullet-proof pattern for accessors for my Daimyo library that I use to hook Herd Hound to Samurai payment gateway.

Now, before I begin, it should be noted that method that I’ll be showing you here actually covers only simple types (String, Number, RegExp, and Date). Composite types like Object or Array require a different approach, which I will only briefly mention, with a link to useful code that can get you there.

So, let’s get started. First the normal way to do accessors:

function Constructor() {
    var self = this;

    self.__defineSetter__('prop', function(val) {
        self._prop = val;
    });

    self.__defineGetter__('prop', function() {
        return self._prop;
    });
}

These are simply setting and returning a different property that will hold the real value. Obviously, you’d do more complex stuff to the incoming or outgoing value, which is the point of accessors.

Let me demonstrate one teeny-weeny problem with this:

var myObj = new Constructor();

myObj.prop = 'foo';
console.log(myObj.prop); // => 'foo'
console.log(myObj._prop); // => 'foo'
myObj.prop = 'bar';
myObj._prop = 'foo';
console.log(myObj.prop); // => 'foo'

So obviously, our _prop property is exposed, and anyone can play with it. It’s all cool as long as you are dealing with a well-behaved developer who will respect the API and never touch stuff that starts with an underscore (except, obviously, the __defineGetter__(), and __defineSetter__()). But what if, in a very unlikely but still remotely possible scenario, malicious attacker (or just some kid armed with too much free time and super-human curiosity) manages to compromise the app that uses this API? In that case, it is quite possible that the app is tricked into setting the _prop instead of prop, and circumventing any carefully written validation logic in our setter function.

Naturally, our instinct tells us (or is it paranoia?) that we should tuck away the _prop property somewhere where nobody can play with it (not even the kids). A good solution for this is to use closures.

function Constructor() {
    var self = this;

    (function(self) {
        var propVal;

        self.__defineSetter__('prop', function(val) {
            propVal = val;
        });

        self.__defineGetter__('prop', function() {
            return propVal;
        });

    }(self));
}

Before you go ‘WTF’ on me, I’ll explain what happened. So, first thing first:

(function(args) {

}(args));

This pattern is called immediate function invocation. What we do is, we define a normal function, and then call it immediately. Note that the function we’ve defined has a closing curly bracket, followed immediately by a set of round brackets with arguments in them (last line of the last snippet). This tells it to immediately call itself with the specified arguments, which are passed to it (surprised?). The whole thing is then wrapped in round brackets to signify that it’s an expression that has to be evaluated on the spot.

One cool thing about this is that this function is sealed from the rest of the world. What’s uncool about it is that it’s sealed from the rest of the world. You can take that any way you want. What it means is that it cannot access variables defined outside (like self in our example), so you have to pass them to it.

Closure pattern is this one:

(function(args) {

    var someVar;

    return function() {
       // We can access someVar from here, but nobody else can!
    };

}(args));

The outer function (the immediate function expression) closes over the one defined inside it. Note that, as long as we have one function closing over the other, we don’t actually have to use the immediate function expression, but in this case it came quite handy, because we needed a one-off function that won’t be used again.

Now, the cool thing about closures is that they are so awesome. Seriously. Anything defined in the outer function is accessible to the inner function even after the outer function is done. The inner function can then be called at some later stage in execution, while still maintaining the knowledge of the stuff that was defined outside it.

So, in our accessor example, we define the variable that will store the property’s value in the closing function, and the inner functions (accessor functions) will use this variable to set and return the value of the property. The variable is visible only to accessors, and nobody can tamper with them (not even the kids).

There’s one exception to the above tamper-free claim. If the values are objects, then they can still be tampered with, because, as we’ve seen before, objects are passed by reference, not by value. This allows anyone to manipulate the properties of the value stored in the closure after it’s returned by the getter function. To avoid this, you need to return a cloned object. Code to do that is available in a Gist by Brian White, so I won’t go into details of how it works. As an aside, if it’s a simple object whose properties only contain non-object values, you can do this in your getter:

var clone = {};
Object.keys(valueInTheClosure).forEach(function(key) {
    clone[key] = valueInTheClosure[key];
});
return clone;

That’s called shallow copying.

Classic Web Form POST in AJAX apps

Sat, 13 Aug 2011 23:21:00 +0200

So, you have an AJAX app that has to do a normal POST (not an AJAX post) using a regular web forms for whatever reason? While the solution outline here is not complete (it does not cover doing cross-domain requests), here’s a simple pattern I used to make Ashigaru jQuery plugin work with the Samurai payment gateway.

The problem I was facing was that Herd Hound is a single-page web application. You basically load a single page, and any change on that page is done using JavaScript. All requests to the backend are made via XHR (commonly known as AJAX), and the frontend can only talk to the backend (i.e., it has to talk to the same place from which the JavaScript was loaded). This restriction is known as same origin policy, and it’s been giving pain to anyone wanting to do a mashup.

The same origin policy lead to invention of various techniques to get data from other sites to the AJAX app, including some fancy stuff I’ve only heard about in the newer versions of major browsers. One classic technique is JSONP. This technique has a major drawback in that it cannot be used for sensitive data. Because JSONP requests must be GET requests, any parameters that you want to send must be sent as URL parameters which are not encrypted even over SSL. Another solution was to POST a web form to a hidden iframe using the target attribute.

The target attribute used to be deprecated in HTML 4.01, but in HTML 5, it is no longer deprecated, so you can safely use it knowing that it will be supported for quite some time. What this attribute does is it tells the browser to open the results of the form action in a different target, and not in the current page. The target can be an iframe, so if you have a hidden iframe on your page, the page will not be reloaded, and the user will never know what was loaded in the iframe.

This behavior is what I expoloited for writing Ashigaru.

Basically, you have a form, or you create one using JavaScript, and you add a target attribute that has the same value as the name attribute on the iframe. You also make sure the iframe is hidden. Ashigaru creates a form which has all input elements of type hidden so the form itself is also hiddent. It then takes an object with data that you want to POST, and it populates the form. You can either do that, for a generic solution, or you can use the existing form, and let it submit to the iframe.

Once you submit the form to the iframe, you can use the load event on the iframe to read the data from it once it’s loaded.

Before you attempt to use this technique, here’s a few tips.

It doesn’t matter what the action URL is as long as what ends up in the iframe is coming from your server. Otherwise you cannot access it.
You can access data in the iframe, but iframe can also access data on the page, so beware of script injection attacks! Make sure there are no embedded malicious scripts.
You cannot prevent scripts in the iframe from executing.
If you want to do cross-site requests, you can make a CNAME subdomain that points to that site, and then manipulate window.document.domain on both the current page and the iframe, and strip out the subdomain parts so the browser thinks it’s the same domain.

Here’s the main function of the Ashigaru:

$.ashigaru = function(data, merchantKey, redirectURI, callback) {
  var form;

  var fullData = {
     // ... defaults
  };

  $.extend(fullData, data);

  var formHTML = '...';

  // Replace non-data field placeholders and form attribute placeholders
  formHTML = formHTML.replace('$requestURI', samuraiURI);
  // ....
  // Replace placeholders in form HTML with real data
  for (var key in fullData) {
    if (fullData.hasOwnProperty(key)) {
      formHTML = formHTML.replace('$' + key, fullData[key]);
    }
  }

  // Inject this into DOM
  form = $(formHTML);

  // Attach the form to body
  form.appendTo('body');

  // Submit the form
  form.unbind('submit');
  form.submit(function(e) {
    e.stopPropagation();
    return createIframe(callback);
  });

  form.submit();
};

I’ve stripped out a fair bit of error-handling and other little pieces that are not relevant to this post. A few things to note is e.stopPropagation(), adding iframe creation routine into form submit action (so the iframe is created on the fly when the form is submitted), and form.submit() call, which forces form submission (form is not submitted manually by the user).

To be honest, I saw e.stopPropagation() used somewhere in a similar context, and I left it there because it works. I’m slightly ashamed to say that I haven’t tested the code without it. In theory, it will prevent the submit event from being handled by any handlers that are trying to do event delegation. Only the submit action on this particular form will be handled. So I guess that’s why we have it there. This should be a completely isolated form, created for the purpose of making this method of POSTing work, so we don’t want anyone else on the page reacting to the event.

Another thing I did was first make sure submit event is not handled by anything other than my code. For the same reason as e.stopPropagation(), obviously.

Here’s the createIframe method:

function createIframe(callback) {
  var iFrame = $('<iframe src="" name="samurai-iframe" '+
                 'style="display:none">' +
                 '</iframe>');
  // Attach iframe to <body>
  iFrame.appendTo('body');
  // Do the right thing when iframe loads
  iFrame.load(function() {
    onResultLoad(this, callback);
    removeForm();
  });
  return true;
}

Fairly simple. It creates the HTML for the iframe, attaches it to body, and adds the load event handler whic extracts the data from the iframe and deletes the form. I’m not going to show you the code that removes the form. It’s not relevant. You can check out the Ashigaru source if you want to know more.

There’s one thing to note here, and that’s the fact that we return true as the last thing in the createIframe method. Remember that we said return createIframe(callback) in the form’s submit handler? Well, this is the thing. If you return false in a handler, the event will be prevented from happening, and the form will not submit. I haven’t written the code yet, but I will do basic sanity check: see if the iframe was created successfully, and return false if it wasn’t (so that the form is not submitted).

I’ll just show you one last snippet, and that’s the onResultLoad handler:

function onResultLoad(iFrame, callback) {
  var resultDocument = $(iFrame).contents();
  var jsonData;

  // Remove the form
  removeForm();

  // The result document should contain JSON data in a <script> tag
  jsonData = $.trim(resultDocument.find('body script').text());
}

To get the DOM tree of iframe contents, you can use the contents() method. It will return a jQuery object that you can use to access the DOM inside the iframe. Note that this only works if the iframe satisifes the requirements of the same-origin policy. The reason this method works for Ashigaru is that Samurai redirects the user back to a page on my server, so the final result of making the POST request always satisfies the same-origin policy.

One you have access to the iframe’s DOM, it’s a matter of doing your usual $(..) to get whatever you want from it.

Specifically in Ashigaru, I have a single <script> tag that contains JSON data enclosed in round brackets:

<script>({"status": "ok"})</script>

The round brackets are there to prevent the browser from thowing SyntaxError exceptions because the object literal is required to be inside brackets.

The Best Git UI for Vim: Fugitive

Sat, 13 Aug 2011 22:31:48 +0200

Tim Pope, known to many Vim (and Rails) users as just tpope, very well known for his many vim- projects (including the infamous pathogen plugin), has this to say about his Fugitive plugin for Vim:

I’m not going to lie to you; fugitive.vim may very well be the best Git wrapper of all time. Check out these features: […]

And he didn’t lie. Fugitive is so awesome it may well be the reason for you to switch to Vim right fucking now. But of course, you are already using Vim, aren’t you? Oh and if you’re not using Git, you absolutely suck. :P

So, I’m not going to go all in-dept. Instead, I’ll point to you vimcasts.org, where Drew is showing you how to use it in his typically thorough, and awesome style.

What makes Fugitive really great is the fact that it actually does more for you than what you’d get using just Git.

One of the awesome features that I enjoy using is the vimdiff view of unstaged changes. (I started using vimdiff because of Fugitive, too.) What Fugitive does is it gives you a side-by-side of changes that are staged, and the working version of your file. Not only can you clearly see what’s changed since last git commit (or git add), you can also hand-pick changes you want to move into the staged version. Of course, just being able to preview the changes is quite awesome on its own.

Another cool thing is it makes the git-status output interacitve. So you can go to a file mentioned in git-status output inside a Vim buffer, and you can press - to stage or unstage it. And then you can just press S-C (Shift + C) to commit the staged changes.

Merge conflicsts have also been my pain point. It’s quite easy to loose track of those markers, isn’t it? Well, with Fugitive, you get a three-way vimdiff, with target on the left, and source on the right, and the conflict in the middle. You can pick and merge parts of the source or target just like when you want to stage small chunks of changes.

Ok, so those are my three favorites, but Fugitive actually packs a whole lot more, so I encourage you to give it a spin.

Passing by Value vs Passing by Reference in JavaScript

Fri, 12 Aug 2011 14:52:00 +0200

A thing that I’ve heard desktop application developers say about web developers is that web developers don’t know what phrases passing by value and passing by reference mean. I’m sure most of us, web developers, know what those mean. If you don’t, you should learn them ASAP!

To recap, passing by value means that the value that you pass to a function gets cloned and independent, and that changes to that value made within the function are not reflected on the original value. On the other hand, passing by reference means that the object passed to a function is linked to the one that is passed, so the original is affected by any changes that happen in the function.

I’ve done some testing to see when the values are passed by value, and when they are passed by reference in JavaScript, and here’s what I’ve found out.

tl;dr

Literal values are always passed by value. Object properties are always passed by reference (including array members, because arrays in JavaScripts are objects with numerical properties).

Now the long version.

Suppose we have a few values assigned to a bunch of variables.

var int = 1;
var str = 'a';
var obj = {
    int: 1,
    str: 'a'
};
var arr = [1,2,3,4];

Now, suppose that we have a function like this:

function passme(passed) {
    passed = 'foo';
}

What this does is change the value of the incoming argument passed to 'foo'. If we run int through this, interesting thing happens:

passme(int);
alert(int);

The value of int before and after calling passme is still 1, and it will not change to 'foo'. The exact same thing applies to all other variables, inclluding obj, and arr. They are all passed in by value.

So how do we get the new value assigned to the variable? We have to return it:

function passme(passed) {
    passme = 'foo'
    return passme;
}

And then, we can assign the return value:

int = passme(int); // int is now === 'foo'

What would you expect to happen if we did this:

var obj1 = {
    a: 'foo',
    b: 'bar'
};
var obj2 = {};

function cloneAndAddC(o) {
    o.c = 'baz';
    return o;
}

obj2 = cloneAndAddC(obj1);

Honestly, I expected obj2 to have proerty c, and obj1 to not have it. But what happens in reality is that both have property c. (You can see a demo of failed object cloning in JSFiddle).

To explain it:

[obj1][.property]
  |            \------------> by reference
  |
  \------> by value

So while the actual object is passed by value (if you assign something else to it, you won’t kill the original), but the properties of the object are passed in by reference (if you modify a property, original property is modified). I hope that makes sense.

So, what if an object’s property is an object? I bet you can’t guess unless you’ve done tried this before! So check this out:

var obj1 = {
   a: 'foo'
};
var obj2 = {
   a: obj1
};

function modme(o) {
    o.a.a = 'bar'; // supposedly this is obj1.a
    o.a = 'bar'; // o.a should be obj2.a, meaning it's obj1
}

modme(obj2);

So what most of us though this would do is: 1. change obj1.a to 'bar', and then change obj1 itself to 'bar'. In fact, something else happens which is also very logical (and I seriously envy anyone who got this without peeking or knowing in advance): obj1.a is 'bar'```, but instead ofobj1itself turning into‘bar’with the second statement,obj2.abecomes‘bar’, andobj1“ is left intact. (There is a demo for this on JSFiddle, too.)

So now, let’s go back to our cloneAndAddC example for a second. How do we make that one work? Before you can do that, you first need to add a bit of boilerplate code:

// For browsers that don't support ECMA5
if (!Object.keys) {
    Object.keys = function(obj) {
        var keys = [];
        for (key in obj) {
            if (obj.hasOwnProperty(key)) {
                keys.push(key);
            }
        }
        return keys;
    };
}

With the above snippets, all non-ECMA5-compliant browsers will get the Object.keys function which can be used like this:

var obj1 = {a: 1, b:2};
Object.keys(obj1); // => ['a', 'b']

Up next, another piece of boilerplate:

if (!Array.prototype.forEach) {
    Array.prototype.forEach = function(callback, thisObject) {
        for (var i = this.length, l = this.length; i; --i) {
            if (!thisObject) {
                callback(this[l - i]);
            } else {
                callback.apply(thisObject, [this[l-i]]);
            }
        }
    };
}

So, you now have access to forEach, which loops thrhough array members and executes a callback function on each element.

Finally, we can define our clone function:

function clone(obj) {
    var clone = {};
    Object.keys(obj).forEach(function(key) {
        clone[key] = obj[key];
    });
    return clone;
}

The above does shallow copying. You could do some type detection and do recursive cloning. In fact, that’s just a few more lines at most, but I’ll leave that to you.

Now, we can make our clone thingy work:

var obj1 = {
    a: 'foo',
    b: 'bar'
};
var obj2 = {};

function cloneAndAddC(o) {
    var cloned = clone(o);
    cloned.c = 'baz';
    return cloned;
}

obj2 = cloneAndAddC(obj1);

Now wasn’t that fun? You can see it in action on JSFiddle. Happy hacking!

UPDATE: You don’t really have to write the boilerplate code. There’s this cool little thingy called es5-shim (ECMAScript 5 shim) by Kris Kowal. Just plug it in, and you’re ready to rock. You can see the last example using this shim on JSFiddle.

Profilejs: V8 profiling for Express framework

Tue, 02 Aug 2011 02:01:04 +0200

I’ve started using Danny Coates’ node-inspector recently. It works very well as a graphical debugger (runs in a webkit browser), and it can also serve as a profiler in conjunction with v8-profiler.

I wanted a tool that would profile all requests for me, so I could browse and pick through the logs afterwards, but I couldn’t find one. So I decided to write a middleware for Express that would pipe all requests throug the v8-profiler. The result is Profilejs.

Currently at version 0.0.2, Profilejs is still immature and probably buggy, and most certainly underfeatured. But the good news is, it’s not meant to replace node-inspector, and it works very well with it (for now).

I won’t detail installation and usage here, as it’s documented on the Github page.

Installation is fairly simple:

npm install profilejs

It will pull in v8-profiler as dependency, though, so be aware of that.

Typical usage involves setting up the middleware, and starting the profiler:

var profilejs = require('profilejs');
app.use(profilejs.profiler);
profilejs.start();

You can start either in silent mode or without the silent mode. The difference is you get a lot less noise in standard output in silent mode. Default is to run in extremely loud mode.

Each request results in a single profile. However, profiles are all named by the request URL that resulted in a call to the profiled handler. If the same URL gets hit multiple times, it will add runs to the profile with the same name. So you’ll have 5 runs for the url ‘/foo’ if you make 5 requests to ‘/foo’. In node-inspector UI, you will see a single profile named ‘/foo’, and five subitems under it which show different data from each run.

Once your app is hooked up, you can sart it with the debug option:

node --debug app.js

Preferably, you will make a separate environment for your profiling needs, and run the app in that environment:

NODE_ENV=profiling node --debug app.js

Once your app is running, you can start the node-inspector:

node-inspector --profile

The --profile option enables profiling. Now, you can open your WebKit browser (I use Chromium), and direct it to 0.0.0.0:8080. Once there, you need to go over into the Profiles tab and enable them.

With profiles enabled, you can click the refresh icon in the bottom left corner, and your profiles will be loaded. There won’t be any profiles until you actually inteact with the application, though.

I hope this tool is useful to you. If you have any issues with this library, you can file a bug, or drop by the nodejs mailing list.

Iframe ad killer

Sat, 30 Jul 2011 03:41:00 +0200

Most ads I’ve seen are inside an iframe. That makes them very easy to target and murder with a bit of JavaScript. Here’s a small bookmarklet that will do just that:

iframe killer

Drag that bookmarklet (link) to your bookmarks toolbar, and click it whenever you see too many flash or image banners on the page.

Mind you, this also kills legitimate iframes, so it’s not 100% clean. It will also miss any ads that are not inside iframes.

This bookmarklet is provided to you on works-for-me basis, no guarantees, and no warranties. ;)

How it works is very simple. It fetches a list of all iframes in a document:

t = document.getElementsByTagName('iframe');

Then it loops through them (in reverse for speed), and clears the src attribute:

for(i = t.length;i;--i) {
  t[i-1].src='';
}

(Keep in mind that removing the iframes breaks the page layout, which is not what we want. Clearing the src attribute is enough.)

The code is then wrapped in a closure so that it doesn’t leak stuff (if it returns anything other than undefined, even by accident, the browser will try to go to the ‘URL’ defined by that something).

(function() {
  t = document.getElementsByTagName('iframe');
  for(i = t.length;i;--i) {
    t[i-1].src='';
  }
}());

Now, to make a bookmarklet of this, just strip all spaces, and prepend javascript: (see the bookmarklet link at the beginning of the article).

The Cost of User Input Sanitizing

Fri, 29 Jul 2011 12:59:56 +0200

You might have heard of ‘user input validation’. That’s all cool and dandy, but have you heard of ‘user input sanitizing’? It is a concept as important as validation. The former checks for validity of the user input, and the latter prepares the user input for further processing by the software.

Now, you might think that proper validation might mostly eliminate the need for sanitizing. But that’s the wrong mindset. In the age of UX, software is expected to fit human needs, and not other way around, so sanitizing input becomes even more important than just validating it. It’s not only done for security reasons, but also to make users’ lives easier, too.

Here I’ll illustrate on a real-life example, why this is so important, and why it costs a lot if you do not care.

I use Moneygram from time to time to receive money from friends and relatives.

As with all money-related services, Moneygram uses software to manage the transactions. And software is operated by clerks. Now, clerks aren’t exactly the brightest folks on the planet when it comes to working with software systems. They may or may not have read manuals, but they tend to make mistakes. And if the UX of the softwar is not top-notch, this can lead to unpleasant delays and mistakes that cost people time and money.

Recently, I went to pick up some cash. I was asked to fill in a form that was otherwise printed out by the Moneygram software, because the clerk could not figure out error in data input. 15 minutes after I’ve filled in the form, the clerk managed to get ahold of support staff and it turned out the error was software getting stuck on dashses entered into phone number field. It also has to be noted that the phone number is an optional field. I sometimes get asked for my phone number, and sometimes not. It should also be noted that clerk was not able to explain the problem to the support staff. The only reason they figured it out is that clerk spelled out the contents of each field to the support person.

It all boils down to user input sanitizing. In most programming languages it takes one line of code to remove any non-digit characters (including spaces) from the form. For example:

# Python
>>> phone_number = '123-456-7890' 
>>> phone_number = re.sub('[^\d]+', '', phone_number) # this line
'1234567890'

#JavaScript
> phoneNumber = '123-456-7890'
> phoneNumber = phoneNumber.replace(/[^\d]+/g, '') # this line
'1234567890'

Similar examples may include trimming input. I’ve seen it too many times where leading and/or trailing space triggering all sorts of problems for the end user. Make it a habit to strip leading/trailing spaces.

There are more common pitfalls, but they are outside this article’s scope.

It’s unbelievable that the Moneygram software fails to do proper sanitizing. The above one-liners take 5 seconds to write with decent typing skills in any language. And it saves customers anywhere from a minute (if clerk has seen it before but is confused again) to 15 minutes (if it happens for the first time and clerk needs to call support without any clue as to what the problem is) each time a clerk gets stuck on this. Moneygram is a world-wide service, so you can imagine how much time is lost on this single mistake word-wide every day.

Blocking vs Non-blocking: Node vs Bottle

Wed, 27 Jul 2011 10:50:00 +0200

Node.js boasts real-time performance. Of course, depending on hardware, it may show some level od delay in responses, but it is usually able to handle requests as soon as they come in. I have built two simple hello world apps, one running on top of Node.js, and one running on top of Bottle/Bjoern. Bjoern is a non-blocking WSGI server written in C, and it should perform very well on its own, but Python is blocking, so we want to see how it stacks up against a pureluy non-blocking app written on Node.js.

Concept

We will use two applications, one written on Node.js, one written on Bottle/Bjoern stack. Both applications will return a ‘Hello World’ after a 1 second delay.

If everything works fine, the apps should be able to serve an arbitrary number of parallel requests in about one second total.

Bottle/Bjoern

Up first, Bottle/Bjoern. Bottle is version 0.9.6 (stable), and Bjoern is version 1.2.0 (stable). The stack runs on top of Python 2.7.2. The code looks like this:

from bottle import run, route
from time import sleep

@route('/')
def hello():
    sleep(1)
    return 'Hello world'

if __name__ == '__main__':
    run(server='bjoern')

Node.js

I won’t introduce you to Node itself this time. You can read one of my previous posts for that. Node is currently version 0.4.10. The test application looks like this:

var http = require('http');
http.createServer(function (req, res) {
    setInterval(function() {
      res.writeHead(200, {'Content-Type': 'text/plain'});
      res.end('Hello World\n');
    }, 1000);
}).listen(3000, "127.0.0.1");
console.log('Server running at http://127.0.0.1:3000/');

Before jumping to conclusion about syntax, and other aesthetical issues, you should be aware that Bottle is a web framework, whereas Node is a networked application framework. They have different priorities and this reflects itself in the API style used for our example code.

Testing tool

For this test, we are using ab utility. If you don’t have one on your system, it’s part of apache package. On some distros, it may be part of some apache-related package, I’m not sure. On Arch Linux, you should just install apache package, and you’ll have the tool.

Let’s get the show on the road

First, let’s start Bottle/Bjoern combo. I’ve put the test app inside a virtualenv.

So, we start the server and app with:

(bottlehello) $ python helloworld.py
Bottle server starting up (using BjoernServer())...
Listening on http://127.0.0.1:8080/
Use Ctrl-C to quit.

We will test with 10, 1000, and 1000 concurrent requests launched all at once. If X is the number of requests, the command is:

ab -c X -n X http://127.0.0.1:8080/

At X=10, Bottle/Bjoern show appaling performance (average ms / request):

10019.282 [ms] (mean)

At X=100, Bottle/Bjoern fails, so we didn’t even attempt X=1000.

Let’s move on to Node.js.

$ node helloworld.js
Server running at http://127.0.0.1:3000/

Using the same ab command as before but with port 3000, we get the following results for X=10, X=100, X=1000 respectively:

1011.443 [ms] (mean)
1029.896 [ms] (mean)
2148.086 [ms] (mean)

As you can see, at 1000 requests, it took 2 seconds for Node.js to handle the requests on average, but it did not fail. (It also ran at 1.5s/req on some earlier trials.)

Bottle/Cherrypy

To do Bottle some justice, we’ll try the test again using Cherrypy server. We will modify the run() line like so:

run(server='cherrypy')

And install Cherripy, obviously. Unlike single-threaded bjoern, Cherrypy is multithreaded, so it should be able to take advantage of all 8 cores on my developer machine.

And the test results for X=10, X=100, and X=1000 look like:

1325.132 [ms] (mean)

…well, it fails at 51 requests.

Bottle/Tornado

Just to be on the safe side, I’ve also tested with Tornado, and got similar results as in Bjoern’s case: 10s request time with 10 concurrent requests, and failed attempts to test X=100 and X=1000.

Conclusion

A single blocking call like time.sleep can make all the difference between scalability and failure. Now, you might argue that it is possible to make the above work with Bjoern/Python, and you are, indeed correct. If you replace the blocking time.sleep call with something non-blocking. This demo was meant to demonstrate the power of non-blocking, though, and I think the demo is successful at that.

Serial Expresso Testing with Mongo Fixtures

Tue, 26 Jul 2011 20:07:54 +0200

Expresso is a test runner for Node.js. It boasts “high speed parallel testing”. It means it can run your whole test suite of a couple of thousands of tests in less than a second… but this post is not about that. This post is about taking it slow, and running tests one by one like you might be used to from other environments (Python, anyone?).

Isn’t slow… err… bad?

So, why use a high-speed parallelized test runner to run tests slowly and serially? There are some cases where this might be useful.

In particular, serial testing is very useful when running tests with database operations where you have no transaction support. In such a case, database operation in one test (for example, removal of a record) migth conflict with operation in another test (for example, fetching of a record that should have been removed). Database operations take time, and on a non-blocking platform such as Node.js, this leads all sorts of problems if you cannot allow operations to end before next ones are executed.

Regardless of how much you value speed, sometimes hasty means wrong test results. For the past 4 or 5 days, I’ve battled the non-blocking nature of Node.js to get a good work flow for testing database-related operations in an app that I’m writing, and here I’ll recap my final solution.

The basic setup

Here’s a basic setup for the test module.

var mongoose = require('mongoose'),
    db = mongoose.connect('mongodb://localhost/mymodeltest'),
    fixture = require('../helpers/fixture'),
    MySchema = require('../../models/mymode).MySchema,
    Model,
    test = exports;

Let’s assume that tests are all located in appdir/tests/unit. This test will be located in appdir/tests/unit/mymodel.tests.js. I’ll also use Mongoose ORM for this test. If you don’t use Mongoose, you will find the fixture module uninteresting, but it still might give you an idea about how to make your own setup.

For the moment, ignore the var statements above. You can look at it later as we progress to see what comes from where. Just note that we have aliased exports as test to make the module look a bit nicer, but that’s completely optional.

Also note that we are using a seaparate database for this unit test. I generally opt to use separate database for each unit, and only use a project-wide database for integration testing. This allows me to run multiple unit test modules at once in parallel (even though individual tests in each module are run serially), which speeds things up a little.

First thing first:

Model = db.model('Model', MySchema);

This sets up the model to use the connection. Next, you need to add a few test fixtures. I usually specify fixtures as an array of objects. The objects are formulated in a way that will make a whole document if passed to the model constructor. In our example, if I pass such object as new Model(testData[0]), it should be ready to save without any further intervention. So we specify the fixtures like so:

testData = [
  { ... },
  { ... },
  { ... }
];

This is the minimal setup that will get us started.

`fixture` module

You can get the code for the fixture module from a gist. (I give you permission to use, modify, copy, and redistribute, yada yada… IOW, I won’t sue.) It’s a single method which cleans up the database, loads the fixtures specified as an array (as discussed in previous section), and allows you to specify a callback that will be executed once all fixtures are loaded.

Your callback may accept data, and another callback (I usually call it next like in middlewares). Data is a collection of document objects that were inserted into the database. The next callback cleans up the database once your tests are run, if you choose to call it when you’re done testing. The next callback will also take a callback that… is it too confusing? Ok, let’s just put down basics and move on.

fixture.load(Model, testData, function(data, next) {
  // Your tests go here
});

Oh, and the load path for this was ../helpers/fixture, because I keep the test helpers in appdir/tests/helpers directory. You should adjust the path according to your situation.

Using `exports['test name'] = function() {...}` format

As you know, Expresso allows you to specify tests in one of two ways. One way is to specify all your tests in an object literal:

module.exports = {
  'test1': function() {...},
  'test2': function() {...},
  'test3': function() {...}
};

But… you cannot use this in this scenario. You have to use the other format:

exports['test1'] = function() {...};
exports['test2'] = function() {...};
exports['test3'] = function() {...};

As you rememeber we have aliased exports as test, so you can do this:

test['test1'] = function() {...};
test['test2'] = function() {...};
test['test3'] = function() {...};

If you use the latter method, you can specify tests withing a callback of an asynchronous operation (such as inserting fixtures, right?), and that’s exactly what we’re aiming for.

fixture.load(Model, testData, function(data, next) {
  test['test1'] = function() {...};
  test['test2'] = function() {...};
  test['test3'] = function() {...};
});

Using the before-exit hook

All test functions accept a before-exit callback that you can use to tell Expresso when your test is done. Do use them do this:

test['my asynchronous test'] = function(exit) {
  Model.find(function(err, docs) {
    // test data
    exit(); // Calls exit to signify that test is done.
  });
};

Tearing down

Finally, after all your tests are run, it’s time to destroy the test data and disconnect from the database. Disconnecting from the database allows Expresso to terminate correctly instead of you having to terminate it manually with Ctrl+C.

Tear-down is done by adding one more test to the end of your test module that will call the next callback from fixture.load call, and disconnect the database. I usually call this last test simply end.

fixture.load(Model, testData, function(data, next) {
  test['test1'] = function() {...};
  test['test2'] = function() {...};
  test['test3'] = function() {...};
  // ... more tests, and finally:

  test.end = function(exit) {
    next(function() { // ``next`` is from the ``fixture.load`` call
      db.disconnect(exit); // We let ``db.disconnect`` call exit
    });
  };

});

As noted in the comments next to the code, db.disconnect call takes a callback which is called when the connection to database is closed. We conveniently pass the before-exit callback to it. This is done within a call to next. The callback function we passed to next will be executed once all test fixtures are removed from the database.

Running the tests

To run the tests serially, you can use the -s switch:

expresso -s tests/unit/*.tests.js

Aaand… that’s it. Happy serial killing… err… I mean testing.

Mouse and Vim for the Lazy

Sat, 23 Jul 2011 23:17:22 +0200

Have you ever noticed that, when you have line numbers enabled in Vim, you can’t select just the code? You know, the numbers get selected as well. I was too lazy to fix that problem for a while now, and today I took a stab at it finally. Solution was deceivingly simple.

One method mentioned involved adding mouse button bindings to vimrc, and then adding more options to Xdefaults, but I wanted none of that. So I found another solution:

gvim -v

Yup. Gvim. Invoking Gvim with the -v switch makes it behave and look just like normal Vim, but with the awesome mouse support. Now you can scroll, select, copy, paste, and move separators with mouse, and still enjoy the text-version look and feel. How awesome is that! (^_^)

Idea Machine

Getting started with modular JavaScript

NodeJS (we’ll need it later)

Setting things up

Baby step

Let’s build our first AMD modules

Privacy

Loading text templates

Adding some style

Final steps

Conclusion

The New Resistance: Aftermath of the Serbian Revolution

Apple laptop parody featuring Plum D series, an imaginary...

After watching the Trigun series and the movie edition, I...

Node Web Development

Herman's Thought Soup

We have a new family member, the lovely Labrador Retriever we...

Packt launch sixth annual Open Source Awards

About Packt

Show Me Your Email!

New Node.js Book from Packt

Bullet-proofing Setters and Getters

Classic Web Form POST in AJAX apps

The Best Git UI for Vim: Fugitive

Passing by Value vs Passing by Reference in JavaScript

Profilejs: V8 profiling for Express framework

Iframe ad killer

The Cost of User Input Sanitizing

Blocking vs Non-blocking: Node vs Bottle

Concept

Bottle/Bjoern

Node.js

Testing tool

Let’s get the show on the road

Bottle/Cherrypy

Bottle/Tornado

Conclusion

Serial Expresso Testing with Mongo Fixtures

Isn’t slow… err… bad?

The basic setup

fixture module

Using exports['test name'] = function() {...} format

Using the before-exit hook

Tearing down

Running the tests

Mouse and Vim for the Lazy

`fixture` module

Using `exports['test name'] = function() {...}` format