There are times on your website when you want to allow users to upload content from their local drive to your server e.g. user profile pictures. In this post I will show you how to do this using node.js, Express and jquery.
A fully functional example is available on my github account Note: You will have to follow the instructions below after fetching the code to install Express and all of it’s dependencies.
Creating your webserver - Express
Express is a web framework build on top of node. It provides functionality such as creating a webserver to handle network requests to your app, view rendering, static file serving etc. In this post we will use it as our webserver to host our client html file and also handle the file uploads to the server.
To install express, we will use NPM, a package manager for node. From the command line (after installing npm), run the following command:
npm install express
After this completes you will have a node_modules folder and inside that the express node module folder. To create your new express app, run:
node_modules/express/bin/express [the_name_of_your_app] e.g. upload-example-app
Express will then create the application skeleton, an app file, folders for static content etc. The final step is to make sure you have all of the dependencies Express relies on. Change to your applications folder and run:
npm install -d
Now we should be ready to run the server, simply type (I am assuming you have node installed :) ): node app.js, you then have a webserver listening on port 3000, you can view it in your browser at http://localhost:3000
The webpage
In HTML, we can specify an input element with a type of “file”, this then causes the browser to show an upload button that the user can use to choose a file from their computer. The upload buttons visual appearance is dependent on the browser you are using. I’m not going to talk about how you can style this button, but it can be done: http://www.clipboard.com/clip/LQtdg6YHzcYKLyZbAkF5hASUu7kvuDya_ZLe
Our example HTML page will have the basic structure:
Go ahead and create this file under the public folder in your express app, e.g. public/index.html
Limitations
The title of this post has the word “asynchronous” in it, the default implementation of the file input element causes a page reload when the form in submitted, it is not possible to use an AJAX request i.e. XmlHttpRequest, to upload the file to the server. In order to be able to upload a file to the server without causing a page reload you need to perform some browser trickery using hidden iframes, textareas and a sprinkling of magic. Luckily for us there is a jquery plugin we can use that will hide all of this trickery from us, it’s called jquery.form. Download the JavaScript file to your /public/javascripts directory and include the jquery library in your page (good developers will make sure they serve a minified and zipped version of these files), we will also add a javascript file that will contain all of our file upload specific code (upload.js)
We also need a way to initiate the upload of the file to the server, we could add another button to our page, so the user would first choose the file, then click on the upload button, but that seems kind of lazy, so what I do here is to add a setInterval call, that is continually checking the value of the input element, once it has a value we can assume the user has selected a file and we kick off the upload.
jquery.form
Using the jquery.form plugin is pretty simple, we’ll add a submit handler to our form, using jquery, then on the submit event, we use the jquery.form plugin to submit the form asynchronously.
In order for this to work correctly, we need to make sure the form enctype field is set and the action attribute points to our API end point on the server, so our final HTML looks like:
The code in upload.js is shown below, I haven’t filled in the success handler, we will do that once we have implemented the server side part of the code:
Now we have the client side code, we need to implement the server code which will handle uploads to the /api/photos API. If we open app.js in your express app folder, we need to define a route for the API call and a handler. Express makes file uploads amazing easy, when you define the handler function, it has two parameters, req and res, which are the HTTP request and response objects. In our HTML we defined an input element named userPhoto, on the req object there is a files field that contains an object with information about all of the files that were uploaded. If we look at req.files we will see that Express has retrieved all of the uploaded data and saved it to a file in the /tmp directory. Using:
For this sample app, I’m just going to move the image from the /tmp directory to the /public/images folder so that we can access the file via an URL. The response we send back to the client will be the path to the image. Note: It is your responsibility to delete the files in the /tmp directory in the request handler, Express will not delete these files once they have been created, don’t forget otherwise your server will mysteriously run out of hard disk space eventually.
Our final server side handler code is shown below, it’s just moving the file and returning the path to the client:
Back on the client side we, we now add the code to our success handler to show the image we uploaded on the page. Note, even though the function is called success, we still need to check for the error parameter that might exist if there was a problem on the server:
Awesome, we’re all done, time to do a git push, high fives around the office and go home like a hero … but wait, there is that heavy feeling in your stomach, you should really test this in IE8, you don’t want to, but the good dev in you makes you fire up IE8 and test the code. Nearly everything works, but after you upload the data IE8 will open a save file dialog to save the response of the server, sigh (at least it was for me, your milage may vary.
To work around this annoyance, I had to change the response datatype to ‘text’ instead of JSON, you can do this by passing a ‘dataType’: ‘text’ field in the options object we pass to the ajaxSubmit method in the client code. Then on the server side, when calling res.send(), stringify the response e.g. res.send(JSON.stringify({ path: ‘foo/img.jpg’})); and finally before sending back the data using res.send, set the content type to be text/plain i.e. res.contentType(‘text/plain’)
Obviously now in the client code, instead of the response parameter being a JSON object it is a string, so we have to turn it back into a JSON object, using something like jqueries parseJSON method.
Further improvements
For production level code, make sure you:
Limit the size of the uploads you permit (if applicable). In production you are probably using something like nginx as a frontend webserver proxying to node, so you can set limits in your nginx.conf file
Never trust content users upload, make sure files that are uploaded are an expected type and don’t give the files more permissions than absolutely necessary
node.js is a great platform for building network based applications (I’ll jump into details why in another post). For the rest of this post I’ll assume you know what node is and have some experience using it.
If you have ever used node to make a number of simultaneous requests, what do you think the following code does:
If you are thinking that the code will kick off 100 simultaneous requests to your server, you would be wrong. I see code where people need to do a number of network operations and write code to actually batch and limit the number of requests they are making at once, but node.js networking is already doing this.
In the example above, 100 requests are created and added to an internal network queue, but the default node behaviour when using http.request is to have only 5 requests happening simultaneously to a given socket (a socket is the combination of an IP address and port number, so 127.0.0.1:8080 is a different socket to 127.0.0.1:9090), once a request completes, node then processes the next waiting request in the queue
To look at this in action, let’s create a server on our local machine (127.0.0.1) that is listening to port 1337, then we will run the above code. The server is keeping track of the number of concurrent requests made from the client, stored in concurrentServerRequests and the total number of requests made by the client in totalServerRequests
$node maxsockets.js
Server running at http://127.0.0.1:1337
Concurrent active server requests:1, total received:1
Concurrent active server requests:2, total received:2
Concurrent active server requests:3, total received:3
Concurrent active server requests:4, total received:4
Concurrent active server requests:5, total received:5
Concurrent active server requests:5, total received:6
Concurrent active server requests:5, total received:7
Concurrent active server requests:5, total received:8
Concurrent active server requests:5, total received:9
Concurrent active server requests:5, total received:10
...
Concurrent active server requests:5, total received:98
Concurrent active server requests:5, total received:99
Concurrent active server requests:5, total received:100
As you can see, the server never receives more than 5 simultaneous requests from the client. To change this behaviour, globally we can modify the http.globalAgent.maxSockets value, this allows us to specify how many open sockets we have at one time, this change will then apply to all http.requests made:
Running the code above, our output now show 10 simultaneous requests:
$ node maxsockets.js
Server running at http://127.0.0.1:1337
Concurrent active server requests:1, total received:1
Concurrent active server requests:2, total received:2
Concurrent active server requests:3, total received:3
Concurrent active server requests:4, total received:4
Concurrent active server requests:5, total received:5
Concurrent active server requests:6, total received:6
Concurrent active server requests:7, total received:7
Concurrent active server requests:8, total received:8
Concurrent active server requests:9, total received:9
Concurrent active server requests:10, total received:10
Concurrent active server requests:10, total received:11
Concurrent active server requests:10, total received:12
Concurrent active server requests:10, total received:13
Concurrent active server requests:10, total received:14
...
Concurrent active server requests:10, total received:98
Concurrent active server requests:10, total received:99
Concurrent active server requests:10, total received:100
To demonstrate how the connections are pooled per socket (IP address + port combination), lets create 2 servers, listening on port 1337 and 1338, in this case we can see that the client code is sending 20 simultaneous requests, 10 to each server (I’m leaving in the maxSockets = 10 line) :
$ node maxsockets.js
Server running at http://127.0.0.1:1337
Server running at http://127.0.0.1:1338
Concurrent active server requests:1, total received:1
Concurrent active server requests:2, total received:2
Concurrent active server requests:3, total received:3
Concurrent active server requests:4, total received:4
Concurrent active server requests:5, total received:5
Concurrent active server requests:6, total received:6
Concurrent active server requests:7, total received:7
Concurrent active server requests:8, total received:8
Concurrent active server requests:9, total received:9
Concurrent active server requests:10, total received:10
Concurrent active server requests:11, total received:11
Concurrent active server requests:12, total received:12
Concurrent active server requests:13, total received:13
Concurrent active server requests:14, total received:14
Concurrent active server requests:15, total received:15
Concurrent active server requests:16, total received:16
Concurrent active server requests:17, total received:17
Concurrent active server requests:18, total received:18
Concurrent active server requests:19, total received:19
Concurrent active server requests:20, total received:20
Concurrent active server requests:20, total received:21
Concurrent active server requests:20, total received:22
Concurrent active server requests:20, total received:23
...
Concurrent active server requests:20, total received:198
Concurrent active server requests:20, total received:199
Concurrent active server requests:20, total received:200
As well as defining a global maxSockets value, http.request takes an optional Agent instance in the options parameter, if you pass in false connection pooling is turned off:
Did you notice what I did above? No, it’s subtle but I just embedded an arbitary part of another webpage in my webpage - freaking awesome, no it’s not magic, it’s clipboard.com. If you want to try it out, just send me an email to mark@clipboard.com and I’ll be happy to send you an invite :)
If you want to specify the image that is shown when your webpage is shared on Facebook, like in the example below:
You need to add a bit of Open Graph metadata in your page:
Gotchas
I ran into two gotchas, the first was that my image was not showing up, it turns out Facebook doesn’t like it when you leave the protocol off the image URL, for example I had something like //www.clipboard.com/foo.png the reason for this is that having the URL start with // is a neat trick that will make the browser load the image from http is the page is http or https if the page is https, it stops you having to litter your code with checks for the protocol your page is on. So you have to explicitly list http:// or https:// in your URLs. I found this was causing the problem by using the Facebook Linter, you specify your URL and it shows you if there are any problems with your page - pretty handy!
The second problem was that the image I specified in the og:image meta value was not being picked up by Facebook, it had cached the previous share image. The way I found to refresh the cache was to actually run the Facebook Linter with the URL of the page and that seemed to refresh the cache Facebook was using for the page.
One of the benefits of writing your server code in JavaScript is the ability to easily share code between your frontend and backend. In the rest of this post I will describe how you can share JavaScript code, using a technique that I use at clipboard.com.
First, a quick intro into how you include JavaScript in your node code and client code. In a browser you can inline JavaScript in a page or include a reference to a JavaScript file that should be loaded. Typically it is best practice to not litter the global namespace with functions and variables (which could cause problems if it was included in a page that had other JavaScript using the same variable or function names), so you will normally see a top level object hanging off the window object that contains all of a libraries code e.g.
In node there is a module loading system, basically when you define a module you export functions from the module, which can then be accessed from other parts of the code that load the module. For example, if I have a module that contains common code, I would create a file called common.js, then inside common I would export my functions by attaching them to the exports object, like so:
With the above in mind, it is obvious that for common code we can’t use objects that are only available in node e.g. the process object, or functionality only available in the client code e.g. the window object, so shared code is pure JavaScript code, just logic, it must be agnostic to the hosting environment under which the JavaScript is being run.
To share code, we will make the client code look like node code, with respect to the export process, so we will put all of our functions onto an export object. If the code is loaded from node, nothing changes, if the code is loaded from a browser, the export object will actually just be an object we define like the window.common object in the example above. To know if the code has been loaded in node or in a browser, you do a simple check to see if the process object exists (only exists in node), I do a double check to see if the process object has a version property, just incase a “process” object exists in the client code, it is less likely to also have a version field like the node process object. Below is an example showing how this all comes together:
Imagine you have a file called common.js that contains some helper functions, to use this file in both your client side JavaScript and node.js you would …
One of my aims for this year was to blog more and to share some of the knowledge I have gained using node.js in production at www.clipboard.com/mark. I feel blogging is a good way to distill your thoughts, so if your interested in node hopefully I will have a nice series of posts that you might find useful, tracked here: http://www.markdawson.org/nodejs.
If there are any particular areas of node you think are interesting to write about, shoot me an email to markdawson@live.com, I plan to write about networking and look at the V8 internals, to really dive into the heart of the JavaScript engine in node
Wow - it’s been a while since I last posted to this blog. I’ve been meaning for quite a while to get back to blogging and writing. It’s interesting to see how my life has changed since I last blogged, back in december 2010 I was working at Microsoft, living a life of C# and Silverlight, working with the Bing Maps team. Fast forward to 2012, I quit Microsoft and joined a great startup called www.clipboard.com and now work with a completely different technology stack, JavaScript, Node.js, redis, riak, not a hint of Microsoft .
A big change of pace and an exciting challenge, I plan to write some posts about working at a startup, so stay posted.
A very cool invention, a robot that can climb walls using a technique to produce electrical adhesion, think of it as a temporary sticky tape that doesn’t leave any residue.
The music video was shot without a traditional camera, instead the whole video was captured using two technologies, Geometric Informatics and Velodyne LIDAR to capture 3D points. Watch the video and see the “how it’s made” video for more info, you can also play with the actual data in a 3D viewer: http://code.google.com/creative/radiohead/viewer.html
An insightful blog post about the future or Silverlight and the role it plays on the web, from the Silverlight team. Interesting to note that in half of the time it has taken to get the HTML5 spec we have today, Silverlight has had four major releases!
Safari has a debug menu that lets you set a number of options such as showing Javascript error logs, disable caching, debug the webpage etc. To enable it, close Safari and in a terminal window type:
JSONP (JSON with padding), can be used to get around the cross domain problem, where JavaScript running on one URL cannot access a webservice running on another domain. Here is an interesting article explaining how to use JSONP to get around this problem: