TwitterChimes: How do they work?

May 1st 2011

My last project using CoffeeScript and Processing.js had me wanting more, so I created TwitterChimes. As one might possibly maybe expect from the name, TwitterChimes uses a stream of tweets from the Twitter Streaming API to create music, like windchimes… but with more CoffeeScript.

I really enjoy taking music/sound data and creating visual representations, so I figured why not go the other way.

In this first part we’ll cover creating a server with Express that uses Twitter-Node to connect to the Streaming API and Socket.IO to broadcast tweets of interest to the browser. In the next tutorial I’ll walk you through visualizing the tweets and making music.

Here’s a video of it in action. And here’s a live demo. I find Chrome to work best.

And of course, for your enjoyment, the final product of this exercise is on github

A little help from our friends

There are some really awesome libraries out there. Let’s use a couple to make great things happen:

  • Node.js + npm – Javascript ftw.
  • Express – This lets us quickly create a webserver, think of it as Sinatra for Node.
  • Socket.IO – “The cross-browser WebSocket for realtime apps” – polling for server changes is lame.
  • Underscore.js – If Batman’s utility belt was made out of Javascript
  • Twitter-node – A nice interface to the Twitter Streaming API
  • CoffeeScript – It’s just like writing Javascript, but more fun.

And on the client side we’ll want:

  • jQuery – makes dealing with the DOM enjoyable

Installation

Since we’re relying on Node.js for the server, this project has a bit more setup involved.

  • 1) Install Node.js. For reference I’m using version 0.4.2

  • 2) Install npm. This is Node’s package manager (like Gems for Ruby). It makes installing libraries for Node a snap.

  • 3) Using npm, install Express, Socket.IO, and Underscore

      npm install express socket.io underscore 
  • 4) CoffeeScript is slightly different, we install it with the -g flag

      npm install -g coffee-script 
  • 5) Install Twitter-node. At the time of writing, the version of Twitter-node in npm doesn’t want to cooperate. Luckily, installing from git is a snap. Let’s do that instead:

      npm install https://github.com/technoweenie/twitter-node/tarball/master 

Hopefully that was easy and problem-free. Next up we’ll create a skeleton of our app.

Some Structure

  • 5) Set up the project directories:

    • Here’s how the directory will be organized:

      twitter_chimes/ 
        |-- public/
        |   |-- css/ 
        |   |   |-- reset.css [ style reset from html5boilerplate ]
        |   |   `-- style.css [ we'll put our custom styles in here ]
        |   |-- index.html [ our html markup goes here ]
        |   `-- js/ 
        |       |-- libs
        |       |   `-- jquery-1.5.2.js
        |       |-- script.coffee [ we'll be writing our client code in coffeescript ]
        |       `-- script.js [ and it will compile to js ]
        |-- server.coffee [ we'll write our server code here ]
        `-- server.js [ and it will compile here] 
    • Or, how it looks in a Textmate drawer:

    • Textmate Drawer

    • 5a) Create the directories in the structure above

    • 5b) Our reset.css will be the stylesheet from html5-boilerplate on github. This goes in /public/css/reset.css
    • 5c) Put jQuery in /public/js/libs/jquery-1.5.2.js
    • 5d) Create blank text files for the rest:
      • /server.coffee
      • /public/index.html
      • /public/js/script.coffee
      • /public/css/style.css

Don’t worry about server.js and /public/js/script.js, CoffeeScript will handle those in a minute.

A Map

Now that the initial setup is out of the way, we can get to some coding.

Here’s what we want:

A page with a form (nothing fancy, just a text field and a submit button). When you type a word into the form and hit submit, we want to see tweets that contain that word populate the page in real-time.

Here’s what we need:

  • Server:

    • 1) Serve a static directory containing our client-side code, html and css (/public)
    • 2) Connect to the Twitter Streaming API
    • 3) Change the track word(s) at the request of the client
    • 4) Broadcast tweets to connected clients using Socket.IO
  • Client:

    • 1) Send desired track words to the server
    • 2) Respond to tweet events
    • 3) Display the tweets

The responsibilities of the server will primarily live in /server.js compiled from /server.coffee. We’ll code our client stuff in /public/coffee/script.coffee which will compile to /server/public/js/script.js

Ready? Let’s do it!

A Coffee Break

CoffeeScript is awesome. We want to write our code in CoffeeScript. Sadly, Node doesn’t understand CoffeeScript as we write it, and neither does the browser. They need a little help. Our .coffee files need to be compiled into .js.

Fortunately for us, that part is super simple. CoffeeScript’s coffee tool will watch a directory, look for changes in .coffee files, and automatically compile them to .js!

It’s as easy as typing the following into your terminal:

coffee --watch --output <directory for .js> --compile <directory with .coffee> 

Let’s do this for server.coffee in our root directory (/twitter_chimes):

coffee --watch --output . --compile .
   

You should see something like:

20:44:20 GMT-0700 (PDT) - compiled server.coffee
  20:44:20 GMT-0700 (PDT) - compiled public/js/script.coffee 

In fact, it should look eerily similar if you do it at 8:44pm with your computer’s clock set to Pacific time.

Boom, all our .coffee now have .js siblings in their directories! Super easy.

Best of all, if we leave this running, any time we save changes to a .coffee file, its compiled .js sibling will be updated as well.

The Twitters

Open up /server.coffee, it’s time for action!

We’ll start by requiring all of the fun node libraries we’ll be relying on:

# /server.coffee

  # get those libraries!
  sys           = require 'sys'
  _             = require 'underscore'

  io            = require 'socket.io'
  express       = require 'express'
  {TwitterNode} = require 'twitter-node'
   

Let’s set up our connection to Twitter using Twitter-Node. To do this we instantiate a new TwitterNode object, passing in a couple things: username, password, and an array of words we’d like to track. To make sure things are working, we’ll start with the array [“why”]. Then, we just need to tell our new twitter object what to do when it gets a tweet. For now we’ll just use console.log() to print it to the terminal:

# /server.coffee
  ...
  {TwitterNode} = require 'twitter-node'

  # create our interface to the Twitter Streaming API

  twitter = new TwitterNode
    user: 'yourtwitterusername'
    password: 'yourtwitterpassword'

    track: ["why"]

  # tell our interface what to do when a tweet comes in
  twitter.addListener 'tweet', (tweet) ->
    # print it.

    console.log tweet.text

  # start it up.
  twitter.stream() 

Let’s fire up the server and see how we’re doing:

node server.js 

You should now see all kinds of tweets. Perhaps your terminal will look like mine:

Terminal Stream Test

If you see tweets streaming into your terminal, you’re doing great. Ctrl-C that bad boy and lets continue.

Webserver Express

The next step is to get our webserver up and serving files. And by files I mean the html, js, and css sitting in that lovely /public directory of ours.

To do this, we’ll (1) create an Express server in our /script.coffee and (2) configure it to serve /public as static files.

To make sure that it’s working we’ll (3) throw some text into /public/index.html.

Starting with /server.coffee again:

# /server.coffee
  ...
  {TwitterNode} = require 'twitter-node'


  # We'll put this new stuff above the twitter code

  # (1) create our express webserver
  app = express.createServer()

  # (2) expose our public directory
  app.configure ->
    app.use express.static(__dirname + '/public')


  # This will be the port that our server listens on
  # I'm choosing 3010, but feel free to get creative
  app.listen 3010


  # create our interface to the Twitter Streaming API
  ... 

Since we just want to make sure that express is working, let’s put in a little of text in /public/index.html. Nothing crazy, just enough to see that it’s doing what we want.

<!-- /public/index.html -->

  <!doctype html>
  <html>
    <head>
      <meta charset="utf-8">

      <title>
        TwitterChimes
      </title>
    </head>

    <body>
      <!-- (3) -->

      <h1>Chunky bacon.</h1>
    </body>
  </html> 

Ok, let’s run that server again and see what we got.

node server.js 

Use a web browser of your choice and open up http://localhost:3010. Your browser should spout off some nonsense about lumpy pork products.

Sockets

Real-time tweets? Check.

Web server? Check.

Web sockets? Socket.IO time!

Socket.IO does the heavy lifting for real-time communication between our server’s ever-flowing twitter stream and the browser. We need to do three things for this to happen: (1) create a socket object on our express server, (2) tell our twitter object to broadcast the tweet (instead of just printing to the terminal), and (3) add a bit of client code to handle the tweets coming in.

Add one line for the socket object in /server.coffee

# /server.coffee
  ...
  app.listen 3010

  # (1) create that socket object

  socket = io.listen app

  # create our interface to the Twitter Streaming API
  ... 

And just a bit further down, let’s tell our ‘tweet’ listener to use our new fancy socket

# /server.coffee
  ...
  twitter.addListener 'tweet', (tweet) ->

    # print it.
    console.log tweet.text

    # (2) not only will we print it, we'll broadcast it too.
    socket.broadcast tweet.text

  # start it up.
  twitter.stream() 

<3 Socket.IO.

Now we need to tell the client what to do when it receives a message.

This involves (1) adding a script tag to load Socket.IO client javascript, (the server-side code will automatically make this available for us at http://localhost:3010/socket.io/socket.io.js), (2) initializing a client-side socket object to send and receive messages, and (3) telling that socket message what to do when a message comes in.

In our html (/public/index.html) we’ll get rid of our test string and add some script tags for jQuery, Socket.IO (client-side). In our own client-side code /public/js/script.js (which is really just /public/js/script.coffee in disguise), we’ll tell socket to take the message as an argument and prepend its value to a div (#messages).

<!-- /public/index.html -->
  ...
    <body>
      <!-- No more Chunky Bacon. -->
      <div id="messages"></div>

      <!-- Instead we'll get jQuery in on the fun -->
      <script src="/js/libs/jquery-1.5.2.js"></script>

      <!-- and link to Socket.IO's client-side library -->

      <script src="/socket.io/socket.io.js"></script>

      <!-- (1) link to our client-side code -->
      <script src="/js/script.js"></script>

    </body>
  </html> 

And now the client side code that will receive the tweets/socket messages:

# /public/js/script.coffee

  # standard jQuery document ready listener
  $(document).ready ->

    # (2) create the client socket with our server's port
    socket = new io.Socket "localhost",  port: 3010

    # (3) tell it what to do on the "message" event
    socket.on "message", (message) ->
      $('#messages').prepend("<p>" + message + "</p>")

    # and connect!

    socket.connect() 

Restart the server and point your browser at http://localhost:3010. You should now see that tweet-torrent pouring into your browser. Victory!

Ok, maybe that’s a strong choice of word, but we’re doing great =)

Interaction

Right now the words we’re tracking on twitter are set in stone, up front. Honestly, we shouldn’t have to settle for that. I’m not going to, and frankly, neither should you.

Here’s what we’re going to do about it. On the server: (1) we’re going to create a new route in server.coffee. By passing a parameter in we can tell our server to which words to track. We’ll do this by (2) splitting the parameter string into an array, (3) giving it to our twitter object, and (4) restarting our twitter stream.

On the client: (5) we’ll create a text field and form for our new words, and (6) ajaxify it in /public/js/script.coffee.

For example, if we hit http://localhost:3010/track?q=fox,+elf,+ham our server should start tracking those words.

Let’s do the server:

# /server.coffee
  ...
  app.configure ->
    app.use express.static(__dirname + '/public')

  # (1) Create a new "/track" route
  app.get '/track', (req, res) ->

    # (2) Get the words from the "q" parameter
    track_words = req.query.q.split(", ")

    # (3) Give them to the twitter object as an array
    twitter.trackKeywords = track_words

    # (4) Restart the stream
    twitter.stream()

    # and I'm just going to redirect back to the root for fun

    res.redirect '/'
  )

  # start the server on port 3010
  ... 

and now the client html:

<!-- /public/index.html -->

  ...
    <body>
      <!-- (5) a simple text input field with some ids -->
      <form method="get" accept-charset="utf-8">

        <p><input type="text" id="track_words"/></p>

        <p><input type="submit" id="track_button" value="Track!"></p>

      </form>

      <div id="messages"></div>
  ... 

and client coffee/js:

# /public/js/script.coffee

  ...
  socket.connect()

  # (6a) create the form click handler
  $('#track_button').click ->
    # (6b) get the words from the text field

    track_words = $('#track_words').val()

    # (6c) hit that route on the server with the words as the query
    $.get('/track?q=' + track_words)

    return false
   

End of Side 1

I hope this tutorial gave you an idea of how you can use CoffeeScript and friends to harness a fun data stream.

What we have here so far is fun, but there’s definitely room for much more awesome. Here are three simple ways to go above and beyond the above:

  • 1) Style the incoming tweets to highlight the track word
  • 2) Measure and display tweet velocity for each track word
  • 3) Build up tag/word clouds for each track word

In the next part, we’ll use the tweets to make music when we turn this app into a nice set of TwitterChimes.