June 16, 2010

Arduino Tiny Web Server - part 2

 Arduino | Hardware | Open Source

Update (December 30, 2010): Latest version of Arduino TinyWebServer: arduino-tinywebserver-20101230.zip.

Update (December 8, 2010): The below picture of the required Arduino hardware is obsolete. Look at this newer post for updated information on the new hardware.

In part 1 of the Arduino Tiny Web Server I presented some hardware modifications and changes to the Arduino Ethernet shield and the Adafruit Data Logging shield.

In this part I present the Arduino TinyWebServer library or TWS.

TinyWebServer allows you to provide a Web interface to your Arduino-based project. You can get very creative with this, and add a full Ajax web interface to your project. This is possible because it's the Web browser doing all the UI work, while your Arduino board interacts with the hardware connected to it.

I'm using TWS in a remotely controlled projection screen that I'm currently building to replace an existing system. The end goal is to be able to control the projection screen from an Android phone, and let my kids choose to watch movies either on TV or on the big screen. More on this is a later post, until then read below to see this works.

The library has been developed on MacOS X and should most likely work fine on Linux. No guarantees about Windows, but I'd love to hear if it works for you.

As I mentioned in part 1, there are several hardware modifications, as well as software modifications that need to be made. Make sure you have those modifications done to your hardware before proceeding further.

To make things easy, I've decided to bundle the TWS library with the modifications to those libraries, as well as with two additional libraries that TWS depends on: Georg Kaindl's EthernetDHCP and Mikal Hart's Flash library.

After you download and unzip the package, copy the contents of the directory in the directory where you store your Arduino libraries.

The library comes with few examples, look in TinyWebServer/examples. The simplest one is SimpleWebServer, which shows how to write a basic HTTP server with a GET handlers. The more complex one, FileUpload shows how to implement a PUT handler to implement file uploads and write them on the SD card, and how to serve the files in GET requests.

Basic web server

To make use of the TWS library, you need to include the following your sketch:

#include <Ethernet.h>
#include <EthernetDHCP.h>
#include <Flash.h>
#include <Fat16.h>
#include <Fat16util.h>
#include <TinyWebServer.h>

EthernetDHCP is optional, but it makes acquiring an IP address a lot easier if you have a DHCP server in your network.

TWS is implemented by the TinyWebServer class. The constructor method takes two arguments. The first one is a list of handlers, functions to be invoked when a particular URL is requested by an HTTP client. The second one is a list of HTTP header names that are needed by the implementation of your handlers. More on these later.

An HTTP handler is a simple function that takes as argument a reference to the TinyWebServer object. When you create the TinyWebServer class, you need to pass in the handlers for the various URLs. Here is a simple example of a web server with a single handler.

static uint8_t mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };

boolean index_handler(TinyWebServer& web_server) {
  web_server.send_error_code(200);
  web_server << F("<html><body><h1>Hello World!</h1></body></html>\n");
  return true;
}

TinyWebServer::PathHandler handlers[] = {
  // Register the index_handler for GET requests on /
  {"/", TinyWebServer::GET, &index_handler },
  {NULL}, // The array has to be NULL terminated this way
};

// Create an instance of the web server. No HTTP headers are requested
// by the HTTP request handlers.
TinyWebServer web = TinyWebServer(handlers, NULL);

void setup() {
  Serial.begin(115200);
  EthernetDHCP.begin(mac);
  web.begin();
}

void loop() {
  EthernetDHCP.maintain();
  web.process();
}

In the loop() function we need the call to the process() to make sure HTTP requests are serviced. If there is no new request, the method returns immediately. Otherwise the process() method blocks until the request is handled.

For a complete working example look in TinyWebServer/example/SimpleWebServer.

Serving files from the SD card

Now that we've seen the basics, let's see how we can extend this web server to serve files stored on the SD card. The idea is to register a handler that serves any URLs. Once the handler is invoked, it interprets the URL path as a file name on the SD card and returns that.

boolean file_handler(TinyWebServer& web_server) {
  char* filename = TinyWebServer::get_file_from_path(web_server.get_path());
  if (!filename) {
    web_server.send_error_code(404);
    web_server << "Could not parse URL";
  } else {
    TinyWebServer::MimeType mime_type
      = TinyWebServer::get_mime_type_from_filename(filename);
    web_server.send_error_code(mime_type, 200);
    if (file.open(filename, O_READ)) {
      web_server.send_file(file);
      file.close();
    } else {
      web_server << "Could not find file: " << filename << "\n";
    }
    free(filename);
  }
  return true;
}

We can now register this in the handlers array:

TinyWebServer::PathHandler handlers[] = {
  {"/" "*", TinyWebServer::GET, &file_handler },
  {NULL},
};

Note how the URL for the HTTP request is specified. We want it to be /*, very much like a regular expression. However Arduino's IDE preprocessor has a bug in how it handles /* inside strings. By specifying the string as "/" "*" we avoid the bug, while letting the compiler optimize and concatenate the two strings into a single one.

The * works only at the end of a URL, anywhere else it would be interpreted as part of the URL. If the * is at the end of the URL, the code in TinyWebServer assumes the handler can process requests that match the URL prefix. For example, if the URL string was /html/* then any URL starting with /html/ would be handled by the specified handler. In our case, since we specified /*, any URL starting with / (except for the top level / URL) will invoke the specified handler.

Uploading files to the web server and store them on SD card's file system

Now wouldn't it be nice to update Arduino's Web server files using HTTP? This way we can focus on building the actual interface with the hardware, and provide just enough HTTP handlers to interact with it. After we implement a minimal user interface, we can iterate it without having to remove the SD card from the embedded project, copy the HTML, JavaScript and/or image files on a computer, and plug it back in. We could do this remotely from the computer, using a simple script.

TinyWebServer provides a simple file upload HTTP handler that uses the HTTP 1.0 PUT method. This allows you to implement an Ajax interface using XMLHttpRequest or simply use a tool like curl to implement file uploads.

Here's how you add file uploads to your Arduino web server:

TinyWebServer::PathHandler handlers[] = {
  // `put_handler' is defined in TinyWebServer
  {"/upload/" "*", TinyWebServer::PUT, &TinyWebPutHandler::put_handler },
  {"/" "*", TinyWebServer::GET, &file_handler },
  {NULL},

Note that the order in which you declare the handlers is important. The URLs are matched in the order in which they are declared.

This is where the headers array mentioned before comes into picture. The put_handler makes use of the Content-Length. To avoid unnecessary work and minimize precious memory usage, TinyWebServer does not do any header processing unless it's instructed. To do so, you need to declare an array of header names your handlers are interested in. In this case, we need to add Content-Length.

const char* headers[] = {
  "Content-Length",
  NULL
};

And we now initialize the instance of TinyWebServer like this:

TinyWebServer web = TinyWebServer(handlers, headers);

The put_handler method is really generic, it doesn't actually implement the code to write the file to disk. Instead the method relies on a user provided function that implements the actual logic. This allows you to use a different file system implementation than Fat16 or do something totally different than write the file to disk.

The user provided function take 4 parameters. The first is a reference to the TinyWebServer instance. The second is a PutAction enum which could be either START, WRITE or END. START and END are called exactly once during a PUT handler's execution, while WRITE is called multiple times. Each time the function is called with the WRITE param, the third and fourth parameters are set to a buffer and a number of bytes in this buffer that should be used.

Here is a small example of a user provided function that writes the PUT request's content to a file:

void file_uploader_handler(TinyWebServer& web_server,
			   TinyWebPutHandler::PutAction action,
			   char* buffer, int size) {
  static uint32_t start_time;

  switch (action) {
  case TinyWebPutHandler::START:
    start_time = millis();
    if (!file.isOpen()) {
      // File is not opened, create it. First obtain the desired name
      // from the request path.
      char* fname = web_server.get_file_from_path(web_server.get_path());
      if (fname) {
	Serial << "Creating " << fname << "\n";
	file.open(fname, O_CREAT | O_WRITE | O_TRUNC);
	free(fname);
      }
    }
    break;

  case TinyWebPutHandler::WRITE:
    if (file.isOpen()) {
      file.write(buffer, size);
    }
    break;

  case TinyWebPutHandler::END:
    file.sync();
    Serial << "Wrote " << file.fileSize() << " bytes in "
	   << millis() - start_time << " millis\n";
    file.close();
  }
}

To activate this user provided function, assign its address to put_handler_fn, like this:

void setup() {
  // ...

  // Assign our function to `upload_handler_fn'.
  TinyWebPutHandler::put_handler_fn = file_uploader_handler;

  // ...
}

You can now test uploading a file using curl:

curl -0 -T index.htm http://my-arduino-ip-address/upload

For a complete working example of the file upload and serving web server, look in TinyWebServer/examples/FileUpload.

Advanced topic: persistent HTTP connections

Sometimes it's useful to have an HTTP client start a request. For example, I need to be able to enter an IR learning process. This means that I cannot afford TinyWebServer's process() to block while serving my /learn request that initiated the IR learning process. Instead I want the handler of the /learn request to set a variable in the code that indicates that IR learning is active, and then return immediately.

If you noticed the HTTP handlers return a boolean. If the returned value is true, as it was the case in our examples above, the connection to the HTTP client is closed immediately. If the returned value is false the connection is left open. Your handler should save the Client object handling the HTTP connection with the original request. Your code becomes responsible with closing it when it's no longer needed.

To obtain the Client object, use the get_client() method while in the HTTP handler. You can write asynchronously to the client, to update it with the state of the web server.

In my remotely controlled projection screen application, I have another handler on /cancel that closes the /learn client forcibly. Otherwise the /learn's Client connection is closed at the end of the IR learning procedure. Since the Ethernet shield only allows for 4 maximum HTTP clients open at the same time (because of 4 maximum client sockets), in my application I allow only one /learn handler to be active at any given time.

Posted by ovidiu at June 16, 2010 01:56 PM |
Comments

This looks to be a bit more advanced than my Webduino library (http://wedduino.googlecode.com), although possibly using more memory. How much form processing logic is in the library -- that was one area I focused on with my code and its examples?

Posted by: Ben Combee on June 17, 2010 12:33 PM

Grr, that should be http://webduino.googlecode.com... typo!

Posted by: Ben Combee on June 17, 2010 12:33 PM

I'm not doing any form processing at the moment. Uploads are handled using PUT instead of POST. Though at some point I'll add code to handle GET and POST parameters. Also the library only supports HTTP/1.0 not HTTP/1.1.

Also since you pretty much have unlimited storage with the SD card (well, from an Arduino perspective :), you can store pretty complicated HTML/JavaScript files on the filesystem. As a result, TinyWebServer doesn't implement any methods to generate HTML pages.

Posted by: Ovidiu Predescu on June 17, 2010 12:50 PM

You project seem very interesting and I was looking forward to it as not many people have well documented on how to integrate SD card Logging/Storage and Ethernet connection at the same time with a single Arduino board. Thank you so much for posting.

I just wanted to ask you if you think is there enough space remaining (as well as port pins) on the arduino's flash to write the code for an LCD Display?

Thank you

Posted by: George on June 28, 2010 07:22 PM

I haven't played with an LCD display yet. But you should still have plenty of pins left to drive the LCD, assuming you're going to use a serial one. I have such an LCD I got from sparkfun, I'll try to put something together and report back with results.

Posted by: Ovidiu Predescu on June 29, 2010 12:02 AM

Cool stuff Ovidiu! I've got the Arduino experimenter's kit from Adafruit since last week, and have a lot of fun with it... and plenty of ideas of stupid and not so stupid projects ;-)

Posted by: Sylvain Wallez on June 30, 2010 03:20 AM

@ Ovidiu Predescu:

Yes. I was also thinking on serial LCD Display. The only thing is that the guy in my team has already wrote pages of code (Literally) for the display.
I just wouldn't like com say "Hey pal you are gonna have to do it again" (Just kidding). I have done work on writing code for the parallel LCD. But never for a serial LCD. Do you know if there is a way to convert it or adapt it?

Thank you!

Posted by: George on June 30, 2010 05:06 PM

George, I just took a look at the SerLCD software provided by SparkFun on their web page for the serial LCD:

http://www.sparkfun.com/commerce/product_info.php?products_id=9066

SerLCD seems to be the software that drives the actual LCD on the provided board.

The PDF documentation on that page seems to indicate a pretty straightforward serial API to control what's displayed on the LCD. Writing a small library to make it easy to display stuff on the screen should be pretty straightforward.

Posted by: Ovidiu Predescu on July 1, 2010 11:59 AM

i still cant believe this was made using arduino. i wonder if these babies are going to be mass produced anytime soon... would be perfect for running information databases or a low spec site. endless possibilities, cant wait to see more development.

Posted by: Anon83 on July 2, 2010 01:45 PM

@Anon83 what I think is more interesting is the fact that you can provide a web interface for your project, in addition to the more traditional interfaces. This allows you to control your project using your cell phone or web browser, with a much richer interface than was possible before.

Posted by: Ovidiu Predescu on July 2, 2010 02:38 PM

Awesome project! Just a thought, Rabbit Semiconductor makes some AWESOME micros that have webservers built in. They're more expensive than an Arduino (~$70 IIRC) but overall it'd be less expensive once you factor in the costs of the shields you're using. They use Dynamic C which is cool b/c you can have co-functions to allow parallelism and the API allows you to code C methods and by simply adding a compiler directive that method gets transformed into a .CGI script. They also have wireless modules so if you have a wireless router your project essentially becomes mobile and retains connectivity. Just thought it was worth mentioning since I've been looking for an adequate Arduino substitute for these controllers for a while and have yet to find anything that blows them away.

Cheers!

Posted by: Paul on July 5, 2010 04:33 PM

Thanks for the pointer, Paul! The Rabbit Semiconductor boards look pretty good.

Unfortunately their Dynamic C compiler is Windows only, and all my development happens on Linux and Mac OS X. I'll give their stuff a try though on a Windows machine to see if it's worth going through the pains of developing on Windoze.

What I'd really like to see is an Atmel-based board that has Ethernet and SD card incorporated. Or a combination of such options, similar to how Rabbit does it. The Atmel/Arduino platform has a great advantage in that all the host software is open source and runs on all the major platforms.

Posted by: Ovidiu Predescu on July 6, 2010 01:34 AM

The Ethernet shield from DFRobots has the SD card incorporated. It works just fine for me, as I've pointed out a few days ago in my comment to part 1 of your article.
My experiments with this shield are described here:
http://ro-duino.blogspot.com/search/label/ethernet
(Romanian language only...)
It seems to work better with MMC cards than with SD cards, but it's up to you to test it.

Posted by: iard on July 8, 2010 03:48 AM

Nice utilization of various Arduino add-on modules. Could you please disclose what's the price of this 'sandwich'? Have you considered to use Bifferboard instead? As storage option it's possible to use USB Mass Storage Device (http://www.bifferos.com)

Posted by: ikon on July 8, 2010 02:02 PM

Iard, awesome find! I'll buy one of those Ethernet cards and see how they work. Avoiding the separate SD card shield saves both space as well as money.

Ikon, thanks for the link! The Bifferboard looks like an interesting option if all you want to do is build a tiny web server.

My interest however is to provide a web interface to some hardware that's controlled through an Arduino. In my current project I have 2 stepper motors and 2 DC motors, an IR sensor, few switches and 2 range sensors. I need the input/output pins provided by the Arduino board.

The price of the current configuration is pretty high at US $95. Using the board pointed out by Iard, you can reduce this to $64. I wish somebody made an AVR board that incorporates the AVR, the Ethernet chip and the SD card.

Posted by: Ovidiu Predescu on July 9, 2010 01:39 AM
Post a comment
Name:


Email Address:


URL:


Comments:


Remember info?



  
Copyright © 2002-2016 Ovidiu Predescu.