garykac

Which book API to use for a mobile book app? (Part II)

2010-06-21T07:48:00.000-07:00

(...continued from Part I...)

Here is an evaluation of the various Book APIs available with regard to their suitability for use in a standalone Android application. Please see Part I for more background information - this post contains just the raw evaluation and conclusions.

Amazon

Previously called "eCommerce Service API" (or ECS), Amazon's Product Advertising API is part of the Amazon Web Services (AWS), although the correct link is challenging to find if you start from the top AWS link.

Developer Guide

Access
Provides basic Item Search/Lookup and List Search/Lookup. Lists aren't exactly the same a bookshelves, but I suppose that they're close enough.

Read/Write
As far as I can tell in the API, there is no way to add a new list or add an item to an existing list. You can, of course, edit/modify the shopping cart, so write access is provided by the API. It seems that Amazon wants the API to be read-only unless you're trying to buy something.

Auth
Each app that wants to use the API needs to register for an AWSAccessKeyId and then use that ID in all requests.

I didn't investigate how one would access non-public lists using this API. Presumably this would require some further authentication, but the API restrictions made me stop evaluating.

Restrictions
From the license agreement, the purpose of the API is to "permit you to advertise items offered on the Amazon Site". Each app that wants to use the API requires enrollment and approval by Amazon (it's apparently not an automated process).

The license mentions that

Unsuitable applications include those that:
(a) do not have as their principal purpose advertising and marketing the Amazon Site and driving sales of products and services on the Amazon Site;

!!

Conclusion
Given the access limitations and the license requirements, this API does not seem suitable for use for this purpose. It might be worth considering in the future if I want to extend this app by adding a link to an Amazon page.

aNobii

aNobii API Home

Access
aNobii has 3 API calls: item.getInfo, shelf.getSimpleShelf and contributor.getInfo.

These seem like they would be enough for a basic read API, but I was unable to get them to work correctly (I kept getting getting a code 401 "Missing parameter(s)" error - was my api_key wrong or was I passing the wrong book id, I have no idea).

Read/Write
Read-only.

Auth
Requires a devkey with an extra "secret" id that you combine and then run through a MD5 hash to create your real API key.

Restrictions
The terms of service include the following:

b. You shall not:
Use aNobii APIs for any application that replicates or attempts to replace the essential user experience of aNobii.com.

Since I'm sure a mobile book management app could easily be construed as replacing the "essential user experience" of aNobii.com, it's probably
not valid to use the API for this.

Conclusion
As with all of the APIs that require a devkey, there is an issue with assigning a devkey to an application that is distributed (like a phone app). Even though each app is run by a separate user, they all are counted toward the same app id.

Perhaps this would work if each user could get easily get their own developer key + private key. The user would enter this info once when they installed the app.

In any case, since this is read-only (and didn't seem to work properly for me), I'll probably pass on this for now since the APIs provided by other sources provide more functionality.

GoodReads

Access
Nice API with access to most things that you could want. Book info, shelf info for a user, list of books in a shelf.

Examples:

Get a list of shelves:
http://www.goodreads.com/shelf/list?format=xml&v=2&key=<devkey>&user_id=<id>

Get list of books in shelf:
http://www.goodreads.com/review/list?format=xml&v=2&key=<devkey>&id=<id>&shelf=<shelf-name>

Read/Write
Both reads and writes are supported. Writes require authentication.

Auth
Most API calls require either a devkey or require that a user be authorized via OAuth.

Restrictions
License is pretty standard. API queries are limited to no more than 1 query/second.

Conclusion
GoodReads has a well thought-out API and handles authorization properly. The only concern is that some are OAuth-based while others are devkey based. Since the devkey based accesses are limited to 1 query/second, this may be a problem when incorporating calls into a standalone app (since the app would be installed on lots of different end-user devices).

However, this definitely seems like something worth pursuing.

Google Books

Google Book Search API - Getting Started Guide
Google Book Search Data API : Developer's Guide

Access
Basic book info and shelves (with shelf editing) are supported.

Read/Write
Book data can be read and user shelf info can be read/written (with authentication).

Auth
General book queries don't require authorization, but to access user-specific data (like bookshelves) it is necessary to have the user authenticate. All writes also require authentication.

Since the Google Books data is exposed using the Google Data protocol, this authentication can be done using either AuthSub, OAuth or ClientLogin, as described in the Google Data Authentication page.

Restrictions
The Terms of Service are long and "legal" and deal mostly with requiring that you don't misrepresent yourself (or your app), that you respect Google trademarks and the privacy of the users and that you abide by all applicable laws. Most of it seems pretty boilerplate for a web service.

No queries/second restrictions, although I suspect if your app started DoSing the service it would get cut off rather quickly.

The only item that stood out is that Google reserves the right to insert advertising in the content. This isn't the case with any of the Google Books APIs that I've ever used and I'm not sure how that would work when using a simple text-based API. In any case, this doesn't seem to be an issue but it's worth being aware of.

Conclusion
In order to securely access user data (like bookshelf info), having a system that allows the end-user (rather than the app developer) provide credentials is essential. This is one of the few APIs that supports this.

ISBNdb

Developer API

ISBNdb crawls various libraries to build/maintain a database of ISBNs.

Access
Only provides basic ISBN book lookup.

Example:

http://isbndb.com/api/books.xml?access_key=<key>&index1=isbn&value1=<isbn>

Read/Write
Read-only.

Auth
Dev key required to access.

Restrictions
None mentioned on site.

Conclusion
Not worth investigating further for this application since this info seems readily available elsewhere with fewer restrictions.

LibraryThing

LibraryThing APIs

LibraryThing has a collection of different APIs rather than one coherent API. They are divided into 4 basic categories: Covers, JSON, Webservices, and Misc.

Covers

Used to get book covers given an ISBN.
Requires a devkey and is limited to 1000 queries/day

JSON APIs

Books API
- This allows you to lookup a book on LibraryThing.
- The Books API page shows 2 different example formats:
- They both return basically the same info, but the "json_books" version includes a bunch of widget-creating JavaScript at the top before the data begins.

Works (book search) API
- This is a book search API and the only documentation relating to it is in this blog post
- This doesn't require a developer key.

WebServices APIs

This appears to be a work-in-progress since it only provides access to a LibraryThing works and authors. I didn't see way to access user lists.
This requires an apikey and is limited to 1000 queries/day.
Example:
- http://www.librarything.com/services/rest/1.0/?method=librarything.ck.getwork&apikey=<apikey>&id=<librarything-id>

Misc APIs

Easy Linking: Link to LibraryThing using ISBN (rather than LibraryThing book id).
ThingISBN: given an ISBN, return a list of related ISBNs
ISBN -> LibraryThing book id
ISBN -> book language
ISBN validate
Title -> list of possible ISBNs

Access
These APIs provide basic access to book data. However, none of the APIs appears to provide access to the user collections. Shelf functionality could be re-created using the tags (which are available in the API), but it seems odd not to use LibraryThing's native "shelf" concept

Read/Write
Mostly read access but write access of sort-of there. You can add a book using:

http://www.librarything.com/addbook/<isbn>

But this assumes that you are already logged in, so it doesn't really work for a standalone app.

Auth
Devkeys are required for the JSON and WebServices API calls (although different keys are needed for each). No real Authentication is provided to access user data.

Restrictions
Most API calls require a devkey and they're all limited to 1000 queries/day.

Some of the license requirements are quite odd. For example, the JSON Books API actually says the following:

We have yet to settle on a license. You agree to abide by whatever license we eventually choose.

!!

In addition, both of the JSON APIs have the following in the license:

Must be run as Javascript on user's browser, not fetched by a server; cannot be stored, except for browser caching. Commercial use requires a hard (non-JS) link to LibraryThing on every page that returns results.

Which is clearly not the case if I want to run call this from a phone app.

Conclusion
Overall, while there are many things to like about what LibraryThing provides, there are too many parts that don't seem mature enough to support a phone app based on the current APIs. In addition, the licensing requirements for some of the APIs are best described as quirky and certainly preclude creating things like phone apps.

Still, I was left with the impression that there's some useful stuff here and it might be useful to add some sort of LibraryThing integration.

OpenLibrary

OpenLibrary APIs:
REST API
Books API
Covers API

Access
Basic access to books. E.g., using:

http://openlibrary.org/api/books?bibkeys=ISBN:<isbn>&details=true

No access to lists since it's a new feature that they're still working on.

Read/Write
Read only.

Auth
OpenLibrary allows you to login automatically by POSTing to http://openlibrary.org/account/login. This sets a login cookie and is apparently intended for web access.

The API doesn't have or require authentication.

Restrictions
None mentioned on the site.

Conclusion
Issues:

Lack of support for lists
Lack of write support

Not worth investigating further for this application.

WorldCat

WorldCat Search API
WorldCat Basic API

Access
Provides basic access to books, but doesn't appear to allow access to user created lists.

Read/Write
Read-only.

Auth
Requires a devkey for all access

For example to do a simple lookup by ISBN:

http://www.worldcat.org/webservices/catalog/content/isbn/<isbn>?wskey=<key>

where <key> is your assigned key.

Restrictions
Limited to 1000 queries per day.

Conclusion
Issues:

Devkey + query limit make this inappropriate for an app unless each user gets their own devkey
Lack of access to user created lists
Lack of write functionality

Not worth investigating further for this application.

Whew! I'm glad that's done. Please let me know if you see any errors or omissions.

I need to say that I'm actually a bit surprised that more APIs didn't make the cut and with how critical the Auth problem ended up being. I was also kinda hoping that LibraryThing would do better than it did.

But now it's time to start playing more with the 2 finalists (GoodReads and Google Books) to see how well they work in practice. Perhaps I'll experiment a bit more with LibraryThing as well.

Which book API to use for a mobile book app? (Part I)

2010-06-21T07:47:00.000-07:00

So I'm starting to look into creating a book management app that will run on my Android phone. There are a few of these apps available in the Android Marketplace, but none of them do quite what I'm looking for.

What am I looking for? Well, I have a few requirements:

Needs to be able to add new books easily (via barcode scanner)
Should automatically get book info from online sources
Should allow books to be arranged in "shelves" or tagged
Should store my list of books online

The first few requirements are fairly easy - it's the last one that makes things interesting. While it'd be far easier for me to create (yet another) local database to store this information, there are some big advantages to having everything stored "in the cloud":

The data is available on other devices (via the website)
The data is backed up (if something happens to my phone)
It's easy to share
...

Given this requirement, I could either write my own online data storage app or I can take advantage of the already-existing services out there.

"Writing my own" is not as bad as it sounds. A simple AppEngine app could be thrown together to handle this. The problem is that while this would work for me with my books, it wouldn't be able to handle other people's books (without running into the free service limits). The code could be open-sourced to make it easy to others to create their own instance of this web app for their own books, but that's starting to become needlessly complicated.

So using an existing online book manager is the right path to take. Fortunately, there are a bunch of decent ones out there to choose from that seem to provide (at least part of) what is needed:

The trick is to find which one (or more) supports the features that I want. Note that there are a few other services out there (like Shelfari) not listed because they don't have a developer API. If you know of any others that are worth investigating, please let me know.

Now, in an ideal world, this (proposed) phone app would be a simple front-end and the user would be able to select from a number of different online book managers to use as a backend. While this is great for users (they can pick/choose their favorite online book management site), the sad reality is that this sort of things is challenging for (mostly) technical reasons. This is where a standard interface for accessing personal libraries would come in handy. But, alas, such a thing does not exist.

So how can I choose from amongst the available options? It seems I need to sit down with each of them and see what they're capable of and then compare that with my requirements listed above.

To do that, I've broken things out into 4 sections and evaluated each API. These four sections are : Access, Read/Write, Auth and Restrictions.

Access
Quite simply this evaluates whether or not the API provides access to the required data.

In its simplest form, a "user library" app needs a list of books (ISBNs or EANs) and way to arrange them into groups (bookshelves or labels).

More generally, my app needs access to an API that provides the ability to:

Get general book info
- Given an ISBN, get the title, author, pub info, thumbnail, ...
Get user specific info
- Get a list of bookshelves (or get a list of labels)
- Get a list of books in a bookshelf (or get a list of all books with a particular label)

Read/Write
Technically, this is part of "Access", but I broke it out for some reason during the evaluation and ended up leaving it that way. Since all APIs (pretty much by definition) provide read access, the real question here is "do they provide write access?".

The write access required is:

Update user specific info
- Add a new bookshelf (or label)
- Add a new book
- Add/update user info for a book

The bare minimum is to be able to add a new book to a shelf. The other 2 are 'would be nice's.

Auth
This is where things start getting tricky. "Auth" is a combination of "authentication" (is the user who they say they are) and "authorization" (does the user have access to the requested resource).

So the question is : "How does the API authenticate/authorize the user to provide access to the data?"

Often there will be different levels of Auth provided by the same API. To get general book info, the API doesn't need to know who you are. But to get (or update) info on a private shelf, it needs to verify that you are the owner of that shelf.

As a preview of things to come, this is the area where most APIs fall down. The most common problem being that they attempt to authorize the application (through the use of a devkey or something similar) rather than the user of the application.

Restrictions
This (almost) final section discusses limits and licensing restrictions on the API.

Most of the restrictions in the license agreements are pretty typical. Things like:

You can't use the API to harvest data
You can't relicense the data
You agree not the break the law
Not for commercial use

So I'll only be breaking out issues that either fall outside what is typical or that may impact the app.

The 2 common issues that pop up here are:

Acceptable use for the API - Does the agreement forbid what I'm trying to do?
QPS (queries/second) restrictions - Are the QPS restrictions low enough to cause concern?

Conclusions
And finally, each sections ends with a brief set of conclusions. It's important to note that these conclusions do not evaluate the service as-a-whole, but rather specifically to their suitability for this particular application.

Public disclosure: I should note that, while I do work at Google, I do not work on the Android or Google Books teams, so I'm not tied to promoting these particular products. My hope with this evaluation is to compare the various options in a fair manner since this Android book-app project falls outside the scope of what I do at work. In addition, I should explicitly state: "The views represented here are my own and not those of my employer."

So, onward with the API evaluations (in Part II)

[Edit: added link to Part II]

Using the Android SDK on a Mac, Eclipse is really slow. How can I speed it up?

2010-06-16T10:32:00.000-07:00

Most useful link I've hit in the past few days:

http://stackoverflow.com/questions/2787055/using-the-android-sdk-on-a-mac-eclipse-is-really-slow-how-can-i-speed-it-up

Mac Eclipse was painfully slow (15 seconds to switch between tabs in the same project). Editing the eclipse.ini file as described made it usable again. Yay!

Using the Google Books API

2010-06-06T17:43:00.001-07:00

I recently started playing around with the Google Books API and ran into some issues that (at least to me) were not clear in the documentation.

Authentication

Perhaps it's just me, but I found the Authenticating section of the Books API to be somewhat difficult to follow. This is unfortunate since this step is absolutely necessary to do anything with the API.

I found the Getting Started page for accessing the Google Website Optimizer API to be a lot easier to deal with. These instructions were based on another document : Using cURL to interact with Google Data services. Basically, you create 2 scripts : one to record your authorization token and another to access the feeds using this token.

I've modified these scripts to work with the Books API. Note that these are shell scripts that work on Linux and Macintosh. For Windows, you'd need to create equivalent batch files to replicate this.

The first script is called getAuthToken.sh and is used to request and record your authorization token :

#!/bin/bash
read -p 'Username: ' username
stty -echo
read -p "Password: " password; echo
stty echo
curl -d  "accountType=GOOGLE&Email=$username&Passwd=$password&source=books-feed&service=print"  https://www.google.com/accounts/ClientLogin | grep 'Auth=' | sed  s/Auth=// > ~/.booksAuthToken
echo "Token: "
cat ~/.booksAuthToken

The second script is called getBooksFeed.sh and is a convenience wrapper to make it easier to access the feeds :

#!/bin/bash
authToken=`cat ~/.booksAuthToken`
url="http://books.google.com/books/feeds/users/me/$1"
curl --silent --header "Authorization: GoogleLogin Auth=$authToken" -L "$url" | tidy -xml -indent -quiet
echo ""

To use these scripts, first use getAuthToken.sh to get your authorization token. Then you can access the Books feeds by using getBooksFeed.sh.

For example, you can get a list of your collections with :

getBooksFeed.sh collections

and a list of books with :

getBooksFeed.sh volumes

Bookshelves

In January of this year, the Google Books team introduced "bookshelves", which replace the labels that could previously be added to books. With this change, your "My Library" now has a set of shelves. Individual books in your collection can belong to one or more bookshelves.

There are 5 bookshelves created automatically for you (and you can't delete them if you don't want them. Grrr..). These default bookshelves are :

"Reviewed" - This is your public bookshelf. It cannot be made private, so anything you place here will always be visible to everyone.
"Favorites" - This is your default bookshelf.
"Reading now"
"To read"
"Have read"

Unfortunately, the documentation for using Google Books API doesn't talk about how to access individual bookshelves. For example, to access your library, both the Google Books API Developer's Guide and the Google Books Data API : Reference Guide say to use :

http://books.google.com/books/feeds/users/me/collections/library/volumes

The problem is that this URL doesn't return all of the books in the user library — it only returns the books in the default bookshelf ("Favorites").

To get a list of the books in a different bookshelf, you need to use the following URL :

http://books.google.com/books/feeds/users/me/collections/<id>/volumes

which requires that you know the <id> of the bookshelf you want to search. You can get these IDs by requesting a list of bookshelves :

http://books.google.com/books/feeds/users/me/collections

This will return a list of all bookshelves in the user library and each entry will contain a URL that contains the bookshelf ID. The pre-defined shelves have numbers between 0 and 5, whereas user-defined shelves seem to start at 1001.

For my account, 0 = "Favorites", 2 = "To read", 3 = "Reading now", 4 = "Have read", and 5 = "Reviewed". Bookshelf ID = 1 doesn't seem to be defined.

So, using "library" as the <id> appears to be an alias for bookshelf "0" which is somewhat misleading since "library" implies all bookshelves, whereas it actually returns only those volumes in the "Favorites" bookshelf. My guess is that this is an artifact of the old pre-bookshelf days that didn't quite translate well.

Searching bookshelves

To search your library, the Developer's Guide says to use :

http://books.google.com/books/feeds/users/me/collections/library/volumes?q=bear

But this will only search the default ("Favorites") bookshelf. The proper way to search your entire library is to use :

http://books.google.com/books/feeds/users/me/volumes?q=bear

which will search all of your bookshelves.

To search a single bookshelf, you can use :

http://books.google.com/books/feeds/users/me/collections/<id>/volumes?q=bear

where <id> is the ID of the bookshelf to search.

Summary

Note: This section gives the parameter that you would pass to the getBooksFeed.sh script. If you need the full URL for these feeds, add the following at the beginning :

http://books.google.com/books/feeds/users/me/

To get a list of all your bookshelves :

collections

To get information about a specific bookshelf :

collections/<id>

To get a list of all the books in a bookshelf :

collections/<id>/volumes

To get a list of all your books :

volumes

To search all your books (in all bookshelves) :

volumes?<search-term>

To search within a specific bookshelf :

collections/<id>/volumes?<search-term>

HP Pavilion dv6

2009-07-06T22:00:00.000-07:00

This is a nice little laptop. The screen is pretty and it's not too big. It runs fairly cool (at least compared with my old HP laptop).

And the keyboard is nice. I wish is were backlit, and while I like the numeric keypad on the right, I don't think that I love it. With the old laptop, I was (subconsciously) used to aligning my right hand with the edge of the laptop, and I can't do that anymore. The touchpad is slightly offset to be more aligned with the main keyboard - a little odd at first, but not a big deal. I find that I'm sitting a little bit to the left when I use the computer so that I'm aligned with the keyboard as well.

But it's very nice so far.

New Laptop with Vista

2009-07-06T21:39:00.000-07:00

I was planning on skipping the whole Vista thing. Not that there was anything terribly wrong with it - just that it didn't really give me anything that I really needed.

We got a netbook earlier this year and I was glad that it came with XP because that meant that "things just worked". This is partially a comment on XP functioning nicely (thankyouverymuch) and partially a comment on family members expectations when using a computer. "Things just work" and "things are where they are supposed to be". Changing to a new OS means that things are going to move around and I had no great need/desire to figure out where everything had moved (and then show everyone else in the household).

Well, my XP laptop is now encountering problems and it'll take a while to get it back in working order. In the meantime, I needed a new laptop and CostCo was having a good deal on a HP Psvilion dv6. I was irritated with CostCo for being closed on the 4th of July (how could they do that to me!), but not irritated enough to go shopping elsewhere for a computer. So I waited an entire day without a computer.

And yesterday I brought it home.

Vista.

Now what? Do I play the "I'm too cool to use Vista" game and install XP on this machine? Or do I go straight to Windows 7 and hope that the various upgrades go smoothly?

Ah, I'm too lazy for that. Let's just see how Vista works. I'm sure it's perfectly usable...

Disabling startup items in WinXP

2009-04-05T07:18:00.000-07:00

Looking at the task manager this morning, I noticed a bunch of old/garbage items running -- like WDBtnMgr, which is a "button manager" for WD hard drives that was left over from when we briefly owned one (we returned it, partially because of the software it insisted on installing everywhere, and partially because it tried to be clever with power management and didn't always connect properly. In any case, why would I want a "button manager" for a hard disk?).

Anyway, while searching I found this site: http://www.wikihow.com/Alter-Startup-Programs-in-Windows-XP which shows how to use msconfig to disable the startup items.

But the most useful information on that page (for me, at least) is the link to ProcessLibrary, which is a list of all those cryptic process names with information about each one. Very useful!

Debugging C# NUnit unit tests

2009-01-14T19:11:00.000-08:00

I'm finally getting around to adding unit tests to a C# project I'm working on and got a chance to play around with NUnit. Yes, I know, I should've started adding these tests long ago (like when I started the project), but better late than never.

Using NUnit is quite easy:

Download the MSI from http://nunit.org (Current version 2.4.8)
Install
Add a Reference to "nunit.framework" in your project

Now for each class that needs a unittest, you add

 using NUnit.Framework;

at the top of the file and add a new class that you mark with a [TestFixture] attribute. This attribute is used by the NUnit GUI app to identify the classes that contain your unit tests.

Within your new [TestFixture] class, add tests by including methods marked with a [Test] attribute, and you can optionally have SetUp/TearDown methods to make your tests easier to manage.

Here is a stub [TestFixture] class with a single [Test]:

 [TestFixture]
 public class Utils_Test
 {
     [Test]
     public void Test_XXX()
     {
         // The unit test.
         Assert.IsTrue(true);
     }
 }

If you want to add the SetUp/TearDown methods, they are marked with their own attributes as follows:

 [TestFixture]
 public class Utils_Test
 {
     [TestFixtureSetUp]
     public void FixtureInit()
     {
         // Initialization for the entire TestFixture.
         // Called once at beginning before any tests.
     }

     [TestFixtureTearDown]
     public void FixtureCleanup()
     {
         // Cleanup for the entire TestFixture.
         // Called once at end of all tests (even if they throw exceptions).
     }

     [SetUp]
     public void TestInit()
     {
         // Init for each test.
     }

     [TearDown]
     public void TestCleanup()
     {
         // Cleanup after running each test.
         // Called even if the test throws an exceptions.
     }

     [Test]
     public void Test_XXX()
     {
         // The unit test.
         Assert.IsTrue(true);
     }
 }

But again, all you really need is the [Test] method.

Once you've done this and built your project, you can launch the NUnit GUI exe, point it at your assembly and it will find and run all (or some, if you choose) of your tests.

One problem is that when you encounter a failing test, you can't just jump in and debug your code. The free Visual Studio C# Express Edition doesn't allow you to attach to a running process like the non-free Professional versions of Visual Studio.

To get around this, I added the following TestSuite class to seek out (using reflection) and call all of the unit tests (just like what the NUnit GUI does).

 using NUnit.Framework;
 using System;
 using System.Collections.Generic;
 using System.Reflection;
 using System.Text;

 namespace MyApplication
 {
     public class TestSuite
     {
         public static void RunTests()
         {
             int nTests = 0;
             int nFailedTests = 0;

             foreach (Type t in Assembly.GetExecutingAssembly().GetTypes())
             {
                 if (t.GetCustomAttributes(typeof(TestFixtureAttribute), false).Length != 0)
                 {
                     // Gather test info.
                     MethodInfo mFixtureSetup = null;
                     MethodInfo mFixtureTearDown = null;
                     MethodInfo mSetup = null;
                     MethodInfo mTearDown = null;
                     List<methodinfo> mTests = new List();
                     foreach (MethodInfo m in t.GetMethods())
                     {
                         if (m.GetCustomAttributes(typeof(TestFixtureSetUpAttribute), false).Length != 0)
                             mFixtureSetup = m;

                         if (m.GetCustomAttributes(typeof(TestFixtureTearDownAttribute), false).Length != 0)
                             mFixtureTearDown = m;

                         if (m.GetCustomAttributes(typeof(SetUpAttribute), false).Length != 0)
                             mSetup = m;

                         if (m.GetCustomAttributes(typeof(TearDownAttribute), false).Length != 0)
                             mTearDown = m;

                         if (m.GetCustomAttributes(typeof(TestAttribute), false).Length != 0)
                             mTests.Add(m);
                     }

                     // Run tests
                     object obj = Activator.CreateInstance(t);

                     if (mFixtureSetup != null)
                         mFixtureSetup.Invoke(obj, null);

                     foreach (MethodInfo m in mTests)
                     {
                         nTests++;
                         if (mSetup != null)
                             mSetup.Invoke(obj, null);

                     try
                     {
                         m.Invoke(obj, null);
                     } catch (Exception) {
                         nFailedTests++;
                         Console.WriteLine(String.Format("Exception thrown in {0} during {1} test. ", t.Name, m.Name));
                     }
      
                     if (mTearDown != null)
                         mTearDown.Invoke(obj, null);
                 }
     
                 if (mFixtureTearDown != null)
                     mFixtureTearDown.Invoke(obj, null);
             }
         }

         System.Windows.Forms.MessageBox.Show(String.Format("{0} tests. {1} failed", nTests, nFailedTests),
                 "Unit test results");
         }
     }
 }

And then I add a call to

 TestSuite.RunTests();

somewhere in my application so that I can run the unittests from within the app.

So now when I encounter a unittest failure in the NUnit GUI, I can set a breakpoint at an appropriate place in the code and then run the unittests in the Visual Studio debugger.

It's not perfect, but it works.

[Note: older versions of NUnit had a [Suite] attribute that could be used to set up test suites, but this doesn't seem to be present in the most recent releases. In addition, you apparently needed to add each unittest to the test suite manually, which is just asking for problems with missing tests.]

Rescue Princess 2.0 Application

2008-11-19T16:45:00.000-08:00

Slides from a talk that Danc gave at the local Seattle chapter of IxDA (Interaction Design Association): "Building a Princess Saving App" and how it compares to application development in general.

http://lostgarden.com/2008/10/princess-rescuing-application-slides.html

Stairway to Satan

2008-11-19T13:05:00.001-08:00

This video from Michael Shermer (director of Sceptics Society, publisher of Sceptic Magazine) discusses why people believe strange things.

The talk is from the 2006 TED Conference. The entire (14 minute) talk is interesting, but skip ahead to 9:30 to hear the section on Led Zeppelin's Stairway to Heaven which supposedly has satanic messages when you play the song backwards. Interesting demonstration of how pre-conditioning can trick you into seeing/hearing/believing things that aren't really there.

http://www.youtube.com/watch?v=8T_jwq9ph8k

Waiting for 2009年3月...

2008-10-18T09:41:00.000-07:00

Dragon Quest IX (ドラゴンクエストIX 星空の守り人)

Developed by Level 5, published by Squenix. This video was taken at this year's Tokyo Game Show:

Dragon Quest IX trailer from TGS 2008 [1up]

Coming March 2009 for Nintendo DS (in Japan. No official date for US).

Ni no Kuni (ニノ国 - The Another World)

Developed by Level 5 with Studio Ghibli.

Ninokuni trailer [Kotaku]

Ninokuni promotional packaging [Kotaku]

Coming 2009 for Nintendo DS (No official dates for Japan/US release).

Have I mentioned how much I love Level 5?

Nintendo DSi and homebrew

2008-10-11T07:10:00.000-07:00

The most interesting aspect of Nintendo's recently announced DSi is the addition of SD slot. According to reports, this card is intended to store photos, video, music and games from the DSi Shop. So, wow! it sounds like Nintendo is actually allowing users to play games from a re-writeable memory card. This sounds perfect for homebrew.

It also sounds perfect for game piracy, which is why I'm doubtful that Nintendo would make this slot anywhere near as useful as it sounds. Aside from maximizing their market share, Nintendo's greatest desire is to eliminate game piracy for their systems.

Nintendo vs. the Aarrr! 4

Earlier this year Nintendo sued a group of companies in Japan for importing "game copying devices" like the R4DS. Because that's how they view them - simply as "game copying devices" with the implication that all game copying is illegal.

Of course, that's how they have to present their case. If they drew attention to the various uses for these cartridges which are clearly non-infringing like Moonshell, DS Organize, and the numerous homebrew titles, then they would be presenting a weaker (but more truthful) case.

Granted, it's likely (although I have no figures) that a significant number of R4 carts have at least one pirated game on them. And the "game backup" defense that is often used is weak because Nintendo made it impossible for normal users to create their own backups. This is important because the courts view a file you made yourself differently from a file you downloaded from the web. One is a backup and the other is not - even if the 2 files are identical and you own a legitimate copy.

Regardless, the legality of these carts (in the U.S. at least) will likely be determined by the precedent established in the Betamax case. This is where the "significant/substantial non-infringing use" defense comes into play. That is, the legality of these devices will depend on whether or not they support a non-infringing use like homebrew. Since all of these devices effectively enable homebrew, it probably more realistically comes down to whether or not there is a vibrant ("significant") homebrew community using these devices.

What about the DSi?

With the addition of music and video playback in the DSi, they're undermining the need for some homebrew like Moonshell. But there will always be a demand for other homebrew apps and games. If they really want to eliminate the R4, M3 and other carts, then they need to eliminate the need for these devices. That means having some Nintendo sanctioned method for writing your own games. With the addition of the SD card in the DSi, they now have a mechanism for doing this.

But will they do it? Sadly, I think that it's unlikely. My guess is that Nintendo has a love/hate relationship with homebrew. Actually, it's probably more of an "indifferent/hate" relationship. Some people in the company probably think it's cool what people are doing with their hardware, while others can only see dollars (and yen) lost to evil pirates.

And even if they do allow user-created games to run from the SD card, there are so many ways for them to screw it up and render it (mostly) useless. The most obvious mistake would be to not grant these SD card-based games full access to the hardware, but they could also require that apps be signed or some other nonsense. Nintendo just needs to look at how Apple is handling approvals for the iPhone App store for inspiration.

But this is all speculation. We know very little about what Nintendo is planning on doing with the DSi. My hope is that they will enable homebrew on the SD card, but I'm not holding my breath.

Converting videos for use with Moonshell (NDS)

2007-12-30T18:11:00.000-08:00

One of the tasks that I needed to do over the holidays was to get some Nintendo DS/GBA flash carts set up for some people who are just getting started in Gameboy/DS homebrew development. Even though the primary use for this card is for homebrew, I thought it would be nice to get some audio/video playback working as well.

So I installed the multimedia player (homebrew) app Moonshell onto the DS flash carts. The latest version of Moonshell can be found in Moonlight's NDS archive. As of this post, Version 1.71+1 (August 2007) is the most recent version.

To use image and music files with Moonshell, you just copy your JPG/MP3 files (or a few other formats) over to your flash card and Moonshell will show/play them just fine.

Video, however, is a different beast as Moonshell won't dynamically convert the format or the size - it expects the video file to be GBA screen-sized and to be encoded in MPG1 video/MP2 audio. This means that you need to convert the videos into DPG format, which is a special format specific to the NDS and Moonshell. To convert videos into this format, there are a few free tools available (presented in the order in which I discovered them):

dpgenc.exe - This is included in the dpgtools distribution that comes with Moonshell. A basic tutorial on how to use (an older version of) this is at can be found in the MaxConsole Forums.
BatchDPG - This is an alternative to dpgenc created by LS5, another homebrew developer. The official site still has the old original 1.0 version (with source), but updated versions from various people are also available (like Firon's 1.2 version). Be sure to review Yee and Firon's BatchDPG Guide because there are a number of pre-requisites (AviSynth, ffdshow, .Net Framework 1.1) that you'll need to have installed for this to work.
SUPER by eRightSoft - "Simplified, Universal Player Encoder and Recorder" - This is a general video/audio converter that contains special support for DPG files. It can be downloaded from the SUPER main page, although to get to the actual download link, you'll have to navigate through a number of links. If you get frustrated trying to find the download, here is a direct download link. The latest build is 2007.23 (4 July 2007).

All of these applications are basically GUI wrappers around ffmpeg or ffdshow, which are standard audio/video encoding/decoding libraries. The first two applications (dpgenc, BatchDPG) are specific for Moonshell DPG files whereas the last one (SUPER) is a general purpose tool that can be used for all sorts of audio/video conversions.

I tried all of these tools and had luck only with SUPER. There are clearly people out there using dpgenc and BatchDPG successfully, but I kept encountering problems with the conversion - probably codec related, but I didn't bother tracking them down since SUPER worked "out of the box" for my tasks.

Using SUPER to create DPG files...

To create DPG video files using SUPER, select "Nintendo - DS" from the dropdown list of output containers. This will simplify the interface by removing options that are not appropriate for DPG containers.

Select "Nintendo - DS" as the Output Container

Now you can drag a video file into the window to add it to the filelist. Press the "Encode" button and SUPER will chug away and drop the newly encoded video file into your output directory.

Drag your video files into the SUPER window

One nice thing about SUPER is that, while the interface is a bit busy, there is plenty of context-sensitive help that pops up when you hover over the elements in the UI. This makes it fairly easy to get started with the program and get things working. To set the output directory, for example, just hover over the Output box and a message will pop up telling you how to set the output options.

Bedlam DL3 : 10 years and 1 month ago today

2007-11-14T17:07:00.000-08:00

October 14th was the 10th anniversary of Bedlam DL3. These images are from the "I survived Bedlam DL3" t-shirt designed by Frank Lee, who was also in the Outlook group at the time Bedlam occurred.

The back of the Bedlam shirt is far more interesting - it contains choice extracts from some of the emails messages that were sent to the list.

...I see that I'm a member of this alias. I never subscribed to it, and would like to know:
a) am I supposed to be on it?
b) what is it?

Ditto!
ditto????
Ditto to the response below.
How big is this alias anyway?
Wider distribution...
same here...
Here, as well..
Here too.....
Yep. Over here, too.
yep
and here...
Myself also.........
And me.
Very good question
very very good question
in fact an excellent question
couldn't agree more
Yes, I also think so.
I concur...
Likewise...
Me too!
Me too. Any ideas?
Me, too. I have no idea, either.
me too? What is this?
Yeah, me too.
me three...
moi, aussi.
do not "me too" this is apparently a huge list
it would probably be a good idea to halt the "me too's" before they get out of hand.
At the risk of adding to the avalanche of "me too's:" Me too!
What does it mean?
It's everywhere...
I think there are a few of us...
And me.
spooky...
How odd.
Count me in on the puzzlement.
I am also not related to this alias.
I'm curious also.
OK - I am on it to. Thank you
I am now curious also.
I'm sort of liking it.
Can I be part of the fun too?
Ummm...why am I on this mail?
I'm taking care of this....
I'm still stumped.
Dear sir. Or madam.
How you doing?
what was the original question?
Uhhhh huhu huhu huhu....They said.."Member"

I don't know how I got on here either...and I'm in Tampa.
In Montreal too....
what is this message about?
Maybe if we keep this up we'll figure out what we all have in common...
Maybe someone else is trying to figure out what we all have in common......
Maybe we should all come up with a reason for being on this undecipherable alias
any explanations ???
Are we all redheads??
it has something to do with the seattle bedlam society.
It is a new High Society Club??
I think that someone's messing with us.
Perhaps this is a CIA plot...
You all have been selected by our race because or your special computer software skills and abilities
It's a conspiracy
It is Greatly Annoying.
I'm not certain how all of you got on our little football pool
CAN you all please STOP sending mail to the whole alias????
I can't resist--I have to add.
please do not announce your membership to the alias.
we are wasting many people's time!
Please stop replying all or emailing Bedlam DL 3.
PLEASE - no reply all.
Please, let's all stop sending messages on this alias
Would everyone please stop replying to all on this thread
I suggest that we all stop replying to this alias.
lets don't blast back with reply alls
hummm we all seem to be thinking the same thing.
exclude my name please
Exlude my name too, please!!!
It is unnecessarily cluttering inboxes of everyone on the alias.
I do not know why I am on it of what it is.
It's forced me to learn how to use the Inbox Assistant to auto-delete.
I guess I've joined the club too.... are there dues? :)
I'm shocked at the incompetence of all those how have sent silly responses to the list.
Please spam me when I'm done with this mail. I deserve it.
My mom doesn't know what the Bedlam DL3 alias is for.
I'm thinking of having a party at the Speakeasy for all members of the "Bedlam" alias.
How come we're Bedlam 3? Aren't we good enough to be Bedlam 1?
"Bedlam" relates to insanity
"Stop the insanity"

PLEASE, NOBODY SEND ANY MORE MAIL TO THIS ALIAS. EVER.

Reply to All.
October 14, 1997

Ah. Those were the days... For a snapshot of what it was like to be in the Outlook group at that time, check out Frank's Our LTD Album pages - the LTD was our official Outlook vehicle for a while.

OK. If I'd've been paying attention, I would've posted this exactly on the 10th anniversary. But I'm not cycling though all my t-shirts as often as I used to (probably because I have more of them now), so I didn't see my reminder t-shirt until late October.

[It looks like someone on Channel9 already posted images of the shirt back in 2005 (although all I see are broken images). Oh well.]

[Update: Fixed a typo and the image links]

Milking goats on your DS

2007-10-03T09:43:00.000-07:00

Yaknow, I was thinking just the other day that there weren't enough goat-milking games for the DS and lo! I see these new screenshots for Konami's upcoming "Lost in Blue 3".

From some of the other released screenshots (see gamebrink or nintendic), it looks like this game also affords you the opportunity to be attacked by leopards or to sing with dolphins.

I wonder if there's a Wii version coming out...

Anime Shopping Guide to Tokyo

2007-05-13T06:15:00.000-07:00

Last month, Google launched My Maps, which provides a basic point-n-click interface so you can create map annotations and then save them as custom maps. For the launch of My Maps, I created a couple of demonstration maps. One of them, Les Arrondissements de Paris, is easy to find in the Mes Cartes tab of maps.google.fr.

But the other map I created was (I felt) far more interesting and useful. It's the Tokyo Anime Shopping Guide and it contains markers, links and photos of interesting places to visit on your next anime shopping junket to Tokyo.

If you have any suggestions/updates/contributions that you'd like to see included in this map, let me know. I'll be updating this map whenever I visit Tokyo - which unfortunately isn't as often as I'd like.

XNAExtras - What's going on

2007-04-30T07:27:00.000-07:00

Many people have been asking me about the current status of XNAExtras. The short answer is that it is currently in Microsoft's hands - they own the code and if they decide to release it then I can continue doing some work with it.

Teaching GBA Homebrew to Kids

2007-04-28T08:51:00.000-07:00

This past Thursday (April 26th) was Take Your Offspring to Work Day*. This is where you bring your young-uns in to work and then you show them what you do for your job. Rather than have them come in and watch me browse the web all day, I thought it would be nice to run a class for all the kids that were coming into Google's Kirkland office.

I've been putting together a simple Intro to Programming class for a local Montessori school so this seemed like a perfect opportunity to test things out before the trial run this summer.

The full class targets 10-12 year olds and starts with a basic introduction to computer engineering (binary, bytes, how things are represented in memory, ...), and then some gaming stuff like creating your own sprites, and then some code to move them around the screen, and then either animation or more coding depending on what direction the kids want to go.

For the 1-hour TYOTWD version, I needed to simplify things a bit so I dropped the computer engineering intro and replaced it with a demo of The Legend of Zelda - The Minish Cap running in an emulator. I demoed the game using the debugger version of No$gba so that I could show how the different background layers and sprites were used in the game.

After that, I showed them how to create their own sprites using Spritely, which is a sprite editor that I am writing. Now there are a number of nice tile/map editors out there, but none of them was perfect for what I wanted to do - they are either too complicated, or force you to work with tiles (instead of sprites), or cost money, or are really map/background editors, or have some other problem.

I wrote Spritely with a mind to make it easy to create GBA/NDS sprites, and I made a bunch of simplifying assumptions along the way. It only handles 8x8 pixel tiles, only allows the sprite sizes supported by the GBA/NDS, and restricts you to 16-color palettes. But it does allow you to directly edit multi-tile sprites.

In addition, Spritely has an option to generate the source code for a complete GBA or NDS project (including makefile) that you can compile and see your sprites in action. The generated project doesn't do much - it just allows you to move your sprites around - but it demonstrates the basics. The generated code requires that you have the latest version of devKitPro installed, but that's OK since I pre-installed everything needed on the laptops that the kids were using.

So in summary, I had the kids:

Create some sprites using Spritely
Export the GBA/NDS project from Spritely into a directory
Open a command prompt and "cd" to that directory
Type "make"
Double-click the .gba or .nds ROM to run the program in an emulator (either VisualBoyAdvance or the standard version of No$gba)

Yes, that's right. I had 10 year olds pulling up a command prompt and typing "make" to build their project. And it worked.

I also had a couple of cartridges available so that they could see their creation running on a real GBA/NDS. One 7-year old walked around with an NDS playing her "game" for (what seemed like) half an hour. Google was also nice enough to buy a few NDS cartridges to give out as prizes.

Note that even though I was targetting 10-12 year olds, there were kids ranging in age from 6 to 16 present. The younger kids needed their parents (or siblings) help with the computer, but had no problem creating their own sprites once they were shown the basics. I had hoped to go into the source code a bit with the older kids, but didn't have time.

Now, the TYOTWD class was slated to take an hour, but we kept an extra hour at the end for people who wanted to stay on and keep working. We ended up having to chase the kids off to the next event after nearly 2 and a half hours.

So it seems like it was a success. It started out a bit rough because my laptop wouldn't connect to the projector properly, and we didn't have One Laptop Per Child so the kids had to share, and the room was a bit cramped. But all-in-all it looked like the kids were excited to be there. It was nice to overhear some of them talking to their parents afterwards: "I hope you got all that information dad, because I want to keep doing this when we get home".

Hopefully at least one of the kids will continue their interest in programming. Maybe we'll see some innovative GBA/NDS homebrew coming out a few years from now.

As a final note, the kids were using a preview version of Spritely. I'll be releasing it widely once I fix a few obvious flaws and add a few more features.

* Take Your Offspring to Work Day (TYOTWD) is also known as Take Your (Daughters / Daughters and Sons / Spawn) to Work Day
GBA = Gameboy Advance
NDS = Nintendo DS

Unemployed and uninsured

2006-12-02T06:45:00.000-08:00

"So what do you do for a living?"
"I'm... uh... well, I'm between jobs at the moment."

My last day at Microsoft was yesterday, and I don't start my new job at Google for a whole 2 days. That'll give me plenty of time to fix up all the stuff in the house that needs fixin'.

Start page

2006-12-01T06:00:00.000-08:00

This blog starts here...