Which book API to use for a mobile book app? (Part II)

(...continued from Part I...)

Here is an evaluation of the various Book APIs available with regard to their suitability for use in a standalone Android application. Please see Part I for more background information - this post contains just the raw evaluation and conclusions.

Amazon

Previously called "eCommerce Service API" (or ECS), Amazon's Product Advertising API is part of the Amazon Web Services (AWS), although the correct link is challenging to find if you start from the top AWS link.

Developer Guide

Access
Provides basic Item Search/Lookup and List Search/Lookup. Lists aren't exactly the same a bookshelves, but I suppose that they're close enough.

Read/Write
As far as I can tell in the API, there is no way to add a new list or add an item to an existing list. You can, of course, edit/modify the shopping cart, so write access is provided by the API. It seems that Amazon wants the API to be read-only unless you're trying to buy something.

Auth
Each app that wants to use the API needs to register for an AWSAccessKeyId and then use that ID in all requests.

I didn't investigate how one would access non-public lists using this API. Presumably this would require some further authentication, but the API restrictions made me stop evaluating.

Restrictions
From the license agreement, the purpose of the API is to "permit you to advertise items offered on the Amazon Site". Each app that wants to use the API requires enrollment and approval by Amazon (it's apparently not an automated process).

The license mentions that

Unsuitable applications include those that:
(a) do not have as their principal purpose advertising and marketing the Amazon Site and driving sales of products and services on the Amazon Site;

!!

Conclusion
Given the access limitations and the license requirements, this API does not seem suitable for use for this purpose. It might be worth considering in the future if I want to extend this app by adding a link to an Amazon page.

aNobii

aNobii API Home

Access
aNobii has 3 API calls: item.getInfo, shelf.getSimpleShelf and contributor.getInfo.

These seem like they would be enough for a basic read API, but I was unable to get them to work correctly (I kept getting getting a code 401 "Missing parameter(s)" error - was my api_key wrong or was I passing the wrong book id, I have no idea).

Read/Write
Read-only.

Auth
Requires a devkey with an extra "secret" id that you combine and then run through a MD5 hash to create your real API key.

Restrictions
The terms of service include the following:

b. You shall not:
Use aNobii APIs for any application that replicates or attempts to replace the essential user experience of aNobii.com.

Since I'm sure a mobile book management app could easily be construed as replacing the "essential user experience" of aNobii.com, it's probably
not valid to use the API for this.

Conclusion
As with all of the APIs that require a devkey, there is an issue with assigning a devkey to an application that is distributed (like a phone app). Even though each app is run by a separate user, they all are counted toward the same app id.

Perhaps this would work if each user could get easily get their own developer key + private key. The user would enter this info once when they installed the app.

In any case, since this is read-only (and didn't seem to work properly for me), I'll probably pass on this for now since the APIs provided by other sources provide more functionality.

GoodReads

Access
Nice API with access to most things that you could want. Book info, shelf info for a user, list of books in a shelf.

Examples:

• Get a list of shelves:
  http://www.goodreads.com/shelf/list?format=xml&v=2&key=<devkey>&user_id=<id>


• Get list of books in shelf:
  http://www.goodreads.com/review/list?format=xml&v=2&key=<devkey>&id=<id>&shelf=<shelf-name>

Read/Write
Both reads and writes are supported. Writes require authentication.

Auth
Most API calls require either a devkey or require that a user be authorized via OAuth.

Restrictions
License is pretty standard. API queries are limited to no more than 1 query/second.

Conclusion
GoodReads has a well thought-out API and handles authorization properly. The only concern is that some are OAuth-based while others are devkey based. Since the devkey based accesses are limited to 1 query/second, this may be a problem when incorporating calls into a standalone app (since the app would be installed on lots of different end-user devices).

However, this definitely seems like something worth pursuing.

Google Books

Google Book Search API - Getting Started Guide
Google Book Search Data API : Developer's Guide

Access
Basic book info and shelves (with shelf editing) are supported.

Read/Write
Book data can be read and user shelf info can be read/written (with authentication).

Auth
General book queries don't require authorization, but to access user-specific data (like bookshelves) it is necessary to have the user authenticate. All writes also require authentication.

Since the Google Books data is exposed using the Google Data protocol, this authentication can be done using either AuthSub, OAuth or ClientLogin, as described in the Google Data Authentication page.

Restrictions
The Terms of Service are long and "legal" and deal mostly with requiring that you don't misrepresent yourself (or your app), that you respect Google trademarks and the privacy of the users and that you abide by all applicable laws. Most of it seems pretty boilerplate for a web service.

No queries/second restrictions, although I suspect if your app started DoSing the service it would get cut off rather quickly.

The only item that stood out is that Google reserves the right to insert advertising in the content. This isn't the case with any of the Google Books APIs that I've ever used and I'm not sure how that would work when using a simple text-based API. In any case, this doesn't seem to be an issue but it's worth being aware of.

Conclusion
In order to securely access user data (like bookshelf info), having a system that allows the end-user (rather than the app developer) provide credentials is essential. This is one of the few APIs that supports this.

ISBNdb

Developer API

ISBNdb crawls various libraries to build/maintain a database of ISBNs.

Access
Only provides basic ISBN book lookup.

Example:

http://isbndb.com/api/books.xml?access_key=<key>&index1=isbn&value1=<isbn>

Read/Write
Read-only.

Auth
Dev key required to access.

Restrictions
None mentioned on site.

Conclusion
Not worth investigating further for this application since this info seems readily available elsewhere with fewer restrictions.

LibraryThing

LibraryThing APIs

LibraryThing has a collection of different APIs rather than one coherent API. They are divided into 4 basic categories: Covers, JSON, Webservices, and Misc.

Covers

• Used to get book covers given an ISBN.
• Requires a devkey and is limited to 1000 queries/day

JSON APIs

• Books API
  
  • This allows you to lookup a book on LibraryThing.
  • The Books API page shows 2 different example formats:
  • http://www.librarything.com/api/json_books.php?userid=<user-id>&key=<key>&showstructure=1
  • http://www.librarything.com/api_getdata.php?userid=<user-id>&key=<key>&showstructure=1
  • They both return basically the same info, but the "json_books" version includes a bunch of widget-creating JavaScript at the top before the data begins.


• Works (book search) API
  • This is a book search API and the only documentation relating to it is in this blog post
  • This doesn't require a developer key.

WebServices APIs

• This appears to be a work-in-progress since it only provides access to a LibraryThing works and authors. I didn't see way to access user lists.
• This requires an apikey and is limited to 1000 queries/day.
• Example:
  • http://www.librarything.com/services/rest/1.0/?method=librarything.ck.getwork&apikey=<apikey>&id=<librarything-id>

Misc APIs

• Easy Linking: Link to LibraryThing using ISBN (rather than LibraryThing book id).
• ThingISBN: given an ISBN, return a list of related ISBNs
• ISBN -> LibraryThing book id
• ISBN -> book language
• ISBN validate
• Title -> list of possible ISBNs

Access
These APIs provide basic access to book data. However, none of the APIs appears to provide access to the user collections. Shelf functionality could be re-created using the tags (which are available in the API), but it seems odd not to use LibraryThing's native "shelf" concept

Read/Write
Mostly read access but write access of sort-of there. You can add a book using:

http://www.librarything.com/addbook/<isbn>

But this assumes that you are already logged in, so it doesn't really work for a standalone app.

Auth
Devkeys are required for the JSON and WebServices API calls (although different keys are needed for each). No real Authentication is provided to access user data.

Restrictions
Most API calls require a devkey and they're all limited to 1000 queries/day.

Some of the license requirements are quite odd. For example, the JSON Books API actually says the following:

We have yet to settle on a license. You agree to abide by whatever license we eventually choose.

!!

In addition, both of the JSON APIs have the following in the license:

Must be run as Javascript on user's browser, not fetched by a server; cannot be stored, except for browser caching. Commercial use requires a hard (non-JS) link to LibraryThing on every page that returns results.

Which is clearly not the case if I want to run call this from a phone app.

Conclusion
Overall, while there are many things to like about what LibraryThing provides, there are too many parts that don't seem mature enough to support a phone app based on the current APIs. In addition, the licensing requirements for some of the APIs are best described as quirky and certainly preclude creating things like phone apps.

Still, I was left with the impression that there's some useful stuff here and it might be useful to add some sort of LibraryThing integration.

OpenLibrary

OpenLibrary APIs:
REST API
Books API
Covers API

Access
Basic access to books. E.g., using:

http://openlibrary.org/api/books?bibkeys=ISBN:<isbn>&details=true

No access to lists since it's a new feature that they're still working on.

Read/Write
Read only.

Auth
OpenLibrary allows you to login automatically by POSTing to http://openlibrary.org/account/login. This sets a login cookie and is apparently intended for web access.

The API doesn't have or require authentication.

Restrictions
None mentioned on the site.

Conclusion
Issues:

• Lack of support for lists
• Lack of write support

Not worth investigating further for this application.

WorldCat

WorldCat Search API
WorldCat Basic API

Access
Provides basic access to books, but doesn't appear to allow access to user created lists.

Read/Write
Read-only.

Auth
Requires a devkey for all access

For example to do a simple lookup by ISBN:

http://www.worldcat.org/webservices/catalog/content/isbn/<isbn>?wskey=<key>

where <key> is your assigned key.

Restrictions
Limited to 1000 queries per day.

Conclusion
Issues:

• Devkey + query limit make this inappropriate for an app unless each user gets their own devkey
• Lack of access to user created lists
• Lack of write functionality

Not worth investigating further for this application.

---

Whew! I'm glad that's done. Please let me know if you see any errors or omissions.

I need to say that I'm actually a bit surprised that more APIs didn't make the cut and with how critical the Auth problem ended up being. I was also kinda hoping that LibraryThing would do better than it did.

But now it's time to start playing more with the 2 finalists (GoodReads and Google Books) to see how well they work in practice. Perhaps I'll experiment a bit more with LibraryThing as well.

Which book API to use for a mobile book app? (Part I)

So I'm starting to look into creating a book management app that will run on my Android phone. There are a few of these apps available in the Android Marketplace, but none of them do quite what I'm looking for.

What am I looking for? Well, I have a few requirements:

• Needs to be able to add new books easily (via barcode scanner)
• Should automatically get book info from online sources
• Should allow books to be arranged in "shelves" or tagged
• Should store my list of books online

The first few requirements are fairly easy - it's the last one that makes things interesting. While it'd be far easier for me to create (yet another) local database to store this information, there are some big advantages to having everything stored "in the cloud":

• The data is available on other devices (via the website)
• The data is backed up (if something happens to my phone)
• It's easy to share
• ...

Given this requirement, I could either write my own online data storage app or I can take advantage of the already-existing services out there.

"Writing my own" is not as bad as it sounds. A simple AppEngine app could be thrown together to handle this. The problem is that while this would work for me with my books, it wouldn't be able to handle other people's books (without running into the free service limits). The code could be open-sourced to make it easy to others to create their own instance of this web app for their own books, but that's starting to become needlessly complicated.

So using an existing online book manager is the right path to take. Fortunately, there are a bunch of decent ones out there to choose from that seem to provide (at least part of) what is needed:

• Amazon
• aNobii
• GoodReads
• Google Books
• ISBNdb
• LibraryThing
• OpenLibrary
• WorldCat

The trick is to find which one (or more) supports the features that I want. Note that there are a few other services out there (like Shelfari) not listed because they don't have a developer API. If you know of any others that are worth investigating, please let me know.

Now, in an ideal world, this (proposed) phone app would be a simple front-end and the user would be able to select from a number of different online book managers to use as a backend. While this is great for users (they can pick/choose their favorite online book management site), the sad reality is that this sort of things is challenging for (mostly) technical reasons. This is where a standard interface for accessing personal libraries would come in handy. But, alas, such a thing does not exist.

So how can I choose from amongst the available options? It seems I need to sit down with each of them and see what they're capable of and then compare that with my requirements listed above.

To do that, I've broken things out into 4 sections and evaluated each API. These four sections are : Access, Read/Write, Auth and Restrictions.

Access
Quite simply this evaluates whether or not the API provides access to the required data.

In its simplest form, a "user library" app needs a list of books (ISBNs or EANs) and way to arrange them into groups (bookshelves or labels).

More generally, my app needs access to an API that provides the ability to:

• Get general book info
  • Given an ISBN, get the title, author, pub info, thumbnail, ...
• Get user specific info
  • Get a list of bookshelves (or get a list of labels)
  • Get a list of books in a bookshelf (or get a list of all books with a particular label)

Read/Write
Technically, this is part of "Access", but I broke it out for some reason during the evaluation and ended up leaving it that way. Since all APIs (pretty much by definition) provide read access, the real question here is "do they provide write access?".

The write access required is:

• Update user specific info
  • Add a new bookshelf (or label)
  • Add a new book
  • Add/update user info for a book

The bare minimum is to be able to add a new book to a shelf. The other 2 are 'would be nice's.

Auth
This is where things start getting tricky. "Auth" is a combination of "authentication" (is the user who they say they are) and "authorization" (does the user have access to the requested resource).

So the question is : "How does the API authenticate/authorize the user to provide access to the data?"

Often there will be different levels of Auth provided by the same API. To get general book info, the API doesn't need to know who you are. But to get (or update) info on a private shelf, it needs to verify that you are the owner of that shelf.

As a preview of things to come, this is the area where most APIs fall down. The most common problem being that they attempt to authorize the application (through the use of a devkey or something similar) rather than the user of the application.

Restrictions
This (almost) final section discusses limits and licensing restrictions on the API.

Most of the restrictions in the license agreements are pretty typical. Things like:

• You can't use the API to harvest data
• You can't relicense the data
• You agree not the break the law
• Not for commercial use

So I'll only be breaking out issues that either fall outside what is typical or that may impact the app.

The 2 common issues that pop up here are:

• Acceptable use for the API - Does the agreement forbid what I'm trying to do?
• QPS (queries/second) restrictions - Are the QPS restrictions low enough to cause concern?

Conclusions
And finally, each sections ends with a brief set of conclusions. It's important to note that these conclusions do not evaluate the service as-a-whole, but rather specifically to their suitability for this particular application.


Public disclosure: I should note that, while I do work at Google, I do not work on the Android or Google Books teams, so I'm not tied to promoting these particular products. My hope with this evaluation is to compare the various options in a fair manner since this Android book-app project falls outside the scope of what I do at work. In addition, I should explicitly state: "The views represented here are my own and not those of my employer."

So, onward with the API evaluations (in Part II)

[Edit: added link to Part II]

Using the Android SDK on a Mac, Eclipse is really slow. How can I speed it up?

Most useful link I've hit in the past few days:

http://stackoverflow.com/questions/2787055/using-the-android-sdk-on-a-mac-eclipse-is-really-slow-how-can-i-speed-it-up

Mac Eclipse was painfully slow (15 seconds to switch between tabs in the same project). Editing the eclipse.ini file as described made it usable again. Yay!

Using the Google Books API

I recently started playing around with the Google Books API and ran into some issues that (at least to me) were not clear in the documentation.

Authentication

Perhaps it's just me, but I found the Authenticating section of the Books API to be somewhat difficult to follow. This is unfortunate since this step is absolutely necessary to do anything with the API.

I found the Getting Started page for accessing the Google Website Optimizer API to be a lot easier to deal with. These instructions were based on another document : Using cURL to interact with Google Data services. Basically, you create 2 scripts : one to record your authorization token and another to access the feeds using this token.

I've modified these scripts to work with the Books API. Note that these are shell scripts that work on Linux and Macintosh. For Windows, you'd need to create equivalent batch files to replicate this.

The first script is called getAuthToken.sh and is used to request and record your authorization token :

#!/bin/bash
read -p 'Username: ' username
stty -echo
read -p "Password: " password; echo
stty echo
curl -d  "accountType=GOOGLE&Email=$username&Passwd=$password&source=books-feed&service=print"  https://www.google.com/accounts/ClientLogin | grep 'Auth=' | sed  s/Auth=// > ~/.booksAuthToken
echo "Token: "
cat ~/.booksAuthToken

The second script is called getBooksFeed.sh and is a convenience wrapper to make it easier to access the feeds :

#!/bin/bash
authToken=`cat ~/.booksAuthToken`
url="http://books.google.com/books/feeds/users/me/$1"
curl --silent --header "Authorization: GoogleLogin Auth=$authToken" -L "$url" | tidy -xml -indent -quiet
echo ""

To use these scripts, first use getAuthToken.sh to get your authorization token. Then you can access the Books feeds by using getBooksFeed.sh.

For example, you can get a list of your collections with :

getBooksFeed.sh collections

and a list of books with :

getBooksFeed.sh volumes

Bookshelves

In January of this year, the Google Books team introduced "bookshelves", which replace the labels that could previously be added to books. With this change, your "My Library" now has a set of shelves. Individual books in your collection can belong to one or more bookshelves.

There are 5 bookshelves created automatically for you (and you can't delete them if you don't want them. Grrr..). These default bookshelves are :

• "Reviewed" - This is your public bookshelf. It cannot be made private, so anything you place here will always be visible to everyone.
• "Favorites" - This is your default bookshelf.
• "Reading now"
• "To read"
• "Have read"

Unfortunately, the documentation for using Google Books API doesn't talk about how to access individual bookshelves. For example, to access your library, both the Google Books API Developer's Guide and the Google Books Data API : Reference Guide say to use :

http://books.google.com/books/feeds/users/me/collections/library/volumes

The problem is that this URL doesn't return all of the books in the user library — it only returns the books in the default bookshelf ("Favorites").

To get a list of the books in a different bookshelf, you need to use the following URL :

http://books.google.com/books/feeds/users/me/collections/<id>/volumes

which requires that you know the <id> of the bookshelf you want to search. You can get these IDs by requesting a list of bookshelves :

http://books.google.com/books/feeds/users/me/collections

This will return a list of all bookshelves in the user library and each entry will contain a URL that contains the bookshelf ID. The pre-defined shelves have numbers between 0 and 5, whereas user-defined shelves seem to start at 1001.

For my account, 0 = "Favorites", 2 = "To read", 3 = "Reading now", 4 = "Have read", and 5 = "Reviewed". Bookshelf ID = 1 doesn't seem to be defined.

So, using "library" as the <id> appears to be an alias for bookshelf "0" which is somewhat misleading since "library" implies all bookshelves, whereas it actually returns only those volumes in the "Favorites" bookshelf. My guess is that this is an artifact of the old pre-bookshelf days that didn't quite translate well.

Searching bookshelves

To search your library, the Developer's Guide says to use :

http://books.google.com/books/feeds/users/me/collections/library/volumes?q=bear

But this will only search the default ("Favorites") bookshelf. The proper way to search your entire library is to use :

http://books.google.com/books/feeds/users/me/volumes?q=bear

which will search all of your bookshelves.

To search a single bookshelf, you can use :

http://books.google.com/books/feeds/users/me/collections/<id>/volumes?q=bear

where <id> is the ID of the bookshelf to search.

Summary

Note: This section gives the parameter that you would pass to the getBooksFeed.sh script. If you need the full URL for these feeds, add the following at the beginning :

http://books.google.com/books/feeds/users/me/

To get a list of all your bookshelves :

collections

To get information about a specific bookshelf :

collections/<id>

To get a list of all the books in a bookshelf :

collections/<id>/volumes

To get a list of all your books :

volumes

To search all your books (in all bookshelves) :

volumes?<search-term>

To search within a specific bookshelf :

collections/<id>/volumes?<search-term>