First, the boiler plate for Starboard, one of my Flatiron School projects:
What is it?
A modern-take on an old fashioned gold star board.
Why does it matter?
The Flatiron School is hard. Throughout the semester students are learning new material, working on github projects, writing blog posts, completing online coding tutorials, and much more. Without having much time to realize how much has been accomplished, Starboard becomes the hub to track all of their achievements. From completing Codeschool courses, to committing to an open source project, to giving someone else a star for help with a bug fix, Starboard is an app to collectively celebrate learning to code.
One of my favorite problems to solve when building the application was how to normalize data (creating user achievements) for actions taken on other websites such as completing a Codeschool course or making a commit to an opensource project. The complete gist is below.
Each service we used would return a uniquely formatted set of data. Treehouse and Codeschool also have 0 documentation for their APIs. Figuring out how to manipulate these different data sets was the engine driving our application. Here are the steps I suggest you take when building API wrappers:
Explore the API in your browser
Without writing a line of code, read the documentation (if there is any) and ask for json objects back. For Codeschool, all that was required was a json get request for a user profile’s information: http://www.codeschool.com/users/dolinsky.json
Play with the data in console
Using the json and open-uri gems, test manipulating the data in console. How is the json structured? What keys do you need to chain together in order to get to the values you are looking for? In our case, we were looking for the ‘title’ of the ‘completed’ ‘courses’.
Decide on your output structure
Once you have a solid understanding of what you can get back from each API, begin to think through how you can normalize the data to use in your own application and save to your database. For instance, would a hash work best with sets of key/value pairs or a simpler array? We decided that an array would be best for us. At this point, all we knew was that each wrapper (Codeschool, Treehouse, Github, Blog) would need to return an array of strings. As an example, Codeschool could return [“Rails for Zombies”, “Ruby Bits”], while Github could return [“Committed to an open source project”, “Had a repo forked”]. The individual wrappers will contain their own methods for getting us to the final array, but in the end, an array will be needed.
Before an array is returned from a wrapper, a method needs to be called to ask for data from an API. As you can see in the gist below, each of our wrappers has the ‘get_data’ method. So, we could call things like Treehouse.get_data(username) or Blog.get_data(blog_url). Test the method by itself. Write additional methods needed within the wrapper’s class to help you manipulate the data for your application. For example, Codeschool.get_data creates a hash from the json returned and then calls the get_completed_courses method to iterate through the hash and return an array of completed courses.
Here’s our get_data method for Codeschool:
1 2 3 4 5 6 7 8 9 10 11 12 13
Create services hash
After creation of a user, we call user.get_external_data in ther user model. This method contains a hash of service => identifier
1 2 3 4 5 6 7 8 9
Call individual wrappers
Once you have your hash, iterate through it, calling your predefined get_data methods.
1 2 3 4 5 6 7 8 9 10 11 12 13
Save your data
Now comes the fun part. We have the data needed stored in arrays for each service and we can begin storing the data how we see fit in our application. In our case, we created a method called check_achievement_by_array.
1 2 3 4 5 6 7 8
You’ll see that the code looks slightly different in the gist below, but don’t worry about that. We are using Sidekiq to offload these API calls to a background service (another blog post to come). You can make these API calls work without Sidekiq.
For each element in the returned array, we then call check_achievement_by_string. We are essentially checking to see if we have the given star in our database already, if that user already has that star, and if not, create the ‘achievement’ for that user.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Check your database
Does your data look the way you want it to look? Is it there? It may not be. We fixed many bugs along the way to actually getting our wrappers to work. Whether it’s the data itself, the way you are manipulating it, or an error from the API, you will need to work through these bugs to eventually store the data you need. But, it’s worth it.
I hope this helps as an example for those who are depending on mulitiple API’s to drive their own application. Feel free to comment below with other suggestions!