New Posts Coming

16 01 2012

Ok, so I have been a bit neglectful over here at Perlita Labs. I got a little distracted with work, life, other projects, etc… but I plan on some more posts this year.

Over the past year I spent a bunch of time working with Drupal and Flash. All of my creations were being put on my  Family Art Studio site where you could use the Flash tools to create art.

My interest is now turning to Android development and I figured I would resurrect Perlita Labs for this work.  Here is the link to my first app: The Legend of Chess.  I will add a post on it as things progress.

Using Social Graphs to Visualize Political Factions

21 06 2010

A couple great articles from O’Reilly on using social graphs to visualize data.  Andrew Odewahn revisits and older project in  Visualizing the Senate social graph, revisited – OReilly Radar.  The basic notion is to go through senate voting sessions with each senator as a node.  When there is a pattern of frequently having similar votes a node is drawn.  He does this over an number of different periods giving insight into the political climate.  Definetly watch the video.

Then he goes into how to improve upon the effort.  The second post shows how to code this using Processing for a more interactive social graph analysis.

Add to DeliciousAdd to DiggAdd to FaceBookAdd to Google BookmarkAdd to RedditAdd to StumbleUponAdd to TechnoratiAdd to Twitter

Is MS Pivot a New Tool or a New Paradigm

17 03 2010

Microsoft Pivot is a new data exploration\visualization tool that allows rapid interaction with enormous sets of information.  The tool’s capability is impressive, but some may say inevitable with the convergence of large data volumes, powerful databases, and web technologies.  What is just as interesting is that Pivot, and tools like it, will change how we navigate, explore and come to understand information (why data visualization is important).

Screenshots of Microsoft PivotPivot uses the metaphor of a collection.  It seems each item in the collection has attributes that can be used to filter, or to distribute (think bar chart) the items in that collection.  Collection items can also be linked to one another to rapidly navigate to similar items.  Words really do not do this justice and you should look at the Pivot Demo or Gary Flake’s TED Talk on Microsoft Pivot.  This tool moves at the speed of thought and when you follow a hunch you get a response back quickly that lets you continue without interruption.

When a user is able to work with data without their thought process being interrupted the “tool” starts to disappear.  We have all heard the cliché that a car disappears and the driver is one with the road.  This happens to users and analyst working with data as well.

This is not a complete new concept.  Corporate analyst have had Cubes and other Data Warehousing systems provide them with similar capabilities.  In general these have been expensive and developer intensive solutions.  What Pivot and tools like offer is a way to bring this to the masses.

We will all become comfortable immersing ourselves in profession and personal data.  It may be how you navigate your bank transactions or choose a school for your children.  Your employer may expect you to use this skill for your job, or at least to clean out your inbox.

Data exploration with the help of new technologies will become mainstream in ways we cannot imagine today.


Pivot Demo

TED Talk on Microsoft Pivot

Sea Dragon Showcase

Add to DeliciousAdd to DiggAdd to FaceBookAdd to Google BookmarkAdd to RedditAdd to StumbleUponAdd to TechnoratiAdd to Twitter

Mapping TIGER Stimulus Grants with Simile

19 02 2010

Today the U.S. Department of Transportation announced the cities that would be awarded with Transportation Investment Generating Economic Recovery (TIGER) Grants.  This seemed like a great opportunity to apply some visualization tools to a new data set.  The data was manually pulled out of the official PDF announcing the TIGER Grants and put into a Google Spreadsheet.  Then it was quick to together a SIMILE Exhibit that pointed to the data.

You can find the resulting map and table at Blldzr.


I create a similar TIGER map using Tableau Public.

Add to DeliciousAdd to DiggAdd to FaceBookAdd to Google BookmarkAdd to RedditAdd to StumbleUponAdd to TechnoratiAdd to Twitter

Importing Google Spreadsheet Data into Processing

17 02 2010

It is really powerful to be able to connect your Processing applet to external data sources.  This enable art infused with data and other great mash-ups.  I have done a couple of different projects that connect to Amazon Web Services, YQL and Google Spreadsheets.  The Google solution is perhaps the most flexible, because if the data does not reside in your spreadsheet you can import it via the spreadsheet functions.

Writing the Processing code to connect is pretty straight forward.  You can use functions like loadStrings to reference URLs that have your data.

String items[];
String[] pieces;
PFont  fontA = loadFont("StylisticSFBold-48.vlw");
//load from Google Spreadsheet Url
int itemCount=items.length;
for(int a=0;a<itemCount;a++){
pieces = split(items[a], '|');
textFont(fontA, 32);
text(pieces[0], 50, 25+a*(height/itemCount));

This works really well in the IDE… but when you upload it you will see that it will not work.  The problem is that the Java applet’s security does not allow it to access urls and content that are not on the same server.  If it could there could be viruses that downloaded malicous content to your machine.

Standard Approach – Sign the Applet

The standard option is to sign your applet and create a certificate for the user to approve.  To do this you must make sure you have the Java KeyTool and Jarsigner downloaded.  Then you need to issue the follow commands in ms-dos or the terminal on a Mac:

  • “C:\Program Files\Java\jre6\bin\keytool” -selfcert -keystore pKeyStore -alias yourKeyName
  • “C:\Program Files\Java\jre6\bin\jarsigner” -keystore pKeyStore GoogleImporter2.jar yourKeyName
  • “C:\Program Files\Java\jdk1.6.0_06\bin\jarsigner” -keystore pKeyStore GoogleImporter2.jar yourKeyName

You will need to replace the keytool and jarsigner location with the right ones on your machine, and replace “GoogleImporter2.jar” with your applets name.  Finally make sure to use your own key name.

If all goes well you will have successfully signed your app.  While attempting this for the first time I found the following two links to be very useful:

Alternative Approach – Redirect to a Local File

As an alternative you can use an intermediate script file to bring the external sites data to your server.  This is the approach I took in building my Flickr Kaleidoscope.  To create the same result as above, but without the signing the app I created a small php file that I can reference from Processing:

$response = file_get_contents($link.'&output=txt');
echo $response

To call a Google spreadsheet from this you simple call a url like the following:

There is an additional advantage to this approach is that if the data needs to be further processed and “massaged.”  In this case you can use the PHP functions to parse the data, parse xml, create\sort arrays and much more.

Whichever approach is pursued their is a lot of power in connecting Processing with data from the web.  The Processing visuals can be driven from data and have a dynamic element that comes from refreshable data.

Add to DeliciousAdd to DiggAdd to FaceBookAdd to Google BookmarkAdd to RedditAdd to StumbleUponAdd to TechnoratiAdd to Twitter

Book Review: The Visual Miscellaneum

24 01 2010

So I did get the books on my holiday wishlist and have had a great time going through all the amazing thinking on data visualization.  The first one I have completely finished is The Visual Miscellaneum by David McCandless.

This book is a great sources of inspiration and entertainment.  David McCandles shows off that he is able to take any data set and make it into a beautiful infographic or data visualization.  His layouts and color selections make the information really pop.

The Visual Miscellaneum - Rising Sea Levels

Topics cover many areas including science, pop culture, history, music, the internet, food and more.  The variation helps to make the topic of visualization stand out.  By seeing the techniques and visuals applied to different data sets in different areas, it helps the reader realize that data visualization has value in everything from dealing with climate change (page 74) to ordering the right cup of coffee (page 156).The Visual Miscellaneum - Simple Part I

On the down side is that some of the beauty is only skin deep.  I found certain illustrations would have been served better by a simple table or alternative visualization.  There were also a few cases where a typo existed or it seemed a legend must have been left out.

I would still say this book great addition to a coffee table or library.  If you are a data geek you will love to study the visualizations.  If you are not a data geek you will enjoy the interesting facts and comparisons represented by beautiful images.


  • Beautiful book with great graphic design, color schemes, etc.
  • A lot of variety in how data can be presented… some are the same old standards and others are really creative new approaches.


  • The illustrations are not always an improvement in understanding the data.

I will keep this book close at hand for best practices in making data look awesome and as a source of inspiration when stuck for a way to represent information. Click here to get it from Amazon.

Add to DeliciousAdd to DiggAdd to FaceBookAdd to Google BookmarkAdd to RedditAdd to StumbleUponAdd to TechnoratiAdd to Twitter

Blldzr: What are they Building?

16 01 2010

I have written before about the power of applying mash-up and web 2.0 technologies to government data.  For me there was a bit of put up or shut-up to the whole effort as well.  What could I do to make my community better using my development, analytical and visualization knowledge.  I wanted to find something that I had an interest around and was relevant to my community (Tucson, AZ).


What I came up with was (pronounced Bull-Dozer).  Blldzr is a wiki that allows members of a community to enter information about development and construction projects in their area.  In a addition to the factual information there are also comments to express opinions about the developments.

Blldzr:  Answering the Question... What are they building?

After being added each development has a page that can be updated and subscribed to.

Blldzr:  A page for each development

Being informed and having conversations about what is being built in your neighborhood is one way to create community.  My hope is that Blldzr can foster more of that.

There is still much to do.  The site needs content, activity, moderators and to be promoted.  If you are interested in any or all of this feel free to pitch in (e.g. add some entries for you city).  If you would just like to be kept up to date on progress feel free to follow Blldzr on Twitter.

I will post more in the future on progress and the technology that was used in building the site.

Add to DeliciousAdd to DiggAdd to FaceBookAdd to Google BookmarkAdd to RedditAdd to StumbleUponAdd to TechnoratiAdd to Twitter

Interactive Resume using Simile Exhibit

11 01 2010

Simile Exhibit is a JavaScript framework that enable a data sets to be visualize, sliced and embedded into any web page.  It is fun and easy to quickly see your data brought to life with all of Exhibit’s features.  Some of the included views are a basic table, timeline, map, gallery and more.  It is a great solution for data with less that 1,000 rows.  The data can be integrated in a number of ways, including Google Spreadsheets, JSon files, etc.

In addition to plugging the same data set into multiple visualizations, Exhibit provides Facet filters.  These automatically provide multiple choice filter list that dynamically modify the visualizations.

With all of these features and easy data integration I plugged it into my resume which I set up in a Google Spreadsheet.  First I came up with a basic table view (I applied a little extra CSS):

Garys Resume in an Exhibit Table

Then I added the map view to see all of the locations that I have worked at:

Garys Resume as Exhibit Map

And lastly I added a timeline view:

Garys Resume in an Exhibit TimelineAs you will notice on each of the pages there are a list of filterable facet list on the left and right side (this is configurable as well).

For more examples check out this Exhibit Examples 1Exhibit Examples 2, and Exhibit Examples 3.

So far I have only identified 3 real downsides to the software.

  1. First since the content is published through Javascript you cannot tell what the page is about by looking at its source code.  This will impact some spiders and search engines, when crawling, who cannot call the Javascirpt.  There are work-arounds as the content can be output as html, xml, tab delimited, json, etc, but the dynamic data and filtering is lost.
  2. One of the reasons the software is so easy to deploy is that the server side of Simile is hosted at MIT.  So there is no need to host the library on your site.  This can be limiting since you become dependent on MIT’s uptime and it may not be possible to deploy on an intranet, or with sensitive internal data.  The good news is that is a way to set up your own server, so it can be done, but looks to be a bit of work.
  3. Exhibit is not for large data sets.  It works well for small list that can be managed in a text file or Google Spreadsheet, but it is not something you would want to plug into a data warehouse.

Overall, Exhibit is a great way to interact and visualize data and allows you to be up and running quickly.  Check out my resume, some examples and go build your own.

Add to DeliciousAdd to DiggAdd to FaceBookAdd to Google BookmarkAdd to RedditAdd to StumbleUponAdd to TechnoratiAdd to Twitter

Panoramic Pictures with Windows Live and Image Composite Editor

1 01 2010

The kids and I went Geocaching yesterday.  We had a blast.  We found the location and coordinates at

The cache was at the top of a pretty steep hill near Madera Canyon in the Coronado National Forest (Southern Arizona).  It was a small metal tea can in a buried steel pipe.

I started taking pictures as Isabelle and Jasper were removing the cache.  I circled them from a number of angles to get enough pictures to stitch together for a panorama.  Usually you stay in one spot and spin, but I wanted multiple images of the explorers, so I actually circled them (look at the shadows to get a sense of the different angles).

I was planning on using the image stitching that came with my Canon Camera, but after googling for the software (and getting frustrated with Canon’s website) I came to a recommendation of Windows Live Photo Gallery and it’s panorama feature (see below).

Then I also found a link to a more advance version (same software engine) from Microsoft Research called Image Composition Editor(ICE).  The results were pretty much the same, but the ICE program offered a little more flexibility in determining the perspective of the panoramic(see below).

What a great way to end the year.

Add to DeliciousAdd to DiggAdd to FaceBookAdd to Google BookmarkAdd to RedditAdd to StumbleUponAdd to TechnoratiAdd to Twitter

Data Activism: Geeks for Good

21 12 2009

Data Activism is a new an emerging trend with designers, developers and artist utilizing technology to help society understand and address the issues of our times.

The new U.S. Administration has established initiatives to make government data available to the public.  This is converging with the growing number of developers and designers that are familiar with mash-up’s and data visualization technologies.  The convergence of these trends create the ability for developers, designers and artist to create apps, sites and art that educate and provide transparency to the government.

Creating Government Transparency

The Open Government Directive is a memo sent by the White House on December 8th, 2009 directs federal agencies to:

  1. Publish Government Information Online
  2. Improve the Quality of Government Information
  3. Create and Institutionalize a Culture of Open Government
  4. Create an Enabling Policy Framework for Open Government

Each of these are follow with specific actions that must start to show results in the next couple of months.  The main hub for this information will be, providing Federal data sets, and links to State and local efforts as well. Screenshot
This is amazing progress for the government that will bring both innovation and efficiency.  Possibilities will be opened up as great designers get creative on representing the governments data.  The government agencies will be pressured to improve as their performance becomes public knowledge.  All of this is a win for Americans with more services, useful information and government efficiency.

Everyone can participate by using the data, giving feedback, building visualizations, building reports, etc.  You can get involved by yourself, or join a team of like minded individuals.  What are you waiting for?



If you have other resources or examples you recommend, add them as comments and I will update this article.

Add to DeliciousAdd to DiggAdd to FaceBookAdd to Google BookmarkAdd to RedditAdd to StumbleUponAdd to TechnoratiAdd to Twitter