Automating CartoDB administration tasks

CartoDB is a mapping and visualisation platform we are using at REA to help us analyse and visualise our data. If you want to read more about how to set up a local CartoDB environment read here.

When you create a CartoDB account you can access the CartoDB dashboard which allows you to perform a stack of administrative functions such as loading data and creating visualisations.

CartoDB currently has a feature which allows you to synch tables. This is a nice feature, but the level of control is pretty coarse at the moment. You can register a table as being synched against a data source (e.g. a CSV in AWS S3) and schedule it to synch every hour, day, week or month.

As our use of CartoDB has scaled up it’s important that we can automate some of the administrative operations so we can do stuff like schedule data loading and post processing. We wanted finer grained control than that currently provided by the synch feature and also wanted to ensure we can monitor our regular data loading processes to ensure our maps are kept up to date.

As a result we’ve create a small python client module to help manage some of the common administrative functions we currently do manually via the CartoDB dashboard. You can grab the code here or alternatively you can install the module directly from PyPi

$ pip install cartodb_dashboard

Let’s walk through a simple example of loading a dataset (the dataset we’re going to use is included in the test directory in the repository). When creating this module we’ve tried to follow a similar pattern to the existing cartodb client python module which allows you to interface with the CartoDB SQL API.

• Firstly lets create an instance of our client:

client = CartoDbDashboard(cartodb_domain, cartodb_user, cartodb_password, cartodb_host+':'+cartodb_api_port,
cartodb_protocol, cartodb_version)

• Now we can import some data:

client.import_data('test/testdata/localities.zip')

• You’re done 🙂

So far the module implements the following functions:

• Import tables
• Delete tables
• Update column data types (handy if you’re uploading csv’s)

When used together with the CartoDB SQL API we think this is a useful module so give it a whirl and let us know what you think?