Here in the UK, postcode geodata was recently released as part of the OS Opendata initiative. Although not the full postcode address file (PAF), it’s enough to be useful for visualisation purposes!

I thought it’d be interesting to plot the density of postcodes in a heatmap, as a way of getting a visualisation of population density.

I downloaded a copy from MySociety. The raw data uses the Ordnance Survey coordinate systems (Eastings and Northings), but MySociety also provide a version with coordinates converted to the WGS84 datum (latitude and longitude). It gets better; one file per outcode (so all the EH postcodes go in one file, for example).

So how to plot this?

I’ve been using R with the ggplot2 library for visualisations; it’s an amazing toolset, and you’d be surprised how few lines of code it takes to get results.

First of all, download the data.

Now, extract the data for the EH postcode area.

The first step is to load the extracted CSV file into a data frame. There are no column headers (header=F) and the file is comma separated (sep=”,”)

inp <- read.table("c:\\infoviz\\ehpostcode.csv",header=F, sep=",")

Next, a quick hack to remove certain postcodes which seem way out of the region of Edinburgh. I’ve yet to work out what these stray postcodes are.

inp <- subset(inp,inp$V10 != "")

Now we bring in the ggplot2 library..

library(ggplot2)

Now the plot itself.

m <- qplot(xlab="Longitude",ylab="Latitude",main="EH Postcode heatmap",geom="blank",x=inp$V3,y=inp$V4,data=inp)  + stat_bin2d(bins =200,aes(fill = log1p(..count..))) 

The stat_bin2d does the binned heatmap plot. It splits the area into a grid of 200×200 (‘bins’) and counts the number of postcodes in each grid square. These counts are then scaled logarithmically (fill = log1p(..count..)).

Finally, we plot this puppy.

m 

That’s it! Five lines of code.

Here’s the whole chunk..

inp <- read.table("c:\\infoviz\\ehpostcode.csv",header=F, sep=",")
inp <- subset(inp,inp$V10 != "")
library(ggplot2)
m <- qplot(xlab="Longitude",ylab="Latitude",main="EH Postcode heatmap",geom="blank",x=inp$V3,y=inp$V4,data=inp)  + stat_bin2d(bins =200,aes(fill = log1p(..count..))) 
m 

If you set the boundaries of the x and y axes, you can then use the resulting image as an overlay in Google Earth.

I found that the heatmap is a reasonably good proxy for population density – but with exceptions. There are hotspots that turn out to be postal sorting offices – PO boxes and post restante.

This map shows the location of pictures in a Flickr group superimposed on an OpenStreetMap (OSM) export.

If you try plotting points directly on an OSM map, you’ll find that points are all over the shop. The reason is that OSM exports use the Mercator projection; you need to change the latitude and longitude coordinates into the Mercator projection.

Basemap to the rescue!

If you use the code in the previous post, you can change the plotting code.

from basemap import Basemap 
import matplotlib.pyplot as plt
import matplotlib.pylab as pylab
import numpy as np
import string
import matplotlib.cm as cm

x=[] #longitudes
y=[] #latitudes

fi=open(r'C:\infoviz\scotland_photos.csv','r')

linenum=0
for line in fi:
    if linenum>0:
        line=string.replace(line, "\n","")
        try:
            fields=string.split(line,",")
            lon,lat=fields[0:2]
            x.append(float(lon))
            y.append(float(lat))
        except:
            print "Error!"
    linenum+=1
fi.close()

m = Basemap(llcrnrlon=-8.0,llcrnrlat=54.5,urcrnrlon=1.5,urcrnrlat=59.5,lat_ts=20,
            resolution='h',projection='merc',lon_0=-4.36,lat_0=54.5)
x1,y1=m(x,y)
m.drawmapboundary(fill_color='white') # fill to edge
m.scatter(x1,y1,s=5,c='r',marker="o",cmap=cm.jet,alpha=1.0)

Rather than using basemap to draw the outline of Scotland, this script simply creates a scatter plot on a white background, like so:-

Now, you need to export the map from OpenStreetMap.

The corners have been set as follows..

… llcrnrlon=-8.0,llcrnrlat=54.5,urcrnrlon=1.5,urcrnrlat=59.5 …

llcrn stands for the lower-left coordinate, and and urcrn for the upper-right coordinate.

So if you if you export from OpenStreetMap using these coordinates…

.. you’ll have a map of Scotland with the Mercator projection.

The two images can then be composited in Photoshop or another graphics app like the Gimp.

In this post I’ll show how to produce a simple pinboard map of geotagged photos in a flickr group pool, using Python and Basemap/Matplotlib. You’ll need:-

• Python
• Beej’s Flickr API
• A flickr API key
• Matplotlib
• Basemap

There are two short scripts here:-

• A script to find the longitude and latitude of geotagged photos in the group pool
• A script to generate the plot

The first script produces a CSV file; the second uses this CSV file to produce the plot.

Here’s the script to produce the CSV file with photo locations:-

# -*- coding: UTF8 -*-
'''
Created on 12 May 2009
Based on beejs flickr API
Produce a list of photo locations for a given
group's pool on flickr
@author: Steven Kay
'''

import flickrapi
import string
import datetime
import string
import time

# Enter your API key below
# You can apply for an API key at 
# http://www.flickr.com/services/apps/create/apply
api_key = '' 

# paste group NSID below
group = '1124494@N22'

# sample... fetch your latest images with 
# a count of the views, faves and comments

if __name__ == '__main__':
    flickr = flickrapi.FlickrAPI(api_key)
    response_photos = flickr.groups_pools_getPhotos(group_id=group,per_page=500,extras='geo')
    root=response_photos.findall('.//photos')
    pages=int(root[0].get('pages'))
    if pages>8:
        # stop after 8 pages of 500 images
        # not sure if groups.pools.getPhotos has the same
        # 4000 image limit as photos.search..?
        pages=8
    
    fo=open(r"C:\infoviz\scotland_photos.csv","w")
    print "Longitude,Latitude"
    fo.write("Longitude,Latitude\n")
    for page in range(0,pages):
        response_photos = flickr.groups_pools_getPhotos(group_id=group,per_page=500,page=str(page),extras='geo') 
        for photo in response_photos.findall(".//photos/photo"):
            try:
                lat=photo.get('latitude')
                lon=photo.get('longitude')
                st="%s,%s" %(lon,lat)
                if not st=="0,0":
                    # ignore the odd buggy 0,0 coords
                    print "%s,%s" %(lon,lat)
                    fo.write("%s,%s\n" %(lon,lat))
            except:
                pass
        time.sleep(1)
    fo.close()

You’ll need to find the NSID of the group as an input; you can find this with the flickr API call flickr.group.search.

Now, you have a simple CSV file with the latitude and longitude of each geotagged image in the pool.

Longitude,Latitude
-5.167792,58.352519
-4.024359,57.675544
-4.230251,57.497356
-4.2348,57.501045
-4.84703,56.646034
-4.306168,55.873986
-3.586263,56.564732
...

This demo uses the Photography Guide to Scotland pool.

The next step is to plot the map.

'''
Simple Matplotlib/Basemap pinboard map for
Flickr Groups.

Need to provide a CSV file in following format

Longitude,Latitude
20.1,-3.25
20.225,-3.125
.. etc..

Created on 10 Oct 2009

@author: Steven Kay
'''

from basemap import Basemap 
import matplotlib.pyplot as plt
import matplotlib.pylab as pylab
import numpy as np
import string
import matplotlib.cm as cm

x=[] #longitudes
y=[] #latitudes

fi=open(r'C:\infoviz\scotland_photos.csv','r')

linenum=0
for line in fi:
    if linenum>0:
        line=string.replace(line, "\n","")
        try:
            fields=string.split(line,",")
            lon,lat=fields[0:2]
            x.append(float(lon))
            y.append(float(lat))
        except:
            pass
    linenum+=1
fi.close()

# cass projection centred on scotland
# will need to replace with a projection more suited
# to the group you're plotting

m = Basemap(llcrnrlon=-8.0,llcrnrlat=54.5,urcrnrlon=1.5,urcrnrlat=59.5,
            resolution='h',projection='cass',lon_0=-4.36,lat_0=54.5)
x1,y1=m(x,y)
m.drawmapboundary(fill_color='cyan') # fill to edge
m.drawcountries()
m.drawrivers() # you may want to turn this off for larger areas like continents
m.fillcontinents(color='white',lake_color='cyan',zorder=0)
m.scatter(x1,y1,s=5,c='r',marker="o",cmap=cm.jet,alpha=1.0)

plt.title("Photography Guide to Scotland in FlickR") # might want to change this!
plt.show()

This script uses a projection centred around scotland; you’ll need to change the following line…

m = Basemap(llcrnrlon=-8.0,llcrnrlat=54.5,urcrnrlon=1.5,urcrnrlat=59.5,
            resolution='h',projection='cass',lon_0=-4.36,lat_0=54.5)

Here are a few tasters… click through to see the animation (needs Java) and the source.

You can see the rest of my Processing portfolio on Openprocessing.org