Saturday 19th May saw the UK OpenStreetMap community and Open Data Manchester join forces for an afternoon of exploring central Manchester. Several mappers from the Midlands made the trip and boy, did we have fun!
Our hosts had prepared some great “Explorer Backpacks” and we soon broke off in to teams for our expeditions into the unknown. From the waterways to historic relics, the missions were wide ranging – we even had a Geiger counter on hand to find the spot with the highest background radiation!
After soaking up the sun we headed back to Federation House to share our findings before heading for a well earned drink.
Dan S: It was ace, in particular the slightly whimsical mapping trip we did, following psychogeographic suggestions prepared by the Joy Diversion organisers. That was a good way to get new mappers and experienced mappers working together on an equal footing.
Samantha: I really enjoyed the event on Saturday. I’d love to get out and about around Manchester and learn more about the city in general. If anyone else is local and wants to go exploring, to add to the map or otherwise please let me know.
Tony Shield: Really enjoyed the ability to meet like minded people and get a deeper understanding of the OSM project. Going out and surveying then placing the information on the map was great. Seeing some of the uses the data could be used for was really interesting.
Martyn: Joy Diversion was a great afternoon, I’ve just updated OSM with the Team Crocodile survey (cycle and car parking, playgrounds, water taxi, footpaths and more). Keep on improving the map.
A big success then! Partnering with Open Data Manchester worked really well. Nice folk and a good opportunity to spread the word about OSM without having to arrange a venue and drum up attendees from scratch.
Post content: OpenStreetMap United Kingdom CIC (with minor amends). Local mappers Brian and Rob are Directors of the community interest company, which focuses on supporting OSM in the UK.
When Ilya approached the UK OpenStreetMap community about incorporating third party data (Shell petrol stations) into OSM he had already ran the data through his “OSM Conflator” tool. As part of the project he also created a “Community Validation” tool. I decided to take a look at both of these using Asda petrol station data as a test case.
In a series of posts I will share my experience with conflation and validation. This first post covers just OSM Conflator, with a follow on posts in the coming days covering the Community Validation tool and some reflections on the process.
Intro to OSM Conflator
OSM Conflator is a command-line tool written in Python 3 that compares a third party dataset against OpenStreetMap. It does not directly edit OpenStreetMap but instead gives you two outputs based on what it finds. The first, preview.json, can be loaded into an online tool such as geojson.io to visualise the differences. The second, a OSM change file, can be opened in JOSM for uploading in to OpenStreetMap. In both cases it assumes the third party data is correct and more up to date than any OSM data it is replacing. As such it is worth using the Community Validation tool to check the results before uploading.
For now the third party data must be point (node) data but it can match to both nodes and ways in OpenStreetMap; downloading the most recent data each time you run the script. The matching is initially done by distance and you can set the maximum tolerance (e.g. 100 meters). If the third party data has a unique reference key (e.g. a store ID number) then this can be added to OpenStreetMap the first time you merge the data. The next time you run the comparison, for example if a retail chain has changed their opening hours, then OSM Conflator relies on this reference key rather than having to undertake a proximity search.
Inputs
OSM Conflator requires two inputs: a profile and the third party data. The profile sets the search criteria and which third party tags should always replace values on matched OSM objects. In the case of the Asda petrol stations the search criteria was for ‘amenity=fuel’ objects within 100 meters. The tags to upload were ‘brand’, ‘opening_hours’, ‘website’, ‘phone’, ‘addr:street’, and ‘addr:postcode’. The data included a unique reference ID so I set the profile to write this to OSM.
My OSM Conflator profile file.
The profile is actually created as a Python file but is simple enough that you don’t need any Python experience. If you are however a pro at Python you can add to the profile. Example additions include tag transformations (e.g. reformatting telephone numbers in to ‘+44 <Area Code> <Number>’ format) or even code to download the source data direct from the third party’s website. If like me you are not a Python pro then you will need to provide OSM Conflator with a separate file including the third party data. This must be in a JSON file format.
Extract of the Asda source data (rows and additional tag columns omitted).
Getting the third party data into a JSON file is easy when you know how. Prepare a table like the one above with columns for latitude (lat), longitude (lon) and, if you have it, the unique reference key (id). For the tags you wish to add to OpenStreetMap name the columns according to the standard tag usage within OSM adding ‘tags/’ in front of the column name. To convert your table into a JSON file simply copy and paste the data in to www.convertcsv.com/csv-to-json.htm making sure to select “First row is column names” and “Recreate nested objects and arrays” in the options. Copy the output into a blank notepad and save as ‘data.json’.
Running the tool
With the profile and third party data now prepared the final step is to run OSM Conflate. As noted this is a Python 3 command-line tool. I assume you have already installed this and have also installed ‘pip’ which is a package management system used to install and manage software packages written in Python. With both of these installed go ahead and open a command/terminal window and run “pip install osm_conflate” to install OSM Conflate. Finally to run OSM Conflate using your profile and data file run “conflate -i data.json -v -o result.osm -c preview.json profile.py”. When it has finished running try opening preview.json in geojson.io to get a visualisation of the results.
In the next post we will look at how to load the outputs in to the Community Validation tool.
As we described when we kicked off this quarters UK project to map petrol stations, recent community validation of third party (import) data has highlighted much outdated and erroneous data within OpenStreetMap :-(. For the Shell import we captured some of this and added it to a “fixme” tag. If you have not yet come across this tag then it is one in which OpenStreetMap users can flag something that looks wrong or requires additional information to improve the map.
It is possible to view petrol stations with a fixme tag by using Overpass Turbo scripts. We have done this already so all you have to do is click this link and press run. Results are visual as shown in the example below. There are approximately 130 petrol stations with a fixme tag so with some effort we should be able to review them all. Happy mapping!
When we did the highly successful quarterly project to map schools we tracked the progress using a script that collated data on a daily basis from TagInfo UK. This enables us to see how we are getting on, both in terms of total features mapped, but also the split between nodes, ways and relations. We can then produce charts like the one for schools shown below.
UK Schools mapped in OpenStreetMap
This script was also used on the following quarterly projects up to the end of Q1 2017. Unfortunately I failed to set it up for the following quarterly projects, and worse still, failed to notice that the script stopped running in May 2017. It looks like the reason behind this was a change in the server that we are hosted on causing a 32bit vs 64bit error – but I’m no expert!
The important thing is that I have it up and running again and have added tracking for petrol stations. You can view the data by following this link.
Following a lengthy discussion on the talk-gb mailing list and several false starts, Ilya recently imported UK Shell petrol station data into OpenStreetMap.To confirm the quality of the third party data a brand new community validation tool was developed. Use of this tool highlighted a lot of inconsistency in the way we map – as such let’s make petrol stations the quarterly mapping project for Q1 2018.
According to Statistica there are some 8450 petrol stations in the UK. Compare this to OpenStreetMap, where TagInfo shows that we have mapped 7200. Not bad – just 1250 more to go! Let’s see if we can get to the magic 8450 by the end of the quarterly project (or show that the real number is indeed different). This also gives us a great opportunity to review the existing data, updating old tags to reflect on-the-ground change and converting petrol stations mapped as points (nodes) to ones mapped as areas (closed ways).
Last October I had the pleasure of visiting the National Library of Scotland. I was up in Edinburgh at State of the Map Scotland and couldn’t pass up on the opportunity to have a guided tour of the library’s maps reading room. Our host, Chris Fleet, had laid out a number of fantastic items from their collection – each with it’s own story.
Being a bit of a map geek I took way too many photos during the hour or so we had before closure. Here are some of the best to whet your appetite. The NLS are long term supporters of OpenStreetMap, have scanned and georeferenced many thousands of maps, and host zoomable maps on their amazing website. If you find yourself in Edinburgh, be sure to check out their collection.
Russian military maps (link: secret cold war maps).Textured maps for the blind and partially sighted.The Blaeu Atlas of Scotland, 1654 (view online)Precision tools of the map maker.Map corrections: tracing paper and sticky tape!A map of WW1 trench positions in Belgium.Now you see it, now you don’t. Early day ‘airbrusing’ to remove an airfield from aerial photos.
Several years ago we created JOSM paint styles to help with the task of mapping gritting routes. The styles made it easier to identify which roads had the gritting tag.
The recent versions of JOSM have changed the requirements of these style plugins therefore breaking our code. If you still use the style you will see the following message that gives the reasoning and how to fix the problem. The style plugin would need to be updated to the latest MapCCS code.
This applies to both our josm-style.xml and josm-preset.xml code. Currently we have no plans to update these style plugins to the latest MapCCS code, however we welcome support from others to complete this update. If someone can come forward we will upload the new code so the functionality continues to work. For now, to prevent the warning message popping up each time you start JOSM, simply go to “View -> Map paint styles -> Map paint preferences” and delete “Winter Gritting” and “Winter Gritting2”.
Back in June of last year we shared with you Richard Fairhurst’s State of the Map US talk in which he set out the case for more mobile OpenStreetMap apps. One of the key message was that “we need to think about smartphone editors” as a way to continue growing OpenStreetMap. This month we got a new mobile editor and a new app to help you explore your surroundings.
Firstly the new editor. This comes in the form of an update to the popular MAPS.ME. Even before this update MAPS.ME was a essential app for any OpenStreetMapper. It provides a simple to use offline map with voice over guidance for travel directions by car or on foot. It’s interface is simple and effective: for example, clicking on a map feature brings up a slider at the bottom of the screen providing more information.
On April 5th 2016, MAPS.ME developer Ilya Zverev explained how you can now edit the map information direct from the app. This is perfect for adding addresses, phone numbers and opening hours to existing map features whilst out and about.
Editing OpenStreetMap with MAPS.ME
With more than 7 million monthly active users MAPS.ME is aiming to be the number one OpenStreetMap editor. Although OpenStreetMap’s editor usage stats are a little out of date, it’s already clear that MAPS.ME is the most used mobile editor measured by number of users.
The second addition to mobile comes in the form of Geopedia. This is a neat little app that takes the OpenStreetMap base map and overlays the wikipedia database. Clicking on the map shows you all the nearby features that have a wikipedia article (we understand this is based on wikipedia’s latitude/longitude data rather than the wikipedia tag stored within OpenStreetMap).
Searching for nearby attractions with geopedia.
As this links to wikipedia you can view the articles in multiple languages and view an image if one is available. It works well to find new and exciting places to explore in your neighbourhood, or whilst away travelling. And of course you can use it as a prompt to add text and photos to wikipedia, and map details to OpenStreetMap – perhaps via MAPS.ME!
Yesterday the mapping community in the UK held an online “Night School” mapathon to hep map schools. So how did we get on?
We did it! Yesterday we set the record for the highest number of edits to amenity=school features in the UK. Between us we edited 405 schools beating our previous high from 18th January.
Looking at the detail we created 202 new school polygons (a 33% increase on our previous daily best) and modified 191 ways. The other edits (12) were to nodes.
We are now at 75.5% of all UK schools mapped as ways (according to Robert W’s fantastic progress tool). This is up from just 62.8% when we started this task at the start of January. All but 10 postcode areas are above 50% complete. When we started there was 30 below 50% (8 below 40%). At the top end we have 24 postcode areas 95+ percent complete. Compared to zero when we started!
A total of 38 people edited during the day – again, another record! And the edits of those who used the #OSMschools comment in their changesets can been seen here.
Thanks for the amazing effort today. I hope the MapRoulette task is proving popular and will encourage more edits in the least loved parts of the UK OSM map.
This quarter the UK OpenStreetMap community is mapping schools. So far over 250 people have made 10,000 edits – a hugely impressive feat! There’s still time to get involved and on Tuesday 1st March we are holding “Night School”, an online mapathon.
Night School is experiment for the UK Quarterly Project; on the evening of Tuesday 1st March let’s join together as one big community to map schools. There are no rules, simply start when you like and finish when you like and map where you like! Follow along with the progress on the school edit tracker and our live map of edits (to show up on the map simply add “#OSMschools” to the comment box when you save your map edit).
If you don’t know how to map schools in OpenStreetMap check out our handy picture tutorial. You can always contact us for more support should you need it.
Ideas for making the evening more fun are welcome! Don’t forget the discussions on Schools Quarterly Project that are taking place here on Loomio as a trial for a possible platform for UK member decision-making.
Even at this stage of the project new people are joining, so spread the word on other channels to see who we might attract.
