Conflation & Validation: OSM Conflator

posted in: Participate | 3

When Ilya approached the UK OpenStreetMap community about incorporating third party data (Shell petrol stations) into OSM he had already ran the data through his “OSM Conflator” tool. As part of the project he also created a “Community Validation” tool. I decided to take a look at both of these using Asda petrol station data as a test case.

In a series of posts I will share my experience with conflation and validation. This first post covers just OSM Conflator, with a follow on posts in the coming days covering the Community Validation tool and some reflections on the process.

Intro to OSM Conflator

OSM Conflator is a command-line tool written in Python 3 that compares a third party dataset against OpenStreetMap. It does not directly edit OpenStreetMap but instead gives you two outputs based on what it finds. The first, preview.json, can be loaded into an online tool such as geojson.io to visualise the differences. The second, a OSM change file, can be opened in JOSM for uploading in to OpenStreetMap. In both cases it assumes the third party data is correct and more up to date than any OSM data it is replacing. As such it is worth using the Community Validation tool to check the results before uploading.

For now the third party data must be point (node) data but it can match to both nodes and ways in OpenStreetMap; downloading the most recent data each time you run the script. The matching is initially done by distance and you can set the maximum tolerance (e.g. 100 meters). If the third party data has a unique reference key (e.g. a store ID number) then this can be added to OpenStreetMap the first time you merge the data. The next time you run the comparison, for example if a retail chain has changed their opening hours, then OSM Conflator relies on this reference key rather than having to undertake a proximity search.

Inputs

OSM Conflator requires two inputs: a profile and the third party data. The profile sets the search criteria and which third party tags should always replace values on matched OSM objects. In the case of the Asda petrol stations the search criteria was for ‘amenity=fuel’ objects within 100 meters. The tags to upload were ‘brand’, ‘opening_hours’, ‘website’, ‘phone’, ‘addr:street’, and ‘addr:postcode’. The data included a unique reference ID so I set the profile to write this to OSM.

 

My OSM Conflator profile file.

 

The profile is actually created as a Python file but is simple enough that you don’t need any Python experience. If you are however a pro at Python you can add to the profile. Example additions include tag transformations (e.g. reformatting telephone numbers in to ‘+44 <Area Code> <Number>’ format) or even code to download the source data direct from the third party’s website. If like me you are not a Python pro then you will need to provide OSM Conflator with a separate file including the third party data. This must be in a JSON file format.

 

Extract of the Asda source data (rows and additional tag columns omitted).

 

Getting the third party data into a JSON file is easy when you know how. Prepare a table like the one above with columns for latitude (lat), longitude (lon) and, if you have it, the unique reference key (id). For the tags you wish to add to OpenStreetMap name the columns according to the standard tag usage within OSM adding ‘tags/’ in front of the column name. To convert your table into a JSON file simply copy and paste the data in to www.convertcsv.com/csv-to-json.htm making sure to select “First row is column names” and “Recreate nested objects and arrays” in the options. Copy the output into a blank notepad and save as ‘data.json’.

Running the tool

With the profile and third party data now prepared the final step is to run OSM Conflate. As noted this is a Python 3 command-line tool. I assume you have already installed this and have also installed ‘pip’ which is a package management system used to install and manage software packages written in Python. With both of these installed go ahead and open a command/terminal window and run “pip install osm_conflate” to install OSM Conflate. Finally to run OSM Conflate using your profile and data file run “conflate -i data.json -v -o result.osm -c preview.json profile.py”. When it has finished running try opening preview.json in geojson.io to get a visualisation of the results.

In the next post we will look at how to load the outputs in to the Community Validation tool.

Petrol stations with a fixme tag

posted in: Map Improvements, Participate | 3

As we described when we kicked off this quarters UK project to map petrol stations, recent community validation of third party (import) data has highlighted much outdated and erroneous data within OpenStreetMap :-(. For the Shell import we captured some of this and added it to a “fixme” tag. If you have not yet come across this tag then it is one in which OpenStreetMap users can flag something that looks wrong or requires additional information to improve the map.

It is possible to view petrol stations with a fixme tag by using Overpass Turbo scripts. We have done this already so all you have to do is click this link and press run. Results are visual as shown in the example below. There are approximately 130 petrol stations with a fixme tag so with some effort we should be able to review them all. Happy mapping!

Example result from Overpass Turbo.

Q1 2018 mapping project: Petrol stations

posted in: Participate | 1

Following a lengthy discussion on the talk-gb mailing list and several false starts, Ilya recently imported UK Shell petrol station data into OpenStreetMap.To confirm the quality of the third party data a brand new community validation tool was developed. Use of this tool highlighted a lot of inconsistency in the way we map – as such let’s make petrol stations the quarterly mapping project for Q1 2018.

According to Statistica there are some 8450 petrol stations in the UK. Compare this to OpenStreetMap, where TagInfo shows that we have mapped 7200. Not bad – just 1250 more to go! Let’s see if we can get to the magic 8450 by the end of the quarterly project (or show that the real number is indeed different). This also gives us a great opportunity to review the existing data, updating old tags to reflect on-the-ground change and converting petrol stations mapped as points (nodes) to ones mapped as areas (closed ways).

Image: CC-By-Sa 2.0 Betty Longbottom

A guide to mapping Fire Hydrants in the UK

 

In the West Midlands our fire hydrant signs are generally placed on lamp posts:

The black H on a yellow background I believe to be a UK standard. The upper number is the diameter of the underlying water main in mm. The lower number is the distance in feet to the fire hydrant from the sign with the arrow showing the direction to the actual hydrant. There is also a reference number at the foot of the sign. So, having spotted the sign, the actual fire hydrant has to be found. The signs can get swivelled on the lamposts through maintenance interventions or general neglect and the arrows point in the wrong direction!

To complicate matters further, there are some fire hydrants that do not have a sign, and some older variants of the signs. These older signs tend to be attached to buildings or walls and I have no idea what the numbers refer to.

So you have to on the lookout on the ground as well as spotting yellow H  signs.Fire hydrant covers generally have FH on them to identify them, although some older ones can have just H only. Whether the older style ones are still operational I don’t know, but they get mapped anyway.

 

And of course it’s always good to see public organisations collaborating successfully with each other!

The basic tag is emergency=fire_hydrant with more details on the OSM wiki

Why the sudden interest in fire hydrants? In the UK, they’ve languished as an item that gets mapped.

At our last mappamercia pub meeting Andy Mabbett regaled us with his saga of trying to get West Midlands Fire Service to release the locations of fire hydrants under Freedom of Information legislation.

The full saga can be found here

WMFS refused to release the information on the grounds of national security viz: “publishing information about water  networks and other parts of the critical national infrastructure could  expose vulnerabilities in the network and pose a serious risk to public health either through non availability of water resource or
contamination of supplies. There does not have to be any evidence that this is being planned but it is a possibility given the current threat level in the United Kingdom.”

An appeal by Andy against WMFS to the first-tier information rights tribunal mainly on the grounds that other Fire Services in the UK had already released their information on fire hydrants was rejected. The judges agreed with the Information Commissioner’s Office (ICO) assessment:

“The ICO concluded that the withholding of the information was reasonably necessary for national security and a listing of hydrants and their locations would assist in the planning of an attack by poisoning on water supply infrastructure by identifying access points. Such an attack on the infrastructure would be in the domain of national security. While hydrants are visible a comprehensive list of the precise location of every hydrant would place in the public domain more information than is available through hydrants being visible. WMFS supplied a list of attacks and attempted attacks on water supplies.The ICO considered that such an attack was plausible.”

There is a method for contaminating the water supply known as backflow contamination, which is considered to be reasonably easy according to an excellent paper here which quotes extensively from US military and homeland security sources. Fire hydrants are one possible source to generate backflow.

My limited understanding of water networks is that backflow is a general problem for the water supply industry, which can mitigate the effects with backflow prevention devices. Just how extensive and successful this mitigation is in the UK (and thus how safe we are from backflow attacks) will remain a commercial mystery, as our water supply is in the hands of private companies who are under no obligation to reveal such information. ( I have another story about water networks and open information which will be the subject of a later blog)

Given the cult of secrecy that exists at all levels of official Britain, it also seems unlikely there will be any disclosure of the risk levels of such an attack vector so that we might make up our own minds based on the data. We will have to make do with the assurances of “those that know best”. The public’s role seems to be limited primarily to that of potential or actual victims.

This issue does raise some interesting ethical challenges for OpenStreetMap as it seems to be sending us back several centuries when accurate maps were regarded as military secrets. Or to the days of the Cold War when our national mapping agency the Ordnance Survey would obligingly leave blank spaces on their maps at the sites of military installations. Would a map of fire hydrants in the West Midlands be construed as offence under ss57&58 of the Terrorism Act 2000: collecting, possessing or making a record of information of a kind likely to be useful to a person committing or preparing an act of terrorism? Should we be standing up for opendata, one of the founding principles of OpenStreetMap, or protecting  (supposedly) national security? What is the position in other jurisdictions where there are more draconian restrictions about identifying and mapping military installations and “national critical infrastructure”? Are mappers more circumspect there?  What’s the legal postion- is the OSMF liable for prosecution or just the contributing mapper or both? Should OSMF comply with a demand to remove information on the grounds of national security? Or to refrain from collecting it in the first instance?

According to taginfo there are only 1786 fire hydrants mapped in the UK, of which there are now about 300  in the West Midlands. Prior to my interest being piqued by this sorry tale there were only about 5 fire hydrants mapped in the West Midlands. Judging by the density of fire hydrants I’ve discovered so far, there are probably thousands in the West Midlands so I seriously doubt whether we’ll ever crowdsource the location of all of them,(or even many more than we have already).

Nonetheless, it’s been an education in another aspect of urban infrastructure I wasn’t really aware of, and a confirmation of the patrician “you don’t need to know about that” attitude of much of British officialdom.

(all of the images are my own and are published here as public domain)

.

OpenStreetMap UK Local Chapter officially exists

posted in: Participate | 0
On Saturday morning I received from Companies House the Certicifate of Incorporation for OpenStreetMap United Kingdom Community Interest Company Ltd. It is a private limited company, limited by guarantee.

It’s taken us a year to get this far, through a protracted and  often tedious process of agreeing Memorandum and Articles of Association,Community Interest Statement, form-filling, and signature gathering. Thankyou to everyone who participated, especially the volunteers who agreed to be the first directors necessary to get the thing off the ground.

Now we can start doing the fun stuff of how we make this work and transform it into a living organisation.(Although I’m sure we’ll still have our share of bureaucratic process to navigate).

Congratulations!

posted in: Participate | 1

Congratulations to everyone who has participated in the various UK Quarterly Projects. We were nominated for an award at the recent international State of the Map 2016 conference in Brussels.

Unfortunately we weren’t awsome enough to win, but it’s nice to be recognised, even though that’s not why we do things. It’s just fun!

Thanks to everyone who voted for us.

Anyway here’s  the nice certificate that Rob brought back from Brussels

osm-award

UK Summer Quarterly Project

Today saw us pass the 1,000 total for farmyards added during this project(1008 to be precise).

Well done to everyone who has participated. What has been the most unusual farm name anyone has come across?

stone_built_slurry_tank_at_moel_y_mab-_part_of_the_leighton_model_farm

Picture Wikimedia Commons: Rod Trevaskus       cc0

Only a couple of weeks left – time to start thinking about our next quarterly project, while we see how many more farmyards we can add.

Busy days

posted in: Participate, Use The Map | 1

Yesterday I attended the Landor Future Highways Conference at the new iCentrum building on Birmingham Science Park, at the invitation of the Birmingham ODI (Open Data Institute) node. It was good to see so many OSM maps being used by presenters (thankyou Devon County Council and TransportAPI). I  was provided an adhoc slot into the programme to deliver a lightning 5 minute presentation about OpenStreetMap (thankyou Birmingham ODI node). It was a bit daunting presenting to highway professionals, especially in front of a screen the size of an IMAX cinema!

Then it was off to Leicester to train new volunteers in editing with iD, as part of Leicester City Council’s walking-mapping project which is an  extension of their weekly Leicester health Walks. Thanks to Sara for organising the training and to Chris, Alan, Stan and AJ who took their first steps editing OpenStreetMap. We all look forward to you increasing the data coverage in Leicester.

I’ve just about finished the edits from our June mini-mapping party in Kidderminster  from 2 weeks ago, when guess what?  Our July mini-mapping party is tonight in Tamworth – everyone welcome. We’ll be meeting in the Globe Inn Lower Gungate, Tamworth B79 7AT at 8pm.

 

Summer Programme

posted in: Mapping Party, Participate | 1

To take advantage of the good weather and light evenings during the summer months we tour around the region mapping areas which look either interesting or not well-mapped.

So far this year we have visited Stratford-upon-Avon where we attempted to map all the tourist accommodation (it is the 400th annivarsary of Shakespeare’s death and this town is a global tourist destination). The ongoing project is to map all the buildings – help appreciated!

Last week saw us in Atherstone where our mapping effort was eclectic to say the least

We have mapped out the remainder of the venues:

June will be Kidderminster on a Saturday ( pub & date tbc)

July will be Tamworth Wed July 6th (Globe Inn, Lower Gungate)

August is the 10th anniversary of mappamercia so we’re going to map as much of WR10 as possible, centred on Pershore (pub & date tbc)

September will be somewhere in Herefordshire (some input from mappers in Herefordshire appreciated – given the distance involved for most attendees this will probably be a Saturday too)

Becoming mobile: MAPS.ME and Geopedia

posted in: Map Improvements, Participate | 0

Back in June of last year we shared with you Richard Fairhurst’s State of the Map US talk in which he set out the case for more mobile OpenStreetMap apps. One of the key message was that “we need to think about smartphone editors” as a way to continue growing OpenStreetMap. This month we got a new mobile editor and a new app to help you explore your surroundings.

Firstly the new editor. This comes in the form of an update to the popular MAPS.ME. Even before this update MAPS.ME was a essential app for any OpenStreetMapper. It provides a simple to use offline map with voice over guidance for travel directions by car or on foot. It’s interface is simple and effective: for example, clicking on a map feature brings up a slider at the bottom of the screen providing more information.

On April 5th 2016, MAPS.ME developer Ilya Zverev explained how you can now edit the map information direct from the app. This is perfect for adding addresses, phone numbers and opening hours to existing map features whilst out and about.

Editing OpenStreetMap with Maps.Me
Editing OpenStreetMap with MAPS.ME

With more than 7 million monthly active users MAPS.ME is aiming to be the number one OpenStreetMap editor. Although OpenStreetMap’s editor usage stats are a little out of date, it’s already clear that MAPS.ME is the most used mobile editor measured by number of users.

The second addition to mobile comes in the form of Geopedia. This is a neat little app that takes the OpenStreetMap base map and overlays the wikipedia database. Clicking on the map shows you all the nearby features that have a wikipedia article (we understand this is based on wikipedia’s latitude/longitude data rather than the wikipedia tag stored within OpenStreetMap).

Searching for nearby attractions with geopedia.
Searching for nearby attractions with geopedia.

As this links to wikipedia you can view the articles in multiple languages and view an image if one is available. It works well to find new and exciting places to explore in your neighbourhood, or whilst away travelling. And of course you can use it as a prompt to add text and photos to wikipedia, and map details to OpenStreetMap – perhaps via MAPS.ME!