Latest posts for tag osm
While traveling around Germany, one notices that most towns have a Greek or Italian restaurant, and they all kind of have the same names. How bad is that lack of fantasy?
Let's play with https://overpass-turbo.eu/. Select a bounding box and run this query:
node [cuisine=greek] ({{bbox}}); out;
Export the results as gpx and have some fun on the command line:
sed -nre 's/^name=([^<]+).*/\1/p' /tmp/greek.gpx \ | sed -re 's/ *(Grill|Restaurant|Tavern[ae]) *//g' \ | sort | uniq -c | sort -nr > /tmp/greek.txt
Likewise, with Italian restaurants, you can use cuisine=italian and something like:
sed -nre 's/^name=([^<]+).*/\1/p' /tmp/italian.gpx \ | sed -re 's/ *(Restaurant|Ristorante|Pizzeria) *//g' \ | sort | uniq -c | sort -nr > /tmp/italian.txt
Here are the top 20 that came out for Greek:
162 Akropolis
91 Delphi
86 Poseidon
78 Olympia
78 Mykonos
78 Athen
76 Hellas
74 El Greco
71 Rhodos
57 Dionysos
53 Kreta
50 Syrtaki
49 Korfu
43 Santorini
43 Athos
40 Mythos
39 Zorbas
35 Artemis
33 Meteora
29 Der Grieche
Here are the top 20 that came out for Italian, with a sadly ubiquitous franchise as an outlier:
66 Vapiano
64 Bella Italia
59 L'Osteria
54 Roma
43 La Piazza
38 La Dolce Vita
38 Dolce Vita
35 Italia
32 Pinocchio
31 Toscana
30 Venezia
28 Milano
28 Mamma Mia
27 Bella Napoli
25 San Marco
24 Portofino
22 La Piazzetta
22 La Gondola
21 Da Vinci
21 Da Pino
One can play a game while traveling: being the first to spot a Greek or Italian restaurant earns more points the more unusual its name is. But beware of being too quick! If you try to claim points for one of the restaurant with the top-5 most common names, you will actually will actually lose points!
Have fun playing with other combinations of areas and cuisine: the Overpass API is pretty cool!
Update:
Rather than running xml through sed, one can export geojson, then parse it with the excellent jq:
jq -r '.features[].properties.name' italian.json \ | sed -re 's/ *(Restaurant|Ristorante|Pizzeria) *//g' \ | sort | uniq -c | sort -nr > /tmp/italian.txt
I like the idea of matching photos to GPS traces. In Debian there is gpscorrelate but it's almost unusable to me because of bug #473362 and it has an awkward way of specifying time offsets.
Here at SoTM10 someone told me that
exiftool gained -geosync and -geotag
options. So it's just a matter of creating a little tool that shows a photo and
asks you to type the GPS time you see in it.
Apparently there are no bindings or GIR files for gtkimageview in Debian, so I'll have to use C.
Here is a C prototype:
/* * gpsoffset - Compute EXIF time offset from a photo of a gps display * * Use with exiftool -geosync=... -geotag trace.gpx DIR * * Copyright (C) 2009--2010 Enrico Zini <enrico@enricozini.org> * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or * (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ #define _XOPEN_SOURCE /* glibc2 needs this */ #include <time.h> #include <gtkimageview/gtkimageview.h> #include <libexif/exif-data.h> #include <stdio.h> #include <stdlib.h> static int load_time(const char* fname, struct tm* tm) { ExifData* exif_data = exif_data_new_from_file(fname); ExifEntry* exif_time = exif_data_get_entry(exif_data, EXIF_TAG_DATE_TIME); if (exif_time == NULL) { fprintf(stderr, "Cannot find EXIF timetamp\n"); return -1; } char buf[1024]; exif_entry_get_value(exif_time, buf, 1024); //printf("val2: %s\n", exif_entry_get_value(t2, buf, 1024)); if (strptime(buf, "%Y:%m:%d %H:%M:%S", tm) == NULL) { fprintf(stderr, "Cannot match EXIF timetamp\n"); return -1; } return 0; } static time_t exif_ts; static GtkWidget* res_lbl; void date_entry_changed(GtkEditable *editable, gpointer user_data) { const gchar* text = gtk_entry_get_text(GTK_ENTRY(editable)); struct tm parsed; if (strptime(text, "%Y-%m-%d %H:%M:%S", &parsed) == NULL) { gtk_label_set_text(GTK_LABEL(res_lbl), "Please enter a date as YYYY-MM-DD HH:MM:SS"); } else { time_t img_ts = mktime(&parsed); int c; int res; if (exif_ts < img_ts) { c = '+'; res = img_ts - exif_ts; } else { c = '-'; res = exif_ts - img_ts; } char buf[1024]; if (res > 3600) snprintf(buf, 1024, "Result: %c%ds -geosync=%c%d:%02d:%02d", c, res, c, res / 3600, (res / 60) % 60, res % 60); else if (res > 60) snprintf(buf, 1024, "Result: %c%ds -geosync=%c%02d:%02d", c, res, c, (res / 60) % 60, res % 60); else snprintf(buf, 1024, "Result: %c%ds -geosync=%c%d", c, res, c, res); gtk_label_set_text(GTK_LABEL(res_lbl), buf); } } int main (int argc, char *argv[]) { // Work in UTC to avoid mktime applying DST or timezones setenv("TZ", "UTC"); const char* filename = "/home/enrico/web-eddie/galleries/2010/04-05-Uppermill/P1080932.jpg"; gtk_init (&argc, &argv); struct tm exif_time; if (load_time(filename, &exif_time) != 0) return 1; printf("EXIF time: %s\n", asctime(&exif_time)); exif_ts = mktime(&exif_time); GtkWidget* window = gtk_window_new(GTK_WINDOW_TOPLEVEL); GtkWidget* vb = gtk_vbox_new(FALSE, 0); GtkWidget* hb = gtk_hbox_new(FALSE, 0); GtkWidget* lbl = gtk_label_new("Timestamp:"); GtkWidget* exif_lbl; { char buf[1024]; strftime(buf, 1024, "EXIF time: %Y-%m-%d %H:%M:%S", &exif_time); exif_lbl = gtk_label_new(buf); } GtkWidget* date_ent = gtk_entry_new(); res_lbl = gtk_label_new("Result:"); GtkWidget* view = gtk_image_view_new(); GdkPixbuf* pixbuf = gdk_pixbuf_new_from_file(filename, NULL); gtk_box_pack_start(GTK_BOX(hb), lbl, FALSE, TRUE, 0); gtk_box_pack_start(GTK_BOX(hb), date_ent, TRUE, TRUE, 0); gtk_signal_connect(GTK_OBJECT(date_ent), "changed", (GCallback)date_entry_changed, NULL); { char buf[1024]; strftime(buf, 1024, "%Y-%m-%d %H:%M:%S", &exif_time); gtk_entry_set_text(GTK_ENTRY(date_ent), buf); } gtk_widget_set_size_request(view, 500, 400); gtk_image_view_set_pixbuf(GTK_IMAGE_VIEW(view), pixbuf, TRUE); gtk_container_add(GTK_CONTAINER(window), vb); gtk_box_pack_start(GTK_BOX(vb), view, TRUE, TRUE, 0); gtk_box_pack_start(GTK_BOX(vb), hb, FALSE, TRUE, 0); gtk_box_pack_start(GTK_BOX(vb), exif_lbl, FALSE, TRUE, 0); gtk_box_pack_start(GTK_BOX(vb), res_lbl, FALSE, TRUE, 0); gtk_widget_show_all(window); gtk_main (); return 0; }
And here is its simple makefile:
CFLAGS=$(shell pkg-config --cflags gtkimageview libexif) LDFLAGS=$(shell pkg-config --libs gtkimageview libexif) gpsoffset: gpsoffset.c
It's a simple prototype but it's a working prototype and seems to do the job for me.
I currently cannot find out why after I click on the text box, there seems to be no way to give the focus back to the image viewer so I can control it with keys.
There is another nice algorithm to compute time offsets to be implemented: you choose a photo taken from a known place and drag it on that place on a map: you can then look for the nearest point on your GPX trace and compute the time offset from that.
I have seen that there are programs for geotagging photos that implement all such algorithms, and have a nice UI, but I haven't seen any in Debian.
Are there any such softwares that can be packaged?
If not, the interpolation and annotation tasks can now already be performed by exiftool, so it's just a matter of building a good UI, and I would love to see someone picking up the task.
Third step of my SoTM10 pet project: finding the POIs.
I put together a query to find all nodes with a given tag inside a bounding box, and also a query to find all the tag values for a given tag name inside a bounding box.
The result is this simple POI search engine:
#
# poisearch - simple geographical POI search engine
#
# Copyright (C) 2010 Enrico Zini <enrico@enricozini.org>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
#
from pysqlite2 import dbapi2 as sqlite
class PoiDB(object):
def __init__(self):
self.db = sqlite.connect("pois.db")
self.db.enable_load_extension(True)
self.db.execute("SELECT load_extension('libspatialite.so')")
self.oldsearch = []
self.bbox = None
def set_bbox(self, xmin, xmax, ymin, ymax):
'''Set bbox for searches'''
self.bbox = (xmin, xmax, ymin, ymax)
def tagid(self, name, val):
'''Get the database ID for a tag'''
c = self.db.cursor()
c.execute("SELECT id FROM tag WHERE name=? AND value=?", (name, val))
res = None
for row in c:
res = row[0]
return res
def tagnames(self):
'''Get all tag names'''
c = self.db.cursor()
c.execute("SELECT DISTINCT name FROM tag ORDER BY name")
for row in c:
yield row[0]
def tagvalues(self, name, use_bbox=False):
'''
Get all tag values for a given tag name,
optionally in the current bounding box
'''
c = self.db.cursor()
if self.bbox is None or not use_bbox:
c.execute("SELECT DISTINCT value FROM tag WHERE name=? ORDER BY value", (name,))
else:
c.execute("SELECT DISTINCT tag.value FROM poi, poitag, tag"
" WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE ("
" xmin >= ? AND xmax <= ? AND ymin >= ? AND ymax <= ?) )"
" AND poitag.tag = tag.id AND poitag.poi = poi.id"
" AND tag.name=?",
self.bbox + (name,))
for row in c:
yield row[0]
def search(self, name, val):
'''Get all name:val tags in the current bounding box'''
# First resolve the tagid
tagid = self.tagid(name, val)
if tagid is None: return
c = self.db.cursor()
c.execute("SELECT poi.name, poi.data, X(poi.geom), Y(poi.geom) FROM poi, poitag"
" WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE ("
" xmin >= ? AND xmax <= ? AND ymin >= ? AND ymax <= ?) )"
" AND poitag.tag = ? AND poitag.poi = poi.id",
self.bbox + (tagid,))
self.oldsearch = []
for row in c:
self.oldsearch.append(row)
yield row[0], simplejson.loads(row[1]), row[2], row[3]
def count(self, name, val):
'''Count all name:val tags in the current bounding box'''
# First resolve the tagid
tagid = self.tagid(name, val)
if tagid is None: return
c = self.db.cursor()
c.execute("SELECT COUNT(*) FROM poi, poitag"
" WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE ("
" xmin >= ? AND xmax <= ? AND ymin >= ? AND ymax <= ?) )"
" AND poitag.tag = ? AND poitag.poi = poi.id",
self.bbox + (tagid,))
for row in c:
return row[0]
def replay(self):
for row in self.oldsearch:
yield row[0], simplejson.loads(row[1]), row[2], row[3]
Problem 3 solved: now on to the next step, building a user interface for it.
Second step of my SoTM10 pet project: creating a searchable database with the points. What a fantastic opportunity to learn Spatialite.
Learning Spatialite is easy. For example, you can use the two tutorials with catchy titles that assume your best wish in life is to create databases out of shapefiles using a pre-built, i386-only executable GUI binary downloaded over an insecure HTTP connection.
To be fair, the second of those tutorials is called "An almost Idiot's Guide", thus expliciting the requirement of being an almost idiot in order to happily acquire and run software in that way.
Alternatively, you can use A quick tutorial to SpatiaLite which is so quick it has examples that lead you to write SQL queries that trigger all sorts of vague exceptions at insert time. But at least it brought me a long way forward, at which point I could just cross reference things with PostGIS documentation to find out the right way of doing things.
So, here's the importer script, which will probably become my reference example for how to get started with Spatialite, and how to use Spatialite from Python:
#!/usr/bin/python
#
# poiimport - import nodes from OSM into a spatialite DB
#
# Copyright (C) 2010 Enrico Zini <enrico@enricozini.org>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
#
import xml.sax
import xml.sax.handler
from pysqlite2 import dbapi2 as sqlite
import simplejson
import sys
import os
class OSMPOIReader(xml.sax.handler.ContentHandler):
'''
Filter SAX events in a OSM XML file to keep only nodes with names
'''
def __init__(self, consumer):
self.consumer = consumer
def startElement(self, name, attrs):
if name == "node":
self.attrs = attrs
self.tags = dict()
elif name == "tag":
self.tags[attrs["k"]] = attrs["v"]
def endElement(self, name):
if name == "node":
lat = float(self.attrs["lat"])
lon = float(self.attrs["lon"])
id = int(self.attrs["id"])
#dt = parse(self.attrs["timestamp"])
uid = self.attrs.get("uid", None)
uid = int(uid) if uid is not None else None
user = self.attrs.get("user", None)
self.consumer(lat, lon, id, self.tags, user=user, uid=uid)
class Importer(object):
'''
Create the spatialite database and populate it
'''
TAG_WHITELIST = set(["amenity", "shop", "tourism", "place"])
def __init__(self, filename):
self.db = sqlite.connect(filename)
self.db.enable_load_extension(True)
self.db.execute("SELECT load_extension('libspatialite.so')")
self.db.execute("SELECT InitSpatialMetaData()")
self.db.execute("INSERT INTO spatial_ref_sys (srid, auth_name, auth_srid,"
" ref_sys_name, proj4text) VALUES (4326, 'epsg', 4326,"
" 'WGS 84', '+proj=longlat +ellps=WGS84 +datum=WGS84"
" +no_defs')")
self.db.execute("CREATE TABLE poi (id int not null unique primary key,"
" name char, data text)")
self.db.execute("SELECT AddGeometryColumn('poi', 'geom', 4326, 'POINT', 2)")
self.db.execute("SELECT CreateSpatialIndex('poi', 'geom')")
self.db.execute("CREATE TABLE tag (id integer primary key autoincrement,"
" name char, value char)")
self.db.execute("CREATE UNIQUE INDEX tagidx ON tag (name, value)")
self.db.execute("CREATE TABLE poitag (poi int not null, tag int not null)")
self.db.execute("CREATE UNIQUE INDEX poitagidx ON poitag (poi, tag)")
self.tagid_cache = dict()
def tagid(self, k, v):
key = (k, v)
res = self.tagid_cache.get(key, None)
if res is None:
c = self.db.cursor()
c.execute("SELECT id FROM tag WHERE name=? AND value=?", key)
for row in c:
self.tagid_cache[key] = row[0]
return row[0]
self.db.execute("INSERT INTO tag (id, name, value) VALUES (NULL, ?, ?)", key)
c.execute("SELECT last_insert_rowid()")
for row in c:
res = row[0]
self.tagid_cache[key] = res
return res
def __call__(self, lat, lon, id, tags, user=None, uid=None):
# Acquire tag IDs
tagids = []
for k, v in tags.iteritems():
if k not in self.TAG_WHITELIST: continue
for val in v.split(";"):
tagids.append(self.tagid(k, val))
# Skip elements that don't have the tags we want
if not tagids: return
geom = "POINT(%f %f)" % (lon, lat)
self.db.execute("INSERT INTO poi (id, geom, name, data)"
" VALUES (?, GeomFromText(?, 4326), ?, ?)",
(id, geom, tags["name"], simplejson.dumps(tags)))
for tid in tagids:
self.db.execute("INSERT INTO poitag (poi, tag) VALUES (?, ?)", (id, tid))
def done(self):
self.db.commit()
# Get the output file name
filename = sys.argv[1]
# Ensure we start from scratch
if os.path.exists(filename):
print >>sys.stderr, filename, "already exists"
sys.exit(1)
# Import
parser = xml.sax.make_parser()
importer = Importer(filename)
handler = OSMPOIReader(importer)
parser.setContentHandler(handler)
parser.parse(sys.stdin)
importer.done()
Let's run it:
$ ./poiimport pois.db < pois.osm
SpatiaLite version ..: 2.4.0 Supported Extensions:
- 'VirtualShape' [direct Shapefile access]
- 'VirtualDbf' [direct Dbf access]
- 'VirtualText' [direct CSV/TXT access]
- 'VirtualNetwork' [Dijkstra shortest path]
- 'RTree' [Spatial Index - R*Tree]
- 'MbrCache' [Spatial Index - MBR cache]
- 'VirtualFDO' [FDO-OGR interoperability]
- 'SpatiaLite' [Spatial SQL - OGC]
PROJ.4 Rel. 4.7.1, 23 September 2009
GEOS version 3.2.0-CAPI-1.6.0
$ ls -l --si pois*
-rw-r--r-- 1 enrico enrico 17M Jul 9 23:44 pois.db
-rw-r--r-- 1 enrico enrico 37M Jul 9 16:20 pois.osm
$ spatialite pois.db
SpatiaLite version ..: 2.4.0 Supported Extensions:
- 'VirtualShape' [direct Shapefile access]
- 'VirtualDbf' [direct DBF access]
- 'VirtualText' [direct CSV/TXT access]
- 'VirtualNetwork' [Dijkstra shortest path]
- 'RTree' [Spatial Index - R*Tree]
- 'MbrCache' [Spatial Index - MBR cache]
- 'VirtualFDO' [FDO-OGR interoperability]
- 'SpatiaLite' [Spatial SQL - OGC]
PROJ.4 version ......: Rel. 4.7.1, 23 September 2009
GEOS version ........: 3.2.0-CAPI-1.6.0
SQLite version ......: 3.6.23.1
Enter ".help" for instructions
spatialite> select id from tag where name="amenity" and value="fountain";
24
spatialite> SELECT poi.name, poi.data, X(poi.geom), Y(poi.geom) FROM poi, poitag WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE (xmin >= 2.56 AND xmax <= 2.90 AND ymin >= 41.84 AND ymax <= 42.00) ) AND poitag.tag = 24 AND poitag.poi = poi.id;
Font Picant de la Cellera|{"amenity": "fountain", "name": "Font Picant de la Cellera"}|2.616045|41.952449
Font de Can Pla|{"amenity": "fountain", "name": "Font de Can Pla"}|2.622354|41.974724
Font de Can Ribes|{"amenity": "fountain", "name": "Font de Can Ribes"}|2.62311|41.979193
It's impressive: I've got all sort of useful information for the whole of Spain in just 17Mb!
Let's put it to practice: I'm thirsty, is there any water fountain nearby?
spatialite> SELECT count(1) FROM poi, poitag WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE (xmin >= 2.80 AND xmax <= 2.85 AND ymin >= 41.97 AND ymax <= 42.00) ) AND poitag.tag = 24 AND poitag.poi = poi.id;
0
Ouch! No water fountains mapped in Girona... yet.
Problem 2 solved: now on to the next step, [[trying to show the results in some usable way||osm-search-nodes]].
I have a pet project here at SoTM10: create a tool for searching nearby POIs while offline.
The idea is to have something in my pocket (FreeRunner or N900), which doesn't require an internet connection, and which can point me at the nearest fountains, post offices, atm machines, bars and so on.
The first step is to obtain a list of POIs.
In theory one can use Xapi but all the known Xapi servers appear to be down at the moment.
Another attempt is to obtain it by filtering all nodes with the tags we want out of a planet OSM extract. I downloaded the Spanish one and set to work.
First I tried with xmlstarlet, but it ate all the RAM and crashed my laptop, because for some reason, on my laptop the Linux kernels up to 2.6.32 (don't now about later ones) like to swap out ALL running apps to cache I/O operations, which mean that heavy I/O operations swap out the very programs performing them, so the system gets caught in some infinite I/O loop and dies. Or at least this is what I've figured out so far.
So, we need SAX. I put together this prototype in Python, which can process a nice 8MB/s of OSM data for quite some time with a constant, low RAM usage:
#!/usr/bin/python
#
# poifilter - extract interesting nodes from OSM XML files
#
# Copyright (C) 2010 Enrico Zini <enrico@enricozini.org>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
#
import xml.sax
import xml.sax.handler
import xml.sax.saxutils
import sys
class XMLSAXFilter(xml.sax.handler.ContentHandler):
'''
A SAX filter that is a ContentHandler.
There is xml.sax.saxutils.XMLFilterBase in the standard library but it is
undocumented, and most of the examples using it you find online are wrong.
You can look at its source code, and at that point you find out that it is
an offensive practical joke.
'''
def __init__(self, downstream):
self.downstream = downstream
# ContentHandler methods
def setDocumentLocator(self, locator):
self.downstream.setDocumentLocator(locator)
def startDocument(self):
self.downstream.startDocument()
def endDocument(self):
self.downstream.endDocument()
def startPrefixMapping(self, prefix, uri):
self.downstream.startPrefixMapping(prefix, uri)
def endPrefixMapping(self, prefix):
self.downstream.endPrefixMapping(prefix)
def startElement(self, name, attrs):
self.downstream.startElement(name, attrs)
def endElement(self, name):
self.downstream.endElement(name)
def startElementNS(self, name, qname, attrs):
self.downstream.startElementNS(name, qname, attrs)
def endElementNS(self, name, qname):
self.downstream.endElementNS(name, qname)
def characters(self, content):
self.downstream.characters(content)
def ignorableWhitespace(self, chars):
self.downstream.ignorableWhitespace(chars)
def processingInstruction(self, target, data):
self.downstream.processingInstruction(target, data)
def skippedEntity(self, name):
self.downstream.skippedEntity(name)
class OSMPOIHandler(XMLSAXFilter):
'''
Filter SAX events in a OSM XML file to keep only nodes with names
'''
PASSTHROUGH = ["osm", "bound"]
TAG_WHITELIST = set(["amenity", "shop", "tourism", "place"])
def startElement(self, name, attrs):
if name in self.PASSTHROUGH:
self.downstream.startElement(name, attrs)
elif name == "node":
self.attrs = attrs
self.tags = []
self.propagate = False
elif name == "tag":
if self.tags is not None:
self.tags.append(attrs)
if attrs["k"] in self.TAG_WHITELIST:
self.propagate = True
else:
self.tags = None
self.attrs = None
def endElement(self, name):
if name in self.PASSTHROUGH:
self.downstream.endElement(name)
elif name == "node":
if self.propagate:
self.downstream.startElement("node", self.attrs)
for attrs in self.tags:
self.downstream.startElement("tag", attrs)
self.downstream.endElement("tag")
self.downstream.endElement("node")
def ignorableWhitespace(self, chars):
pass
def characters(self, content):
pass
# Simple stdin->stdout XMl filter
parser = xml.sax.make_parser()
handler = OSMPOIHandler(xml.sax.saxutils.XMLGenerator(sys.stdout, "utf-8"))
parser.setContentHandler(handler)
parser.parse(sys.stdin)
Let's run it:
$ bzcat /store/osm/spain.osm.bz2 | pv | ./poifilter > pois.osm
[...]
$ ls -l --si pois.osm
-rw-r--r-- 1 enrico enrico 19M Jul 10 23:56 pois.osm
$ xmlstarlet val pois.osm
pois.osm - valid
Problem 1 solved: now on to the next step: importing the nodes in a database.
The FreeRunner has a headset which includes a microphone and a button. When doing OpenStreetMap mapping, it would be very useful to be able to keep tangogps on the display and be able to mark waypoints using the headset button, and to record an audio track using the headset microphone.
In this way, I can use tangogps to see where I need to go, where it's already mapped and where it isn't, and then I can use the headset to mark waypoints corresponding to the audio track, so that later I can take advantage of JOSM's audio mapping features.
Enter audiomap:
$ audiomap --help Usage: audiomap [options] Create a GPX and audio trackFind the times in the wav file when there is clear voice among the noise Options: --version show program's version number and exit -h, --help show this help message and exit -v, --verbose verbose mode -m, --monitor only keep the GPS on and monitor satellite status -l, --levels only show input levels
If called without parameters, or with -v which is suggested, it will:
- Fix the mixer settings so that it can record from the headset and detect headset button presses.
- Show a monitor of GPS satellite information until it gets a fix.
- Synchronize the system time with the GPS time so that the timestamps of the files that are created afterwards are accurate.
- Start recording a GPX track.
- Start recording audio.
- Record a GPX waypoint for every headset button press.
When you are done, you stop audiomap with ^C and it will properly close the
.wav file, close the tags in the GPX waypoint and track files and restore the
mixer settings.
You can plug the headset out and record using the handset microphone, but then you will not be able to set waypoints until you plug the headset back in.
After you stop audiomap, you will have a track, waypoints and .wav file
ready to be loaded in JOSM.
Big thanks go to Luca Capello for finding out how to detect headset button presses.
I was missing a simple command line tool that allows me to perform basic GPS queries in shellscripts.
Enter getgps:
# getgps --help
Usage: getgps [options]
Simple GPS query tool for the FSO stack
Options:
--version show program's version number and exit
-h, --help show this help message and exit
-v, --verbose verbose mode
-q, --quiet suppress normal output
--fix check if we have a fix
-s, --sync-time set system time from GPS time
--info get all GPS information
--info-connection get GPS connection information
--info-fix get GPS fix information
--info-position get GPS position information
--info-accuracy get GPS accuracy information
--info-course get GPS course information
--info-time get GPS time information
--info-satellite get GPS satellite information
So finally I can write little GPS-aware scripts:
if getgps --fix -q
then
start_gps_aware_program
else
start_gps_normal_program
fi
Or this.
I have it in my TODO list to implement taking waypoints when pressing the headset button of the openmoko, but that is not done yet.
In the meantime, I did some experiments with audio mapping, and since I did not manage to enter waypoints while recording them, I was looking for a way to make use of them anyway.
Enter findvoice:
$ ./findvoice --help
Usage: findvoice [options] wavfile
Find the times in the wav file when there is clear voice among the noise
Options:
--version show program's version number and exit
-h, --help show this help message and exit
-v, --verbose verbose mode
-p NUM, --percentile=NUM
percentile to use to discriminate noise from voice
(default: 90)
-t, --timestamps print timestamps instead of human readable information
You give it a wav file, and it will output a list of timestamps corresponding to where it things that you were talking clearly and near the FreeRunner / voice recorder instead of leaving the recorder dangling to pick up background noise.
Its algorithm is crude and improvised because I have no background whatsoever in audio processing, but it basically finds those parts of the audio file where the variance of the samples is above a given percentile: the higher the percentile, the less timestamps you get; the lower the percentile, the more likely it is that it picks a period of louder noise.
For example, you can automatically extract waypoints out of an audio file by using it together with Geocoding Unix timestamps:
./findvoice -t today.wav | ./gpxinterpolate today.gpx > today-waypoints.gpx
The timestamps it outputs are computed using the modification time of the
.wav file: if your system clock was decently synchronised (which you can do
with getgps), then the mtime of the wav is the time of the end of the
recording, which gives the needed reference to compute timestamps that are
absolute in time.
For example:
getgps --sync-time
arecord file.wav
^C
./findvoice -t file.wav | ./gpxinterpolate today.gpx > today-waypoints.gpx
Geocoding EXIF tags in JPEG images is fun, but there is more that can benefit from interpolating timestamps over a GPX track.
Enter gpxinterpolate:
$ ./gpxinterpolate --help Usage: gpxinterpolate [options] gpxfile [gpxfile...] Read one or more GPX files and a list of timestamps on standard input. Output a GPX file with waypoints at the location of the GPX track at the given timestamps. Options: --version show program's version number and exit -h, --help show this help message and exit -v, --verbose verbose mode
For example, you can create waypoints interpolating file modification times:
find . -printf "%Ts %p\n" | ./gpxinterpolate ~/tracks/*.gpx > myfiles.gpx
In case you wonder where you were when you modified or accessed a file, now you can find out.
The FreeRunner can record audio. It is nice to record audio: for example I can run the recording in background while I keep tangogps in the screen, and take audio notes about where I am while I am doing mapping for OpenStreetMap.
Here is the script that I put together to create geocoded audio notes:
#!/bin/sh
WORKDIR=~/rec
TMPINFO=`mktemp $WORKDIR/info.XXXXXXXX`
# Sync system time and get GPS info
echo "Synchronising system time..."
getgps --sync-time --info > $TMPINFO
# Compute an accurate basename for the files we generate
BASENAME=~/rec/rec-$(date +%Y-%m-%d-%H-%M-%S)
# Then give a proper name to the file with saved info
mv $TMPINFO $BASENAME.info
# Proper mixer settings for recording
echo "Recording..."
alsactl -f /usr/share/openmoko/scenarios/voip-handset.state restore
arecord -D hw -f cd -r 8000 -t wav $BASENAME.wav
echo "Done"
It works like this:
- It synchronizes the system time from the GPS (if there is a fix) so that the timestamps on the wav files will be as accurate as possible.
- It also gets all sort of information from the GPS and stores them into a file, should you want to inspect it later.
- It records audio until it gets interrupted.
The file name of the files that it generates corresponds to the beginning of the recording. The mtime of the wav file obviously corresponds to the end of the recording. This can be used to later georeference the start and end point of the recording.
You can use this to check mixer levels and that you're actually getting any input:
arecord -D hw -f cd -r 8000 -t wav -V mono /dev/null
The getgps script is now described in its own post.
You may now want to experiment, in JOSM, with "Preferences / Audio settings / Modified times (time stamps) of audio files".