OSM Tiles

Preface

This Document overviews the process of generating OpenStreetMap (OSM) tiles for use with Leaflet.

Obtaining Map Data

The OSM Wiki provides a list of downloads of OSM data. In general, it is not advisable to get data for the entire planet. Geofabrik provides data clustered by individual regions. Pick the subset of the data that you need and download the PBF file.

Rendering Tiles

The OSM tile rendering stack is very involved, rooted in incredibly stale and unsupported versions of python and several libraries, and horribly documented to boot. You may be tempted to set up a rendering stack on bare metal. Don't. Just don't. You will spend the next week combing through dependencies, compiling old versions, and end up completely frustrated.

The NCAR Earth Observing Laboratory has wrapped all of the relevant code/dependencies in a Docker archive. Save yourself and your hair and use this.

Install docker if it's not already installed.
Put your PBF file in a directory by itself (e.g., /tmp/tile_data/osm_data_file.osm.pbf)
Create a directory that will preserve postgresql state needed by the renderer

$> docker run --name osm_tiles\
              --env OSM_IMPORT_CACHE=12000 \
              --env OSM_IMPORT_FILE="/data/{{osm_data_file.osm.pbf}}" 
              -v {{/path/to/postgresql/data/dir}}:/var/lib/postgresql 
              -v {{/osm/data/file/directory}}:/data 
              ncareol/osm-tiles import

For example, if I have:

/data/osm/pbf_files/new_york-latest.osm.pbf
/data/osm/postgresql

$> docker run --name osm_tiles\
              --env OSM_IMPORT_CACHE=12000 \
              --env OSM_IMPORT_FILE="/data/new_york-latest.osm.pbf" 
              -v /data/osm/postgresql:/var/lib/postgresql 
              -v /data/osm/pbf_files:/data 
              ncareol/osm-tiles import

Depending on the size of the region you're importing, this may take a few dozen minutes or more.

Also note that once you run the command once, you don't need the full set of docker flags. From that point, you can achieve the same effect. If you get the flags wrong, you'll need to remove the image.

$> docker run osm_tiles import

I had a bit of a struggle getting the above to work at first. I think it should work, but if not, you may need to initialize the database first (this is definitely the case if you get error messages about missing users, in which case, please tell me to update this doc). The following sequence of commands should fix it:

$> docker run osm_tiles initdb
$> docker run osm_tiles createuser

Once the data is imported, you should be ready to render the tiles:

$> docker run osm_tiles render

This command will take many hours and docker will block until it completes. I strongly suggest running it in a screen or tmux session.

Once it's done, you'll need to get the tiles out

$> docker container cp osm_tiles:/var/lib/mod_tile {{path/to/where/you/want/your/tiles}}

Extract Metatiles

What... you thought you were done? LOL, no. One more step. The docker container's renderd creates "metatiles", 8x8 agglomerations of png files that leaflet won't read directly. Renderd normally breaks up the tiles itself, but if you want to host the files out of a bare directory, you need to do this.

In particular,

You can find some utilities for mucking with renderd's output here. In particular, you want unpackmetatile.pl (local mirror).

The following ruby script will break apart your stuff. Be warned that it will take multiple hours.

require "fileutils"

base_dir = "{{/path/to/your/tile_dir}}/default"
extract_metatile_script = "perl {{path/to/unpackmetatile.pl}}"

levels = Dir["#{base_dir}/*"].sort

# levels = [levels[0]]

levels.each do |l|
  puts("Processing #{l}")
  Dir.chdir(l)
  Dir.mkdir("png") unless File.exists? "png"

  Dir["**/*.meta"].each do |metatile|
    puts("Extracting #{metatile}")
    system("#{extract_metatile_script} #{metatile}")
    Dir["*.png"].each do |tile|
      scale, x, y = tile.sub(/\.png/) { "" }.split("-")
      x = x.to_i.to_s
      y = y.to_i.to_s
      dir = "png/#{x}"
      Dir.mkdir(dir) unless File.exists? dir
      FileUtils.mv(tile, "#{dir}/#{y}.png")
    end
  end
end

Using Leaflet

The resulting tiles will be 256x256, so the following leafletJS initializer will work

  L.tileLayer('https://my.server/path/to/tiles/{id}/{z}/png/{x}/{y}.png', {
      attribution: 'Map data &copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors, Imagery © <a href="https://www.mapbox.com/">Mapbox</a>',
      maxZoom: 12,
      id: 'default',
      tileSize: 256,
      zoomOffset: 0,
  }).addTo(mymap);