CentOS 7 Open Street Map Tile Server

Introduction

One of the challenges that I often encounter when designing and developing graph-based systems is how to render entities (graph vertices) onto a map in an environment where there is no internet access or the customer does not have a 3rd party commercial map provider. We are all familiar with Google Maps which is free for apps or websites that are free for anyone to use. However for commercial applications, access to Google Maps and its API requires you to sign up to a pricing plan based on a usage quota. Other commercial Map providers include:

• Mapbox
• Mapzen
• Geofabrik
• Carto
• MapQuest

But what to do when your infrastructure does not connect to the internet or you do not wish to pay for a commercial provider? Well, you just build your own Map server!

What is a Tile Server

There is a really good website called switch2osm.org that provides an overview of what tile servers are and even how to build your own using Ubuntu. Essentially a map is made up of many tiles usually 256 x 256 pixels in size. These tiles are rendered from a database on a server to the client where libaries like Leaflet enable you to integrate them into interactive maps on websites and apps. For example, when you use Google Maps, you are using Google's own map tiles rendered from their own servers. By building your own tile server, you can generate your own tiles which you can store and render from self-hosted servers.

CentOS 7 Tile Server

The website switch2osm.org provides in-depth tutorials of how to build your own tile server using Ubuntu. However, fans of CentOS (RHEL) will find that information on standing up a CentOS 7 tile server is very saturated to say the least. This guide aims to go through the steps to standing up your own CentOS 7 Tile Server.

Software Dependencies

• Open Street Map (OSM) - Database of world maps that are free to use under an open licence. Raw map data of most regions around the world can be downloaded from Open Street Map. This data is then processed by your tile server generating tiles that are rendered and served to the client.
• PostgreSQL - Open source relational database into which OSM map data will be imported.
• PostGIS - Spatial database extender for PostgreSQL supporting geographic objects and location queries in SQL.
• osm2pgsql - Converts Open Street Map data to PostGIS-enabled PostgreSQL databases.
• Mapnik - Geospatial visualisation and processing library that renders tiles.
• Apache HTTP Server - Open source HTTP server conforming to HTTP standards.
• mod_tile - Program to render (and re-render) and serve map tiles using the Apache HTTP Server and Mapnik as the rendering backend.
• Renderd - Priority queueing system for rendering requests.
• Boost C++ - Portable C++ source libraries.
• GEOS C++ - Open source Geometry Engine for C++.
• HarfBuzz - Text shaping library to convert Unicode text into glyph indices and positions.
• Carto - CSS-like Map styling language allowing you to edit the style of maps including shapes, polygons, fonts and colours.
• OSM Carto - Open source Carto Stylesheet for the standard map layer for Open Street Map-based maps.
• OSM Bright - Open source Carto Stylesheet for creating Open Street Map-based maps using Mapnik.

CentOS 7 Minimal ISO Dependencies

There are some pre-requisite packages that need to be installed on your CentOS 7 server before we can get started installing the dependencies listed above. This guide assumes that you have installed the minimal ISO for CentOS 7 (command-line only). If not, or depending on other packages you may have installed historically, some of the pre-requisitie packages listed below may already be installed on your server.

# Dependencies
yum install libpng libtiff libjpeg freetype gdal cairo pycairo sqlite geos boost curl libcurl libicu bzip2-devel libpng-devel libtiff-devel zlib-devel libjpeg-devel libxml2-devel python-setuptools proj-devel proj proj-epsg proj-nad freetype-devel libicu-devel gdal-devel sqlite-devel libcurl-devel cairo-devel pycairo-devel geos-devel protobuf-devel protobuf-c-devel lua-devel cmake proj boost-thread proj-devel autoconf automake libtool pkgconfig ragel gtk-doc glib2 glib2-devel libpng libpng-devel libwebp libtool-ltdl-devel python-devel harfbuzz harfbuzz-devel harfbuzz-icu boost-devel cabextract xorg-x11-font-utils fontconfig perl-DBD-Pg mesa-libGLU-devel

# GCC++ 14 standards are required for Mapnik so we shall install the Dev Toolset from the CentOS Software Collections
yum install centos-release-scl
yum install devtoolset-6
scl enable devtoolset-6 bash

# Carto can be installed via NodeJS so we will install NodeJS too
yum install nodejs
npm -v

# Git Version Control is required to clone relevant dependency source code repositories
yum install git

We are now ready to proceed with installing the sofware dependencies as listed above.

PostgreSQL and PostGIS

# Install PostgreSQL and PostGIS
rpm -Uvh https://yum.postgresql.org/9.4/redhat/rhel-7-x86_64/pgdg-centos94-9.4-3.noarch.rpm
yum install postgresql94-server postgresql94-devel postgis2_94 postgis2_94-docs postgis2_94-utils pgrouting_94

# Initialise PostgreSQL and Basic Setup
/usr/pgsql-9.4/bin/postgresql94-setup initdb
systemctl enable postgresql-9.4.service
cd /var/lib/pgsql/9.4
vi data/postgresql.conf

    # Add the IP addresses on which the server should listen for connections
    listen_addresses = 'localhost,192.168.1.1'

systemctl start postgresql-9.4.service

# Setup a default Postgres Password (as the 'postgres' user)
su - postgres
psql
postgres=# \password
postgres=# \q
exit

# Basic Authentication Methods
vi data/pg_hba.conf

    # Update your client authentication methods as appropriate
    local   all all                 md5
    host    all all 127.0.0.1/32    md5
    host    all all ::1/128         md5
    host    all all 192.168.1.0/24  md5

# Tile Server OSM Processing Performance
vi data/postgresql.conf

    # Update to suit your server capabilities
    shared_buffers = 128MB
    checkpoint_segments = 20
    maintenance_work_mem = 256MB
    autovacuum = off

# Restart PostgreSQL Server
> systemctl restart postgresql-9.4.service
> systemctl status postgresql-9.4.service

# Check that PostgreSQL is listening on port 5432 by default
> netstat -an | grep 5432

# Create the GIS database (Basic Setup)
su - postgres
psql
postgres=# CREATE DATABASE gis WITH ENCODING = 'UTF8';
postgres=# \q

# Execute PostGIS SQL Installation Files
export PATH=$PATH:/usr/pgsql-9.4/bin
psql gis < /usr/pgsql-9.4/share/contrib/postgis-2.1/postgis.sql
psql gis < /usr/pgsql-9.4/share/contrib/postgis-2.1/spatial_ref_sys.sql
createuser osm -W # No to all questions
createuser apache -W # No to all questions
echo "grant all on geometry_columns to apache;" | psql gis
echo "grant all on spatial_ref_sys to apache;" | psql gis
echo "grant all on geometry_columns to hyperlearningai;" | psql gis
echo "grant all on spatial_ref_sys to hyperlearningai;" | psql gis
exit

# Kernel Configuration change for PostgreSQL OSM Processing Performance
vi /etc/sysctl.conf
    kernel.shmmax=268435456
sysctl -p
sudo sysctl kernel.shmmax

Apache HTTP Server

# Basic Installation with default configuration
yum install httpd

Boost C++

# Boostrap and install
JOBS=`grep -c ^processor /proc/cpuinfo`
wget http://downloads.sourceforge.net/boost/boost_1_63_0.tar.bz2
tar xf boost_1_63_0.tar.bz2
cd boost_1_63_0
./bootstrap.sh
./b2 -d1 -j${JOBS} --with-thread --with-filesystem --with-python --with-regex -sHAVE_ICU=1 --with-program_options --with-system link=shared release toolset=gcc stage
./b2 -d1 -j${JOBS} --with-thread --with-filesystem --with-python --with-regex -sHAVE_ICU=1 --with-program_options --with-system link=shared release toolset=gcc install
sudo bash -c "echo '/usr/local/lib' > /etc/ld.so.conf.d/boost.conf"
sudo ldconfig

Harfbuzz

# Build and install
wget https://www.freedesktop.org/software/harfbuzz/release/harfbuzz-1.4.5.tar.bz2
tar xf harfbuzz-1.4.5.tar.bz2
cd harfbuzz-1.4.5
./configure
make
sudo make install
sudo ldconfig

Mapnik

# Clone and Bootstrap
vi /etc/profile.d/pgsql.sh
    $ export PATH=$PATH:/usr/pgsql-9.4/bin:/usr/pgsql-9.4/lib:/usr/local/lib
source /etc/profile.d/pgsql.sh
git clone git://github.com/mapnik/mapnik
cd mapnik
./bootstrap.sh
./configure

# Handle mapbox/variant.hpp: no such file or directory error - https://github.com/mapnik/mapnik/issues/3246
git submodule sync
git submodule update --init deps/mapbox/variant

# Build and install
make && sudo make install
sudo ldconfig

GEOS C++

# Build and install
wget http://download.osgeo.org/geos/geos-3.6.1.tar.bz2
tar xf geos-3.6.1.tar.bz2
cd geos-3.6.1
./configure && make && sudo make install
sudo ldconfig
vi /etc/ld.so.conf
    /usr/local/lib

osm2pgsql

# Clone, build and install
git clone git://github.com/openstreetmap/osm2pgsql.git
cd osm2pgsql
mkdir build && cd build && cmake ..
make
sudo make install

mod_tile

# Clone and configure
git clone git://github.com/openstreetmap/mod_tile.git
cd mod_tile
./autogen.sh
./configure

# From where you cloned the Mapnik source (see above) copy Mapnick libraries to /usr/include/mapnik
cp -rf mapnik/include/mapnik/* /usr/include/mapnik
cp mapnik/include/mapnik/geometry/box2d.hpp /usr/include/mapnik

# Build and install mod_tile
make
sudo make install
sudo make install-mod_tile
sudo ldconfig

Carto

# Install Carto using NodeJS that we installed earlier
npm install -g carto

OSM Carto Stylesheet

# Clone
git clone git://github.com/gravitystorm/openstreetmap-carto.git
cd openstreetmap-carto
git checkout `git rev-list -n 1 --before="2016-12-04 00:00" master`

# Compile and download shape files
carto project.mml > mapnik.xml
scripts/get-shapefiles.py

Renderd and mod_tile

# Configure Renderd
vi /usr/local/etc/renderd.conf

    # Edit where your paths and number of threads differ
    socketname=/var/run/renderd/renderd.sock
    num_threads=1
    plugins_dir=/usr/local/lib/mapnik/input
    font_dir=/usr/local/lib/mapnik/fonts
    XML={OSM Carto Git Clone Path}/openstreetmap-carto/mapnik.xml # See OSM Carto Stylesheet section
    HOST=192.168.1.1 # Host IP Address

# Configure mod_tile
vi /etc/httpd/conf.d/mod_tile.conf

    # Edit the ServerName and ServerAlias to suit your server
    # Also update LoadTileConfigFile and ModTileRenderdSocketName if this differs on your server
    LoadModule tile_module /etc/httpd/modules/mod_tile.so
    <VirtualHost *:80>
        ServerName map1.earth.dev.hyperlearning.ai
        ServerAlias a.map1.earth.dev.hyperlearning.ai b.map1.earth.dev.hyperlearning.ai c.map1.earth.dev.hyperlearning.ai d.map1.earth.dev.hyperlearning.ai
        DocumentRoot /var/www/html

        # Specify the default base storage path for where tiles live. A number of different storage backends
        # are available, that can be used for storing tiles.  Currently these are a file based storage, a memcached
        # based storage and a RADOS based storage.
        # The file based storage uses a simple file path as its storage path ( /path/to/tiledir )
        # The RADOS based storage takes a location to the rados config file and a pool name ( rados://poolname/path/to/ceph.conf )
        # The memcached based storage currently has no configuration options and always connects to memcached on localhost ( memcached:// )
        #
        # The storage path can be overwritten on a style by style basis from the style TileConfigFile
        ModTileTileDir /var/lib/mod_tile

        # You can either manually configure each tile set with the default png extension and mimetype
        #    AddTileConfig /folder/ TileSetName
        # or manually configure each tile set, specifying the file extension
        #    AddTileMimeConfig /folder/ TileSetName js
        # or load all the tile sets defined in the configuration file into this virtual host.
        # Some tile set specific configuration parameters can only be specified via the configuration file option
        LoadTileConfigFile /usr/local/etc/renderd.conf

        # Specify if mod_tile should keep tile delivery stats, which can be accessed from the URL /mod_tile
        # The default is On. As keeping stats needs to take a lock, this might have some performance impact,
        # but for nearly all intents and purposes this should be negligable ans so it is safe to keep this turned on.
        ModTileEnableStats On

        # Turns on bulk mode. In bulk mode, mod_tile does not request any dirty tiles to be rerendered. Missing tiles
        # are always requested in the lowest priority. The default is Off.
        ModTileBulkMode Off
        ModTileRequestTimeout 3

        # Timeout before giving up for a tile to be rendered that is otherwise missing
        ModTileMissingRequestTimeout 10

        # If tile is out of date, don't re-render it if past this load threshold (users gets old tile)
        ModTileMaxLoadOld 16

        # If tile is missing, don't render it if past this load threshold (user gets 404 error)
        ModTileMaxLoadMissing 50

        # Sets how old an expired tile has to be to be considered very old and therefore get elevated priority in rendering
        ModTileVeryOldThreshold 31536000000000

        # Unix domain socket where we connect to the rendering daemon
        ModTileRenderdSocketName /var/run/renderd/renderd.sock

        # Alternatively you can use a TCP socket to connect to renderd. The first part
        # is the location of the renderd server and the second is the port to connect to.
        #   ModTileRenderdSocketAddr renderd.mydomain.com 7653

        ##
        ## Options controlling the cache proxy expiry headers. All values are in seconds.
        ##
        ## Caching is both important to reduce the load and bandwidth of the server, as
        ## well as reduce the load time for the user. The site loads fastest if tiles can be
        ## taken from the users browser cache and no round trip through the internet is needed.
        ## With minutely or hourly updates, however there is a trade-off between cacheability
        ## and freshness. As one can't predict the future, these are only heuristics, that
        ## need tuning.
        ## If there is a known update schedule such as only using weekly planet dumps to update the db,
        ## this can also be taken into account through the constant PLANET_INTERVAL in render_config.h
        ## but requires a recompile of mod_tile

        ## The values in this sample configuration are not the same as the defaults
        ## that apply if the config settings are left out. The defaults are more conservative
        ## and disable most of the heuristics.

        ##
        ## Caching is always a trade-off between being up to date and reducing server load or
        ## client side latency and bandwidth requirements. Under some conditions, like poor
        ## network conditions it might be more important to have good caching rather than the latest tiles.
        ## Therefor the following config options allow to set a special hostheader for which the caching
        ## behaviour is different to the normal heuristics
        ##
        ## The CacheExtended parameters overwrite all other caching parameters (including CacheDurationMax)
        ## for tiles being requested via the hostname CacheExtendedHostname
        #ModTileCacheExtendedHostname cache.tile.openstreetmap.org
        #ModTileCacheExtendedDuration 2592000

        # Upper bound on the length a tile will be set cacheable, which takes
        # precedence over other settings of cacheing
        ModTileCacheDurationMax 604800

        # Sets the time tiles can be cached for that are known to by outdated and have been
        # sent to renderd to be rerendered. This should be set to a value corresponding
        # roughly to how long it will take renderd to get through its queue. There is an additional
        # fuzz factor on top of this to not have all tiles expire at the same time
        ModTileCacheDurationDirty 900

        # Specify the minimum time mod_tile will set the cache expiry to for fresh tiles. There
        # is an additional fuzz factor of between 0 and 3 hours on top of this.
        ModTileCacheDurationMinimum 10800

        # Lower zoom levels are less likely to change noticeable, so these could be cached for longer
        # without users noticing much.
        # The heuristic offers three levels of zoom, Low, Medium and High, for which different minimum
        # cacheing times can be specified.

        #Specify the zoom level below  which Medium starts and the time in seconds for which they can be cached
        ModTileCacheDurationMediumZoom 13 86400

        #Specify the zoom level below which Low starts and the time in seconds for which they can be cached
        ModTileCacheDurationLowZoom 9 518400

        # A further heuristic to determine cacheing times is when was the last time a tile has changed.
        # If it hasn't changed for a while, it is less likely to change in the immediate future, so the
        # tiles can be cached for longer.
        # For example, if the factor is 0.20 and the tile hasn't changed in the last 5 days, it can be cached
        # for up to one day without having to re-validate.
        ModTileCacheLastModifiedFactor 0.20

        ## Tile Throttling
        ## Tile scrappers can often download large numbers of tiles and overly straining tileserver resources
        ## mod_tile therefore offers the ability to automatically throttle requests from ip addresses that have
        ## requested a lot of tiles.
        ## The mechanism uses a token bucket approach to shape traffic. I.e. there is an initial pool of n tiles
        ## per ip that can be requested arbitrarily fast. After that this pool gets filled up at a constant rate
        ## The algorithm has two metrics. One based on overall tiles served to an ip address and a second one based on
        ## the number of requests to renderd / tirex to render a new tile.

        ## Overall enable or disable tile throttling
        ModTileEnableTileThrottling Off
        # Specify if you want to use the connecting IP for throtteling, or use the X-Forwarded-For header to determin the
        # IP address to be used for tile throttling. This can be useful if you have a reverse proxy / http accellerator
        # in front of your tile server.
        # 0 - don't use X-Forward-For and allways use the IP that apache sees
        # 1 - use the client IP address, i.e. the first entry in the X-Forwarded-For list. This works through a cascade of proxies.
        #     However, as the X-Forwarded-For is written by the client this is open to manipulation and can be used to circumvent the throttling
        # 2 - use the last specified IP in the X-Forwarded-For list. If you know all requests come through a reverse proxy
        #     that adds an X-Forwarded-For header, you can trust this IP to be the IP the reverse proxy saw for the request
        ModTileEnableTileThrottlingXForward 0
        ## Parameters (poolsize in tiles and topup rate in tiles per second) for throttling tile serving.
        ModTileThrottlingTiles 10000 1
        ## Parameters (poolsize in tiles and topup rate in tiles per second) for throttling render requests.
        ModTileThrottlingRenders 128 0.2

        ###
        ###
        # increase the log level for more detailed information
        LogLevel debug

    </VirtualHost>

Import Map Data into PostgreSQL

We are now ready to download OSM Map Data and import it into our PostGIS-enabled PostgreSQL Database.

# Download OSM Map Data
# In this example, I am downloading the Greater London OSM Map Data from geofabrik.de
export PATH=$PATH:/usr/pgsql-9.4/bin
wget http://download.geofabrik.de/europe/great-britain/england/greater-london-latest.osm.pbf

# Process this OSM Map Data into the PostGIS-enabled PostgreSQL Database
osm2pgsql --slim -d gis -C 1600 --number-process 1 -S /usr/local/share/osm2pgsql/default.style greater-london-latest.osm.pbf

Test the Tile Server

Now that we have processed our first set of Map data, let us test our Tile Server i.e. test Renderd, Mapnik and mod_tile with the OSM Carto Stylesheet

# Enable and start Apache HTTP Server
systemctl enable httpd
systemctl start httpd

# Start Renderd
mkdir /var/run/renderd
mkdir /var/lib/mod_tile
renderd -f -c /usr/local/etc/renderd.conf

In a client browser, try navigating to the following resource (replace with your server's hostname): http://192.168.1.1/osm_tiles/0/0/0.png

If all goes well, you should see the first tile that represents the map of the world, as follows:

Pre-render Tiles

The final thing to note is that currently tiles are rendered on the fly - meaning that tiles are generated at the time of the request. Depending on your hardware and network specifications this may not be performant. Luckily, we can pre-render tiles for a specified geographical area and zoom level range which are then stored in /var/lib/mod_tile. It is possible to pre-render tiles for the entire world and for all zoom levels, however dependent on your hardware this may take a very long time!

# Start Renderd
renderd -f -c /usr/local/etc/renderd.conf &

# Render all tiles of Greater London (the only OSM Map Data we have loaded thus far) between Zoom Levels 0 - 17
render_list -m default -a -z 0 -Z 17 -s /var/run/renderd/renderd.sock

Leaflet

Now that our Tile Server is fully functional, we can use a client-side Javascript library such as Leaflet to display interactive maps in our apps or websites. You can download the Leaflet CSS and Javascript dependencies using the link provided or using NPM. Assuming that you have included the Leaflet library in your project, below is a simple Javascript function that expects the ID of the DIV in which to render the Map, the latitude and longitude co-ordinates to focus the map at and a caption for the marker.

// Define the Tile Server URL - Replace with your Tile Server Hostname
var mapTileServerUrl = "http://10.1.7.1/osm_tiles/";

// Define the OSM Tile Co-ordinate Placeholders
var osmTileCoordinatesPlaceholder = "{z}/{x}/{y}.png";

// Define the Zoom Levels
var mapTileLayerMinimumZoomLevel = 8;
var mapTileLayerMaximumZoomLevel = 18;
var defaultZoomLevel = 15;

// Define the Attribution
var mapTileAttribution = "Map Data &copy; <a href='http://openstreetmap.org'>OpenStreetMap</a> contributors";

// Render a single marker on a Map
function renderMapSingleVertex( divId, vertexLatitude, vertexLongitude, vertexCaption ) {

    // Set up the Lefalet Map
    var map = new L.Map( divId );

    // Create the Tile Layer with correct attribution
    var osmUrl = mapTileServerUrl + osmTileCoordinatesPlaceholder;
    var osm = new L.TileLayer( osmUrl, {
        minZoom: mapTileLayerMinimumZoomLevel,
        maxZoom: mapTileLayerMaximumZoomLevel,
        attribution: mapTileAttribution
    });

    // Navigate to the requested co-ordinates
    var latitude = parseFloat( vertexLatitude );
    var longitude = parseFloat( vertexLongitude );
    map.setView( new L.LatLng( latitude, longitude ), defaultZoomLevel );
    map.addLayer( osm );

    // Render a Marker and Popup
    var marker = new L.marker( [latitude, longitude] )
                        .addTo( map )
                        .bindPopup( vertexCaption )
                        .openPopup();

}

Testing the Leaflet function with Latitude 51.501476 and Longitude -0.140634 should provide you with the following Map centered on Buckingham Palace in central London:

We have now setup a fully functional end-to-end self-hosted tile server capable of rendering map tiles from any region in the world and serving them to your app or website without the need to pay for a commercial provider.