noTile – The project database

Here I will describe how the project database defines the map project. As we have seen earlier data is stored in SQLite databases. So is also the information about the map project. That is information about where to find the databases with map data. At what scales to render the different layers. What layer is on when project loads. How to style the layers and some more.

The idea about a project database

Many map clients uses config-files or javascript files to define the mapping project. That is convenient. But here is my 2 reasons for not storing the project definition in a text file:

  1. We already handle SQLite in the project, so I didn’t have to search and evaluate any other formats and techniques when chosing SQLite.
  2. It is easy to maintain from many different interfaces. By SQL-query updating values direct. Or by sending another SQLite DB to the client, attach it to the one in production and copy rows between them. Or just exchange the whole db of course.

So, when starting the noTile client it takes the project db as an argument. By reading that project db it finds the other databases with data in.

 

The tables in the project database is:


Tables to manipulate when adding a new layer to the project

table “dbs”:
In this table paths to databases with data is stored.

table “layers”:
In this table a lot of information about the layers is stored. There is information about geometry type as integers as:

1 – Point
2 – Line
3 – Polygon

In this table is also information about at what scales the layer shall be rendered. For simplicity the value is just “units per pixel”. That means that if you use a meter based projection and sets maxScale to 30, the layer will only show when the scale gives less than 30 meters per pixel.

This table also keeps information about what dataabse to get data from of course, and the name of the table to read from. It also keeps information about the geometry column and if polygon about the triangle index column (more about that later).

We also keep information about what “program” to use for rendering. A program in this context is a combination of vertex shader and fragment shader in openGL. More about this later too.

The layer table also have information about what field to use for styling and in what order the layers shall be rendered (first layers gets in the back)

table “styles”

Here all styles is stored. Styles is very simple so far. It is just a color, an outline color, a linewidth and on what value the particular style is to be used. The value referenced here is the value in the column that is given in the layer table for styling. So, for now the client supports very few options for styling.


Tables that most times don’t need to be modified

table “shaders”

This is a table holding the source code for the shaders. maybe this is overkill to make this flexible. But when I have been reading about openGL I have understood that shaders is the heart of openGL. Shaders is small programs that gets loaded into the gpu at runtime. Those programs is what the gpu actually execute in the rendering process.

So, what I have done is to put the source code of the shaders into this project db. That means that there can be different shader programs for different layers. Maybe that can be usefull. But so far I have ily used simpliest possible shaders to reproject the vertex points to pixels and to set the right colors.

But, anyway, the possibilities is there to do something crazy with it, for someone with that knowledge.

table “programs”

Here we combine vertex shaders and fragment shaders to a “program”. Since vertex shaders and fragment shaders always is compiled together it makes sence to give the combination a unique id. This id is what the layers get. So a layer using “program 1” gets sources to vertex shader and fragment shader from table shaders referenced in table programs where programID is 1.


Ok, that was the story about the project database.

You can find an example project db in the repository https://github.com/nicklasaven/noTile
It is called “norge_proj.sqlite and is used to render the data in norge.sqlite in http://twkb.jordogskog.no/maps/

Next post will probably be about how the clients selects and uses geometry data and how it gets rendered.

noTile – intro

I haven’t written a blog post in a few years. But I think it is time to give it a try again.

Why?

Well, I have spent too much time in a map client technique that will need some explanation.

Some history, in non-chronological order

This goes back to 2011 or 2012 when I started to think about a new format for transporting geometry data between a server (PostGIS) and a client (a web client). I wanted a compact but fast format to send data around. The result is TWKB  (Tiny WKB) now implemented in PostGIS. When I started this there was no MapBox Vector Tiles (at least I had not heard of it), so compact geometry formats was quite unknown.

The background for this wish for a compact format originated from a decade before. When I worked at Skogsstyrelsen in Sweden (Swedish Forest Agency) we started to use pda with ArcPad. Back then I was able to put shapefiles with all data I needed for may daily work into that pda. Detailed data for Torsby kommun, about 4000 km2. A few years later things like that was just a dream. ArcPad maybe existed, but everything went online, and as maps became something everyone demanded, the user focus changed from field work to “search for a restaurant”. On the new platforms like smartphones and tablets there was no good tools for detailed offline work. Then there came some clients that could download “offline areas”. But mostly just smaller areas.

Knowing that the pda back in 2003/2004 had 64 mb of RAM and a sd-card of 1 GB, I think it s quite strange that we don’t have a lot of techniques for storing whole countries with detailed data today.

But I have some more requests than just rendering. I also want to be able to keep data updated, without changing whole data sets. With the future in mind I also think that access to geometries in an easy way for analyzing and processing is a quite natural request.

The command from the daydream part of me to the doer part of me

So, from this, my conclusion is that we need structured data, compressed and fast to access. That is great but how to balance the trade offs? Some basic rules:

First thing to trade away is “eye candy”. If you have been out in the bush trying to do a job (of course in the rain, and hungry) eye candy is one of the most provoking things in the world as all you see is a well designed circle going around. I am supprized that anyone using a computer professionally asks for smoth transitions before snappy transitions. That is like a carpenter asking for the color of the power drill instead of tourqe. So,performance is more important than eye candy

When performance is descent more performance is classified as eye candy. This means that when we have enough performance, optimizing for small storage is more important.

Data can never be too small. There is always more data to bring that might be “good to have”.

So, to summarize:
Forget about eye candy. It just eats the power of your device, and melts down our planet. Enjoy snappy over smooth!

A funny thing I have found when working with this. There is no contradiction between performance and smallish. Often the smaller size in itself gives a better performance, even when local. Both reading and writing big amounts of data from disc have a cost.

Disclaimer of what is coming

All this is still in a very early stage. Big parts of it is also in domains that I have only learnt about from tutorials. So ideas and techniques is very naive and some parts is just a result of trial and error. So, here is a lot of room for more competent brains to take a look.

Diving into

This is it. Data is stored in SQLite at the client. The layers and styling is defined in a separate SQLite db. There “project”- db can get data from many “Data” db, to form a project.

systemet

The client is written in C.
SDL2 is used to abstract away platform differences and to easily get user input.

The code for the client and link to prepacked map-data can be found at github:
https://github.com/nicklasaven/noTile

 

I will get back with info describing the project technically. I will also write about how to pack the map-data.

If this sounds interesting, please join in the effort to make something good out of it.

I have put the GPL v2 license on the code. But that can be discussed if there is other opinions.