Deal with geo_point types on ElasticSearch

Rationale TL;DR

I’ve been trying to ship some logs from Nginx to ERK (Elastic + Rsyslog + Kibana). The process is easy enough (I’ll post about that eventually). But, I ran into some problems when using the mmdblookup rsyslog module to do geolocalization because I wanted to position the remote clients on Kibana.

The solution

The issue is that out of the box ElasticSearch is unable to figure out that a JSON key like this:

"geo": {
	"loc": {
		"lat": 0.0,
		"lon": 0.0,
	}
}

The issue here is that ElasticSearch will simply figure out those fields are floats (lat and lon will be floats). If that’s the case, Kibana won’t be able to use them because it needs the loc field to be geo_point.

Every index uses a mapping that helps ES to figure out what to do with the data. Changing the mapping applied to loc would fix the issue.

However, using a mapping to set the loc field did not fix the problem, it make things even worse, because:

• We can’t just set the mapping of one field, if we do that ES will just use that mapping for that field and will not process the other. That is to say that we’re saying ES that we’re just worried about on field.
• Since ES 6.x we can’t delete a mapping without deleting the index itself. That means we need to reindex all the data, drop the index (and the mapping) and the reindex back to our original index.

So, we need something else. That’s what index templates are used for. They allow us to set the defaults applied to an index, including the mappings to apply. We can have multiple templates, deciding in which order they will be applied. The only drawback is that they need to applied at creation time.

Let’s suppose we want to apply a template to an index named nginx_logs-*. We want to set the field geo_ip.loc so that ES is able to use that field as a geo_point field:

PUT _template/nginx_geo_point
{
  "order": 0,
  "index_patterns": "nginx_logs-*",
  "settings": {
    "number_of_shards": 5
  },
  "mappings": {
    "_default_": {
      "properties": {
        "geo_ip.loc": {
          "type": "geo_point"
        }
      }
    }
  }
}

The syntax is clear. This creates a new template called nginx_geo_point. The index will use 5 shards and a new mapping will be applied, setting geo_ip.loc to geo_point.

Also, templates are applied in order. The order field controls that. In this example, this template will be applied the first one. If multiple templates have the same order number, there’s no guarantee of the order they will be applied.