7 min read apache-superset

How to Correctly Load Your Country in to Apache Superset

TLDR
  • The geoJSON specification published in 2016 requires the polygons to follow the right-hand rule.
  •  All updated geoJSON files now follow the right-hand rule.
  •  However, Superset requires your geoJSON file polygons to follow the opposite direction, breaking the specification.
  •  Covert your geoJSON files so polygons are drawn in opposite directions using the script included in Step 5.
  •  Include the geoJSON file inside the legacy-plugin-chart-country-map plugin.
  •  Import the geoJSON file and rebuild your Superset frontend.

Apache Superset is a free and open-core data analysis tool that offers a wide range of really neat features for a business. It’s a good piece of software where you don’t have the budget to spend on an expensive business intelligence tool.

Sadly their documentation is not up to date, the questions and solutions on StackOverflow are limited and customizing Superset can be a pain. GitHub support is very limited by the community. And most of the time you will have to figure things out by yourself.

Recently I had to load a country map to Apache Superset to generate country map charts. Apache Superset does not include all the country maps by default and provides only a handful of countries.

The Apache Superset documentation provides you with the steps needed to load a country into Superset. This sounds straight forward, however the reality it’s far from that.

In this tutorial, I’m going to write about my experience and how you can include your country on Apache Superset. And take you through the treasure hunt like I did which literally drove me near insanity.

Step 1 – Finding the right geoJSON file of the country

First I had to find the right geoJSON file. Luckily I found the geoJSON file for Sri Lanka at – https://data.humdata.org. The website includes a lot of geoJSON files from different countries that you can freely download.

If your country is not listed on the humdata website, you can create a geoJSON file using QGIS software. You can download the QGIS software from their website https://qgis.org/en/site/.

I downloaded and included the geoJSON file in the legacy-plugins-chart-map-country plugin as mentioned in the Superset documentation and rebuilt the Superset frontend.

I was hoping everything will go according to plan and I will nicely get the Sri Lankan map loaded into the Superset website. However, the map rendered was completely different from what I expected,

The initial rendering of the map, you can faintly see a small indentation on the right border.

The map is rendered as a rectangle, and if you look closely you can see a small indentation on the right border of the rectangle with some sort of rendering caused by the geoJSON file.

Apparently, this has been asked as an issue on GitHub multiple times, but no one seems to share a working solution that fixed the issue.

Can this be an issue with Superset? Or is something wrong with the geoJSON file as mentioned in both GitHub issues?

I initially thought this has to be an issue with Superset, therefore I used an existing geoJSON file in the Superset plugin for Sri Lanka and reloaded the frontend and that geoJSON file rendered correctly.

To get to the bottom of this, I downloaded a different country geoJSON file from humdata and loaded it to Superset.

This also rendered a rectangle as you see above, and by then I knew this has to be something wrong with the geoJSON file or how Superset use the geoJSON file that was downloaded from humdata or created by QGIS.

Step 2 – Trying parse the geoJSON file

I then went to https://geojson.io website and pasted the geoJSON file of Sri Lanka and see if there is something wrong with the GeoJSON file, and the file rendered correctly on geojson.io website.

The geoJSON file for Sri Lanka also gets rendered correctly on GitHub and this meant there is nothing wrong with the geoJSON file I downloaded from humdata. So what can be the issue here?

I even drew a simple polygon over the Sri Lankan map (a large triangle) and tried to render it on Superset. And this also rendered a large rectangle as you see above.

Step 3 – Trying see if the coordinates are correct

Being frustrated I googled the latitude and longitude of one of the points in an existing map inside Superset [113.13181294000367, -25.951992185759366], this is a point in Western Australia, and to my surprise, google did not show a place matching these coordinates.

Then I reversed the coordinates and googled [-25.951992185759366, 113.13181294000367], and this correctly showed a point in Western Australia.

So does this mean I have to invert the coordinates correctly and display it on Superset? This is not abnormal, as mentioned on Stackoverflow, the popular javascript map rendering plugin Leaflet also uses coordinates inverted. Apache Superset might be doing the same, so there is nothing wrong with that.

How hard inverting the coordinates can be, all I have to do is write a simple javascript to invert the coordinates within the geoJSON.

I wrote a simple javascript and inverted the coordinates of the geoJSON file that I downloaded from humdata. And loaded it to Superset, and this was the result,

Rendering the map with inverted coordinates.

So this gave me a wired map of Sri Lanka, located in a completely geographically off place. But this was an improvement from the initial rectangle that I received. So I’m definitely making progress here.

Step 4 – Parsing a geoJSON file included in Superset

Since GitHub correctly renders all the geoJSON files included in Superset, I initially thought these geoJSON files are correct. But why my geoJSON file is not working? There has to be something wrong somewhere.

I copied one of the geoJSON files from the superset repository and parsed it on geojson.io, and to my surprise, it did not render on geojson.io.

I got the following error, “Polygons and MultiPolygons should follow the right hand-rule”. So there is something not right with the geoJSON file inside Superset and not with the one I downloaded from humdata.

Searching for this error I landed on a post on StackOverflow about the geoJSON specification published in 2016 which requires all polygons to should be ordered in a clockwise manner. The old 2008 geoJSON informal specification did not have this rule.

A linear ring MUST follow the right-hand rule with respect to the area it bounds, i.e., exterior rings are counterclockwise, and holes are clockwise.

This meant that the Superset geoJSON files breaks this rule, and were ordered in a counter-clockwise manner. Even though current geoJSON files are published following the right-hand rule, while apache Superset requires geoJSON files going against the right-hand rule, and GitHub doesn’t care whether a geoJSON file follows the right-hand rule or not and renders them irrespectively.

And I had to reverse the order of the coordinate and not to reverse latitudes and longitudes as I did earlier. There are plenty of online tools that allow you to correct the issue, to make a geoJSON that does not follow the right-hand rule to follow the right-hand rule. But I didn’t find a tool that breaks this rule.

So as the rule breaker that I always am, again I had to write a simple javascript to reverse the order of the coordinates. I’m sharing below for anyone to use and to convert a geoJSON file following the right-hand rule to go against the rule and go counterclockwise.

Step 5 – Converting geoJSON file to go against right-hand rule

I’m sharing the javascript that I used to convert a geoJSON file to go against right-hand rule. If anyone wants to include a country map in Superset, they can use this script to convert the geoJSON files to go against the right-hand rule and import them into Superset.

// import the geoJSON file
const data = require("./lk.json");
const { writeFileSync, readFileSync } = require("fs");

// reverse the order of the coordinates 
for (let index = 0; index < data.features.length; index++) {
  const element = data.features[index];
  const coordinates = element.geometry.coordinates;
  data.features[index].geometry.coordinates[0].reverse();
}

// save the file onto disk
writeFileSync("./srilanka.geojson", JSON.stringify(data), "utf-8");
console.log("written");

Now I included this geoJSON file which is in counter-clockwise order onto Superset and it finally rendered the map correctly.

Step 6 – Importing the geoJSON file that

Save the geoJSON file you just created inside the countries folder of the legacy-plugin-chart-country-map plugin. This is located in the superset/superset-frontend/plugins/legacy-plugin-chart-country-map/src/ folder.

Then import the geoJSON file you just saved by modifying the country.ts file inside the legacy-plugin-chart-map-country plugin which is located inside the superset/superset-frontend/plugins/legacy-plugin-chart-country-map/src/ folder.

Step 7 – Build the Superset frontend and run Superset

Now build the superset frontend

cd superset/superset-frontend/
npm run build

Then restart the superset server

cd superset
source venv/bin/activate
superset run -p 8088 --with-threads --reload --debugger

Now the country you included will be available for country map charts.

Rukshan

Rukshan