Approaches to handling ~200,000 raster tile sources

[tombh]

I'm looking into how to serve up the output from a total viewshed project. Each raw output tile will be a raster of ~6000x6000 points covering ~600km². The data is just a single band of f32s, so ~144MB per tile. I'll display it as a heatmap.

My current thinking is to use Martin and PMTiles. So naively, could I just point martin at an S3 bucket of 200,000 .pmtiles? I feel not. What bout adding all the raw tile data into one big .pmtile? Or should I start thinking about a database?

Another factor is that it could take days, weeks or months to calculate the entire planet. Let's say that I'm producing new raw tiles at a rate of 100 per day, it'd be nice to add them to a classic z/x/y slippy map website piecemeal as they come.

I'm relatively new to all this so any and all ideas are very welcome.

PS: completely unrelated question, whilst I can get Martin to work I've never been able to see anything in the Inspect view of the Web UI, and no errors in the browser's developer tools inspector.

[CommanderStorm]

So naively, could I just point martin at an S3 bucket

You cannot currently point us to a bucket and let us do the autoconfiguration.
We can autoconfigure local directories, but we don't have a testcase for remote autoconfiguration, so I am unsure if this works.

This would be a natural exstension, so PRs welcome 😉

[CommanderStorm]

No, we don't have such a feature.

You can request composite sources by using a comma and we will look in them for the requested tile in parallel.
You cannot specify 200.000 sources in an URL though, that does not work.
=> You will need to tell the client (MapLibre/..) about the sources that exist

[tombh]

Oh, that's a big misunderstanding I had about Martin then. Seeing as PMTiles are designed to be used without a tile server, then I'm curious, what benefits does Martin offer for serving PMTiles with it?

[nyurik]

pmtiles can be downloaded as a whole because it is public - and you get to pay for that. So if you publish a whole pmtiles, anyone using it can just get the whole thing because they see where it is - and if your files are multi-GBs, all the worse. Running Martin as a lambda or service next to that data makes it much easier to deploy, while also making it simple to serve proper optimized tiles.

Would be great to update https://github.com/maplibre/martin/blob/main/docs/src/sources-files.md with this explanation.

[CommanderStorm]

well..
There may also be a real performance benefit to using martin, at least for the initial load.

If you go with S3 and are reasonably far from the client, you are bound by how fast your network roundtrips. This might be not very fast.
If you use martin, the lamda/server is in the same datacenter, so much faster.

[tombh]

These are great explanations, thank you both.

[CommanderStorm]

200,000 .pmtiles? [..] What bout adding all the raw tile data into one big .pmtile

Given that each of the remote pmtile' files needs its own directory cache (where in the file is XYZ), this would not be optimal.

Merging those is thus going to be much simpler to manage (only one source in the catalog) and better for caching.

[tombh]

I do not know what merging means in this context. What is the difference between merging and creating one big .pmtile?

[CommanderStorm]

Merging means creating a big pmtile. No difference.

You don't need to go all the way down to one pmtile technically but that is another idea

[tombh]

Great, thank you.

[CommanderStorm]

so ~144MB per tile

Shipping 144MB per tile generated from a tile server is.. suboptimal.
You really do not want to do this, that would be very slow.

You want to reduce the data to be more managable.

I think what you would like to do instead is using COGs.
Our support for this is a bit suboptimal currently, but if you check out the following PR and compile it from the new source, that should work.

• Add auto_webmercator to COG #1893

[CommanderStorm]

Or should I start thinking about a database?

We currently support image-type sources for .mbtiles or .pmtiles.
We do not currently support serving rastered data from Postgres:

• Using PostgreSQL for raster tiles #1894

[CommanderStorm]

PS: completely unrelated question, whilst I can get Martin to work I've never been able to see anything in the Inspect view of the Web UI, and no errors in the browser's developer tools inspector.

I have not experienced this...
What is in the tiles you are trying to inspect?

The feature is relatively new, so there might be bugs..

[tombh]

I use the Odessa sample GeoTiff from here: https://www.planetobserver.com/geospatial-data-samples And prepare it with this command: rio pmtiles ~/Downloads/PlanetObserver_PlanetSAT_10m_Ukraine_Odessa_UTM36.tif --output output.pmtiles --rgba. It appears perfectly in a little Leaflet.js map I made. I have tried the Martin Inspect UI in both Firefox and Chromium.

[tombh]

Thanks for all the responses 😅

You cannot currently point us to a bucket and let us do the autoconfiguration.

The issue is the _auto_configuration? If I manually add all the files to the Martin config then it would work?

Merging those is thus going to be much simpler to manage (only one source in the catalog) and better for caching.

I used the word "adding" to refer to having all the data in a single file. And you use the word "merging". I'm afraid I don't understand the difference: adding and merging are not the same?

Shipping 144MB per tile generated from a tile server is.. suboptimal.

Can you briefly explain why that would be slow?

I'll have a look at #1893, thank you.

What is in the tiles you are trying to inspect?

I use the Odessa sample GeoTiff from here: https://www.planetobserver.com/geospatial-data-samples And prepare it with this command: rio pmtiles ~/Downloads/PlanetObserver_PlanetSAT_10m_Ukraine_Odessa_UTM36.tif --output output.pmtiles --rgba. It appears perfectly in a little Leaflet.js map I made. I have tried the Martin Inspect UI in both Firefox and Chromium.

Thanks again for all your feedback.

[CommanderStorm]

Shipping 144MB per tile generated from a tile server is.. suboptimal.

Can you briefly explain why that would be slow?

Assuming fast 4G Internet (6.25 MB/s), loading 144MB of data takes 144/6.25 = 23.04s
Having a map where a single tile (multiple tiles will be even slower) takes this long would not be very fun.

[tombh]

I think I may have confused you. The 144MB is for the raw .tiff data from which I generate the .pmtile. My understanding is that the process of creating a PMTile requires cutting up the large original image into many smaller ones never more than a few KBs. So for any given zoom level there will never be a need to download all the data, only ever reduced resolution versions or small sections of the zoomed-in original.

[CommanderStorm]

Another factor is that it could take days, weeks or months to calculate the entire planet. Let's say that I'm producing new raw tiles at a rate of 100 per day, it'd be nice to add them to a classic z/x/y slippy map website piecemeal as they come.

This is a hard one.
Pmtiles are immutable by design.
You can

• produce them in a separate folder and
• every few generations merge them into a new pmtile database,
• spin up a new instance of martin pointing to that,
• shut down the old martin instance and
• remove the old pmtiles instance

Martin will not handle an pmities source changing its content gracefully, or at least we don't have a testcase for this.
In any case, we don't "watch" for new sources:

• Automatic (catalog, cached tiles, ..) update after data change (WATCH_MODE=info) #288

The cost of doing so might be fine or not (calculate the $ for reads/writes of updates you anticipate in advance).

If you go via an mbtiles database, you can skip a part of this, but this means running an traditional server at something like hetzner.
There is no reason that I can see of using postgres over mbtiles for your usecase.

Using COGs is also possible, but calculate your costs too if you want to use cloud storage.
In terms of image formats, I suspect this matches the profile you are looking for the most, but is also the most experimental.

[CommanderStorm]

I think you are misunderstanding the COG vs GeoTiff.

COG is a valid GeoTIFF file with some requirements for efficient reading. That is, all COG files are valid GeoTIFF files, but not all GeoTIFF files are valid COG files. For quick access to tiles in TIFF files, Martin relies on the requirements/recommendations(like the requirement about Reduced-Resolution Subfiles and the content dividing strategy) so we use the term COG over GeoTIFF in our documentation and configuration files.

Access efficiency should thus be about the same (excluding projection) as far as I understand.

COG stores pixel data once at multiple resolutions (overviews). Thus it may have less duplication than fully pre-rendered tile pyramids, so for very high zooms COG often uses less storage than pre-rendered tiles.
How much is this for you? No clue, you will have to benchmark this.

support for HTTP range requests

Our COG backend does not currently support this, but that should be fairly simple to add. Feel free to do a PR if above you notice that the storage size means this is beneficial.

[tombh]

So the potential benefit of COGs over PMTiles is that they may use slightly less storage? That's worth bearing in mind, but I don't see how that relates to Martin's support for 200,000 tiles. As far as I understand both COGs and PMTiles at the scale of 200,000 present the same issues to Martin.

[nyurik]

Not exactly, pmtiles are supported from s3 bucket, ie via http range requests, whereas cogs are not yet supported like that

[CommanderStorm]

If your estimate holds true globally, the raw size of your data is 28.8 TB (144MB* 200.000).
So size and the cost of storing it is likely a major decision point for you (if you want to go with the cloud variant in the first place.. local HDDs are very cheap, but still expensive if you want redundancy).
This is why I suggested you look into the size difference.

Number of tiles in each of the sources is not very relevant in choosing COG vs Pmtiles.

[tombh]

Thank for taking costs into consideration! But right now I'm more interested in the technical practicalities. I can always just decrease resolution if it starts getting too expensive.

[tombh]

I think I've realised something important: tile server != mosaicing server. I thought that all tile servers were also mosaic tile servers, but that's not at all the case.

So now I'm thinking that I could actually do all this without any tile server (mosaic or otherwise) whatsoever! What if I consistently place all my output data (whether they're COGs or PMTiles) in mathematically known locations? Then I could just use JS to calculate which COG or PMTile to display depending on the viewport of the browser.

There are 2 downsides to this:

1. It's still hard to render the lower zoomed-out levels because it could involve querying up to tens of thousands of COGs/PMTiles. However I can at least imagine a pre-deploy step that took all the, say, zoom level 3 tiles for the planet, and made a single COG/PMTile for that.
2. One-off outputs from my project (so outputs that fall outside the predefined mathematical grid) can't be arbitrarily mixed in with the standard output. Again not a huge problem as I could just add them to another layer accesible via a toggle in the web app or something.

Depending on how this goes, I can also see myself making a thin, custom server that returns a list of COGs/PMTiles for the frontend to use for its tile queries. Or indeed exploring making a basic mosaicing server that converted a single z/x/y endpoint into a reverse proxy that returned the correct tile from the catalog of 200,000. If I did do either of those I would of course be interested in seeing what would be involved in submitting PRs for them in Martin. Do they seem like features that could sensibly fit into the codebase?

[nyurik]

just fyi, martin is also a library - you can reuse most of it in your own app

[tombh]

Ohhh, I didn't know that, I'll certainly bear that in mind, thanks.