How to Incorporate Mesh Blocks into Datasets

How to Incorporate Mesh Blocks into Datasets

Mesh Blocks in real estate and proptech applications

Mesh Blocks are useful for geospatial and proptech applications, providing granularity and accuracy for understanding local real estate markets, demographics and land use.

The integration of Mesh Blocks into datasets can enhance the precision and relevance of analyses within the proptech and real estate sectors.

Useful in geospatial data and census analyses, embedding Mesh Blocks into digital boundaries enhances their usability in various applications.

We will cover the steps to incorporate mesh blocks into data sets below.

What are Mesh Blocks and how are they used in real estate?

Mesh Blocks are foundational building blocks for geospatial and proptech applications, providing granularity and accuracy for understanding local real estate markets, demographics and land use.

How to incorporate Mesh Blocks into datasets

Incorporating Mesh Block into datasets involves several steps to ensure seamless integration and effective utilisation of geographical information. Here’s a guide on how to incorporate Mesh Blocks into datasets:

Step 1: Data Collection

Gather relevant data that aligns with Mesh Blocks.

This may include demographic information, property values, land use details, or any other dataset that can be associated with specific geographical areas.

 

Step 2: Download Mesh Block Boundaries

Mesh Block boundary files can be downloaded from authoritative sources, such as the Australian Bureau of Statistics (ABS) or relevant statistical agencies.

For ease, The Proptech Cloud has a free comprehensive dataset Geography – Boundaries & Insights – Australia ready for access and immediate use.

Geography – Boundaries & Insights – Australia

This free dataset from The Proptech Cloud is available for seamless access from Snowflake Marketplace.

Step 3: Geospatial Data Processing

Use Geographic Information System (GIS) software or programming libraries (e.g., Python with geospatial libraries like GeoPandas) to process and manipulate the mesh block boundaries.

Tip:

Geographical boundaries can be imported using Python libraries including Geopandas and shapely.

Many data warehouses including Snowflake, BigQuery and PostgreSQL have in-built geospatial functionality to allow for the processing of geospatial data.

QGIS – Loading in Geospatial files in QGIS

1. From the toolbar at the top of the page click Layer > Add Layer > Add Vector Layer

2. Make sure the Source Type is clicked to File

3. Load in the Source Data by using the three dots button at the side of the Vector Dataset(s) toolbar

QGIS - Loading in Geospatial files in QGIS

Geospatial Formats

The two most common ways geospatial data are represented in files are Well-Known-Text (WKT) which is a textual representation of a polygon and the geojson format which shows the coordinates and type of geojson format.

Both Python and Snowflake have capabilities to work with these 3 formats and parse them so they can be used in geography functions

WKT Format

#Example 2 using WKT format

from shapely import wkt

brisbane_bbox = “POLYGON ((153.012021 -27.471741, 153.012021 -27.462598, 153.032931 -27.462598, 153.032931 -27.471741, 153.012021 -27.471741))”

brisbane_poly = wkt.loads(brisbane_bbox)

Python – Loading in GeoJSON

The libraries geojson, shapely and json need to be installed.

#EXAMPLE 1 working with a geojson format

import json

import geojson

from shapely.geometry import shape

geojson_example = {

"coordinates": [[[153.01202116, -27.47174129], [153.01202116, -27.46259798], [153.03293092, -27.46259798], [153.03293092, -27.47174129], [153.01202116, -27.47174129]]],

"type": "Polygon"

}

geojson_json = json.dumps(geojson_example)

# Convert to geojson.geometry.Polygon

geojson_poly = geojson.loads(geojson_json)

poly = shape(geojson_poly ))

Snowflake

GeoJSON and WKT format can also be loaded into snowflake and converted to a geometry using the following commands:

#converting Well-Known-Text into geography format

SELECT ST_GEOGRAPHYFROMWKT('POLYGON ((153.012021 -27.471741, 153.012021 -27.462598, 153.032931 -27.462598, 153.032931 -27.471741, 153.012021 -27.471741))');

#Converting Geojson to geography format

SELECT TO_GEOGRAPHY('{

"coordinates": [[[153.01202116, -27.47174129], [153.01202116, -27.46259798], [153.03293092, -27.46259798], [153.03293092, -27.47174129], [153.01202116, -27.47174129]]],

"type": "Polygon"

}

')

Step 4: Data Matching

Match the dataset records with the appropriate mesh blocks based on their geographical coordinates. This involves linking each data point to the corresponding mesh block within which it is located.

Tip:

Geospatial functions which are supported in big data warehouses and Python can be used to match geospatial data.

A common way to match two geographical objects is to see if the coordinates of the two objects intersect. An example of how to do this in Python and Snowflake is shown below.

In Python

Data matching can be done using the shapely library intersects function.

from shapely import wkt, intersects

shape1 = wkt.loads("POLYGON ((153.012021 -27.471741, 153.012021 -27.462598, 153.032931 -27.462598, 153.032931 -27.471741, 153.012021 -27.471741))")

shape2 = wkt.loads("POLYGON ((153.012021 -27.471741, 153.032931 -27.462598, 153.012021 -27.471741))")

shape_int = intersects(shape1, shape2)

print(shape_int)

 

In Snowflake

Data matching can be done using the ST_Intersects function. One of the advantages of using big data warehouses including Snowflake to geospatially match data is that it leverages its highly scalable infrastructure to quickly complete geospatial processing.

WITH geog_1 as (

SELECT ST_GEOGRAPHYFROMWKT('POLYGON ((153.012021 -27.471741, 153.012021 -27.462598, 153.032931 -27.462598, 153.032931 -27.471741, 153.012021 -27.471741))') as poly

),

geog_2 as (

SELECT ST_GEOGRAPHYFROMWKT('POLYGON ((153.012021 -27.471741, 153.022021 -27.465, 153.032931 -27.462598, 153.012021 -27.471741))') as poly

)

SELECT

g1.poly, g2.poly

FROM geog_1 as g1

INNER JOIN geog_2 as g2

on ST_INTERSECTS(g1.poly, g2.poly)

Step 5: Attribute Joining

If your dataset and mesh blocks data have common attributes (e.g., unique identifiers), perform attribute joins to combine information from both datasets. This allows you to enrich your dataset with additional details associated with mesh blocks.

Step 6: Quality Assurance

Verify the accuracy of the spatial integration by checking for any discrepancies or errors. Ensure that each data point is correctly associated with the corresponding mesh block.

Tip:

geojson.io is a handy website that can help with visualising geojson data and ensure it is correct.

If you’re using Snowflake, the ST_AsGeojson command can be used to convert geography into a geojson which allows you to quickly visualise the shapes created.

Step 7: Data Analysis and Visualisation

Leverage the integrated dataset for analysis and visualisation. Explore trends, patterns, and relationships within the data at the mesh block level. Utilise geospatial tools to create maps and visual representations of the information.

Tip:

It’s worth mentioning that Snowflake has the option to create a Streamlit app within the Snowflake UI which allows for the cleaning and processing of data using Python and SQL and the interactive visualisation of data through the Streamlit App.

Read our blog which demonstrates how to predict migration patterns and create forecasts using Snowpark and Streamlit>

Snowflake also integrates really well with local Python development environments so all the initial data processing and cleaning can be done through a Snowflake API, then geography can be converted to a GeoJson or Text formal. Thereafter, libraries like plotly, folium, pydeck can be used to do complex geospatial visualisations.

Step 8: Data Storage and Management

Establish a system for storing and managing the integrated dataset, ensuring that it remains up-to-date as new data becomes available.

Consider using databases or platforms that support geospatial data.

Tip:

Geospatial datasets are usually very large and complex datasets due to the number of attributes included in a geospatial dataset, the resolution of the data and the number of records.

Cloud-based big data platforms can be an excellent option for storing geospatial data due to the low-cost of storage. Many of these platforms including also have spatial clustering options so that geospatial data in a similar location are grouped together, meaning queries for data in certain areas run more efficiently.

Snowflake (Enterprise Edition or Higher) also has an option to add a search optimisation to geospatial data tables to optimise the performance of queries that use geospatial functions.

Step 9: Documentation

Document the integration process, including the source of mesh block boundaries, any transformations applied, and the methods used for data matching. This documentation is essential for transparency and replicability.

By following these above steps, you can effectively incorporate mesh blocks into your datasets, enabling a more detailed and location-specific analysis of the information at the mesh block level.

 

Geography – Boundaries & Insights – Australia

This free dataset from The Proptech Cloud is available for seamless access from Snowflake Marketplace.

All rights are reserved, and no content may be republished or reproduced without express written permission from Data Army and The Proptech Cloud. All content provided is for informational purposes only. While we strive to ensure that the information provided here is both factual and accurate, we make no representations or warranties of any kind about the completeness, accuracy, reliability, suitability, or availability with respect to the blog or the information, products, services, or related graphics contained on the blog for any purpose.

Subscribe to our newsletter

Subscribe to receive the latest blogs and data listings direct to your inbox.

Read more from The Proptech Cloud

What Is A Property?

The definition of a property is something that has troubled many proptechs and users of property data for some time. The concept seems relatively simple until you try to write a definition.

What’s The Difference Between Proptech and Contech?

Proptech and Contech is driving innovation across industries. We uncover the nuances and distinctions between Proptech and Contech here.

What is a Coordinate Reference System (CRS)?

We break down the concept of Coordinate reference systems (CRS) and discuss the different types, their purpose and uses in the world of real estate.

The Property Lifecycle

Discover the extensive property lifecycle and how it connects with other sectors. This infographic highlights the key phases and processes involved.

Why Is It So Difficult To Parse Addresses?

This blog explores what address parsing is and why it presents such unique challenges. Discover the intricacies behind address parsing and why getting it right is more complicated than it first appears.