Spatial SQL for the Geographer – Part 3 – More Basic Spatial SQL Scripts

The tutorial continues…

If you haven’t already read the first three parts of this guide check out:

The first three parts of the guide give some background information on SQL Server, covers some basic spatial and database topics, and provides links to download some software and data. Once the user has completed the first three part of this guide they will be able to understand this tutorial, which will provide some additional basic spatial SQL scripts.

Tutorial

In the last tutorial several scripts were introduced and examples were given.  In the last tutorial I covered the process to validate shapes, calculating centroids, and generate buffers.   In this tutorial several additional scripts will be covered, including generating the bounding box/extent of a feature, intersection and union operations, and difference operators.

Calculate Extent

Calculating the extent of a feature is something I use from time to time.  To find these values, such as the northern bounding latitude or eastern bounding longitude, can be calculated through Spatial SQL.  Spatial SQL has a built in method that will calculate the bounding box of a record, STEnvelope().  In its simplest context the user can generate the bounding box for any feature in a spatial table:

use Spatial_Database
go

select
geom.STEnvelope() as boundingbox
from dbo.States_Provinces
where NAME_1 = 'Massachusetts'

This example will return a field called “boundingbox” for the record where NAME_1 equals ‘Massachusetts’ that contains the minimum area rectangle that encompasses the shape.    The data in the “boundingbox” field is relatively useless the user adds some additional code to the query.  Building on the example provided by Microsoft the user can add the ToString() method to the query:

use Spatial_Database
go

select
geom.STEnvelope() as BoundingBox,
geom.STEnvelope().ToString() as BoundingBoxString
from dbo.States_Provinces
where NAME_1 = 'Massachusetts'

Here the user will see a new column, “BoundingBoxString” that returns a polygon with the five points that represent it.  You may ask yourself, why are there five points when you only need four to create the bounding box?  Well, for SQL Server to create a polygon the first point needs to be represented twice, so that the polygon closes itself properly.

Intersect

The intersect table operator is one of the most popular tools in the GIS work belt. Intersect is an overlay operation, where the area of the input feature is used as the basis to select features from the second feature, the intersect feature.  The following illustration will explain what an intersect is better than I can with words…

Source: Esri

In spatial SQL the STIntersects() tool is straight forward and easy to use.   However, it does not return a shape as other overlay methods do.  When run, STIntersects() will return a value of one if the two records spatially intersect and a zero if they do not.  Also, this method only tests the relationship between a single record in one table against a single record in another table.  The following provides an example of intersecting a single line object against a polygon object:

use Spatial_Database
go

DECLARE @poly_geom geometry
select @poly_geom = geom.STAsText()
From dbo.States_Provinces
where dbo.States_Provinces.name_1 like '%Massachusetts%'

DECLARE @line_geom geometry
select @line_geom = geom.STAsText()
from dbo.Roads
where ID = 2974

SELECT @poly_geom.STIntersects(@line_geom) as IntersectValue, @poly_geom as input1, @line_geom as input2

The Massachusetts record form the State and Provinces table is set as the polygon of interest and a single line object from the roads table is also selected.  Now, I know that the line object is in Massachusetts so the value returned in IntersectValue field should be a value of one.

Look, it worked!

Contains

Another spatial overlay method available in spatial SQL is STContains().  This is a yes/no operation.  If a geometry instance is completely within another geometry a value of one is returned. If the geometry instance is not wholly inside the other, a value of zero is returned.  The following provides a simple example of STContains():

use Spatial_Database
go

DECLARE @poly_geom geometry
select @poly_geom = geom.STAsText()
From dbo.States_Provinces
where dbo.States_Provinces.name_1 = 'Massachusetts'

DECLARE @pointgeom geometry
select @pointgeom = geom.STAsText()
from Spatial_Database.dbo.Populated_Places
where NAMEASCII = 'Boston'

SELECT @poly_geom.STContains(@pointgeom) as ContainValue, @pointgeom as input1, @poly_geom as input2

In this query I am simply testing if the point that represents the city of Boston is within the polygon that represents the state of Massachusetts.  After all the variables are declared the select statement tests the point against the polygon.  The results, in the ContainValue field returns a one, meaning that the selected point is within the selected polygon.

Union

The union operator is an overlay analysis, like the intersect, however the results of a union are the combination of the two features, with overlapping areas obtaining the attributes of both.  Again, let’s turn to Esri for a nice graphic:

Source: Esri

Like the intersect operation, the STUnion() operation analyzes the spatial relationship between two different records.  The basic union query will return the area represented in the union.  In the following example the query will select two countries from the world countries table to union.  The results of the query return the unioned geomerty of the two selected polygons.

use Spatial_Database
go

DECLARE @poly_geom geometry
select @poly_geom = geom.STAsText()
from dbo.world_countries
where dbo.world_countries.ADMIN = 'Germany'

DECLARE @poly_geom2 geometry
select @poly_geom2 = geom.STAsText()
from dbo.world_countries
where dbo.world_countries.ADMIN = 'Switzerland'

SELECT @poly_geom2.STUnion(@poly_geom) as UnionGeo

The initial input are two distinct polygon records, one representing Germany and the other, Switzerland. If the user were to query these two records from the table they would see the following spatial result:

Two distinct records

However, during the union the geometry of these two records are morphed into a single feature, as seen below:

I don’t think Switzerland will union with anyone anytime too soon.

Using the example script the user could then add additional fields to the select query or perform any number of additional operations on the unioned record.

Difference

The final spatial operator that will be covered will be STDifference().  The difference overlay operation uses two layers with the area of one being removed from the overlapping area of the other.

Source:Esri

In spatial SQL the difference method, STDifference(), returns the geometry of the difference.  In the following example the user will find the shape that is left behind between the difference of the United States and Massachusetts:

use Spatial_Database
go

DECLARE @poly_geom geometry
select @poly_geom = geom.STAsText()
From dbo.States_Provinces
where dbo.States_Provinces.name_1 = 'Massachusetts'

DECLARE @USA geometry
select @USA = geom.STAsText()
from dbo.world_countries
where dbo.world_countries.ID = 236

SELECT @USA.STDifference(@poly_geom) as Part1andPart2

The results, as seen below, displays the area that Massachusetts represents in the world_countries table.

I bet there are some parts of the country that wouldn’t mind this…

Conclusion

Well, that is it for this tutorial.  If I have some time over the next month I’d like to explore spatial indexes within spatial SQL and perhaps cover some more in-depth spatial queries.  All these queries will work within 2008 R2 and the user can, if desired, use tables from ArcSDE within the queries.

The tutorial and the examples are very simple and I do not profess to be an expert in spatial SQL, so, if there is a way to represent any of these queries more efficiently let me know.

GISDoctor.com has gone @social

GISDoctor.com is now up to 2009 standards.  I have started a Twitter account for the site to spread the message, keep up with what others are doing, and to follow the trends.

Follow GISDoctor.com at @GISDoctor on Twitter or check out Twitter.com/GISDoctor.

Now, I’m a big fan of the real world so I may not tweet fifteen times a day, but I’ll keep the blog and twitter feed in sync.  As of today I have one follower and guess who it is…@Esri!  They have probably read the comment sections of this post and this post

Happy GIS tweeting!

ArcGIS Viewer for Silverlight – That was easy

This past Wednesday, on the ArcGIS Server Blog, Esri informed the geo-nerd world that their ArcGIS Viewer for Silverlight v1.0 is now available for download to the world.  The viewer is essentially a wizard for creating Silverlight based ArcGIS mapping sites with a number of easy to use features.

Side note: They also released a updated version of ArcGIS for SharePoint, but since I don’t use that I’m not going to write about it.

I downloaded the ArcGIS Viewer for Silverlight and gave it a spin.  Now, I haven’t built a Silverlight app in a while as I have been focusing my attention on building Javascript applications, you know, because Silverlight is dead (jk, not really).

I had seen demonstrations of the demo version a couple times, but I had never tried it out.  Also, I had never used the ArcGIS Viewer for Flex so this was a brand new experience for me.

Here is what I thought:

  • Installation was pretty quick and I had the application up and running in a couple minutes.
  • I followed the quick start guide until I got into the application.  Once I was in the application I used the guide that was available within the tool.  I thought this was a great feature, as it can walk anyone through how to create a mapping application.
  • Following the in-program guide, I selected my basemap, easily established the “look and feel” of the site and started to add data.  I even added a logo and a couple links to the menu bar.  This literally took two minutes.  Real easy and really fast.
  • At this point I could have launched the application, but I decided to play around with the layers in the app.  For my test application I added a US counties layer, a US states layer and a few thousands points from my company’s ArcGIS Server.  I was easily able to set a number of options for the polygon data, including the altering the symbology, the attribute table options, and pop-up info.  However, when I was working with the point data the options to configure the attribute table and symbology were not available.  Now, I’m sure there is a reason for this and if I had read more the of documentation I’m sure I would have been able to figure it out.
  • One of the nice features of this application is that the page updates while you are working on it.  This is a great feature if you are not a developer or GIS expert and you just want to get a map to the web.
  • Once I was happy with my test page I simply clicked the deploy button and the application was launched on my local machine.  I sent the link to a couple coworkers and they were able to view the map and data and asked how they could create their own mapping apps.  They thought the ArcGIS Silverlight Viewer was super awesome.
Two Minutes, Pretty Easy

All this took about 12 minutes while I was waiting for a geoprocessing task to complete. I didn’t even scratch the surface of the options available in the application.  When I get a few more minutes I’ll test out other features including adding tools and geoprocessing tasks, data from ArcGIS online, and working with the layout and display options.

Traditionally I have written the code for these types of applications, which at times can be cumbersome, especially if you are managing a variety of sites.  If a user needs a quick, light weight, Silverlight mapping app and has access to well developed ArcGIS Server I would recommend this tool.

That was easy... Is this a SOPA violation?

I have to say that using tool was refreshing, especially after a couple of tough GIS software days.  You know what I am talking about.

Good work Esri!  When does the ArcGIS Viewer for JavaScript with HTML5 support come out?

Tip of the Day – Check the Code

Some of the most popular pages on GISDoctor.com (shameless self-plug) are those that provide examples of how to read WMS data into Google Map applications from a number of sources including ArcGIS Server, OpenLayers, and MapServer.  People take the code from these pages, embed the scripts into their own applications, make some modifications, and publish their own mash-ups.  I think this is awesome and I encourage it on the pages themselves.  I’ve seen a few good applications of this code and I hope more come in the future.

However, I need you, the mash-up artist, to make sure you do one little thing when copying the code into your own applications. Do not copy the Google Analytics code into your own page.  This is the last two blocks of script at the end of each page.  To aid the user I have included a comment to remind them to not to include the Google Analytics script in their application.

Thanks!

A few times a week I notice pages that have used the code from GISDoctor.com (second plug of the post) that still include my Google Analytics information.  Most of the time the user will eventually remove the code or embed their own tracking info, but there are hold-outs.  I’m talking about you, person in Spain.

Thanks, and happy GIS-ing!

2012 – The GISDoctor.com Year of Spatial Analysis

With GISDoctor.com entering its second year in a blog format I have some plans for the site that will hopefully keep readers interested and bring in new readers.  Here is what I am planning for 2012!

  • Spatial Analysis – As a geographer by training, and professional geographer by occupation, I do a lot of spatial analysis and spatial statistics.  I will be developing a series of spatial analysis and spatial statistics posts over the year.  I will start with the basics and move into more complicated subjects.  Like the Intro to Spatial SQL guides, I will include test data and examples.  The tutorials will be targeted towards the geographer and GIS user, as that is my area of expertise.  I really want to focus this site towards the technical GIS professional and technical geographer during 2012.  Too many GIS blogs review the news (including this blog).  I want GISDoctor.com to become a technical resource for people who have questions like I do.
  • Spatial SQL – During January and February I will be adding some more posts on Spatial SQL and its use in geographical analysis.  I will be adding another series of more technical Spatial SQL how-to guides later in the year.
  • Online Mapping – I will hopefully add a few how-to guides on developing map-mash ups using a variety of APIs.  I’m interested in learning more about Bing Maps as well as Open Layers and Map Server.  I hope to get a few examples out sometime this spring.
  • Software Reviews – In 2012 ArcGIS 10.1 will be released (Hooray! or shucks. It depends on your point of view).  I’ll post a detailed review a few weeks after I get to use the software intensely.  Also, when I get SQL Server 2012 (Denali) I’ll review the improved spatial components.
  • Conference Reviews – I’ll make sure to review the 2012 Esri UC and any other conferences I make it to.
  • Finally, when applicable to the readers of this site, I’ll post news stories and items that I find interesting.

I’m sure there will be a number of other topics that I write about.  So make sure you subscribe to the feed to get the latest updates.

Thanks for being a reader.  The site has done a lot better than I could have imagined and I hope 2012 goes just as well.

Happy GIS New Years!

 

Maps in the News!

There was an article today on Slate.com in regards to David ImusCaGIS’s “Best in Show” award-winning map of the United States.  The story was also picked-up by Uber-nerd tech blog Gizmodo!

Even though this map has been out for a while (I think the map had to be published in 2010 to be eligible for the 2011 award) there are a couple items to take away from the Slate article.   As the readers of this blog know cartography is both a science and an art.  The Slate article discusses how Mr. Imus spent a tremendous amount of time placing labels and focusing on the details.  For anyone who has every been involved with a large cartography project knows the best GIS label placement algorithms will not place all the labels in the best locations from a design perspective.  I’ve personally seen drastic differences in how ArcGIS places labels compared to Quantum, or how Quantum places labels compared to CadCorp.  I’ve personally spent hours fixing the location of labels and graphics, but no where near the time Mr. Imus dedicated to this project!.

Beyond proper cartographic construction (the article discusses the use of a conic projection, label placement theory, feature identification, color usage, etc.), the author brings up what I believe is the key to creating a beautiful and effective map.  Design.  Without having an eye for design a cartographer will often create a map that is often boring or hard to understand.  Combine a well designed map with proper cartographic techniques and you get a map like Mr. Imus’.  One that is elegant and effective.  The map reader should be both interested in the content of the map and the “look and feel” of the map.  The design will pull the reader in and the content will keep them interested.

From a technical perspective I’d be interested in getting more information on a couple things, which may be answered when Mr. Imus’ website is back-up and running:

  • What set of programs did he use to create the map.  We all know there was some component of GIS in this effort, but how much?  At what point was this project moved into Adobe Illustrator or a similar graphics program?
  • What were the data sources?  Did he do any verification?
  • How long did this image take to draw on screen?  I know that when I have a ton of labels and graphics in a map the drawing time can be horrendous.
  • Do you think he has generated tiles of this map and will make them available as a map service?  Hey, Esri, call this guy up and add this map to ArcGIS.com!

I believe we need more high quality cartography like this in the world.  There is a lot of really good cartography happening in the world today, and people are now more familiar with maps, cartography, and GIS than ever before.  Unfortunately, their point of reference of what is a “good map” is the most bland and cartographically boring map I can think of.

Currently, the CaGIS site has the 2010 awards posted.  Go and check out some of the past winners.  Some are truly impressive maps.  Also, Mr. Imus’ website is down (*Update – It was back up as of 1/3/2012).  I think he got a lot of traffic today and his hosting company didn’t like it.

 

Spatial SQL for the Geographer – Part 2 – Basic Spatial SQL Scripts

The tutorial continues…

If you haven’t already read the first two parts of this guide please read:

The first two parts of the guide give some background information on SQL Server, covers some basic concepts, provides some links to download some software and data.  Once the user has down completed the first two part of this guide they will be able to understand this section of the guide, which will provide some basic spatial SQL scripts.

Tutorial

Here the user will learn about a few basic Spatial SQL scripts.  Nothing too complicated, but enough to get the user started.  Hopefully the user will start to think about ways to expand the scripts into their own ideas.

This tutorial will cover the MakeValid, calculating centroids, and buffering a polygon.

Checking the Validity of Shapes

Before the user starts to calculate buffers, generate centroids, union or intersect tables or perform any other Spatial SQL methods the user should first run the MakeValid command. MakeValid will check the shape to see if the data set breaks any of SQL’s rules for storing the geomerty of the shape. As mentioned in the first part of this guide the geometry datatype is very flexible, but the user should run MakeValid first before doing any analysis because there still might be self intersecting polygons, bad ring order or any other number of topoligical errors within the dataset.

Many GIS programs won’t worry about these problems on the surface, but SQL will and users should perform these queries on all spatial datasets, especially polygons data, as it is usually full of errors.

This first example will update the State_Provinces table, checking the geometry of each record in the table.  This query will update the spatial data type column in a table so that the shape is valid, meaning that the shape meets SQL’s requirements for the datatype

In this example the State_Provinces table is being checked. The user will need to
run the MakeValid command against the column that stores the spatial data type, in this case, the geom column.

use Spatial_Database
go 
update dbo.States_Provinces
set geom = geom.MakeValid()

This query make not catch all errors in the data set.  The following scripts will check the data for validity as well.  The third script will check to see if any of the records still contain any invalid geometries.  These scripts were modified for the geometry datatype from Beginningspatial.com, a great resource the user should check out.

--Fix invalid shapes
update dbo.States_Provinces
set geom = geometry::STGeomFromWKB(geom.MakeValid().STAsBinary()
,geom.STSrid)

--Correct the order of the points in the polygon ring
update dbo.States_Provinces
set geom = geometry::STGeomFromWKB
(geom.STUnion(geom.STStartPoint()).STAsBinary(),geom.STSrid)

--Check to see if any records still have any invalid geometries. 
--The results should return an empty table.
select  ID, geom.STIsValid() as ValidCheck,
geom, geom.ToString() as Spatial_Text_String
from dbo.States_Provinces
where geom.STIsValid() != 1

When these queries are run the geometry in the table be updated.  From there the user will be able to repeat this process for each of the tables loaded in the first tutorial.  If the user does not run the MakeValid query some of the examples in this tutorial may not work correctly.

Calculating Centroids

The next set of queries will take a polygon layer, calculate the centroids, and convert the centroids into latitude and longitude.  Using the world_countries table in the Spatial_Database database the user can run this query and return the name of the feature and the X and Y of the polygon’s centroid.  This query uses a subquery where the centroid is first calculated.  Once this is done the outer query converts the centroid into latitude (Centroid.STX) and longitude (Centroid.STY).

select U.Region_Name, Centroid.STY as Longitude,
Centroid.STX as Latidude
 from
(	 select geom.STCentroid() as Centroid, NAME as Region_Name
	 from dbo.world_countries
)U

The following will expand on the previous query.  In the next example the user will calculate the centroid for a specific polygon and then buffer the centroid.  In the subquery a where statement is set to select a specific record that meets a certain requirement. In this example the user will select records from the name_1 column that equal ‘Massachusetts’.  Also, when the STCentroid() method is called no parameters are needed, unlike other spatial methods which can require several methods.

In the outer query the centroid is buffered by 0.05 decimal degrees, set in the STBuffer method. Why are decimal degrees used here?  Well, it is because the data were loaded in as WGS84.   There will be a future tutorial on how to tackle this problem.

The user can run this query and return a buffer around the centroid for the state of Massachusetts.

select U.Centroid, Centroid.STBuffer(0.05) as buffer, U.StateBound
from
(	select geom.STCentroid() as Centroid, geom as StateBound
	from dbo.states_Provinces
	where name_1 =  'Massachusetts'
) U

Calculating a Buffer

The last query that will be demonstrated will introduce the buffer, even though I just gave an example. The buffer operation is a common operation within GIS tools. The following script will give a basic example of how to generate a buffer in Spatial SQL.

The user will notice that the shape column, geom, is set and the buffer method, STBuffer, is called. STBuffer accepts a parameter, the size of the buffer.  In this example the buffer size will be 0.05 decimal degrees.  Also, in this example a where statement is set to query a specific record from the table. When this is done only that record would be buffered. Again, the States_Provinces table is used.

select geom.STBuffer(0.05) as buffer
from dbo.States_Provinces
where name_1 =  'Massachusetts'

What’s Next

That is it for this section of the tutorial. As I said earlier, pretty basic stuff. In the next tutorial I will get into some other topics including unions, intersects, spatial indexes, joins and other methods and operations will be covered. And, it looks like I may end up creating several more tutorials than I originally planned!

Unitl next time…

Spatial SQL for the Geographer – Introduction

During WhereCamp Boston one of the impromptu break-out sessions was on spatial SQL. In WhereCamp fashion the person who proposed the session was looking to learn more about spatial SQL and its many uses.  With about 20 people in the session I volunteered to give a few examples of spatial SQL within Microsoft SQL Server.  Now, I was not prepared to give an in-depth or very informative session.  In fact, it was probably the worst presentation I have ever given.  I went through a number of examples of of some basic principals and I said I would put together a “Hello World” example of spatial SQL sometime after the meeting.

But, to complete a “Hello World” example we need to develop some background first.  In order to develop the guide I will be putting it together from the ground-up.  I use Microsoft SQL Server everyday, but I am GIS professional first, and many of the readers of this site are also GIS professionals.  Therefore, I will write the how-to from this point of view.  If you are a database professional this guide might be a little boring, but if you are a GIS professional I hope you’ll get something out of it.

The guide will be broken into three additional posts.  The first will be cover some background and vocab.  Part two will discuss basic spatial SQL syntax, and the third part will walk through how to expand basic scripts into larger sql processes. Once the guides have been finished I will provide links below.

Part 1Welcome to Spatial SQL – 11/14/2011

Part 2Basic Spatial SQL Scripts – 11/21/2011

Part 3 – More Basic SQL Scripts – 1/2012

Part 4 – Working Spatial SQL into Larger Processes – 2/2012

*Update – 12/4/2011- with the holidays and other commitments I am going to push back the last two installments!

This guide is being developed for use with Microsoft SQL Server.  That’s the DBMS that I use on a daily basis.

Before I release these posts feel free to browse some resources on spatial SQL and Microsoft SQL Server: