One Million Points

I am working on a couple projects and I need to generate some random points across defined bounding boxes.  I have the basic code worked out in python and I am testing the results.  Just for kicks here is a million points generated in Python to a csv file (in 10 seconds) and drawn (rather quickly) in Quantum.  Sweet.

One Million Points

I’ll share the code on Thursday.

 

Spatial SQL for the Geographer – Part 3 – More Basic Spatial SQL Scripts

The tutorial continues…

If you haven’t already read the first three parts of this guide check out:

The first three parts of the guide give some background information on SQL Server, covers some basic spatial and database topics, and provides links to download some software and data. Once the user has completed the first three part of this guide they will be able to understand this tutorial, which will provide some additional basic spatial SQL scripts.

Tutorial

In the last tutorial several scripts were introduced and examples were given.  In the last tutorial I covered the process to validate shapes, calculating centroids, and generate buffers.   In this tutorial several additional scripts will be covered, including generating the bounding box/extent of a feature, intersection and union operations, and difference operators.

Calculate Extent

Calculating the extent of a feature is something I use from time to time.  To find these values, such as the northern bounding latitude or eastern bounding longitude, can be calculated through Spatial SQL.  Spatial SQL has a built in method that will calculate the bounding box of a record, STEnvelope().  In its simplest context the user can generate the bounding box for any feature in a spatial table:

use Spatial_Database
go

select
geom.STEnvelope() as boundingbox
from dbo.States_Provinces
where NAME_1 = 'Massachusetts'

This example will return a field called “boundingbox” for the record where NAME_1 equals ‘Massachusetts’ that contains the minimum area rectangle that encompasses the shape.    The data in the “boundingbox” field is relatively useless the user adds some additional code to the query.  Building on the example provided by Microsoft the user can add the ToString() method to the query:

use Spatial_Database
go

select
geom.STEnvelope() as BoundingBox,
geom.STEnvelope().ToString() as BoundingBoxString
from dbo.States_Provinces
where NAME_1 = 'Massachusetts'

Here the user will see a new column, “BoundingBoxString” that returns a polygon with the five points that represent it.  You may ask yourself, why are there five points when you only need four to create the bounding box?  Well, for SQL Server to create a polygon the first point needs to be represented twice, so that the polygon closes itself properly.

Intersect

The intersect table operator is one of the most popular tools in the GIS work belt. Intersect is an overlay operation, where the area of the input feature is used as the basis to select features from the second feature, the intersect feature.  The following illustration will explain what an intersect is better than I can with words…

Source: Esri

In spatial SQL the STIntersects() tool is straight forward and easy to use.   However, it does not return a shape as other overlay methods do.  When run, STIntersects() will return a value of one if the two records spatially intersect and a zero if they do not.  Also, this method only tests the relationship between a single record in one table against a single record in another table.  The following provides an example of intersecting a single line object against a polygon object:

use Spatial_Database
go

DECLARE @poly_geom geometry
select @poly_geom = geom.STAsText()
From dbo.States_Provinces
where dbo.States_Provinces.name_1 like '%Massachusetts%'

DECLARE @line_geom geometry
select @line_geom = geom.STAsText()
from dbo.Roads
where ID = 2974

SELECT @poly_geom.STIntersects(@line_geom) as IntersectValue, @poly_geom as input1, @line_geom as input2

The Massachusetts record form the State and Provinces table is set as the polygon of interest and a single line object from the roads table is also selected.  Now, I know that the line object is in Massachusetts so the value returned in IntersectValue field should be a value of one.

Look, it worked!

Contains

Another spatial overlay method available in spatial SQL is STContains().  This is a yes/no operation.  If a geometry instance is completely within another geometry a value of one is returned. If the geometry instance is not wholly inside the other, a value of zero is returned.  The following provides a simple example of STContains():

use Spatial_Database
go

DECLARE @poly_geom geometry
select @poly_geom = geom.STAsText()
From dbo.States_Provinces
where dbo.States_Provinces.name_1 = 'Massachusetts'

DECLARE @pointgeom geometry
select @pointgeom = geom.STAsText()
from Spatial_Database.dbo.Populated_Places
where NAMEASCII = 'Boston'

SELECT @poly_geom.STContains(@pointgeom) as ContainValue, @pointgeom as input1, @poly_geom as input2

In this query I am simply testing if the point that represents the city of Boston is within the polygon that represents the state of Massachusetts.  After all the variables are declared the select statement tests the point against the polygon.  The results, in the ContainValue field returns a one, meaning that the selected point is within the selected polygon.

Union

The union operator is an overlay analysis, like the intersect, however the results of a union are the combination of the two features, with overlapping areas obtaining the attributes of both.  Again, let’s turn to Esri for a nice graphic:

Source: Esri

Like the intersect operation, the STUnion() operation analyzes the spatial relationship between two different records.  The basic union query will return the area represented in the union.  In the following example the query will select two countries from the world countries table to union.  The results of the query return the unioned geomerty of the two selected polygons.

use Spatial_Database
go

DECLARE @poly_geom geometry
select @poly_geom = geom.STAsText()
from dbo.world_countries
where dbo.world_countries.ADMIN = 'Germany'

DECLARE @poly_geom2 geometry
select @poly_geom2 = geom.STAsText()
from dbo.world_countries
where dbo.world_countries.ADMIN = 'Switzerland'

SELECT @poly_geom2.STUnion(@poly_geom) as UnionGeo

The initial input are two distinct polygon records, one representing Germany and the other, Switzerland. If the user were to query these two records from the table they would see the following spatial result:

Two distinct records

However, during the union the geometry of these two records are morphed into a single feature, as seen below:

I don’t think Switzerland will union with anyone anytime too soon.

Using the example script the user could then add additional fields to the select query or perform any number of additional operations on the unioned record.

Difference

The final spatial operator that will be covered will be STDifference().  The difference overlay operation uses two layers with the area of one being removed from the overlapping area of the other.

Source:Esri

In spatial SQL the difference method, STDifference(), returns the geometry of the difference.  In the following example the user will find the shape that is left behind between the difference of the United States and Massachusetts:

use Spatial_Database
go

DECLARE @poly_geom geometry
select @poly_geom = geom.STAsText()
From dbo.States_Provinces
where dbo.States_Provinces.name_1 = 'Massachusetts'

DECLARE @USA geometry
select @USA = geom.STAsText()
from dbo.world_countries
where dbo.world_countries.ID = 236

SELECT @USA.STDifference(@poly_geom) as Part1andPart2

The results, as seen below, displays the area that Massachusetts represents in the world_countries table.

I bet there are some parts of the country that wouldn’t mind this…

Conclusion

Well, that is it for this tutorial.  If I have some time over the next month I’d like to explore spatial indexes within spatial SQL and perhaps cover some more in-depth spatial queries.  All these queries will work within 2008 R2 and the user can, if desired, use tables from ArcSDE within the queries.

The tutorial and the examples are very simple and I do not profess to be an expert in spatial SQL, so, if there is a way to represent any of these queries more efficiently let me know.

2012 – The GISDoctor.com Year of Spatial Analysis

With GISDoctor.com entering its second year in a blog format I have some plans for the site that will hopefully keep readers interested and bring in new readers.  Here is what I am planning for 2012!

  • Spatial Analysis – As a geographer by training, and professional geographer by occupation, I do a lot of spatial analysis and spatial statistics.  I will be developing a series of spatial analysis and spatial statistics posts over the year.  I will start with the basics and move into more complicated subjects.  Like the Intro to Spatial SQL guides, I will include test data and examples.  The tutorials will be targeted towards the geographer and GIS user, as that is my area of expertise.  I really want to focus this site towards the technical GIS professional and technical geographer during 2012.  Too many GIS blogs review the news (including this blog).  I want GISDoctor.com to become a technical resource for people who have questions like I do.
  • Spatial SQL – During January and February I will be adding some more posts on Spatial SQL and its use in geographical analysis.  I will be adding another series of more technical Spatial SQL how-to guides later in the year.
  • Online Mapping – I will hopefully add a few how-to guides on developing map-mash ups using a variety of APIs.  I’m interested in learning more about Bing Maps as well as Open Layers and Map Server.  I hope to get a few examples out sometime this spring.
  • Software Reviews – In 2012 ArcGIS 10.1 will be released (Hooray! or shucks. It depends on your point of view).  I’ll post a detailed review a few weeks after I get to use the software intensely.  Also, when I get SQL Server 2012 (Denali) I’ll review the improved spatial components.
  • Conference Reviews – I’ll make sure to review the 2012 Esri UC and any other conferences I make it to.
  • Finally, when applicable to the readers of this site, I’ll post news stories and items that I find interesting.

I’m sure there will be a number of other topics that I write about.  So make sure you subscribe to the feed to get the latest updates.

Thanks for being a reader.  The site has done a lot better than I could have imagined and I hope 2012 goes just as well.

Happy GIS New Years!

 

Spatial SQL for the Geographer – Part 2 – Basic Spatial SQL Scripts

The tutorial continues…

If you haven’t already read the first two parts of this guide please read:

The first two parts of the guide give some background information on SQL Server, covers some basic concepts, provides some links to download some software and data.  Once the user has down completed the first two part of this guide they will be able to understand this section of the guide, which will provide some basic spatial SQL scripts.

Tutorial

Here the user will learn about a few basic Spatial SQL scripts.  Nothing too complicated, but enough to get the user started.  Hopefully the user will start to think about ways to expand the scripts into their own ideas.

This tutorial will cover the MakeValid, calculating centroids, and buffering a polygon.

Checking the Validity of Shapes

Before the user starts to calculate buffers, generate centroids, union or intersect tables or perform any other Spatial SQL methods the user should first run the MakeValid command. MakeValid will check the shape to see if the data set breaks any of SQL’s rules for storing the geomerty of the shape. As mentioned in the first part of this guide the geometry datatype is very flexible, but the user should run MakeValid first before doing any analysis because there still might be self intersecting polygons, bad ring order or any other number of topoligical errors within the dataset.

Many GIS programs won’t worry about these problems on the surface, but SQL will and users should perform these queries on all spatial datasets, especially polygons data, as it is usually full of errors.

This first example will update the State_Provinces table, checking the geometry of each record in the table.  This query will update the spatial data type column in a table so that the shape is valid, meaning that the shape meets SQL’s requirements for the datatype

In this example the State_Provinces table is being checked. The user will need to
run the MakeValid command against the column that stores the spatial data type, in this case, the geom column.

use Spatial_Database
go 
update dbo.States_Provinces
set geom = geom.MakeValid()

This query make not catch all errors in the data set.  The following scripts will check the data for validity as well.  The third script will check to see if any of the records still contain any invalid geometries.  These scripts were modified for the geometry datatype from Beginningspatial.com, a great resource the user should check out.

--Fix invalid shapes
update dbo.States_Provinces
set geom = geometry::STGeomFromWKB(geom.MakeValid().STAsBinary()
,geom.STSrid)

--Correct the order of the points in the polygon ring
update dbo.States_Provinces
set geom = geometry::STGeomFromWKB
(geom.STUnion(geom.STStartPoint()).STAsBinary(),geom.STSrid)

--Check to see if any records still have any invalid geometries. 
--The results should return an empty table.
select  ID, geom.STIsValid() as ValidCheck,
geom, geom.ToString() as Spatial_Text_String
from dbo.States_Provinces
where geom.STIsValid() != 1

When these queries are run the geometry in the table be updated.  From there the user will be able to repeat this process for each of the tables loaded in the first tutorial.  If the user does not run the MakeValid query some of the examples in this tutorial may not work correctly.

Calculating Centroids

The next set of queries will take a polygon layer, calculate the centroids, and convert the centroids into latitude and longitude.  Using the world_countries table in the Spatial_Database database the user can run this query and return the name of the feature and the X and Y of the polygon’s centroid.  This query uses a subquery where the centroid is first calculated.  Once this is done the outer query converts the centroid into latitude (Centroid.STX) and longitude (Centroid.STY).

select U.Region_Name, Centroid.STY as Longitude,
Centroid.STX as Latidude
 from
(	 select geom.STCentroid() as Centroid, NAME as Region_Name
	 from dbo.world_countries
)U

The following will expand on the previous query.  In the next example the user will calculate the centroid for a specific polygon and then buffer the centroid.  In the subquery a where statement is set to select a specific record that meets a certain requirement. In this example the user will select records from the name_1 column that equal ‘Massachusetts’.  Also, when the STCentroid() method is called no parameters are needed, unlike other spatial methods which can require several methods.

In the outer query the centroid is buffered by 0.05 decimal degrees, set in the STBuffer method. Why are decimal degrees used here?  Well, it is because the data were loaded in as WGS84.   There will be a future tutorial on how to tackle this problem.

The user can run this query and return a buffer around the centroid for the state of Massachusetts.

select U.Centroid, Centroid.STBuffer(0.05) as buffer, U.StateBound
from
(	select geom.STCentroid() as Centroid, geom as StateBound
	from dbo.states_Provinces
	where name_1 =  'Massachusetts'
) U

Calculating a Buffer

The last query that will be demonstrated will introduce the buffer, even though I just gave an example. The buffer operation is a common operation within GIS tools. The following script will give a basic example of how to generate a buffer in Spatial SQL.

The user will notice that the shape column, geom, is set and the buffer method, STBuffer, is called. STBuffer accepts a parameter, the size of the buffer.  In this example the buffer size will be 0.05 decimal degrees.  Also, in this example a where statement is set to query a specific record from the table. When this is done only that record would be buffered. Again, the States_Provinces table is used.

select geom.STBuffer(0.05) as buffer
from dbo.States_Provinces
where name_1 =  'Massachusetts'

What’s Next

That is it for this section of the tutorial. As I said earlier, pretty basic stuff. In the next tutorial I will get into some other topics including unions, intersects, spatial indexes, joins and other methods and operations will be covered. And, it looks like I may end up creating several more tutorials than I originally planned!

Unitl next time…

Spatial SQL for the Geographer – Part 1 – Welcome to Spatial SQL

Before we begin I would like to make it clear that learning all the capabilities and limitations of SQL Server and Spatial SQL cannot be covered in three short lessons.  If you believe you might need some more training please visit the Microsoft SQL Transact reference guide.

What is Microsoft SQL Server?

Microsoft SQL Server is a relational database system that stores data and provides several mechanisms to retrieve stored data.  One of the beauties of relational database systems is that they allow you to effectively store a variety of data sets and they are both scalable and securable, and in my experience, much more trustworthy than any flat file.

There are a number of good references out there explaining SQL Server in much more detail than I would attempt.  I recommend that you check a couple of them out:

Explanation of Spatial Data Types

As you probably know by now a column in a table or view can be assigned one of many different data types.  Most are familiar with the traditional data types such as varchars, integers, bits, floats, or decimals.  With SQL Server 2008 two new spatial data types were introduced.  These data types support the storage of spatial data, and are broken up into two different types.  Their characteristics are important to understand:

  • Geometry:  Primary characteristic – spatial data stored in a planar system.
  • Geography: Primary characteristic – spatial data stored using the ellipsoid.
  • Here is a great blog post that describes the spatial data types in more detail.

Tutorial

Step One: Getting Started

The user first needs to download some software and data to their local machine.

Downloading Microsoft SQL Server 2008 R2 may take a few minutes.  The user needs to make sure that they read the download and installation instructions before they proceed. Also, when installing the software the users needs to pay particular attention to the installation instructions.

Note:  This tutorial was completed on a 64 bit machine running Windows 7.  I cannot guarantee that any of this will work on anything different.

Notes about the Data.  The populated places, countries, and states and provinces data sets were downloaded from Natural Earth and the roads data set came from Data.gov. The data represent the three different vector data types, points, lines and polygons. They also provide a variety of query options to work with during for the tutorials. Also, I renamed the data from the original names so for clarity during the tutorials.

Once you download the data I recommend first viewing it in a GIS program to make sure you downloaded it correctly.  Here the user will see the four data sets in Quantum:

Tutorial Data

Step Two: Create a Database

Now that the software has been downloaded the user can open Microsoft SQL Server Management Studio.  To do this the user will select to Start–> All Programs–> Microsoft SQL Server 2008 R2–>SQL Server Management Studio.  Since this will probably be the first time the user opens MSSQL it may take a couple minutes to configure and load.

Next, the user will be prompted to connect to a server.  In this case the server is the local machine.   In this example I connect to my local machine instance and use Windows Authentication.  Unless you configured your SQL instance differently you will connect the same way.  Once these options are selected click connect.

connect to server

Now that the user is connected to the server they can create a new database.  First, the user will expand the database instance.  Then the user will right click on the database folder and select New Database from the menu.

Create Database

The new database window will now open.  Here the user can configure a number of settings before the database is created.  In this tutorial we want to name the database Spatial_Database which is set in the Database Name text box.  Another option I like to configure is the location of the data and log files.  This is done by clicking the … button under the Path column.  I recommend storing the database in a location that has plenty of room to grow, as the default location on the C drive may not have space to expand on your local machine.

Change Location

Once the settings are all set the user will click OK to create the database.  The user can refresh the Databases folder to see the new, empty database.

Spatial Database

The user is the system admin on this sever and this database be default.  This means the user has permissions to create databases, delete databases, create tables and load data into them.  With these privileges already in place the admin user can load data into the database without any further configuration.

Step Three: Load Data into the Database using Shape2SQL

Using Shape2SQL is really easy.  The user will first open Shape2SQL and add a shapefile that will be loaded into the database as an individual table.  To do this the user will first navigate to shapefile stored on the local machine by clicking on the Select Button.

Select  Shapefile

The next step is to select the server and database where the data will live.  The user will click on the Configure Button to do this.  In the Configure window the user enter the name of the server in the Server Name window.  The user will then select Use Windows Authentication and then select the appropriate database.  In this tutorial the database has been named Spatial_Database.  Once this information has been configured the user will click OK.

Configure Server

Now, back in the uploader window the user has several additional options to configure before loading the data into the database.  First, the user needs to select the data type. The user will select either the geometry or geography data type.  In this tutorial we load the data as the geometry data, as it is more flexible, in terms of what it will accept, than the geography data type.

The second setting that needs to be set is the name of the geometry column.  The default for this setting is ‘geom’, but the user can change this if desired.  Remember, all the tutorials will use the ‘geom’ column.

The user will then make sure the Create Spatial Index option is selected.  This will ensure that a spatial index is built, which will improve the performance of the table in SQL Server.  Finally the user will set the name of the table in the Table Name window.

There are a few settings that do not need to be selected or altered.  Since this tutorial is using the geometry data type the user does not need to set the SRID.  The user can also change which columns are loaded into the table.  By default all columns are taken from the shapefile and loaded into the table. If a user does not want certain columns they will simply uncheck the columns they don’t want.  Once all the settings are selected the user will select Upload to Database.  Again, I may use all the columns in future tutorials, so you are best off keeping all of them.

Once the user clicks Upload to Database the shapefile will be loaded into the database as a table and the spatial index will be created.  Depending on the size of the shapefile and the power of the user’s machine it may take several minutes to load the table into the database.

The user will repeat these steps and load all four shapefiles into their database.  Once that is completed the user can proceed to the next section of the tutorial.

Step Four: Perform a Basic Query

Now that the shapefiles have been loaded into SQL Server the user can start to perform some queries.  Now, if the user does not have any SQL experience at all it may make sense to read through this tutorial from Microsoft.

The first example query is probably the most simple SQL query one could perform. In this first example I will use the dbo.States_Provinces table.  The user will open SQL Server, connect to their server, and then click the New Query button.  The user will then paste the following into the query window:

use spatial_database
go

select top 100 * from dbo.States_Provinces

The results will return the first 100 records from the dbo.States_Provinces table.  In SQL Server the user will notice there is a Results tab and a Spatial results tab.  The results tab will display the tabular data and the spatial results will display the spatial data.  The Spatial results tab will only appear if there is spatial data returned in the query.

Step Five: Set a Where Statement to Query Specific Data

The user can narrow down what is selected by incorporating some basic SQL statements, such as a where statement.  The basic premise of a where clause is to limit the data returned from a query. The user is setting a condition in regards to what they want returned.  For example, let’s say we only want the return records from the States_Provinces table that are for the United States.  If the user explores the table they will set that this information is stored in the Name_0 column. However, we are unsure of the exact syntax of “United States” so we can use a like operator in the where clause. For example:

use spatial_database
go

select * from dbo.States_Provinces
where NAME_0 like '%United States%'

The user will see only those records that have ‘United States’ in the Name_0 column. If the user clicks on the spatial results tab they will see the spatial representation of those records.

Spatial Results

Alright, that is it! We have completed the first part of this tutorial. The user has downloaded the appropriate software and data, installed the software, loaded a number of shapefiles using Shape2SQL into SQL Server, and have completed two basic queries.

Before the next tutorial I recommend the user, if they are light in their SQL experience, read up on some basic queries from either the W3 Schools or from Microsoft.

The next part to this tutorial will cover some more Spatial SQL queries.  I’ll go into more detail about some spatial operations and provide a number of examples.  If there is anything the you want to see let me know!

Spatial SQL for the Geographer – Introduction

During WhereCamp Boston one of the impromptu break-out sessions was on spatial SQL. In WhereCamp fashion the person who proposed the session was looking to learn more about spatial SQL and its many uses.  With about 20 people in the session I volunteered to give a few examples of spatial SQL within Microsoft SQL Server.  Now, I was not prepared to give an in-depth or very informative session.  In fact, it was probably the worst presentation I have ever given.  I went through a number of examples of of some basic principals and I said I would put together a “Hello World” example of spatial SQL sometime after the meeting.

But, to complete a “Hello World” example we need to develop some background first.  In order to develop the guide I will be putting it together from the ground-up.  I use Microsoft SQL Server everyday, but I am GIS professional first, and many of the readers of this site are also GIS professionals.  Therefore, I will write the how-to from this point of view.  If you are a database professional this guide might be a little boring, but if you are a GIS professional I hope you’ll get something out of it.

The guide will be broken into three additional posts.  The first will be cover some background and vocab.  Part two will discuss basic spatial SQL syntax, and the third part will walk through how to expand basic scripts into larger sql processes. Once the guides have been finished I will provide links below.

Part 1Welcome to Spatial SQL – 11/14/2011

Part 2Basic Spatial SQL Scripts – 11/21/2011

Part 3 – More Basic SQL Scripts – 1/2012

Part 4 – Working Spatial SQL into Larger Processes – 2/2012

*Update – 12/4/2011- with the holidays and other commitments I am going to push back the last two installments!

This guide is being developed for use with Microsoft SQL Server.  That’s the DBMS that I use on a daily basis.

Before I release these posts feel free to browse some resources on spatial SQL and Microsoft SQL Server:

Upcoming Boston Spatial Unconferences and Meet-ups

October is a busy time for spatial unconferences and meet-ups in the Boston area!

The folks at ISpatialBoston are hard at work organizing Boston’s first WhereCamp, which will be taking place October 29th and 30th at the Microsoft NERD office in Cambridge’s technology hub, Kendall Square .  WhereCamp is an “unconference“, meaning that the attendees develop sessions based on what they are interested in.  They have three keynote speakers lined-up and I’m sure there will be many interesting sessions that develop over the course of the weekend.  Registration is open and it’s not too expensive. If you are in the area and are interested in “spatial” check out WhereCamp Boston.  I’m looking forward to my first unconference.  I can imagine that there will be a lot of open-source and crowdsourcing talk.  I’m interested in finding out more on how to, if possible, validate crowd-sourced data in relation to the impacts of natural catastrophes.

The other upcoming spatial gathering is Esri’s Boston DevMeet-Up taking place on October 20th.  The location is TBD, but last year’s Boston meet-up had about 50 to 60 people show-up and there were a few good talks.  I even won an EDN license!  The Esri DevMeet-Ups bring together Esri geospatial developers in an informal setting to talk shop.  It is also a good time to meet other geospatial developers and pick their brains!

Unlike the weekend long WhereCamp, the Boston Esri DevMeet-Up is only for a couple hours on a weeknight where Esri provides the food and drinks.  I hope there are some good talks.  I’d be interested to see some code examples or talks about HTML5 integration, web-based spatial analytics, or interactive visualization methodologies for very large datasets.

Again, if you are in the area, check out these non-traditional spatial meetings.  I know there are a ton of geospatial folks who can take public transit to these events.  Hope to see all the locals there!

GIS Doctor Mailbag

This blog has been up and running for a while now and I have gotten a number of email questions, responses to blog posts, and search engine keyword questions.  Let’s answer a few of them!

How do geographers collect spatial data?

  • By many means.  There are the arm chair geographers (like me!) who collect, develop and analyze spatial data from a variety of published data sources and there are those geographers who go out into the field.  These rugged souls create and administer surveys, collect data in the field in all conditions, and develop scientific studies to derive their own data.  Depending on the nature of the study a geographer may use a mix of data collection methodologies.

How do I use more memory with ArcGIS 10?

  • I hear this question more and more, especially as more users have 64 bit machines and automatically assume that everything will be faster.  ArcGIS 10 will use all the memory available in a 32 bit environment, which is 4 gigabytes. However, not all the tools and processes in ArcGIS will use all the memory.  To work around this problem users can chunk their analysis to try to speed things up, or they can review their analysis and try to develop a set of processes that make the analysis smaller, therefore faster.  In the past I have split up large input data sets (20 million plus points) and used python and ArcGIS tools to have the analysis run effectively.

How much ArcGIS Server 10?

  • Not cheap.  If you want free check out MapServer.

How can I get Bing Maps in my GIS?

  • If you are not behind a firewall a user can click on the Add Data button in ArcGIS and select Add Base Map.  The Bing data should then be a base map option. Since my instance of ArcMap is behind a firewall I cannot add the Bing Service. This is a bummer, since I am a big fan of Bing Maps.

How can I add Google Maps into ArcMap?

  • There isn’t a quick tool to load Google Maps into ArcMap.  However, there are a few applications out there that allow a user to port in the imagery from Google into ArcMap.  One well known tool is provided by Arc2Earth.  Now, if the user is running ArcServer Esri provides a number of tools to publish data in the Google Maps interface.

How do I kill a geoprocess ArcGIS 10?

How do I speed up ArcGIS 10 on 64 bit machine?

  • Wait until Esri releases a 64 bit version of ArcGIS, or start using GIS software that is native 64 bit.

Is it possible to use embed a WMS data source into a Google Maps mash-up?  

  • Why, yes it is.  Check some examples here

Is ArcGIS difficult to learn?

  • I’ve been in the field of GIS for close to 10 years and I’m learning something new about ArcGIS everyday. GIS is complicated and the theory behind it is deep.  Many want GIS software to be simple, but users need to understand the details to properly frame an analysis and understand the results.  If GIS were very easy I believe you would get many people, more so than now, making poor decision from poor analysis.

Why did Esri skip version numbers?

  • They did?

What is wrong with ArcGIS 10?

  • Unfortunately, plenty.  Check out the bug fixes for SP1 and SP2.

Why do ArcGIS tools take so long?

  • Another question I hear all the time,  “why is process XYZ so slow?”.  The answer is never as direct as the person asking the question wants it to be.  There are a number of reasons why a particular process or processes may be slow.  To try to understand the issue I always work through a series of questions.  First, what are the specs of the machine running the process?  What other processes are running on the machine?  What are the dimensions of the data being processed?  Are the data sets in the analysis in the same projection? How large is the data, both spatially and in memory?  How intensive is the process itself?  How detailed is the data being processed?  The point is that there isn’t always a smoking gun to solve all processing questions.  Sometimes the process will be slow.

OK, that is it for now.  If you have any additional questions drop me a line!  Until next time.

The GIS of Hurricane Irene

Hurricane Irene is upon us here in Boston.  Thankfully, I prepared yesterday so I can spend some time this morning blogging during hurricane (or tropical storm)!  Also, I’d like to get this blog out before we lose power, since Boston is on the windy side of the storm.

Now, geographers and GIS pros are all over this storm.  Unlike the Tohoku earthquake and tsunami, which had a ton of reactionary GIS development, hurricanes provide the opportunity to develop datasets, applications, and analysis before the storm arrives.  There are a number of great applications and datasets that have been generated in the past four days, with many more to come over the next several days.

Tracking Applications: Everyone loves online maps

New York Times hurricane tracker – Great, clean application.

MSNBC hurricane tracker – Built by Stamen.  Great information, and a great look, but no data downloads.  Open the data!

Esri – The Worldwide Leader has a pretty nice mapping application (as they should) tracking the storm. They have a number of social media links, which will be great to view for damage and impact information after the storm.

CNN – Old school, like 1998.  CNN, get with it and build a better mapping application.

Yahoo and ABC are both displaying a mash-up of the storm that appears to be developed by the AP.  The application has a number of different tabs with information related to the storm.

GIS Data Sources: What you are really here for.

NOAA -A number of technical data sets are available and have been updated throughout the storm.  Click on a map and look for the “Download GIS Data” option.  Shapefiles are available and a number of Google based mash-ups are included.  The NOAA site may not be as flashy as others but the data available is very valuable.

Weather Underground – This site has gotten a lot of press this week as the storm has approached Megalopolis.  They have a number of tools and data sets available including a slick tracking mash-up, and number off data sets which are not necessarily GIS ready, but GIS-“able”.

Esri – Yes, Esri is on this storm.  Earlier this week they published a site with a number of data links to GIS Web Services, data providers, and scientific data sources.

FEMA data sources and data feeds.  FEMA is providing data from both internal and external sources.

CrisisCommons – A number of links to data sources related to the storm.  This source will grow as the storm passes.

Maps and data are already available on Geocommons.

After the storm there will be a number of data sets that become available through the Army Geospatial Center.

Global Disaster Alert and System– Hurricane Irene information is available here.

What’s next?  As the storm passes and people are able to survey the damage you will start to see impact analysis data sets, loss estimation maps, and analysis on the storm itself including better measurements of rainfall, windspeed and the track of the storm.  Once that data starts to roll out I’ll update the page with some more links to data.

Good luck, stay safe, and stay dry northeasterners! These storms are nothing to mess with. Take them seriously!

Great story in the New York Times

In case you missed it there was a great story in the New York Times on Tuesday in regards to the use of GIS in historical analysis. The article, Digital Maps are Giving Scholars the Historical Lay of the Land” , by Patricia Cohen, discusses the evolution of the spatial humanities and historical GIS/geography, which are growing disciplines in the humanities at colleges and universities around the country.

The article provides a nice overview of how historians, archaeologists, and other non-geographers have embraced spatial analysis and GIS in their research.  I think this is a great article on a trend in the humanities that has been growing for years.  I remember as an undergrad ten years ago developing GIS tools to visualize historical settings.  In grad school I routinely helped non-geographers develop spatial analysis methodologies and visualization techniques to process and analyze historical GIS data.  Much of that work ended up in scholarly publications.  The spatial component really gave the authors an edge over other papers at that time.

So, if you get a few minutes check the article out.  Anytime that GIS gets mentioned in the New York Times is great for our field!

A couple of notes from the article:

  • The author references www.gis.com.  I’m sure the marketing department at Esri liked the link.
  • The article links to David Rumsey’s site.  If you are a map junkie like myself you will love this site.  An amazing map collection.  This site has really influenced the development of online map libraries around the world.
  • I wonder if the growth of GIS and spatial analysis in the humanities (which has been happening for a number of years) has increased enrollment and/or developed programs in GIS and geography at schools where the spatial humanities are strong.  The AAG should get on this.
  • Does anyone remember the digital landscape history of Manhattan that was put together a couple of years ago?  The project gained some press and buzz when it came out.  I’m surprised this article didn’t mention it.  Oh well…