Data, Analysis, Spatial, Code, Running, Boston
AvidGeo, Boston’s #1 social geo group, will be meeting up this coming Thursday, July 18th, at 6:30 at Cambridge Brewing Company in Kendall Square (map).
Swing by for a beer (or water, or soda, or orange juice) with some friendly geo-nerds, talk shop, and pitch in ideas for fall AvidGeo events. It should be a good time!
For more information check out AvidGeo’s on twitter or at Meetup.com
I’ll be out in San Diego for the 2013 Esri UC this week. Check out my twitter feed, @GISDoctor for updates!
My goal for the week – set a PR in the Esri 5k on Wednesday!
A few years ago my old boss at MAGIC gave me benjaminspaulding.com and some hosting space as a going away gift. I Didn’t do much with it early on (I built this site instead using the hosting), but eventually I set up a static page with some basic info.
Well, after a few months of poking around I finally got the new benjaminspaulding.com up and running. Being a geographer, I have some mapping components on the site (powered by Leaflet) along with links to my twitter, linkedin, and GISDoctor sites. The site is a hobby of mine and by no means do I profess to be a professional web developer. I did test the page with the latest versions of IE, Chrome and Firefox and everything I wanted to do worked. However, if something is broken for you let me know.
Thanks!
A few weeks ago everyone on earth learned about Map Box’s cloudless atlas, a cloud free mosaic dataset, generated from freely available Lansdat data. It was all over the twitterverse and Wired magazine even wrote about it.
But, did you know Google came out with a cloudless imagery layer last week too? They did! Read about it here! Sometimes it’s good being the second one to the party, but I think only a couple people noticed that Google showed up at the cloudless atlas party. Whoever does the marketing and PR at MapBox should get a bonus and a raise. It’s not everyday you get more media traction than Google.
It is pretty well known that cloudless imagery is nothing new. I remember working on projects in grad school several years ago on this same problem and researchers have been developing tools and processes to analyze cloudless imagery for years, as a quick search on Google Scholar will demonstrate. Regardless, these are great datasets for visualization, and the fact that they are easily accessible to those making “beautiful maps”, not just GIS nerds, is excellent.
We need to give these organizations a lot of credit. They took a phenomenal, freely available data source, added some intelligent and creative processing, developed some incredible datasets, and got the spatial and non-spatial world to notice. Kudos to them.
I usually don’t respond to comments on my blog post, but I feel the need to respond to a couple items from a recent comment posted on everyone’s favorite article on this blog. Before I start my response let me say that I agree with some parts of the comment, which I will discuss later. However, I really disagree with other passages and I’ll talk about those first.
But first, the comment…
From Chris, posted on 6/18/2013
Now, my response:
The “open vs. commercial GIS” mentality is getting old. I am getting sick of it, and you should be too.
We should all be vested in the development of GIS as a science, as a tool set, and as a way of thinking, whether it is commercial or free, open or closed. GIS is aided by the growth of both free and open source and commercial software. It is pretty well known that the commercial sector now has real competition and they need to respond. This is a good thing. Saying that open source GIS is going to kill commercial GIS software is like saying that Linux has killed Windows or PostgreSQL killed Oracle. These commercial GIS shops are embedded deep in many organizations and they aren’t going to be dropping them any time soon. That is a choice that they made, and when the business case dictates a change they will make it.
Now, will organizations who are new to GIS or at a point of transition choose to go with open source platforms that are light-weight, reliable, and free and easy to use when they design their next implementation? Probably. I would, especially if cost was a factor.
The more scientists, engineers, planners, civic leaders, decision makers, concerned citizens, business leaders and educators who use GIS the better it is for GIS community as whole. The GIS community should be working together to move technology forward – not digging trenches and setting in for battle. It is a horribly counter productive strategy. Any users of GIS, online mapping, or spatial analysis would logically want the field to grow and evolve. Competition helps drive that growth and evolution and I am all for it.
And, why do people keep saying/thinking that Google should buy Esri, or that Google Earth is going to overtake ArcMap? I’ve never understood this argument. Never. Why would Google, who has really failed at commercializing their current geo-stack, go after such a small market when compared to their other endeavors? If you were a smart company that makes a lot of money what market would you focus on as revenue driver? Millions and millions of mobile users, or a few thousand specialized (and picky) GIS software users? How many of you are paying for Google Maps or have bought Google Earth Pro? Not many? That’s what I thought. Let’s just drop this train of thought.
Now for the second statement in the comment that drives me absolutely crazy:
This statement is so horribly misguided I don’t even know how to respond.
Let’s try. There is a calculator on every computer and smartphone made, and we use them all the time. Does this mean we need to eliminate math as a discipline at the university level? Anyone can download a content management system and build a website. Time to get rid of computer science departments! Turbo Tax! Get rid of accounting majors! WebMD. Who needs pre-med?
See where I am going with this? Just because a tool exists does not mean that a particular discipline should be eradicated. You still need some background to understand what you are looking at.
I have three degrees in geography. A bachelors, masters, and Ph.D. . During those years of schooling and research I did far more than make maps and use online mapping tools. Geographers study far more than just cartography and learn about place names, check out my dissertation for proof.
Geographers have been crucial in the development of the theory, logic, and science behind the G in GIS. The field of geography has also provided countless contributions to spatial analysis, policy and planning, environmental science, economics, anthropology, sociology, biology, civil engineering, and many more and will continue to so. In fact, we need more geography being taught at all education levels.
Geography matters. It always has and it always will.
To say that geography doesn’t matter displays a lack of understanding that is all to common in the GIS community. Sometimes I am floored with the lack of understanding of the basic principles and fundamentals of geography among those who use GIS, online mapping tools, or any other type of spatial decision making system. Without a basic foundation in geography how do you expect to make the correct decisions using a GEOGRAPHIC information system? I’m not saying that everyone who uses GIS needs a Ph.D. in geography, but taking a couple geography courses during your undergraduate years isn’t going to hurt.
Now, what do I agree with from the comment?
That’s enough for this post. I didn’t do a god job summarizing my thoughts at the end, but it is getting late and I want to go to bed. More rambling and ranting will come in later posts
As always, thanks for reading.
I have been using Spatial SQL for a while now. I like it. A few lines of code can do a lot of analysis or data processing. I’ve covered a number of basic topics but there are always more to do. Here is a script to take a series of points and convert them into a single line. This script will use a few SQL commands, including using a cursor, developing a linestring, and STLineFromText.
The sample data comes from NOAA and the National Hurricane Center. The points represent some sample tropical cyclone forecast points for Hurricane Sandy.
The basic idea of the script is to link a set of points together using a common attribute using a cursor, combine the coordinate pairs into a line string, and use the STLineFromText method to convert the coordinate pairs into a single line. The script works pretty well, but since I am using a cursor it can slow down with larger (a few hundred thousand rows) datasets. A good SQL programmer probably wouldn’t use a cursor here since they can be slow and cumbersome to use. In fact, I’m sure a good SQL programmer wouldn’t use a cursor. I am investigating ways to not use a cursor, so if you have a suggestion let me know!
The sample script includes a sample dataset that is available here. Take a look at the script. If you have any suggestions to make this script faster or more efficient let me know.
/*################################################# Script Name: multi_points_to_Line.sql Purpose: This script will take pairs of coordinates of the same line and convert them into a LineString that can be used to generate lines of the geometry data type. User will need to update the database, tables, and column names relevant to their own analysis. Sample Sandy data tracks represent five day models from the National Hurricane Center. Prepping the data - The user will need to download the following file and load into their SQL database: http://www.gisdoctor.com/downloads/Line_Parts.txt Here is a quick script to take the text file and load it into a table generated for this exercise: create table Spatial_Database.dbo.Distinct_Points ([ADVISNUM] varchar(3), [lat] float, [lon] float, [MaxWind] int) BULK INSERT Spatial_Database.dbo.Distinct_Points FROM 'Path to Line_Parts.txt file' WITH (FIELDTERMINATOR =',',FirstRow = 2); ###################################################*/ --Set the database to process in use Spatial_Database --Drop temporary table drop table #Sandy_hur_tracks --Create new temporary table create table #Sandy_hur_tracks ([EventID] int, [line] geometry) --Declare cursor variable DECLARE @eventID varchar(10) --Declare text string that will store coordinate pairs to --populate the LineString DECLARE @coordString VARCHAR(MAX) --Initialize the cursor using the ADVISNUM column DECLARE db_cursor CURSOR FOR select distinct ADVISNUM from Spatial_Database.dbo.Distinct_Points order by ADVISNUM asc OPEN db_cursor FETCH NEXT FROM db_cursor INTO @eventID WHILE @@FETCH_STATUS = 0 BEGIN -- Clear the coordinate string with each iteration of the -- cursor - otherwise the coordinate string will -- append itself each time set @coordString = '' --collect all coordinate pairs and add them to a single row. -- Coordinate pairs are separated by a comma. select @coordString = (COALESCE(@coordString + ', ', ' ') + (cast(Lon as varchar) +' ' + CAST(lat as varchar))) FROM Spatial_Database.dbo.Distinct_Points WHERE ADVISNUM = @EventID --Insert the eventId and coordinate pairs into the table. --Coordinate pairs string is used to build the LineString to --create the line geometry insert into #Sandy_hur_tracks select @eventID as EventID, Geometry::STLineFromText('Linestring (' + right(@coordString,LEN(@coordString)-1) + ')' , 4326) as line FETCH NEXT FROM db_cursor INTO @eventID END --Close and delete the cursor CLOSE db_cursor DEALLOCATE db_cursor --Select results from the temp table. use Spatial_Database select * from #Sandy_hur_tracks
One of the great things about living in Boston is that there is a very active geo-community. Every few weeks there is something interesting happening, whether it is an AvidGeo meet-up, a geo-colloquium at one of the many schools in Boston, or an industry sponsored event.
One of those events happened this past Monday at Space with a Soul in Boston’s Innovation District, organized by Avid Geo and the Eclipse Foundation’s LocationTech group. The event focused on open source geo-based projects. The room was full of geo-thinkers from a variety of backgrounds, and thanks to a number of sponsors (AppGeo, Azavea, IBM, Actuate) there was plenty of food and beer! Added bonus, participants from the PostGIS code sprint were in town!
The lighting talk format – five minutes, 20 slides, auto advancing – works really well for these types of events. The speakers are energized and the crowd stays captivated. I tried in vain to keep up with Twitter during the event. Let’s take a look at my 140 character rundown of the evening.
Quick note – I missed a couple speaker’s names. If you know them please post a comment so I can update accordingly. UPDATE – I only need one more name!
Second quick note- Ignore the grammar mistakes in my tweets. I am a horrible with my thumbs.
Andrew Ross from LocationTech opened up the evening. He talked about the mission of LocationTech and explained how the Eclipse Foundation helps open source projects get off the ground and stay relevant.
Great crowd for #ignitelocationtech at Space with a Soul #boston #geo @avidgeo#locationtech twitter.com/GISDoctor/stat…
— Ben Spaulding (@GISDoctor) March 25, 2013
Max Uhlenhuth from SilviaTerra then gave a great overview how he started contributing to open source projects. Max had a number of great points but a couple really stuck with me. In his professional career he has only used open source software, and he was using FOSS in high school! When I was in high school my parents had just gotten this new thing called the “internet.” His best quote came about halfway through his talk:
“I wanted to contribute back to the open source community so I didn’t feel like a leech” #ignitelocationtech #foss #geo #Boston
— Ben Spaulding (@GISDoctor) March 25, 2013
Michael Evans and his colleague whose name escapes me (if you know, please let me know so I can update this post) talked about ongoing efforts in Boston’s City Hall to improve data sharing, analysis and visualization. They talked about the extremely popular blizzard reporting site that famously crashed and how it was both a success and a failure. It was a success because it was so popular, and it was a failure because it crashed, and crashed hard. They also talked about efforts to make data resources work with more efficiency within City Hall.
100000s of visitors within an hour- “it was really freaking popular…” 2013 #Boston blizzard reporting site #ignitelocationtech #geo
— Ben Spaulding (@GISDoctor) March 25, 2013
I’ve seen Jeffrey Warren from the Public Laboratory give talks a few times over the past couple years and he and his colleagues are always doing something innovative. He didn’t disappoint during his talk on Monday. He talked about the Public Laboratory’s open source spectrometer. It was pretty amazing. I wish he had another 25 minutes to go into more detail.
Another great talk from Jeffrey Warren from public laboratory – open source spectrometers #awesome #ignitelocationtech
— Ben Spaulding (@GISDoctor) March 25, 2013
Leaflet made an appearance at the event as well. Calvin Metcalf, leaflet guru and cat enthusiast, gave a talk titled –Leaflet for some cats. Using Max Ogden’s Javascript for Cats for inspiration, Calvin went through a few examples of how easy Leaflet is to use and customize. If you haven’t tried Leaflet yet you should. Calvin’s slides can be found here.
@cwmma – #leaflet for cats! It is that easy! #ignitelocationtech #opengeo #Boston twitter.com/GISDoctor/stat…
— Ben Spaulding (@GISDoctor) March 25, 2013
Christian Sparning from the Metropolitan Area Planning Council talked about the Hubway’s visualization hackathon. If you are in the Boston area you have probably seen some of the results of this event over the past several months. The Hubway folks released a whole bunch of data – ride numbers, origin/destination data, temporal data – and then held an hackathon. Christian talked about the variety of people who participated and the variety of creative ways they analyzed and visualized the data.
#HubwayMapping hackathon – terrific analysis and visualization @cspanring #ignitelocationtech #Boston @avidgeo
— Ben Spaulding (@GISDoctor) March 26, 2013
UPDATE: Thanks to Andrew Ross I got an update on the evening’s last speaker (I didn’t catch his name at first). The last talk was from Ken Walker from Eclipse, talking about the Orion Editor. I’m not too familiar with Orion, which is a browser-based tool for developing on and for the web, but it looked like something I should learn about, soon. I encourage you to check out the link to learn more. Ken was kind enough to post his slides as well.
Overall, this was a great event. Avid Geo and LocationTech did a great job putting this together and the speakers inspired those in the audience. I really think you are seeing the future of geo at events like this. Geo is no longer just for technicians working in municipal offices. Geo is moving beyond GIS, web mapping, and cartography. Geo is now everywhere and anywhere and that is a great thing. In the near future geo will be even more ubiquitous throughout the business and technology worlds and there will be a growing demand for people trained in the geospatial sciences. Events like this keep the field moving forward!
The old news on the street is that the Google Maps API v2 will be deprecated in just a few weeks in May 19th 2013. I have a few old Google mash-ups that I never upgraded to v3 of the API that are floating around this site and I will retire those pages when the time comes.
However, one of my v2 mash-ups is a nifty transparency slider that I modified to work with WMS data. I will be updating that site to v3 of the API before the last days of v2. I’ve been saying that I was going to upgrade this app to v3 for a while now and I should probably do that soon. Nothing like the deprecation of a API to get you motivated to write some code!
For more info on the v2 deprecation and a whole bunch of Google Maps news visit the Google Geo Developers Blog.
I was thinking about map projections earlier today and I wanted to reference a couple ideas against something other than what I could find through Google. I dug out my copy of the 5th edition of Elements of Cartography by Robinson, Sale, Morrison, and Muehrcke (1989) and I found my answer (when was the last time you used a book’s index?).
This book came into my possession during my undergrad years when I found it in a pile of books that a retiring geography professor was sending to the recycling bin. I also have a 1st edition copy (acquired at the same time as the 5th edition), but that’s on my bookcase at work, next to my copy of David Harvey’s Explanation in Geography and Mark Monmonier’s How to Lie with Maps.
In today’s world of point and click cartography it’s nice to know that geo-visualization has always been deeply rooted in theory and science. If you can find a copy of this book, at any edition, check it out. I believe the way we think of, and use maps, has really changed since the rise of GMaps. Reading through a classic book like lets you appreciate what truly goes into building an effective and beautiful map.
What are our geo-analysis fundamentals?
Fundamentals should be conceptually simple, so that one can learn them and understand them quickly and easily. Lately, three basic fundamentals of geo-analysis have been ringing in my head and I think anyone who works in “spatial” should really understand them. Here they are, in no particular order:
If geo-analysis is your area of expertise, you should be able to discuss all three topics with some intelligence. I know there are many more fundamentals that build the foundation of geo-analysis, but these are the three that have been thinking off lately. Do you have any other geo-analysis fundamentals that you think are crucial to know? If so, leave a comment.