Posted on

WhereCamp Boston 2011 Review

WhereCamp Boston was this past weekend and it was pretty cool.  Even with troubles with the Red Line (typical) about 50 people were there on Saturday.  I was signed-up for both days but other last minute commitments pulled me away from Sunday’s portion of the weekend conference.

There were two keynote talks on Saturday that were both very interesting.

The morning keynote went to Jeffrey Warren from the Public Laboratory.  He talked about a number of projects that they have been working on including the really amazing balloon mapping techniques that they have developed.  I’ve seen him talk a couple times and after I immediately want to create my own aerial photography!

Tom MacWright from Development Seed gave the afternoon keynote talk and discussed a number of projects including MapBox.  The tools he demonstrated were pretty sweet.  I recommend that you take five minutes and check it out.

A few take-aways from my time at Boston’s first WhereCamp:

  • There was a good variety of people.  There was the GIS crowd, the programmers, the open sourcers, and the spatially enabled.  That mix led to some great conversations in the breakout sessions and in the common areas during the breaks.
  • Speaking of open source, there was a definite open source feel to the meeting, which was completely understandable, as many of the participants came from that side of the GIS coin.
  • I need to learn more about open source GIS.
  • There were some great sessions that emerged from the ideas of the group.  I went to a really good Google Android app builder session, and I heard some good talk about some of the other sessions that occurred including those on open source mapping.
  • The Microsoft NERD facility was awesome and it has one of the best views of Boston.
  • I was really impressed with where people came from.  There were attendees from the Boston region, all over New England, the east coast, and even some from California!
  • The spontaneity was great.  People would chat in the meeting areas during the breaks about a particular idea and it would become a session.  Also, people would propose sessions with the hope that someone could come in and show them something they didn’t know.  That’s what exactly happened to me.  Someone proposed an intro to spatial SQL session and I ended up putting together an (but not that great) impromptu workshop!
  • For next year I am going to come prepared with a few pre-canned workshops so that I can contribute more to the sessions.

Finally, Guido Stein and the ISpatialBoston team did a great job planning the event.  I think next year will be even better as more participants will have ideas going into the weekend and be willing to contribute more.

Again, a big thank you to the organizers.  You did a great job!  Thank you!

Posted on

The Query Layer Query

The Problem

I am a big fan of the query layer in ArcGIS 10.  I routinely create and manipulate spatial data that lives in the geography data type in SQL Server.  The query layer tool has opened up a whole new set of analysis and visualization possibilities for a number of large data sets I work with.  Overall, I am a big fan…

But, I’ve run into a problem.  Using the query layer tool I couldn’t connect to a number of SQL Server databases.  When I would try to connect to certain databases I would get this cryptic error.

What Does this Mean?

What is going on? Is this an actual error?  I am missing some font pack therefore displaying the characters I see on screen?

With this useless error I started to trouble shoot.  I knew I had SQL Server databases that worked correctly and some that generated this error.  The only difference between the working and non-working databases was the length of the database name.  As it turns out the length of the database file name that is allowed by the query layer is limited to 31 characters.  Anything greater than 31 characters will generate this error.  This is not listed in the query layer documentation.

What the Heck!

In my job I work with dozens and dozens of databases that have a prescribed naming convention.  Sometimes, these SQL Server database names can get long, but no where near the SQL Server character limit.  Within the Esri software family there are described limits to database names.  When using ArcSDE the name of the database is limited to 31 character, which is understandable.

However, if a user adds a SQL Server database to the “Add OLE DB Connection” tool in ArcCatalog the database name can be more than 31 characters, but if the user adds a SQL Server database with a name greater than 31 characters to the query layer tool it will error out.  What a bummer.

What’s Next

I posted this question in the Esri forums and the response I got from the Esri rep was accurate, but this limit is a pain to those want to use this tool.  In the online documentation for the query layer the text states “Query  layers allow both spatial and nonspatial information stored in a DBMS to be easily integrated into GIS projects within ArcMap.”   I agree with this statement 100%, unless my database name has more than 31 characters…

By limiting the length of the database name Esri is saying that I cannot take full advantage of the tools provided by SQL Server.  Also, if I can only use 31 characters, please document it in the query layer documentation.  Either this is a bug, an undocumented limitation that needs to be addressed, or it is documented somewhere and I can’t find it (which is a real possibility).  I am thinking that this problem is probably a 32 bit issue, in fact a couple of my coworkers are thinking the same thing.  I’m using a 32 bit software in a 64 bit world!

As a geospatial database admin (GeoDBA for short), I need to make a decision that will impact the workflow for my organization.  I have business requirements that limit me from renaming these business critical databases, but I have new business requirements that are greatly benefited by the query layer functionality.  Do I rename databases (and modify the tools that access this information) or do I wait for Esri to fix the problem? Moving forward I’ll try to limit the number of characters in my databases but I’m using a SQL Server database, not an SDE database. Why is Esri limiting what I can do with that piece of software?  Unfortunately, it may be faster for me to rebuild everything than it would be to get a solution from Redlands.

I’ll submit this problem to Esri tech support soon.

 

And…I am running ArcMap 10 with SP 2 using SQL Server 2008 R2 that is fully patched and updated.  The error described has been repeated on several machines within my organization.

Until next time GIS geeks!

Posted on

GIS Doctor Mailbag

This blog has been up and running for a while now and I have gotten a number of email questions, responses to blog posts, and search engine keyword questions.  Let’s answer a few of them!

How do geographers collect spatial data?

  • By many means.  There are the arm chair geographers (like me!) who collect, develop and analyze spatial data from a variety of published data sources and there are those geographers who go out into the field.  These rugged souls create and administer surveys, collect data in the field in all conditions, and develop scientific studies to derive their own data.  Depending on the nature of the study a geographer may use a mix of data collection methodologies.

How do I use more memory with ArcGIS 10?

  • I hear this question more and more, especially as more users have 64 bit machines and automatically assume that everything will be faster.  ArcGIS 10 will use all the memory available in a 32 bit environment, which is 4 gigabytes. However, not all the tools and processes in ArcGIS will use all the memory.  To work around this problem users can chunk their analysis to try to speed things up, or they can review their analysis and try to develop a set of processes that make the analysis smaller, therefore faster.  In the past I have split up large input data sets (20 million plus points) and used python and ArcGIS tools to have the analysis run effectively.

How much ArcGIS Server 10?

  • Not cheap.  If you want free check out MapServer.

How can I get Bing Maps in my GIS?

  • If you are not behind a firewall a user can click on the Add Data button in ArcGIS and select Add Base Map.  The Bing data should then be a base map option. Since my instance of ArcMap is behind a firewall I cannot add the Bing Service. This is a bummer, since I am a big fan of Bing Maps.

How can I add Google Maps into ArcMap?

  • There isn’t a quick tool to load Google Maps into ArcMap.  However, there are a few applications out there that allow a user to port in the imagery from Google into ArcMap.  One well known tool is provided by Arc2Earth.  Now, if the user is running ArcServer Esri provides a number of tools to publish data in the Google Maps interface.

How do I kill a geoprocess ArcGIS 10?

How do I speed up ArcGIS 10 on 64 bit machine?

  • Wait until Esri releases a 64 bit version of ArcGIS, or start using GIS software that is native 64 bit.

Is it possible to use embed a WMS data source into a Google Maps mash-up?  

  • Why, yes it is.  Check some examples here…

Is ArcGIS difficult to learn?

  • I’ve been in the field of GIS for close to 10 years and I’m learning something new about ArcGIS everyday. GIS is complicated and the theory behind it is deep.  Many want GIS software to be simple, but users need to understand the details to properly frame an analysis and understand the results.  If GIS were very easy I believe you would get many people, more so than now, making poor decision from poor analysis.

Why did Esri skip version numbers?

  • They did?

What is wrong with ArcGIS 10?

  • Unfortunately, plenty.  Check out the bug fixes for SP1 and SP2.

Why do ArcGIS tools take so long?

  • Another question I hear all the time,  “why is process XYZ so slow?”.  The answer is never as direct as the person asking the question wants it to be.  There are a number of reasons why a particular process or processes may be slow.  To try to understand the issue I always work through a series of questions.  First, what are the specs of the machine running the process?  What other processes are running on the machine?  What are the dimensions of the data being processed?  Are the data sets in the analysis in the same projection? How large is the data, both spatially and in memory?  How intensive is the process itself?  How detailed is the data being processed?  The point is that there isn’t always a smoking gun to solve all processing questions.  Sometimes the process will be slow.

OK, that is it for now.  If you have any additional questions drop me a line!  Until next time.

Posted on

The GIS of Hurricane Irene

Hurricane Irene is upon us here in Boston.  Thankfully, I prepared yesterday so I can spend some time this morning blogging during hurricane (or tropical storm)!  Also, I’d like to get this blog out before we lose power, since Boston is on the windy side of the storm.

Now, geographers and GIS pros are all over this storm.  Unlike the Tohoku earthquake and tsunami, which had a ton of reactionary GIS development, hurricanes provide the opportunity to develop datasets, applications, and analysis before the storm arrives.  There are a number of great applications and datasets that have been generated in the past four days, with many more to come over the next several days.

Tracking Applications: Everyone loves online maps

New York Times hurricane tracker – Great, clean application.

MSNBC hurricane tracker – Built by Stamen.  Great information, and a great look, but no data downloads.  Open the data!

Esri – The Worldwide Leader has a pretty nice mapping application (as they should) tracking the storm. They have a number of social media links, which will be great to view for damage and impact information after the storm.

CNN – Old school, like 1998.  CNN, get with it and build a better mapping application.

Yahoo and ABC are both displaying a mash-up of the storm that appears to be developed by the AP.  The application has a number of different tabs with information related to the storm.

GIS Data Sources: What you are really here for.

NOAA -A number of technical data sets are available and have been updated throughout the storm.  Click on a map and look for the “Download GIS Data” option.  Shapefiles are available and a number of Google based mash-ups are included.  The NOAA site may not be as flashy as others but the data available is very valuable.

Weather Underground – This site has gotten a lot of press this week as the storm has approached Megalopolis.  They have a number of tools and data sets available including a slick tracking mash-up, and number off data sets which are not necessarily GIS ready, but GIS-“able”.

Esri – Yes, Esri is on this storm.  Earlier this week they published a site with a number of data links to GIS Web Services, data providers, and scientific data sources.

FEMA data sources and data feeds.  FEMA is providing data from both internal and external sources.

CrisisCommons – A number of links to data sources related to the storm.  This source will grow as the storm passes.

Maps and data are already available on Geocommons.

After the storm there will be a number of data sets that become available through the Army Geospatial Center.

Global Disaster Alert and System– Hurricane Irene information is available here.

What’s next?  As the storm passes and people are able to survey the damage you will start to see impact analysis data sets, loss estimation maps, and analysis on the storm itself including better measurements of rainfall, windspeed and the track of the storm.  Once that data starts to roll out I’ll update the page with some more links to data.

Good luck, stay safe, and stay dry northeasterners! These storms are nothing to mess with. Take them seriously!

Posted on

Back to School!

It is that time of year again GIS geeks!  Veteran grad students are awakening from their summer hibernation while new grad students are making their way to campus wondering how they are going to survive on their huge stipend.  School is back in session!

I always loved the beginning of the college school year (I did have ten of them).  I loved seeing people again that I hadn’t seen over the summer, meeting the new crop of grad students who had yet to become jaded with the system, and most of all I loved the energy of the beginning of the school year.

Now here are my unsolicitated tips to you, the GIS grad student, that will guarantee you success in the future (disclaimer, take these tips with a grain of salt and they only apply to students who are in on-campus programs.  Sorry online degree folks…).

  1. Take advantage of technical resources.  If your department has a dedicated grad student computer lab with big, bad machines with every piece of GIS and stats software know to man loaded on them, I recommend you use it.  I was fortunate to have a pretty sweet GIS grad lab where I was able to get a ton of work done.  I also saved a ton of money on not buying a big, bad machine.
  2. If you have an office, use it.  Yes, it may be hard to get any meaningful work done in an office full of other grad students, but it is those other grad students who you want to get to know.  These are the people who you will bounce ideas off of, go out and grab lunch with, and go to conferences with.  The connections you make with your fellow grad students will be the ones you will have throughout your professional career.
  3. Expand your mind and learn something new everyday. Read the abstracts from GIS related journals, check out journal articles that are in your area of research, and read GIS blogs (like mine!).  Stay on the cutting edge!
  4. Treat grad school like an awesome job.  Show up in the morning, work all day while meeting with professors and other students, mix in a two hour lunch break, and head home at the end of the day.  I really believe this is a recipe for success in grad school.  Many GIS based grad students are in grad school to develop a set of professional skills, so they should treat the experience professionally.  I’ve seen one too many students who treated grad school like an undergraduate experience. They would roll in ten minutes before their 11am class, hang around for a little bit after class, and then head back to their apartment or dorm room and take a nap or play some XBox.  Take school seriously and you’ll find success.
  5. Do extra.  Get involved with a real world internship, do some contract work, work on a paper with your advisor or a GIS savvy professor in another department, or get a part time job doing GIS work.  Your department head or advisor may not like that you are partaking in extra-curricular activities, but a degree alone will not get you a job in today’s market.  Experience counts, so grab any chance you can to get it.  But remember, you need to make sure that you keep your grades in check and that you are making progress towards your degree.
  6. Get technical. Take advantage of the resources you have.  Learn how to program, learn about databases, learn about spatial analysis, learn about visualization, and learn about spatial data and the web.  I think you get the idea. I believe there is a strong demand for very technical geographers and GIS professionals.  Use your time in grad school to learn those skills so you can get a job.
  7. Have fun.  Even though I make it sound like grad school is all about work and results it’s not.  Enjoy the campus, the town, the people and have a good time.
Posted on

Great story in the New York Times

In case you missed it there was a great story in the New York Times on Tuesday in regards to the use of GIS in historical analysis. The article, “Digital Maps are Giving Scholars the Historical Lay of the Land” , by Patricia Cohen, discusses the evolution of the spatial humanities and historical GIS/geography, which are growing disciplines in the humanities at colleges and universities around the country.

The article provides a nice overview of how historians, archaeologists, and other non-geographers have embraced spatial analysis and GIS in their research.  I think this is a great article on a trend in the humanities that has been growing for years.  I remember as an undergrad ten years ago developing GIS tools to visualize historical settings.  In grad school I routinely helped non-geographers develop spatial analysis methodologies and visualization techniques to process and analyze historical GIS data.  Much of that work ended up in scholarly publications.  The spatial component really gave the authors an edge over other papers at that time.

So, if you get a few minutes check the article out.  Anytime that GIS gets mentioned in the New York Times is great for our field!

A couple of notes from the article:

  • The author references www.gis.com.  I’m sure the marketing department at Esri liked the link.
  • The article links to David Rumsey’s site.  If you are a map junkie like myself you will love this site.  An amazing map collection.  This site has really influenced the development of online map libraries around the world.
  • I wonder if the growth of GIS and spatial analysis in the humanities (which has been happening for a number of years) has increased enrollment and/or developed programs in GIS and geography at schools where the spatial humanities are strong.  The AAG should get on this.
  • Does anyone remember the digital landscape history of Manhattan that was put together a couple of years ago?  The project gained some press and buzz when it came out.  I’m surprised this article didn’t mention it.  Oh well…
Posted on

It’s MAGIC!

If you are a New England based geospatial professional you have probably visited or downloaded data from the Map and Geographic Information Center (MAGIC) at the University of Connecticut’s Homer Babbidge Library.   MAGIC just recently announced their plans for the summer and I am looking forward to seeing what data and tools they roll out next.  MAGIC was one of the first map libraries to offer GIS data over the web and since it has always been a place of innovation.

I was fortunate to have worked at MAGIC from the fall of 2004 through the spring of 2009.  As a grad student I was able to  work on a number of really interesting projects that expanded my technical skills more than any class ever could (I was one of the last people at MAGIC to program in Perl:( ).  Many GIS professionals in New England and beyond can trace their roots back to MAGIC, whether it was working for Pat, Bill, Sarah, or Michael.

So, if you have downloaded there data, copied their mash-up code, or ever browsed their collection of historical maps on Flickr, send them an email and tell them they are doing a heck of a job!

 

 

Posted on

The death of the GIS guy?

Late last week Esri released their responses to user questions collected from the pre-conference survey.  James Fee has a good response to a number of the questions that Esri posted.  I particularly like his response to FGDC metadata support…

One question from the Esri User Conference Q&A caught my particular attention and it is something that I have been thinking about for a while now.  The question, “Do you see Esri software becoming so easy to use that professional GIS people are edged out?”  The folks at Esri answered that the GIS professional, a.k.a. the GIS guy, will be more important than ever because they will be the ones who connect a variety users to spatial data, applications, and analysis.  I agree with that point, and I understand Esri’s goal of making their software and the concept of GIS as appealing to the masses as possible.  Think about it, would you like to have a potential customer base of a couple hundred thousand, or a few million…

I believe the GIS guy isn’t going anywhere, in fact, if there is a greater demand for the use of GIS then there should be a greater need for GIS experts.  However, to remain relevant in the workplace the GIS guy will need to become a well rounded technical GIS expert.

The profession is changing. No longer is the GIS guy just downloading data sets from a few GIS data warehouses, creating some metadata, doing some basic analysis, and creating a few nice maps for display.  With a variety of web-based mapping and analysis sites and collaborative data analysis and collection sites anybody can collect, visualize, and share spatial data.  And they do.  From Google Maps to GeoCommons a growing number of individuals and organizations who are not trained in the classical arts of GIS are using tools and techniques that at one point were exclusively used by the GIS guy.

How does the GIS guy stay relevant in the office and not be replaced by someone who can develop a slick web-map using a data feed from Twitter?  The GIS guy needs to become a technical expert in a number GIS related of fields they probably never had any training on in college (that’s another blog post).  They need to demonstrate their value by having a solid set of technical and analytical skills as well as a flair data visualization.

The following is a list of skills that I believe that the GIS guy will need to not only be productive, but to stay relevant.  I’ve thought about this list for a while.  I base this list on my own experiences in searching for jobs in the past, being on GIS analyst hiring committees, reading GIS blogs, talking with friends and colleagues, and viewing GIS job posting.  Here it is, in no particular order:

  • The GIS guy needs to be well versed in relational databases.  Data sets are getting larger and larger and the well worn methods of GIS data storage won’t cut it when you have millions of points, lines or polygons to analyze.  Whether it is SQL Server, Oracle, PostGres, or SpatialLite, the GIS guy needs to understand the value of the relational database within GIS.  This is especially true as relational databases become have increasing spatial capabilities and are easily connectable to both GIS and web-based tools.
  • The GIS guy needs to be able to program – Python, C#, C++, SQL, pick your poison…  No matter your daily workflow or GIS software you probably have a number of processes that either don’t exist as an out-of-the-box tool, or would greatly benefit from automation.  Every, and I mean every, GIS professional has to be able to program.  No excuses.
  • The GIS guy needs to know geospatial analysis. I know this sounds a little silly at first but I have met a number of GISPs who couldn’t properly tell me the difference between a union and an intersect, what the Moran’s I measures, or how an IDW works.  As the need for “geospatial” increases so will the need for advanced analysis. This is where the GIS guy can be of real value.  GIS guys need to have training in the art of geospatial analysis and its applications in GIS.
  • The GIS guy needs to know how to integrate web-based technologies into their GIS technology.  The GIS guy has to be able to show value in their work and one of the quickest ways to do this is to share those valuable data sets that they have developed through web-based technologies.  Whether it is a mash-up through Bing, Esri, Google, or OpenLayers the GIS guy needs to understand the benefits and challenges of developing tools in this framework.
  • The GIS guy needs to be a GIS expert. This is related to a prior bullet.  The GIS guy needs to understand raster and vector GIS analysis, proper data editing procedures, the differences between data formats, how projections impact analysis (do you know what a datum is?), the MAUP, and so much more.  Why does the GIS guy need to know all of this?  Well, if the GIS guy is working with geo-enabled individuals who may not be trained in the arts of GIS they have to be able to provide support for any question that may arise.  If the GIS guy can’t or won’t provide this support then the their value and relevance will rapidly deteriorate.
  • The GIS guy needs to continue to learn.  I’ve heard “I didn’t have to do it in my job so I never learned it” one to many times from GIS guys.  If you work with technology you need an evolving set of technical skills.  The skill set I had five years ago isn’t the skill set I have today and the skill set I’ll have five years from now won’t be what I have today.  Be proactive in your learning!
  • The GIS guy needs to know that there is more to GIS than Esri. Go download Quantum, read up on ERDAS, learn about Cadcorp.  The GIS industry is bigger than you think.
  • And finally, the GIS guy needs to know how to make a web-map using a data feed from twitter…Know and understand the trends in the field and be able to communicate in the lingo of what is “next”.

By no means is this list complete, but if you are a GIS professional or an aspiring GIS professional this list might help you get ahead or stay ahead.  Am I calling some GIS professionals out?  Sure, but I do it because I don’t want to see a GIS professional lose out on a job to someone who has more technical training, but not geospatial skills.  The set of skills I just described are attainable, and you don’t need to get a GIS certificate or have to go back to school to get them.

Do you think I’m crazy?  Let me know.   I’d be glad to discuss the list.

 

Posted on

Spring NEArc, Tuesday May 17th!

The annual spring NEArc meeting is tomorrow, May 17th at Smith College in Northampton, Massachusetts.  This is a great one day conference, bringing a variety of talks from education, public, and private sector GIS users.  I’ve spoken at a couple spring NEArc conferences and I have always enjoyed the people who make it out to western Massachusetts for the meeting.  The talks usually represent a good mix of GIS applications, techniques, and theory. So, if you are in the area, check it out.  Onsite registration is will be available on the day of the conference.

The schedule is now online. When you are there make sure to check out “Comparing Neighborhood Change in Connecticut 1934 to Present using Google Maps API” from my friends at MAGIC!

 

Posted on

Upcoming GIS Conference – Harvard Center for Geographic Analysis

I will be attending Harvard’s Center for Geographic Analysis annual conference as an interested spectator this Friday and Saturday, May 6th and 7th.  This year’s conference is focused on the future of web-mapping and geo-collaboration.  Reading through the program it looks like there will be a number of very interesting talks and demonstrations.

From the program:

“This two-day conference will bring web-mapping experts from across the country to Harvard to share their knowledge and experience, and to envision what lies ahead. Speakers will introduce ideas, systems, tools, and visions, and present case studies and discuss challenges. Some will offer live demos and hands-on training. This conference provides a forum for geospatial technologists, developers, academics and end users to engage in dialog and help shape the future of geospatial technology on the Internet.”

source

I am interested in seeing what the world of online mapping is to people who aren’t like me, a trained GIS geek.  Because, let’s face it; many new and innovative web-mapping tools don’t come from geographers, but people who have a particular problem and a new way to solve it.  I am hoping to tap into the web-mapping mojo that will be in abundance at the conference and perhaps incorporate some of the ideas into my current projects.

It looks like registration is still open for the conference. If you are in the Boston area on Friday and Saturday and want to be on the cutting edge of web-mapping and geo-collaboration this could be a great conference for you!