Category: GIS

Posted on

Baaaaad User!

I’m in the process of installing ArcGIS Server 9.3 and moving all our GIS data over to ArcSDE.  It’s been 6 months since the last ESRI UC so most of the time I’m reading over docs trying to remember all that I learned.

 

Anyway, our data is now in SDE.

 

I attempted to publish a locator service to the web.  Here’s what I got:

clip_image002

Baaaaad login user!  No, no!

I’m getting scolded by GIS!

Posted on

ESRI 2008 UC: SQL Server 2008 and ArcGIS

I attended the Intro and Advanced sessions of Administration for SQL Server.  The intro was a pretty straight forward topic of installation and configuration for setting up ArcSDE with SQL Server.

The interesting points were how ArcSDE integrates with SQL Server 2008.  As I mentioned in my talk on SQL Server 2008 at the IE .Net User Group storing spatial data in a SQL database is not a new thing, just new to SQL Server.  ArcGIS already has implementations to take advantage of server level spatial handling in Oracle, DB2 and possibly others.  Only in 9.3 are SQL Server 2008 and PosgresSQL now supported.d

Since SQL Server 2008 hasn’t been officially released yet (all info still points to Q3 2008, which leaves less than 2 more months) ESRI is working with Microsoft on the latest release candidate version.  While you can play with it ESRI of course highly discourages using ArcSDE against SQL 2008 until the official release.  At that time ESRI will certify ArcSDE against SQL 2008 and will most likely release SP1 to deal with any changes since the 9.3 release.

That being said, what exactly is ArcSDE offloading to SQL Server?

Remember that the ArcGIS geodatabase is a very proprietary data model, allowing you to store not only geographic items (features) but also tables, topology rules, data access rules, locking schemes, business rules, etc.  SQL Server 2008 is still at heart simply a data storage platform.  This means that the bulk of the management of your GIS data will still be handled by ArcSDE. 

SQL Server now includes spatial features including two new data types (GEOGRAPHY and GEOMETRY) as well as a whole host of functions to manipulate these data types with.  The data is indexed using new Spatial Indexes on the server.  These are the things that ArcSDE will take advantage of.

ArcSDE will store geographic data in the new GEOGRAPHY and GEOMETRY data types.  This allows SQL to manage it’s own spatial indexes on the data.  Doing this also allows ArcSDE to take advantage of the built-in spatial functions for simple requests.

ArcSDE will take advantage of the indexing schemes built in to SQL Server 2008, however, all the settings are left to Microsoft defaults.  ESRI claims this is because every user’s data is different.  While this is true, I got the impression that it’s more because the indexing scheme is so new that everyone is still coming to terms on it’s quality and how to actually manually maintain it.

Now that you can store your data in the underlying database, several questions come to mind:

  • Can I simply enter my own geographic data in SQL Server and use these in ArcGIS?
    • Yes you can, however, you must register it with ArcSDE before you can integrate it with other ArcSDE data.  If you never plan to process this data using ArcGIS tools then you never have to register it.  However, that doesn’t seem like a likely scenario since you imported the data into your geodatabase.
      To register the data use the sdelayer tool.  This sets up all the geodatabase data and relationships so that the features can be managed using standard ArcGIS tools.
  • Can I manipulate ArcGIS data directly in SQL?
    • Yes!  You can hit the data directly using the built-in spatial methods.  This is pretty awesome for automated tasks that might fire at scheduled moments or for triggers. 
      • For instance, at our school district our Student Information System (SIS) has a field for each student storing what their school of residence is.  Since students may request to attend another school in our district, often it is important to know what school they normally would have gone to.  Currently we autopopulate this data do this once a week or so.  This is a manual process by using ArcGIS Desktop to simply join our students against our school polygons, output the results to a table and import these values back into the SIS.  Once our GIS data is in ArcSDE on SQL 2008 I will be able to setup a trigger on the student record causing it to use the ST_INTERSECT method to find this data on the fly.  Sweet! 🙂
    • However, (you knew there had to be a however right?) you MUST BE VERY CAREFUL.  As with manipulating most SQL data you must know exactly what you’re doing.  If you are inserting new objects you must find what the next ObjectID should be.  If you are working with versioned features it’s more complicated as you have to work with versioned views within a versioning session.  SQL 2008 has no concept of these so ArcSDE installed several utility stored procedures into SQL Server that you will use to manage this process.
  • What about the SRID?
    • If you are new to SQL Server 2008 you may have noticed this weird SRID value that is attached to every spatial value.  This is the equivalent of the Spatial Reference used in ArcGIS such as NAD State Plain 1984 or WGS.  However, the SQL Server SRID corresponds to the EPSD code (European Petroleum Survey Group).  This is a table of spatial references all attached to a unique ID. 
      This is required when registering your imported data with ArcGIS using the sdelayer command.
      What’s different is ArcGIS’s SRID’s do not correspond to the EPSD code.  So what do you do?  ArcGIS stores it’s SRID as auth_SRID.  Here is an example of how to join the ArcGIS srid_auth against the EPSD reference table to find out which SRID to map to:
      select sr.auth_srid from SDE_layers l join SDE_spatial_references sr on l.srid = sr.srid where l.table_name = ‘buildings’ and l.owner = ‘dbo’
      This command joins the SDE_layers table against the SDE_spatial_references table and looks for the layer named buildings owned by dbo.

All in all some really great stuff!  I can’t wait to play with this.

I guess my one complaint is that while you can request an ArcGIS Desktop eval cd from the ESRI website I don’t know that you can get some type of an eval version of ArcGIS Server.

Technorati Tags: ,
Posted on

SQL 2008 RTM’d!!!

I’m in the SQL Server SIG lunch at the ESRI 2008 UC.

Ed Katibah (Spatial Program Manager at Microsoft) just announced that earlier this morning he received an email that SQL Server 2008 has officially gone RTM (Release to Manufacturing).

This is great.  The last few months it’s been stated that SQL Server 2008 would hit the market Q3 of 2008.  I guess this means they are officially on track!

Wahoo!

Posted on

ESRI 2008 UC: ArcGIS Server – The Developers Perspective

All I can say is WOW!  I should have gone to this one instead of the Web Application Interface Design: The Design Process.  This is what I was looking for.

Unfortunately I missed the first 20 minutes but they showed some incredible access to GIS data and tools using REST and SOAP.

lf you are web app developer and deal with a lot of data definitely check out REST.  It’s a really neat strategy for manipulating data.  With .Net 3.5 Microsoft is building it into their data access model.  From my buddy James’ presentation on it it’s still a little immature, but it’s getting there.

Anyway, when you publish a map or other resource to ArcGIS Server it is instantly available via REST and SOAP.

So, just think.  If you need to get access to layer files or perform some sort of processing you can just use a standard url to grab it.

That’s about all I can say on REST because I only saw a bit of the last demo.  Then they switched over to SOAP.

ESRI has a longer investment in SOAP so, while it’s a slightly older technology, it’s a little more mature in its offerings.  ArcGIS Server exposes a few more abilities in their SOAP architecture, and with the built-in web resource features in Visual Studio it’s very easy to add to a web app and use.

I haven’t jumped into SOAP really so I can’t comment that much on it.  I’m jumping on the bandwagon only now and I’ve already dived into the JSON and REST side of things.  I’ll probably delve into SOAP if I can’t get ArcGIS Server’s JavaScript API to do what I want.

I asked the presenters about  security.  They said if it’s an Intranet based app then we can take full advantage of Active Directory.  He seemed to indicate it was built in to the Server and not really a function of the REST or SOAP interface at all.  Hopefully I’ll find out more about this in my ArcGIS Server sessions.

Anyway, as far as public access REST and SOAP can take advantage of token based accessibility.  It sounds like I should be able to roll that in with Microsoft’s ASP.Net MembershipSecurity framwork.

I don’t really relish opening ArcGIS Server directly to the public.  One of the demos used an ImageHandler.ashx handler that retrieved the image from the Image Server and streamed the result to the user.  This way the web app acted as a middle tier between the user and the server, thus allowing you to keep the server internal and manage access to the outside world.

Can’t wait to play with it!

Technorati Tags: ,,

Posted on

ESRI 2008 UC: Web Application Interface Design: The Design Process

Sorry, but this unfortunately was a real waste of time.  I only stayed 20 minutes and that’s because I was answering work email during the presentation.

The presenter had just a few slides each with a single word such as Research, Development, Production and he explained what he does in these phases.  It was the standard fair such as user interviews during the research phase, testing on equipment that models the user’s environment during development, etc.  There were no easy bullet points and there wasn’t any new knowledge at this point.  I didn’t see anything to do with the UI, but then again to be fair, I left early.  There really wasn’t any content of note unless you were an absolute novice in developing a webapp project for a client.

It may have gotten better after I left but I ended up jumping into the Working with GIS Services – The Developer’s View workshop.  Boy this is the one I should have been in!

Posted on

ESRI 2008 UC: Advanced Geocoding

There are a few new features in geocoding in 9.3 but there are also several other fundamental tools I didn’t know about.  So I’ll just go over a few of the options that are pretty great.

I came into our GIS system 2 and a half years ago and it was handed to me the same way it was handed to the last guy.  I got the software, tools and shown the ESRI site with tutorials.  What this means is there are a lot of fundamental practices and features that I simply don’t know about.  It’s sometimes hard to find a better way to work in GIS because you’ve invested time in learning the current process and there is so much data online it’s hard to pinpoint exactly what works for you.

Here are a few features that I will definitely take advantage of:

  • Composite Locators
    Composite Locators are simply an ordered list of locators.  For instance, we purchased parcel data from Riverside County.  We also have a street layer.  I can create a locator for each and then add both of these to a Composite Locator.  Composite locators allow me to tell ArcGIS to match against the parcel locator first and if a good enough match is not found then match against the street layer.
    When viewing the geocoded data a new field is added showing which locator was ultimately used to match against.  This is a good place to look if you start getting a lot of false positives.  If your match score threshold is too low for the parcel locator ArcGIS might be grabbing a “close enough” parcel when it could have gotten an exact match against the street layer.
    When using Reverse Geocoding the data in the tool tip is based on the current locator being used.

    • If you are using a street locator then as you move your mouse you will see the address numbers change based on the distance along the street.
    • If you are using a parcel locator there is no range.  You will see the address numbers jump from one parcel to the next as you move your mouse.
    • If you are using a composite locator the tooltip will show which locator it is using to get the data.  This is pretty sweet

Composite Locators will also come into play in point feature classes for geocoding below.

  • Matching Addresses Interactively
    When you have some addresses that couldn’t be matched (most likely due to incorrect address data or incomplete reference data) you have the option to match these interactively.  During this process ArcMap will show a dialog that lets you manually process each address individually.
    Ideally you would correct your source data (the input addresses) or update your reference data (the parcel or street data).  This way you could automate your geocoding process.  Manually matching addresses can be a time consuming process, and most likely you will have to do it each time you re-geocode your addresses.  Read on to find out how to permanently save your manually matched addresses.
    However, the interactive matching tool is great for pinpointing why an address cannot be located.
    ArcGIS 9.3 offers two new ways to match addresses.  You can now reverse geocode a point, effectively telling ArcGIS, “I know where the address lies on that street.”  This associates the address with that physical location along the street. This is Picked by Address (PA is the MatchType in the underlying data).
    Conversely say you know where on the map the address is located, but it’s not necessarily along a specific street (such as a new development and you do not have streets for the area yet).  You can specify a point on the map and this is known as Picked by Point (PP).
    So the big question, now that I’ve told ArcGIS where these addresses are, how do I make it remember this on the next geocoding?
    You can save these points into a feature class.  Then create a Locator based on this feature class (a point locator).  Then add this locator to your composite locator as mentioned above.  That way ArcGIS will always find exactly where these “manually matched” addresses are located.

Unfortunately there were no demos on actually creating a locator based on a polygon.  I’ll have to play around with this but it sounded like it would be fairly easy to implement.

I’ll try and find some links to the MIT group that demoed during the plenary session.  This was outrageous.  The used 3D Analyst and literally created 3D objects for every single room on the MIT campus.  They then built a locator around these layers and allowed the user to locate a building and room as simply as they would enter an address.  Very nice!  The demo went way beyond that but that’s the extent for using locators.  I would love to develop a 3D interactive map for helping users (and possibly EMS services) to find exact locations on our campuses such as the office, a specific room, the library, etc. However I don’t have a group of students working on their senior project mapping our campus.  🙂  That’s an idea though!

Technorati Tags: ,,

Posted on

ESRI 2008 UC: Plenary Session

The main presentation by Jack Dangermond was pretty good, as was last year’s.  It’s a good mix of what is new in ArcGIS 9.3, what users are doing in the field, what’s coming on the horizon, and an overall impact of GIS in the world. 

From a technical point of view here are just a few of the new features in 9.3.  There are way too many to list so take a look at the ESRI site if you want a comprehensive list.

  • Reverse Geocoding
    This has been a long time coming and a hot item on the request.  This wouldn’t warrant too much discussion, except that the implementation was really well done.  Typically (from my point of view) ESRI is a functional program, but  wouldn’t win any UI or productivity awards for most of it’s features.  The buttons or dialogs for typical tasks aren’t always where you expect them and sometimes you have to drill down through 10 screens just to get to your data. 
    However, with the reverse geocoding, it was a very easy to use tool with a crosshair for your cursor and a small dot that would snap to your street network nearest to your cursor.  Clicking would very quickly give you a geocoded address at that point.  If you clicked on an intersection the geocoding would be the intersection of those streets.
    These reverse geocoded addresses could be saved as pushpins which can later be saved as features I believe.  Very nice!
    • Oh, I just stepped into the Geocoding Intro technical workshop.  Now when you run a geocoding process, if there are any addresses that could not be geocoded, in addition to the standard options to fix this you can now use Pick Location.  This allows you to use the Reverse Geocoding tool to pinpoint exactly where on the map this address should be.  This is great as it might be difficult, if impossible, to change the address in the source data.
  • KML Export
    As the rest of the world jumps online ESRI, arguably the largest provider in the GIS arena, has been a little behind the game.  In the past I have had to resort to 3rd party tools to export our map data to KML.  Invariably this also requires lots of massaging of the data afterwards before it is ready to be published on an online mapping service such as Google. 
    You can see an example of our school district layers pushed to Google through KML here.
    Now ArcGIS will have native KML export built in.  When used in conjunction with ArcGIS Server and other tools this will make offering your GIS data to online mapping systems a very easy process, that will free up maintenance and always hit live data.
  • PDF Support
    For a while now you’ve been able to export GIS maps as PDF.  This is a great feature as ArcGIS Desktop will also export the text as well which is completely searchable and selectable using Acrobat Reader.  I use this all the time when exporting maps of our district.  It’s amazing when I have several hundred streets on a map, go to the Acrobat Reader search box, type in a street name and find it in an instant on a map.  This is really useful when other users download our maps and want to find where they live.  We have an online School Locator tool, however, having a map on your local machine is a great tool for use in offline scenarios. 
    However, other than this ability the PDF version of the map has still been fairly static.  ESRI has been working with Adobe to really exploit the abilities of Reader.  Now you can export a wealth of data to PDF.  This includes data frames, layers and feature attributes.  In the PDF hierarchy you can now see the individual data frames and layers.  When clicking on a feature you can get all the underlying data for that feature.  This is just like using the Info tool in ArcMap.  Also, the data can be georeferenced.  This allows a user to get X,Y coordinates from any area of the map.  There is no geocoding yet, but tis is all pretty neat. 
    This is pretty amazing because now you can get an incredible amount of information just from an offline PDF.  This is not only useful for Internet connected machines.  As more and more users are using mobile devices that may not have direct connection to an online GIS service, having a PDF they can use with this info will be a great step forward short of building an offline app.
  • Virtual Earth Integration
    They went through this area pretty fast so I didn’t get all the details.  It seems that you can pull VE services and resources directly into ArcGIS Desktop now and use in your own maps.  This means that you have full access to the imagery and data.  This is all on demand, which means that you cannot store the resources for your own editing or offline use.  However, this also means that you will always have the latest data.  When you open a map it will retrieve the latest images, including any new ones Microsoft may have published, directly in your maps.  This can offer a wealth of data if you have out of date or no imagery/data for your map content.
    I assume that Google and other map services will be accessible as well, it’s just that ESRI kept touting it’s partnership with Microsoft so I’m a little hesitant to say this.
  • JavaScript API
    This has been a sore point with ArcGIS in the past few years.  As I said above, ESRI has really been playing catchup.  Most of ESRI’s online mapping products have been pretty bad.  The UI design wasn’t great and it was terribly slow.
    I don’t know what the current tools are like (and usually ERSI demos are always running in a perfect world) but ESRI is starting to allow more options to connecting with data.  One of these is the JavaScript API.
    This API, on the surface, seems pretty similar to Google or Microsoft, where you specify a JavaScript file, the resource data and a div to place the contents into.
    When you publish a map to the ArcGIS Server there are now several default options to consume the data.  When you go to the url ArcGIS Server now allows you to open the map in ArcMap, view in the internal viewer, and view using the JavaScript API among others (KML export possibly but not sure).  If you choose the JavaScript API option a new page is opened with a standard web 2.0 map using the ESRI API.  If you view the source you can see that there are only about 10 lines of code that actually retrieve and display the content.  If you simply copy this text you can paste this into your  own apps and very easily add your interactive map resource to your pages.  Pretty nice indeed!
    I have to laugh here because the ESRI rep demoing this function turned a static (and very bad looking) jpeg of a campus map into a fully GIS capable interactive map in about 1 minute.  The crowd cheered.  :)  As any HTML/JavaScript developer might know there are a lot of underlying things being assumed, the first gotcha being make sure your div is properly named either in your DOM or in the JavaScript code referencing the ESRI map resource.  This is of little worry for developers who understand what’s going on, but I know there will be a few business users going back to their organizations saying "Do this, it only takes 1 minute!" and their non-web savvy GIS engineer will be spending a day on it.
    Eh, maybe I’m just pessimistic but you  can see the marketing "woohoo!" all over these demos.  ESRI always operates their demos in the perfect world.  But so does everyone else (i.e. Microsoft). 🙂
  • Mashups
    OK, if you are a web developer and haven’t been in a coma for the past few years, you should know what a mashup is.  In a nutshell a mashup is simply a web page that takes data from one source (i.e. Flickr photos), combine it with another source (i.e. Google Maps) and display the results (i.e. showing where the photos were taken on a map).
    John Grayson from ESRI’s Applications Prototype Laboratory created a great tutorial with 7 different examples of creating mashups using ESRI data and JavaScript API’s.  Each one increases in it’s level of capability and complexity.  Unfortunately all the examples were based on retrievinganalyzing data and not on editing actual data for updating on the server.
    I can’t seem to find these slides or any information on John’s presentation anywhere so hopefully he will publish these soon.  Otherwise in my spare time maybe I can throw a few together.  (Yeah, when do I have spare time!  I stayed up to almost 4am last night!)

Overall it was a great session.

I’ll be adding more posts throughout the conference on anything I see that’s noteworthy.  Those will hopefully be a much shorter read!

Posted on

ESRI 2008 UC: Start up

Well, I’m here at the ESRI User Conference in San Diego, CA.

I’m the GIS admin at Val Verde USD and this is the 2nd time I’ve been here.  I have to say, that this is one of the best conferences I’ve been to.  Not only in content, but the actual logistics of the entire event.  The San Diego Convention Center is a great place, very easy to get around, clean, with lots of nearby restaurants of any food type.  Just the shear amount of people in attendance is a logistical nightmare, but everything is always very orderly and well handled.  Definitely a pleasure to attend.

Anyway, I’ll be writing a few posts here and there about what I think is noteworthy in the conference.

I’m a developer at heart and GIS is just an incredible extension of visualizing data.  That being said, I’m not really the green bio-loving attendee that GIS usually caters too. 🙂  I love the geeky stuff, so I’ll be mainly focusing on the new 9.3 server, SQL 2008 integration and automated development using .Net.  If you want to find out how to map the migration of the Blue Morpho Butterfly or track the degradation of the rainforests over the past 50 years then this may not be the blog for you. 🙂  however, if you’re interested in .Net and GIS then stop by every so often and see if I have anything new for you.

So far I’m in the Plenary session waiting for the main event to start.  The music started out as a nice techno energetic beat, however, now it’s getting a little aboriginal.  That’s not a bad thing, it’s just a little over played.  OK, now we’re into more Last of Mohican’s style, which is definitely a great sound track if you haven’t heard it.  OK, I guess that’s potentially aboriginal as well, but it’s definitely more grounded then the wooden flute stuff that was going on earlier.

OK, I’ll stop.  🙂  I’ll post again when something of actual content comes up.

Take Care!

Posted on

ESRI’s ArcGIS still in the 80’s

This is an argument I’ve had ever since using the ESRI products almost 3 years ago.

ESRI by far is the leader in GIS software. They literally helped to make the industry what it is today.

Unfortunately a lot of the code base still is back in the stone-age (technically speaking).

The ESRI products are your typical “kitchen sink” set of applications. They do anything and everything for everyone. The software we use at our school district is the exact same software used by small business, big business, emergency services, utility services and military. The only differentiation is our data and plug-ins.

That’s nice in one sense. I really like having the power of the “big guys” at my finger tips when I want to use it. However, that’s a problem for customer service and developer support. ESRI makes money by developing the new hot features requested by their customers. When Google Maps came on the scene ESRI had to play catch-up really quick to offer an AJAX ready online product. They still haven’t met the mark but they’re trying.

Anyway, the problem is that no one (AFAIK) is maintaining the old code base. This means that the old bugs and usability issues are going untouched. Unless there is a bug or UI problem that is really significant it is put on the “do later” pile.

This is most evident in performance. We are getting ready to install ArcGIS 9.3 on a new set of servers when it is released in the next month or so. So at the last User Conference in San Diego I asked some of the ESRI techs what our server specs should be. They replied “lots and lots of RAM and a high-end CPU”. Notice that CPU was singular. The ESRI code base has been untouched other than bug fixes for years. Even when they moved to higher precision data storage in 9.2 they didn’t go back and update any of the original foundation code. The tech confirmed that ArcGIS products take no advantage of multiple cores or multiple CPUs at all. Ugh!

There is some performance gains if you use a SQL backend for your data or through your web server since these do have multiprocessor support. But as for actual GIS processing, no such luck.

This last comment is helping to fuel our interest in SQL 2008. If you know anything about me, then you probably know I’ve been playing with the new spatial features of SQL 2008 for several months now. ESRI is going to support SQL 2008 when it launches. Hopefully this means that ESRI will be pushing a lot of the processing work back onto the native SQL platform rather than on my desktop.

We’ll see and I’ll keep you updated as we progress.

If you’re going to the ESRI User Conference this year drop me an email and we can meet up.