There are a million blog posts on this sort of thing but it just bit us so I thought I’d add mine to the mix.
Date calculations can be notoriously difficult depending on the language you are using. However, DO NOT succumb to the desire to "fake" it as our vendor did. Here’s what happened:
In California every school has to submit data for their special ed students/programs to receive proper funding. This is called a CASEMIS report, and reporting day is every June 30th and December 1st.
For CASEMIS if a child is under 6 they are in one category and if 6 and over in another.
In our data there is one student in question who’s date of birth is 12/3/2002. So on the reporting date of 12/1/2008 this student will be just two days shy of 6 years old.
We were getting an error in our validation process because apparently our vendor was calculating ages using 1 year = 365 days. 12/1/2008 – 12/3/2002 yields 2190 days, which at 365 days a year results in 6 years exactly.
Ugh! Sorry, but the "close enough for government work" doesn’t apply here.
I don’t know exactly what language they are using but I believe they use perl on the back end and Business Objects InfoView XI for the reporting. The actual date calculation could be in either of these two places, or somewhere entirely different.
While some languages make calculating proper dates difficult, there is almost always a correct way to do it.
For instance, in Microsoft T-SQL, asking the difference in years between 12/1/2008 and 12/3/2002 using: SELECT DateDiff(yy, ’12/3/2002′, ’12/1/2008′) incorrectly returns 6 years, since 2008 – 2002 = 6.
However, if you change the question slightly and ask if the birthdate + 6 years is less than the reporting date using: SELECT CASE WHEN DateAdd(yy, 6, ’12/3/2002′) < ’12/1/2008′ THEN ‘6 or Over’ ELSE ‘Under 6’ END this results in the correct response “Under 6”.
It’s a matter of finding the proper question and syntax for the language.
I have a pretty decent project at my work that I am using ASP.Net MVC for. It is a great framework and it’s my first real development project in a long time so I’m pretty excited.
I’m really stealing myself to use TDD, with very heavy emphasis on the first D, “Driven”. I’m not allowing myself to write a line of code without a test first. It’s a little hard because IntelliSense is pretty meaningless in tests if the actual objects haven’t even been built. When things get a little muddy I go and create a quick empty controller or something before I finish the actual test. This is just so that I can let VS 2008 and ReSharper work their magic adding namespaces and proper syntax.
Anyway, before I get into the nitty gritty I wrote this post because I was on a search of how to unit test my custom HtmlHelpers. I found some decent posts and looked at the source for MvcContrib, but I didn’t need anything fancy. I’m just creating a file input tag. I don’t need to mock the Context or anything. I didn’t find a easy post anywhere on how to do this. It turns out if you don’t need anything special it’s quite easy indeed. If you’re looking for this then read on.
Anyway, I have a form where the user will upload a couple of files. Using Scott Hanselman’s great blog post on unit testing uploading files using MVC I have a controller that now requires two non-empty files before it returns a success view.
I’ll flesh that out a little more with some actual validation of the upload contents but for now I’m refactoring my initial form.
From Scott’s post what I began with is the following:
This doesn’t make any use of HtmlHelpers so I wanted to refactor this. There is no FileInput HtmlHelper but you could easily use the Textbox extension as in the following:
<%= Html.TextBox("tableA", null, new { enctype = "multipart/form-data" }) %>
And this would product the following correct HTML:
There are two things about this that bug me however. First and foremost, did I save any effort? The HtmlHelper code is actually longer then the straight HTML itself! In my opinion HtmlHelpers should help in some way, not just be used because they are there. So, if it doesn’t save me any extra work or protect me from something such as malformed urls or localization formatting, then there’s no point. Second is that blank value attribute. Why do I need that?
So, being the developer that I am, I thought I’d create my own HtmlHelper for a File Input textbox. Using TDD this requires that I write my test first:
namespace Tests.CECWA.Web.Helpers{
[TestFixture]
public class FileInputExtensionsTests
{
[Test]
public void CanGetCorrectOutput()
{
string html = FileInputExtensions.FileInput(null, "testName");
const string expected = "";
Assert.That(html, Is.EqualTo(expected));
}
}
}
I use NUnit but it’s pretty similar in any testing framework. FileInputExtensions is my extension class containing my custom HtmlHelper FileInput. The null parameter is the HtmlHelper class that is actually extended. Since I’m not making use of the Context or anything else I can simply pass a null in. If I were doing something fancy such as needing the base path of the app then I’d have to mock this out like the other more extensive posts do.
This test simply calls my extension method and stores the result in the html variable. I store what I expect in my expected variable. Then I assert that they are equal. Quite easy actually.
When I build my project it of course doesn’t compile since there is no FileInputExtensions class. So I do the bare minimum to compile:
namespace CECWA.Web.Helpers
{
public static class FileInputExtensions
{
public static string FileInput(this HtmlHelper helper, string name)
{
throw new NotImplementedException();
}
}
}
Everything now compiles but my test fails. So, lets complete the method:
OK, if you compare the pure HTML version and this one you don’t see much difference in the “effort”. However, this is more protected against improper paths in my form action url, typos in my various field ID’s, and just general lousy HTML coding in general.
Multiply this over several dozen pages and you start to feel a real tight management on your html output and maintainability. For instance, if I want to change the way my FileInput extension renders I simply update my test, update my class to pass and now every place a FileInput helper is used has been updated through out the site. Nice!
If you know anything about Microsoft Surface than this will probably peek your interest. Things like this may just be eye candy for now, but you can see in just a matter of years (or less) the way we interact with computers in our daily lives will be drastically different.
I don’t mean the way we all normally use computers, but how we do everything else. Look at the way people are using phones now. They play games, text around the world, get driving directions, take videos and pictures, listen to music and a whole host of other things we never though of 5 years ago. If you went back to just the year 2000 and told everyone what phones would be like today most would never believe you.
That’s where Surface and technologies like it are taking us. When you’re out and about you’ll be interacting with display technology like this every where. Whether it’s ordering off a menu at a restaurant, getting plane tickets, adjusting your hotel amenities and upgrading your rental car all during your layover at the airport, finding where stores are and what sales they have at a mall display, etc information will be much richer and more interactive than we have now.
OK, get the point. 🙂
SecondLight is a technology where a second image is literally projected through the first. It isn’t visible until a translucent item is placed in front of it. This could be something as simple as a sheet of tracing paper.
that isn’t so special in itself, this easily could have been simulated with the older Surface technology. It’s the fact that this second image is projectedthat really beefs things up. Suddenly displays are becoming more "3D". Not in the traditional sense but you can hold the paper above the surface for an easier view. They showed some plexiglass "disks" that were molded with a prism inside. This allowed the light to bend and display the image on the side. It’s all simply amazing.
Check out this article and watch the video. It’s just mind blowing.
This was a pretty fun read. The only oscilloscopes I’ve ever used was during my physics class in high school (we weren’t allowed to touch it) and during my engineering labs at Cal Poly. It’s fun to see something like this. Check out the video towards the middle if you want to see it in action.
This is absolutely amazing. If you’ve never used jQuery definitely check it out. Ever since James Johnson (president of the Inland Empire .Net User’s Group) did a presentation on it last year I’ve been hooked. It’s is an outstanding JavaScript framework that actuallly makes JavaScript a pleasure to use.
As a classically trained developer I’ve always approached JavaScript as a tool to use only when absolutely necessary and as a last resort. Dealing with cross browser compatibility and just plain frustration over the language has made JavaScript a tool of evil in my development toolbelt.
With jQuery I not only now consider JavaScript a valuable asset I actually love to develop in it.
Hearing that Microsoft is now including it in their IDE is pretty exciting. This means that IntelliSense and debugging (while possible with some great workarounds from the jQuery community) will most likely eventually be fully supported for jQuery. I’ve worked with lots of development environments and Visual Studio is by far one of the best IDE’s around.
Probably even more exciting is that this furthers the strategy that MS is really interested in working with developers. Some of my friends are probably tired of me bashing the old-school “Microsoft Way”. Seeing the real encouragement of MS through employees like Scott Gu, Phil Haack and others on projects like MVC and such really make it apparent that MS is offering alternatives for developers who want the ability to code using modern standards.
Actually integrating jQuery into Visual Studio shows that MS is willing to offer alternatives to their own prodcuts such as the ASP.Net AJAX JavaScript framework. MS is no longer in the “We’re Microsoft. Our way or the highway” mentality.
In an ideal world you would design your applications so that your users could never enter bad data. Well, that’s not always possible for some technical reasons and for 3rd party apps you’re at the mercy of the vendor.
One case in point is our Student Information System (SIS) at the Val Verde Unified School District. It does a great job and we love it, but as a developer it does have its cons. The database isn’t as normalized as I would prefer and sometimes their data model just seems weird. Granted it is a Microsoft Access UI that links to a SQL back end, however, it still doesn’t change the way things are.
In this case we need to perform audits on our data. Anyone who is in data management knows that if it’s possible to enter an invalid value (such as a 5 digit phone number) some user will do it someday. In some cases bad data is rare and other cases just plain prevalent. The other issue is how that impacts your organization. Sometimes the invalid data will just be informational and not a big impact, however, in other cases this can impact reporting, automated processes, data exports/imports and many other systems.
So, I came up with our current data auditing system. We started it up back in June, 2006 and it’s been a great help. We currently have 60 checks on our SIS that run various times of the day. Each check is really just a simple SQL statement that looks for errors and stores these in a violations table. After each job a report with the results is sent to the relevant users. Typically we have only a few jobs, most of which run at night and by the morning staff have the latest error report in their email.
This has been a real labor saver as data entry errors are caught within 24 hours. Those who are responsible for the data usually have the information fresh in their minds or the reference paperwork readily at hand. We have to send various reports to the state several times in the year. In the past, before we had audits, many of the reported fields were not actively maintained and we would get thousands of errors that needed to be corrected before we submitted our reports. Now, the errors are corrected by the next day. When it comes time to submit the reports it’s a much faster and less stressful process.
Currently our audit system is in a very 1.0 stage. It only runs on our SIS data since that is what it was originally created for. Also, it’s entire maintained in SQL by our database managers (3 of us). There is no UI at all. It works, and is great for a 1.0 venture if I am allowed to say so. But it’s definitely time for a 2.0 version.
Here are just a few features I’m looking to put into the 2.0 version:
Easy to use web UI
Make use of ASP.Net security and roles
Able to audit ANY data through various data providers:
SQL
Active Directory
Exchange
File System
IIS logs
Etc.
Better reporting system
Easily setup generic audits on fields such as:
Non-empty
Alpha
Numeric
Real-time audits (such as using triggers in SQL)
I’m thinking about putting this on CodePlex once I have the basic framework up and running. I have no ETA on this project since it’s rare that we get a lot of “free” time to work on things like this.
What fun. Looks like a good friend of mine, James Johnson (also the president of the IE .Net User’s Group) got attacked by a potential SQL injection. Fortunately he’s pretty up on his secure coding so it wasn’t a problem. Props to James!
Anyway, he threw it my way because he thought I’d be interested. Was a pretty nasty one. Take a look at his write up and you can also see my comments there.
In my experience (which certainly isn’t exhaustive) a clean up is required when I’ve done something that svn didn’t expect. Typically this is where I’ve manipulated files on my own without using the TortoiseSVN tools. Eventually they get in such a state that svn simply doesn’t know how to resolve things. One example comes to mind where I have deleted my .svn folder in a directory. Now, according to svn, that directory is suddenly missing and an unversioned directory of the exact same name exists, and therefore cannot be replaced.
In many cases the Clean Up tool can fix simple mistakes, but in the example above svn simply needs help. If I can’t fix what’s going on then I’ll typically just delete the entire project folder and check out a brand new copy. This is usually far faster than actually trying to fix what went wrong. If I have made changes that I don’t want to loose then I check out the project into a brand new directory, add the changes to that version (either through VS 2008 or by just manually dragging over files) and do any manipulation such as renames, deleted, etc with Tortoise-svn.
Just yesterday I was working with an SSRS 2005 project. Our set of reports had grown over time so we decided to organize them into folders. VS 2005 cannot do this. Our strategy was to create one solution for the set of reports, containing a separate project for each subgroup. This really helped organize everything, but there was no simple way to do this using VS 2005 and svn. Either we create the new projects and moverename all the reports between projects using VS 2005, and svn was totally lost, or we did everything in svn, which confused VS 2005 and ultimately broke a lot of links.
I decided on the first option, to simply do it all in VS 2005 then commit the changes in svn. svn reported a lot of missing files and a lot of unversioned files after the change but I just marked the missing ones as deleted and committed the unversioned ones. While I technically lost the history in svn of the development of the files I have rarely needed to view changes past a few revisions and I think any intelligent person can see what we did.
I attended the Intro and Advanced sessions of Administration for SQL Server. The intro was a pretty straight forward topic of installation and configuration for setting up ArcSDE with SQL Server.
The interesting points were how ArcSDE integrates with SQL Server 2008. As I mentioned in my talk on SQL Server 2008 at the IE .Net User Group storing spatial data in a SQL database is not a new thing, just new to SQL Server. ArcGIS already has implementations to take advantage of server level spatial handling in Oracle, DB2 and possibly others. Only in 9.3 are SQL Server 2008 and PosgresSQL now supported.d
Since SQL Server 2008 hasn’t been officially released yet (all info still points to Q3 2008, which leaves less than 2 more months) ESRI is working with Microsoft on the latest release candidate version. While you can play with it ESRI of course highly discourages using ArcSDE against SQL 2008 until the official release. At that time ESRI will certify ArcSDE against SQL 2008 and will most likely release SP1 to deal with any changes since the 9.3 release.
That being said, what exactly is ArcSDE offloading to SQL Server?
Remember that the ArcGIS geodatabase is a very proprietary data model, allowing you to store not only geographic items (features) but also tables, topology rules, data access rules, locking schemes, business rules, etc. SQL Server 2008 is still at heart simply a data storage platform. This means that the bulk of the management of your GIS data will still be handled by ArcSDE.
SQL Server now includes spatial features including two new data types (GEOGRAPHY and GEOMETRY) as well as a whole host of functions to manipulate these data types with. The data is indexed using new Spatial Indexes on the server. These are the things that ArcSDE will take advantage of.
ArcSDE will store geographic data in the new GEOGRAPHY and GEOMETRY data types. This allows SQL to manage it’s own spatial indexes on the data. Doing this also allows ArcSDE to take advantage of the built-in spatial functions for simple requests.
ArcSDE will take advantage of the indexing schemes built in to SQL Server 2008, however, all the settings are left to Microsoft defaults. ESRI claims this is because every user’s data is different. While this is true, I got the impression that it’s more because the indexing scheme is so new that everyone is still coming to terms on it’s quality and how to actually manually maintain it.
Now that you can store your data in the underlying database, several questions come to mind:
Can I simply enter my own geographic data in SQL Server and use these in ArcGIS?
Yes you can, however, you must register it with ArcSDE before you can integrate it with other ArcSDE data. If you never plan to process this data using ArcGIS tools then you never have to register it. However, that doesn’t seem like a likely scenario since you imported the data into your geodatabase. To register the data use the sdelayer tool. This sets up all the geodatabase data and relationships so that the features can be managed using standard ArcGIS tools.
Can I manipulate ArcGIS data directly in SQL?
Yes! You can hit the data directly using the built-in spatial methods. This is pretty awesome for automated tasks that might fire at scheduled moments or for triggers.
For instance, at our school district our Student Information System (SIS) has a field for each student storing what their school of residence is. Since students may request to attend another school in our district, often it is important to know what school they normally would have gone to. Currently we autopopulate this data do this once a week or so. This is a manual process by using ArcGIS Desktop to simply join our students against our school polygons, output the results to a table and import these values back into the SIS. Once our GIS data is in ArcSDE on SQL 2008 I will be able to setup a trigger on the student record causing it to use the ST_INTERSECT method to find this data on the fly. Sweet! 🙂
However, (you knew there had to be a however right?) you MUST BE VERY CAREFUL. As with manipulating most SQL data you must know exactly what you’re doing. If you are inserting new objects you must find what the next ObjectID should be. If you are working with versioned features it’s more complicated as you have to work with versioned views within a versioning session. SQL 2008 has no concept of these so ArcSDE installed several utility stored procedures into SQL Server that you will use to manage this process.
What about the SRID?
If you are new to SQL Server 2008 you may have noticed this weird SRID value that is attached to every spatial value. This is the equivalent of the Spatial Reference used in ArcGIS such as NAD State Plain 1984 or WGS. However, the SQL Server SRID corresponds to the EPSD code (European Petroleum Survey Group). This is a table of spatial references all attached to a unique ID. This is required when registering your imported data with ArcGIS using the sdelayer command. What’s different is ArcGIS’s SRID’s do not correspond to the EPSD code. So what do you do? ArcGIS stores it’s SRID as auth_SRID. Here is an example of how to join the ArcGIS srid_auth against the EPSD reference table to find out which SRID to map to: select sr.auth_srid from SDE_layers l join SDE_spatial_references sr on l.srid = sr.srid where l.table_name = ‘buildings’ and l.owner = ‘dbo’ This command joins the SDE_layers table against the SDE_spatial_references table and looks for the layer named buildings owned by dbo.
All in all some really great stuff! I can’t wait to play with this.
I guess my one complaint is that while you can request an ArcGIS Desktop eval cd from the ESRI website I don’t know that you can get some type of an eval version of ArcGIS Server.
I’m in the SQL Server SIG lunch at the ESRI 2008 UC.
Ed Katibah (Spatial Program Manager at Microsoft) just announced that earlier this morning he received an email that SQL Server 2008 has officially gone RTM (Release to Manufacturing).
This is great. The last few months it’s been stated that SQL Server 2008 would hit the market Q3 of 2008. I guess this means they are officially on track!
