GIS Programming - Module 5 - Geoprocessing in ArcGIS

This week's assignment really starts to bring things together, between ArcMap and Python.  We used ModelBuilder from within ArcMap to clip a soil type shapefile to the extent of a basin, then selected the soil features with the attribute "Not prime farmland" then erased those selected features from the clipped soils shapefile to create a new shapefile, shown at left.  This shapefile represents the tracts of soils within the basin that are considered some type of suitable farmland (in other words, everything except "Not prime farmland" tracts).

The Select tool, when operated from within ModelBuilder, has a problem in that it always changed the SQL term 'Not prime farmland' to 'Not Prime Farmland'.  This caused none of those features to be selected, because the lower-case letters had been changed to upper-case, and so were not recognized.  The Erase too
l then had nothing to erase, because the Select tool had found nothing to select.  Strangely enough, when the single quotes around Not prime farmland were changed to double quotes, and the model run (with errors, because it didn't like the double quotes), and then the whole SQL statement was entered again, the lower case letters were preserved in the model, and it could run correctly.  Some other people also had this problem and brought it up on this week's discussion board, and found that removing the quotes completely from Not prime farmland, and then putting them back, also fixed the problem.

The model was then converted to a Python script, and a few adjustments were made, such as adding the full file path to the input shapefiles (soils and basin) so that the script could find them, and also adding the line of script arcpy.env.overwriteOutput = True.  This makes it possible to run the script multiple times, although the outputs already exist in ArcCatelog from previous runs.  The Python script can then be converted into a Script Tool within ArcMap, where it can be run like any other tool.

Process Summary Notes:

Part 1, Step 2: Explain how you set up your SoilErase model.
1. First, open a new ArcMap .mxd, and add the shapefiles soils and basin.
2. To create a model, open ArcCatalog, right-click the folder in which you are working (ex. Module5), select New, and then Toolbox.
3. Rename the toolbox appropriately, then right-click it, select New, and then Model.
4. This will cause a ModelBuilder window to open.  In the top menu, select Model¸ then Save As, then name the model.
5. For this model, the task is to clip the Soils shapefile to the extent of the Basin shapefile, then remove the features within Soils that have the attribute “Not prime farmland.”
6. From the ToC on the .mxd, click and drag both Soils and Basin shapefiles into the ModelBuilder window.  These will be the inputs, and they will show up as blue oval boxes, with the names of the shapefiles inside.
7. The Clip and Select tools are dragged from the ArcToolbox  > Analysis Tools > Extract toolset, into the ModelBuilder space.  They will show up as yellow, rectangular boxes.
8. The Connect tool is used to draw arrows from the input shapefiles, to the tools.  Although Select or Clip can done in either order, I did Clip first.
9. As I drew the arrow from Soils to Clip tool, a box appeared, in which either input feature or clip feature could be indicated.  Because the Soils is being clipped, it is the input feature.
10. Then, another arrow is drawn from Basin to Clip, and in this case, the clip feature option is selected.
11. After this is done, a new box appears for the output, supplied with a default name.  This box is a green oval, indicating that it is an output.
12. The Select tool box is set to the right of the new output from the clip tool, and an arrow is drawn from the clip output, to the select tool box.  Parameters must be set; this can be done by double-clicking the Select box.  In the SQL dialog, enter, “FARMLNDCL” = ‘Not prime farmland”
13. A new output box (green oval) appears, which contains the selected features from the last output.
14. The Erase tool is dragged into the ModelBuilder, and an arrow is drawn from the last (Select) output box, to the Erase tool box.  The input feature is the clipped soil shapefile, and the Erase feature should be the Selection of the last output (those tracts that are Not prime farmland.)
15. The new output feature, and the final one, is the soil tracts that fall within the extent of the basin, with those of attribute “Not prime farmland” removed.
16. After this is done, right click the final output, after the Erase tool, and select “Add to Display.”
17. On each of the 2 inputs (blue ovals) and 3 outputs (green ovals), right-click, and check “Model Parameter.”  A “P” will appear next to each of those boxes. (This is not done to the tools.)  This will make it possible to change the input features in the model, as well as the file paths, and make the model useful for general purposes.
18. From the top menu: Model > Save.

You are also strongly encouraged to include any other notes you have on this week’s lecture and lab or your process.
(These extra notes are optional, and are not required for any part of your grade. However, they can greatly improve your learning experience by reinforcing what you’ve learned, and giving you a reference to look back on in the future. Plus, they can make your blog posts more interesting!)
Other notes:


1. After model is exported to Python script form, you must
2. add the “import arcpy” line of code at the top, and then

3. arcpy.env.overwriteOutput=True

4. you must also make sure each shapefile’s entire file path is specified for the variables.

This problem exists for the Select tool, when it is used from within the ModelBuilder: the attribute in the soils shapefile is "Not prime farmlands," with the p and f in lower case.  However, when you enter that into the SQL statement, inside ModelBuilder, either by typing it in, or selecting it from the "Get Unique Values," it always reverts to "Not Prime Farmland," with capital P and F, even if you change it back to lower case and save it.  This is not the correct name for the attribute, so nothing is selected.    I've included a screen shot.
  I tried running this process manually, using the Clip, Select and Erase tools separately, directly from the toolbox.  In this case, the problem DID NOT happen.  The p and f in Not prime farmland did NOT revert to caps, the selection was created, and it was properly erased from the soils shapefile.

took out the quotes, put em back in again, erased the whole statement and added it in again, and now the select works and the Not prime farmlands gets properly erased!

GIS Programming - Module 4 - Debugging and Error Handling

This week in GIS Programming, we practiced locating, fixing, and otherwise dealing with various types of errors in Python scripts. These skills help make that formidable Python much less fearsome!   

There are generally three types of errors that you might make when writing script:

1) syntax errors include spelling and punctuation mistakes.  Making typos, leaving out a colon (:), or indenting incorrectly are very common.
There are also:
2) exceptions, for example, in which modules and functions (and other parts of Python) don't have what they need to run properly.  There are many, many types of exceptions, and luckily they are identified in a very general sense in the error statements in the results, and the line numbers in which they occur can be found as well.

3) The other type of error can be very insidious: the logic error.  In this one, your code might run fine, but the answer you get is wrong, because you've told Python to do something different than what you intended. Python will not know the difference; it's only following directions, and can't read our minds!  This is an example of GIGO: garbage in: garbage out.

For our assignment, we looked at three scripts with syntax and exception errors.  In Scripts 1 and 2, we repaired the errors, but in Script 3, we added a "try-except" loop, which gives Python an alternative path, in order to step around the error.  We didn't actually fix the error, and the results tell us that it's there.

You don't necessarily have to repair an error right away. It is possible to use a "try-except" statement around a block with an error, which isolates and bypasses the error and goes on through the rest of the script.  It "tries" the problematic line, then has an "except" route, which offers an alternative for Python, such as printing out a statement identifying the nature of the error.  This way Python doesn't have to get hung up in the middle of the script.

I found that there are three excellent ways to find errors. The first two are rather rudimentary.  1) You can "comment out" most of the lines in the script.  This means you type # in front of every line except the first one, then run it.  If it runs okay, take the comment # off the next line or loop (for, while, etc) and run it again.  You keep working your way downward through the script, de-commenting the lines one by one, until it runs properly.   This way, you can eliminate the lines that are okay, and find and fix the ones that aren't.
2) Another way to check and see if a block of script is working right, if there are no results following it, is to insert a "print" statement in a new line after it.  I used "print okay."  This way, if "okay" appeared in the interactive window, I knew that everything above that print statement was okay.

3)PythonWin also has debugging tools that work in a similar way, but without having to comment out or add print statements, which can be cumbersome if you have a long script.  You can turn on the debugger, then run the script in a "step-through" fashion one line or block at a time.  At each step, you'll see that it was okay, or that it wasn't.   This way, you can find the errors or confirm that a line is okay.  You can also insert break points, which isolate the parts you want to run.

Below are screenshots of my results from the three assigned scripts. They are shown in the PythonWin Interactive Window, after I had repaired the errors or added a try-except statement: 

Script 1.  These are the results, after the script was corrected.

Script 1 Results

Script 2: This script had 8 errors.  These are the results, after they were identified and fixed.

Script 2 Results

Script 3:
In this script, the error was not corrected.  Rather, a try-except statement was inserted after the error was identified, and the results printed out the type of error occurring in Part A of the script, then went on to Part B and printed the results.  Part B did not contain any errors, and was not directly affected by the error in Part A.  However, until the error in Part A was dealt with, or "caught," the script would not continue to execute, and would not have been able to proceed to Part B and produce results.  

Script 3 Results

GIS Programming - Module 3 - Python Fundamentals Part 2

Results from the Interactive Window for the Module 3 Scripts

In the second week of learning the fundamentals of Python, we examined a script with an if conditional statement, and identified and corrected a couple of errors that were preventing the script from running.


We also wrote two scripts of our own.  The first script created a list of 20 randomly-generated numbers,with values of 0 to 10, then printed the whole list.








In the second script, we specified one "unlucky" number, then had the script remove all instances of that number, if it was there, from the list we'd made in the previous step.  We also had the script print messages that stated if the number was in the list, or not, and if so, how many times.  At the end, the new list without the unlucky number was printed.

Here are some notes from my Process Summary for this week.  This describes generally how the unlucky number can be removed from the script in the last block.

Removing numbers from the list. 

1.      For this part of the code, you already have the list with the random numbers appended into it.

2.      A variable has already been assigned to the unlucky number.

3.      All instances of the unlucky number are removed from the list with a while loop. 

4.      As long as the list still contains instances of the unlucky number, the while loop will continue to repeat and remove the numbers; this is the condition. 

5.      The indented statement within the while loop that actually removes the number is the remove method: the object is the list, and the argument is the variable assigned to the unlucky number. 

6.      In my script, the method is: SeaseList.remove (unlucky).

This script has a system argument: the user must input the unlucky number variable in the Run Script box.

GIS Programming - Participation Assignment 1

The purpose of this assignment is to find and summarize an article about a real-world application of GIS, programming, or Python.

The article I read (title and link below) discusses the use of a Python script to correct distortion errors in hyperspectral imagery collected by aircraft.

Jensen, R.R., Jackson, M.W. and Lulla, V., 2008: “Single line correction method to remove aircraft roll errors in hyperspectral imagery.” Journal of Applied Remote Sensing 2, pp.1-10.

I found this article to be very interesting and pertinent because it ties together our Remote Sensing course (which I took last fall semester) with the use of Python programming.  The problem being addressed here with a Python algorithm is fairly simple.  Because planes collecting aerial imagery are subject to pitch, roll and yaw as they follow their paths, any data collected for more than just a single instantaneous frame is prone to distortion.  Hyperspectral data is collected in what is referred to as the “push-broom” method:  the sensor instantaneously collects a single continuous strip of data at a time, to each side of the plane, perpendicular to the flight path.  Any amount of roll by the plane will change the relative view angles along the data strips, and cause distortion in the resultant image.
The usual way to correct this distortion is called the reference line method: the wavy, distorted image of a feature known to be linear on the ground (like a straight road) is digitized. The program then calculates and implements the offsets needed in each row of pixels to make that feature appear straight on the image. This process, though simple in theory, is very meticulous, as it has to look at every single row of pixels in the image.  That makes it an excellent candidate for automation by Python scripting.
The correction was accomplished by exporting the coordinates of each pixel in the wavy reference line into an ASCII file.  The inputs for the Python program that will correct the distortion are:

• the uncorrected/distorted imagery file, and
• the ASCII file containing the pixel coordinates of the reference line.

The variables used by the program are also very simple:

• the range of column values for the pixels reference line, and
• the average column value for the entire reference line.

The Python script then just offsets each row by the difference between the column value and the average.  The effect of this is to straighten out the image of the linear feature, so it appears straight on the corrected image.  The positions of the rest of the features in the image are corrected along with it.

Here is a graphic explanation of how pixel rows are offset using a simple Python script to correct the wavy distortion caused by aircraft roll.

Example of Distortion Correction from Jensen, et al. (2008)

GIS Programming - Module 2 - Python Fundamentals Part 1

As we learn more about Python scripting this week, we used functions and methods, which are types of instructions for the program.  We wrote a script in which the input was a string, consisting of our full names.  The final output was our last name, and the number of letters in that name time 3.  My results are shown below:
The last name, Sease, and 15, which is 3 times the number of letters of my last name.

Results of the Python Script

GIS Programming - Module 1 - Introducing Python

In the first week of GIS Programming, we were introduced to the concept of pseudocoding to help us learn how we'll be constructing Python scripts later on.  This is a step-by-step script that is more-or-less like English, that puts together the Input, the desired Output, and a list of the tasks necessary to get from the Input to Output.  We also examined the interactive window of Python, in which script can be run a line at a time, the Pythonwin script editor, in which multiple lines of script can be compiled, saved, then run.  A Python interactive window is also accessible through ArcMap, by clicking the Python Window button on the standard toolbar.  This environment is convenient, because when you start typing, it gives suggestion prompts for possible statements from which you can select. Also, there is help and syntax information in this window for whatever statement you select .

2015 GIS Programming - First Post

This is the initial post for GIS Programming, Summer, 2015

(The blog posts for the partially-completed programming course from summer 2014 are labeled 2014 (old) Programming.)