Saturday, September 8, 2012

Why I Don't Think I Am Too Old To Be Of Value

One of my frustrations with the world is the assumption that employers make that because I have more than 30 years of experience as a software engineer, there is nothing useful that I can do for their organization (or at least that my experience makes me worth less than a fresh college graduate).  Let me give you a couple of recent examples of what I am doing, and why the "you are too old to be of value" assumption seems questionable to me.

1. The project that I am working on right now uses JSPs for the user interface, and Struts to connect the JSPs to Java classes running on the server.  The Java classes use JDBC to query an Informix database through Stored Procedure Language (SPLs), which is an extension of SQL.

I have noticed in the almost three years that I have been working on this project that the SPLs never use boolean type parameters.  Instead, we pass CHAR(1) parameters that are usually "Y" or "N".  In addition, the vast majority of the tables defined in the Informix database that should be boolean are declared CHAR(1).  

Why?  It turns out that the JDBC class responsible for converting parameters in Java to the string to send to Informix did not translate Java boolean objects correctly; it was converting them to a SQL BIT type, and Informix apparently does not know about that type.  It appears that the software engineers (and I use that term loosely) who hacked our current mess together could not figure out how to pass boolean types to SPLs, so they used CHAR(1) for boolean parameters--and did likewise in defining the tables in the database.  It was a one line change to tell the class that formats the strings to convert Java boolean values to something that SPLs could recognize as a boolean parameter.  Now I can pass Java boolean values to SPLs as BOOLEAN parameters.

Our database administrator tells me that Informix is more efficient in both storage space and processing time using booleans rather that CHAR(1).  Every place that we use a CHAR(1) instead of a boolean means that Informix has to do a separate check to make sure that the field is "Y" or "N" (or whatever variant this particular table uses).  These suboptimal uses of CHAR(1) occur in many hundreds of tables, and many of these tables have hundreds of thousands of rows, with vast numbers of transactions on a daily basis.  (A user of our system accesses dozens of tables each time he or she loads an offender's records.)  Pretty obviously, this is an area where changing the existing tables and SPLs to use booleans instead of CHAR(1) is likely to be a big gain for throughput.

2. I have spent much of the last year and a half working on something called the PreSentence Investigation module (PSI).  After an offender has been convicted, Corrections does something called a PreSentence Investigation, which produces a report showing an offender's employment history, family history, previous criminal history, substance abuse history, medical history.  The PSI report is provided to the judge so that he determine the appropriate sentence for this offender.  

You read these reports and it is often quite difficult not to get angry at the bad parenting that clearly set some of these offenders up to fail.  That doesn't mean that they don't need to be locked up--but it makes you realize that a society that cares not at all about values is destined for failure.

Anyway, a recurring problem is that there are dates associated with various events.  When did you start using meth?  When did you start using alcohol?  When did you start using heroin?  And the answers on many of these questions are pretty appalling.  But these are necessarily approximations.  The investigators are asking people to give them a date for events that happened ten or twenty years earlier, and these must necessarily be estimates.

The problem, unfortunately, is that the people who threw together the database definition some years ago assumed that all the dates in our system would be precisely known.  For obvious reasons, the courts need to know which dates are dates, and which are guesses.  But there's no easy way to specify this in an SQL DATE field.  

So I came up with a very clever scheme for solving this problem, without redesigning all the tables: add a column called precision next to each date column in every table where there might be uncertainty.  This value would be an enumeration identifying the precision of the corresponding DATE field: 0=precise date; 1 means, to the month; 2=year; 3=decade.  On the input screens, the investigator can enter a date, and then select how precise this date is.  When it comes time to print the PSI report, we can print various strings, depending on precision: 

6/25/1995
6/1995
1995
1990s

I think this is a very elegant solution to the problem of already having vast quantities of data in the system which you can't discard. 

19 comments:

  1. Great problem-solving skills on display there, Clayton!

    (By the way, Captcha sucks.)

    ReplyDelete
  2. "Our database administrator tells me that Informix is more efficient in both storage space and processing time using booleans rather that CHAR(1)."

    Perhaps this could be translated, "Mr. Cramer, your expertise has exposed the fraud that I am in this position. I will tender my resignation and go get my mop."

    The parenting scandal also reflects the "deck chairs on the Titanic" thinking in the courts and academia. We have painstaking records of the fruits of failure to properly form and nurture society: Oh such joy in our ability to swallow a camel yet fail to grind out a gnat.

    Thanks,

    DM

    ReplyDelete
  3. dmurray: I wouldn't say that. Our DBA has a nearly impossible job--just keeping ahead of the data insanity and new additions that constitute our operation leaves little time for reflection and consideration of what is wrong with the existing database layout.

    ReplyDelete
  4. It's a shame you don't live here, because we're hiring right now, and can't find decent people.

    ReplyDelete
  5. Brother, I'm 36.... I just got through the people thinking I was too young, and NOW, they're starting to think I'm too old ;-)

    ReplyDelete
  6. Congrats on finding and fixing these problems.

    I wish I knew why the programming world seems to insist that only young programmers are useful...though it is an advantage for me, somewhat.

    I just don't know if managers will figure out that age is not necessarily a bad thing by the time I'm a little older.

    ReplyDelete
  7. Chris Byrne: If you're a software developer without a serious security clearance or not in the embedded field, yep, "they're starting to think [you're] too old", at least for a conventional salaried job, unless you're both very good and have very good connections. If you don't have what it takes to be a consultant, it's time to plan a career change.

    ReplyDelete
  8. Isn't ageism wonderful!

    ReplyDelete
  9. I apologize to your DBA.

    Please forgive me.

    The crack about the courts and ivory towers stands.

    DM

    ReplyDelete
  10. As a software developer of 25 plus years, this is exactly why I moved to "consulting". I put it in quotes because it is really temporary SW development work, rather than true consulting. There are now enough companies with a large enough backlog of unstaffed work that IT outsourcing is booming right now. I found out that I was going to be out of work with less than two weeks to go very recently, and I was able to locate a new project at the same bill rate doing the same type of work withing 48 hours. Of course part of this comes from having done this temporary IT work in the same city for over 15 years and having a nice stable of companies on whom to call in such times.

    ReplyDelete
  11. There's little doubt that the young are more flexible and faster at learning completely new things.

    But one of the reasons older candidates are shunned is concern by the hirer that the older individual will know more than he who hires.

    ReplyDelete
  12. Dai Alanye: Perhaps, but how many "completely new things" are really getting developed?

    I can go into details, but almost every intellectual foundation of our current computing infrastructure was established no later than the 1060s, with a few things reaching into the early '70.

    The roots of our operating systems (and very direct ancestors for Linux and Max OSX), computer languages, the G in the GUI actually is in the early '70s but it was a pretty obvious extension to AUGMENT, databases ... I need to reexamine this in the light of the last few years but I'm pretty comfortable with the observation.

    So my point here is that many of the older have more frames of reference to use in learning "new" things, and aren't necessarily slower, it more depends on the person.

    ReplyDelete
  13. "... add a column called precision next to each date column ..."

    Did you actually put it next to each date column, and not just at the end? This is difficult to do in sql, and can cause a problem in queries like this:

    Select * ... order by 4

    asdf

    ReplyDelete
  14. Fortunately, the interface to our tables is through the SPLs. Good practice is RETURNING type AS columnname, and then the code on the server that calls the SPL retrieves the columns by name, not by order.

    ReplyDelete
  15. Clayton, you need to be working with microcontrollers, where every bit, every cycle and every microwatt is still precious.

    The sloppy and profligate don't do really well with them.

    ReplyDelete
  16. Just curious, but:
    (1) You are a very experienced software engineer, and if memory serves, a lot of that experience was in startups.
    (2) You are obviously highly motivated. Folks who follow your blog know that you get income from several sources: a job with the state, teaching, writing articles for PJ Media and law reviews, plus (!) a business on the side (Scope Roller.)
    (3) Just to emphasize, you have already started a small business and kept it alive.

    What am I missing here?

    ReplyDelete
  17. RS: microcontrollers today I suspect are much like the Intel 8085 and Zilog Z80 I programmed back in the 1980s. But who would hire someone like me? No one.

    Joe Shropshire: You are right. But you make it sound like employers care about such things. Not that I can find.

    ReplyDelete
  18. What I meant was: why on earth are you not running your own startup? You know at least one potential employer who does care about these things -- that's you.

    ReplyDelete
  19. Joe, thanks for the vote of confidence, but it doesn't work that way. I was working on a product for a while that increased output of solar panels by about 50%, but discovered that there is no venture capital available for that sort of thing, and it was too big of a startup for me to directly fund.

    ReplyDelete