misc software development snippets

Problem space:

Using Perl, you want to see if XML is ‘valid’.

(it’s actually just a ‘well formed’ check. To check validity, you would need to use the XML documents DTD, if one exists)

my $parser = XML::Parser->new( ErrorContext => 2 );
eval { $parser->parse( $data ); };
# report any error that stopped parsing
if( $@ ) {
    Carp::croak "ERROR: NOT WELL FORMED ELEMENT: $@\n";
}

Additionally, if you have a *NIX box available, xmllint should be installed, is available from the command line.

We are using the Perl module, Spreadsheet::ParseExcel->new to parse XLS files and insert the results into a UTF8 MySQL database.

I came across an interesting situation with a Microsoft Excel file, where only 3 fields of the 12 are encoded as UTF-16BE, the rest are latin1.

Parsing a file with mixed encoding returns unexpected results, since the data in the spreadsheet looks correct, but parsing in it one receives incorrect characters.

Here is some code using Encode::Guess which will resolve the mixed encoding issue.

my $enc = guess_encoding($text); # use Encode::Guess;
if (ref $enc) {
    $descr = decode($enc->name, $text);
}
else { # convert to utf8, does not turn on utf8 flag
  from_to($text, "iso-8859-1", "utf8");
}

or, if you are already expecting UTF8 data, or more specifically, data with the utf8 flag turned on (perl 5.8.* – see perldoc Encode)

use Encode qw/decode from_to/;
use Encode::Guess;
if (! utf8::is_utf8($val)) { # ok, its not utf8, what is it?
    my $enc = guess_encoding($val); # use Encode::Guess;
    if (ref $enc) {
        $val = decode($enc->name, $val); #turns on utf8 flag
    }
}

(you would have to use these on a field-by-field/val-by-val basis obviously)

Searching for examples of ruby/cgi usage didn’t turn up much.

I had to rough it and unfortunately, it took longer than it should have to come up with what I wanted to accomplish. (well, more or less what I wanted, well, actually, it’s not going to work for what I need but anyway, here’s an working ruby/cgi example).

thought I’d share:

require 'cgi'
require 'json'

cgi = CGI.new('html4Tr') # add HTML generation methods

if cgi.params.empty?
  # no request recieved, return html
  cgi.out('charset' => 'UTF-8') do
    cgi.html() do
      cgi.head() do
        "\n" +
        cgi.title{"cgi testing"} +
        CGI.pretty(
          "\n" + %q(<meta http-equiv="cache-control" content="no-cache">
          <meta http-equiv="content-type" content="text/html; charset=UTF-8">
          <meta http-equiv="expires" content="Mon, 22 Jul 2002 11:12:01 GMT">
          <meta http-equiv="pragma" content="no-cache">
          <script type="text/javascript" src="../js/jquery-1.3.2.js">)
        )
      end +
      cgi.body() do
        CGI.pretty(
          "\n" + %q(<input type='hidden' id="ids" value='' />
          <!-- start: show results -->
          <div id="show_hide_container" style="display:none"><a id="hide_n_show">[-]</a></div>
          <div id="action_results"></div>
          <!-- end: show results -->
          <div id="display_results">
            <div class="show_demo" id="unique" style="height:100%"></div>
          </div>)
        ) \
         +
        cgi.form() do
          cgi.textarea("get_text") +
          cgi.br +
          cgi.submit
        end +
        cgi.pre() do
          CGI::escapeHTML(
            "params: " + cgi.params.inspect + "\n" +
            "cookies: " + cgi.cookies.inspect + "\n" +
            ENV.collect() do |key, value|
              key + " --> " + value + "\n"
            end.join("")
          )
        end
      end
    end
  end
else
  # request recieved, return as JSON
  cgi = CGI.new(:encoding => 'UTF-8')
  print cgi.header('type' => 'application/x-javascript', 'charset' => 'UTF-8')
  print cgi.params.to_json
end

Problem Space:

You are submitting a html form, which contains many (hidden) form elements that you do not want (or need) sent to the server for processing.

<script type="text/javascript"> 

// do when submit is pressed
// don't send the 'hidden' textarea content
//  its' un-needed and clutters up the post (and wastes bandwidth)
$('form').submit(function() {

  var elements = this.elements;

  for (i=0; i < elements.length; i++) {

    // if textarea id ends in ".tmpl", exclude from post
    if (elements[i].id && elements[i].id.match(/\.tmpl$/)) {

      // could change the val,...
      //$('textarea[name="'+elements[i].id+'"]').val(' ');
      //console.log($('textarea[name="'+elements[i].id+'"]').val());

      // better, to remove element from submit
      $('textarea[name="'+elements[i].id+'"]').attr('disabled', 'disabled');

    }
   }

  //return false; // if false, doesn't submit, good for testing
});

</script>

This is an example of how to transform a multi-table join result returned by SQL and iterate over it to retrieve the data you are seeking.

Problem Space:

You have a multi-table join result returned by a SQL query which contains the data you need but in a ‘flat’ format which makes it difficult to extract, maintaing the data relationship.

The Data:

This data structure represents the data returned by a multi-table join.

sql_results = [
 {'t_name' => 'Acct & Benft','h_name' => 'Account',  'type' => 'pu_group'},
 {'t_name' => 'Acct & Benft','h_name' => 'Account',  'type' => 'py_group'},
 {'t_name' => 'Acct & Benft','h_name' => 'Account',  'type' => 'c_group'},
 {'t_name' => 'Acct & Benft','h_name' => 'Account',  'type' => 'pu_line'},
 {'t_name' => 'Acct & Benft','h_name' => 'Account',  'type' => 'py_line'},
 {'t_name' => 'Acct & Benft','h_name' => 'Account',  'type' => 'cl_line'},
 {'t_name' => 'Acct & Benft','h_name' => 'Benft',    'type' => 'b_line'},
 {'t_name' => 'Acct & Benft','h_name' => 'Benft',    'type' => 'b_c_line'},
 {'t_name' => 'Acct & Benft','h_name' => 'Benft',    'type' => 'b_c_add'},
 {'t_name' => 'Acct & Benft','h_name' => 'Benft',    'type' => 'b_group'},
 {'t_name' => 'Catg',        'h_name' => 'Catg Pge', 'type' => 'catg_line'},
 {'t_name' => 'Catg',        'h_name' => 'Catg Pge', 'type' => 'catg_group'},
 {'t_name' => 'List Page',   'h_name' => 'All St',   'type' => 's_line' },
 {'t_name' => 'List Page',   'h_name' => 'All St',   'type' => 's_group'},
 {'t_name' => 'List Page',   'h_name' => 'Cpns & S', 'type' => 'cpn_group'},
 {'t_name' => 'List Page',   'h_name' => 'Cpns & S', 'type' => 'cpn_line_a'},
 {'t_name' => 'List Page',   'h_name' => 'Cpns & S', 'type' => 'cpn_line'},
 {'t_name' => 'TLP',         'h_name' => 'Mrch Rand','type' => 'contact'},
 {'t_name' => 'TLP',         'h_name' => 'Mrch Rand','type' => 'tlp_group'},
 {'t_name' => 'TLP',         'h_name' => 'Mrch Rand','type' => 'tlp_line'}
]

From the data, I need to create a table where:

‘t_name’ is a header with one or more sub-nodes, ‘h_name’.
and ‘h_name’ is a sub-header with one or more sub-nodes, ‘type’

For Illustration:

                          'List Page'(t_name)
                               |
               ----------------------------------
               |                                |
           'All St'(h_name)              'Cpns & S'(h_name)
               |                                |
         -------------             --------------------------
         |           |             |            |           |
      's_line', 's_group'     'cpn_group', 'cpn_line_a' 'cpn_line' (type)

Or, simpler

                          'Catg'(t_name)
                             |
                         'Catg Pge'(h_name)
                             |
                        -------------
                        |            |
                   'catg_line', 'catg_group'(type)

It seems clear, to me, that a good data structure for this representation would be
a hash whose key is t_name and whose values are a hash whose key is h_name and
whose value is an array

Example:

{
 "List Page" => {
      "All St"   => ["s_line", "s_group"],
      "Cpns & S" => ["cpn_group", "cpn_line_a", "cpn_line"]
  },
 "Catg"      => {
      "Catg Pge" => ["catg_line", "catg_group"]
  },
 etc,...

}

Here is some code which produces the desired data structure. (I’m sure there are many other approaches)

top_node     = Hash.new
top_sub_node = Hash.new
sql_results.each do |r|

  if top_sub_node[r['h_name']].nil?
    top_sub_node[r['h_name']] = [r['type']]
  else
    top_sub_node[r['h_name']].push r['type']
  end

  if top_node[r['t_name']].nil?
    top_node[ r['t_name'] ] = {r['h_name'] => top_sub_node[r['h_name']]}
  else
    top_node[ r['t_name'] ].merge!({r['h_name'] => top_sub_node[r['h_name']]})
  end
end

require 'pp'
pp top_node

Here are 3 identical ways to capture the content in a div by only the class name.

<div class="error">error message here</div>

@ie.div(:class, 'error').text
@ie.div(:class, 'error').innerText
@ie.element_by_xpath("//div[@class='error']").text

all the above return ‘error message here‘

problem space:
you are trying to click a (click-able) link element, with no real discerning qualities. (i.e. there is no name, no id, a generic css class same as the other links on the page, etc. Luckily, In this case there is a javascript onclick handler which is unique to this element.

<a class="showhide" href="javascript:void(0)" onclick="setDisplay('notes','block');">show

you could accomplish this two ways. (that I know of)
1. find the number of the link in relation to the page, (i.e. it’s the 13th)
ie.link(:index, 13).click
2. specify an xpath.
ie.link(:xpath, "//a[contains(@onclick, \"setDisplay('notes','block')\")]").click

(I just also, ‘discovered’ that firebug (Firefox JavaScript debugging extension) offers an xpath to elements, (in the right click). it offered this: (untested)
/html/body/div[2]/table/tbody/tr/td/table/tbody/tr/td/div/form/table/tbody/tr/td/div[2]/table[7]/tbody/tr[3]/td/a

ref: groups.google.com/watir-general

Nagios/nrpe how to (mini version):

For this session, I am demonstrating a basic Nagios set up of nrpe.

There are two hosts involved.
1. the nagios host. (the master node; where the nagios web ui is)
2. the remote host (the node you want to monitor remotely from the master node)

Software Versions involved:
both hosts: CentOS release 5.3 (Final) (64 bit)

master node:
Nagios® Core™ Version 3.2.0
nagios.x86_64 3.2.0-1.el5.rf (yum install)

remote host:
nagios-nrpe.x86_64 2.12-1.el5.rf (yum install)

(I assume you have a basic nagios set up already – this is just to enable nrpe)

nagios host: (master)

$ sudo yum install nagios-plugins-nrpe  nrpe
$ sudo chown nagios.nagios /etc/nagios/nrpe.cfg
$ sudo vi  /etc/nagios/objects/commands.cfg
# add
define command{
        command_name    check_nrpe
        command_line    $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

$ sudo vi /etc/nagios/objects/myremote_host.cfg
# add
define service{
        use                                      remote-service
        host_name                          example.com
        service_description             Check Remote Users
        check_command                  check_nrpe!check_users
}
# nrpe (remote checks) syntax is check_nrpe(as defined above) ! 'command to
 run' (as defined on remote host, see below)

remote host: (node to be remotely monitored)
(NOTE: nagios itself should not be installed on this host)

$ sudo yum install nagios-nrpe
$ cd /etc/nagios
$ sudo chown nagios.nagios *.cfg
$ sudo vim /etc/xinetd.d/nrpe
# add
service nrpe
{
        flags           = REUSE
        type            = UNLISTED
        port            = 5666
        socket_type     = stream
        wait            = no
        user            = nagios
        group           = nagios
        server          = /usr/sbin/nrpe
        server_args     = -c /etc/nagios/nrpe.cfg --inetd
        log_on_failure  += USERID
        disable         = no    # was yes
        only_from       = NAGIOS_MASTER_IP (master nagios host you want to
connect from) # was localhost
}
$ sudo vim cat /etc/hosts.allow
# add same ip from xinetd above
NAGIOS_MASTER_IP

$ sudo vim  /etc/nagios/nrpe.cfg
#allowed_hosts=127.0.0.1   #
# add same ip from xinetd above
allowed_hosts=NAGIOS_MASTER_IP    

$ sudo vim /etc/services
# append this line to the file
# Local services
nrpe            5666/tcp                        # nrpe (nagios)

$ sudo /sbin/service nrpe start
Starting Nagios NRPE daemon (nrpe):                        [  OK  ]
$ sudo /etc/init.d/xinetd reload
$ sudo /sbin/chkconfig --list |grep nrpe
nrpe           	0:off	1:off	2:off	3:off	4:off	5:off	6:off
	nrpe:          	off
$ sudo /sbin/chkconfig --levels 35 nrpe on
$ sudo /sbin/chkconfig --list |grep nrpe
nrpe           	0:off	1:off	2:off	3:on	4:off	5:on	6:off
	nrpe:          	off

$ netstat -an | grep 5666
tcp        0      0 0.0.0.0:5666    0.0.0.0:*    LISTEN

ok, good, it’s running

now from nagios (master) server:
(can we see it’s running too?)
telnet REMOTE_HOST_WE_JUST_CONFIGURED_IP 5666

$ /usr/lib64/nagios/plugins/check_nrpe -H REMOTE_HOST_WE_JUST_CONFIGURED_IP -p5666 -c check_disk1

to add new commands to be run via nrpe, we must add those to the remote host, nrpe.cfg file:

added new test on remote host:
$ sudo vim /etc/nagios/nrpe.cfg
command[check_mapper]=/usr/lib64/nagios/plugins/check_disk -w 20% -c 10% -p /dev/sda5

test from master node:
$ /usr/lib64/nagios/plugins/check_nrpe -H REMOTE_HOST_WE_JUST_CONFIGURED_IP -p5666 -c check_mapper
DISK OK - free space: / 41180 MB (50% inode=99%);| /=40614MB;68993;77617;0;86242

helpful master node commands:
server config test:
$ sudo /usr/bin/nagios -v /etc/nagios/nagios.cfg
#start/stop/restart:
$ sudo /sbin/service nagios restart
#command line check commands against remote host:
$ /usr/lib64/nagios/plugins/check_nrpe -H REMOTE_SERVER_IP -p5666 -c check_disk1

ref:
http://www.crucialwebhost.com/blog/nagios-howto-using-nrpe-to-monitor-remote-services/
http://ubuntuforums.org/archive/index.php/t-498435.html – bottom comment

I just ran a sudo yum update on my development CentOS 5.3 box. (upgrade to 5.4)

All went well, except two issues.

1. I had to rollback perl-Apache-DBI.noarch 1.07-1.el5.rf
(luckily, I knew about this one, and had an old package available, and there are no deps issues – whew)

The issue manifests itself as:
[Tue Nov 10 10:43:02 2009] [error] Global $r object is not available. Set:\n\tPerlOptions +GlobalRequest\nin httpd.conf at /usr/lib/perl5/vendor_perl/5.8.8/Apache/DBI.pm line 144.\nCompilation failed in require at (eval 2) line 1.\n
[Tue Nov 10 10:43:02 2009] [error] Can't load Perl file: perl_load.pl for server , exiting...

To rollback:
$ sudo yum remove perl-Apache-DBI
$ sudo rpm -ivh perl-Apache-DBI-1.06-1.el5.rf.noarch.rpm

2. synopsis: “Weak references are not implemented in the version of perl ”

The issue manifests itself as:
Weak references are not implemented in the version of perl at /usr/lib/perl5/vendor_perl/5.8.8/SOAP/Lite.pm line 2447\nBEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.8.8/SOAP/Lite.pm line 2447.\nCompilation failed in require,...
...
...
[Wed Nov 11 16:50:01 2009] [error] Can't load Perl file: perl_load.pl for server, exiting...

which seemed me to (incorrectly) initially look in SOAP::Lite. (it’s a Scalar::Util warning) Luckily I had another box I had just updated and knew that the updated did in fact work.

here is a command line test, to check with:
% perl -e"use Scalar::Util qw( weaken );"
%

should return nothing, on my box it returned:
% perl -e"use Scalar::Util qw( weaken );"
Weak references are not implemented in the version of perl at -e line 1
BEGIN failed--compilation aborted at -e line 1.

I couldn’t find Scalar::Util installed!?
$ yum list|grep perl |grep -i scalar
perl-Scalar-Defer.noarch 0.20-1.el5.rf rpmforge
perl-Scalar-List-Utils.x86_64 1.21-1.el5.rf rpmforge
perl-Scalar-Properties.noarch 0.13-1.el5.rf rpmforge
perl-Set-Scalar.noarch 1.24-1.el5.rf rpmforge
perl-Tie-Scalar-Timeout.noarch 1.33-1.el5.rf rpmforge

After some searching, it appears that it’s part or perl core now.

Eventually, I stumbled onto:
% locate Util.pm
/usr/lib/perl5/5.8.8/CGI/Util.pm
/usr/lib/perl5/5.8.8/Hash/Util.pm
/usr/lib/perl5/5.8.8/List/Util.pm
/usr/lib/perl5/5.8.8/Scalar/Util.pm
/usr/lib/perl5/vendor_perl/5.8.8/CGI/Simple/Util.pm
/usr/lib/perl5/vendor_perl/5.8.8/Crypt/DSA/Util.pm
/usr/lib/perl5/vendor_perl/5.8.8/HTTP/Headers/Util.pm
/usr/lib/perl5/vendor_perl/5.8.8/Mail/Util.pm
/usr/lib/perl5/vendor_perl/5.8.8/Mail/SpamAssassin/Util.pm
/usr/lib/perl5/vendor_perl/5.8.8/Net/SFTP/Util.pm
/usr/lib/perl5/vendor_perl/5.8.8/Net/SSH/Perl/Util.pm
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/List/Util.pm
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/Scalar/Util.pm
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/APR/Util.pm
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/Apache/TestUtil.pm
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/Apache2/ConnectionUtil.pm
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/Apache2/RequestUtil.pm
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/Apache2/ServerUtil.pm
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/Apache2/Util.pm
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/ModPerl/MapUtil.pm
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/ModPerl/Util.pm
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/Params/Util.pm

The only difference in the results of this and the box that works,
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/List/Util.pm
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/Scalar/Util.pm are not in the list, of the working box.

(I have no idea how those got there, or where they are from, except for what the obviously look like, rhel5/centos packages,…)

perl INC is loading those libs first too.

$ perl -e 'print join "\n", @INC'
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.8
/usr/lib/perl5/site_perl
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.8
/usr/lib/perl5/vendor_perl
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi
/usr/lib/perl5/5.8.8

so to ‘resolve’ the issue, I did this.

% sudo mv /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/List/Util.pm /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/List/Util.pm-1.19-broken-with-5.8.8-27
% sudo mv /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/Scalar/Util.pm /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/Scalar/Util.pm-1.19-broken-with-5.8.8-27

This worked for me, although is probably not the ‘correct way’. I’m still not sure exactly what happened to get into this state, or I’d file a bug somewhere.

ref: (1)http://www.gossamer-threads.com/lists/modperl/modperl/98113

(1)http://rt.cpan.org/Public/Bug/Display.html?id=36346

(2) http://www.perlmonks.org/?node_id=424737 A different approach to the situation. (in general, we try to do everything through yum/rpm packages)
(2)http://bugs.centos.org/view.php?id=3022 an old bug, which made me realize I had to figure it out
(2) http://www.bluequartz.us/phpBB2/viewtopic.php?t=106577&sid=17481305416e016b01a061acb4ac3f45

When working from the command line in a remote terminal (black background xterm – ssh)

the default directory listing (ls) colors are dark blue, which is hard to read.

here is a hack to make them bright-ish yellow and easy to see. (NOTE this is the color FIFO and SOCK use, so if that is an issue for you, choose a different color)

cp /etc/DIR_COLORS ~/.dir_colors
vi ~/.dir_colors

change this line:
DIR 01;34 # directory

to this:
DIR 01;33 # directory

now log out of your session and log back in again.

viola!

NOTE: if you then work in a text editor such as vim, you may want to set a ligther color scheme as well :colorscheme desert should do it

(NOTE: This hack will change it on a per-user basis, you would have to modify the main /etc/ files for system wide changes. There is also probably a better way to do this,…)