Archive

Archive for the ‘perl’ Category

perl fun with encoding cp-1252

July 30th, 2010 dwright No comments

perl/ruby fun with encoding cp-1252 (windows encoding)

(In each of the following, you can substitute, 'perl' for 'ruby' and the example will work as well.)

[dwright~(74)]$ perl -le 'print "\xe2\x80\x9c"'

[dwright~(77)]$ perl -le 'print "\xe2\x80\xa6"'

[dwright~(78)]$ perl -le 'print "\xe2\x80\xa2"'

[dwright~(79)]$ perl -le 'print "\xe2\x80\xa3"'

[dwright~(80)]$ perl -le 'print "\xe2\x80\xa4"'

[dwright~(81)]$ perl -le 'print "\xe2\x80\xa5"'

[dwright~(82)]$ perl -le 'print "\xe2\x80\xa7"'

http://www.eki.ee/letter/chardata.cgi?ucode=2023
decimal: ‣
UTF-8 (e2, 80, a3)
[dwright~(79)]$ perl -le 'print "\xe2\x80\xa3"'

http://www.eki.ee/letter/chardata.cgi?ucode=2024
decimal:․
UTF-8 (e2, 80, a4)
[dwright~(80)]$ perl -le 'print "\xe2\x80\xa4"'

http://www.eki.ee/letter/chardata.cgi?ucode=2026
decimal: …
UTF-8 (e2, 80, a6)
[dwright~(83)]$ perl -le 'print "\xe2\x80\xa6"'

[dwright~(85)]$ perl -C2 -le 'print "\x{2026}"'

[dwright~(90)]$ perl -C2 -MHTML::Entities -le 'print decode_entities("…")'

[Notice, above is an Ellipses and not '...']

Axe smart quotes from string: (when string is unicode (perl's UTF8 flag is set)
i.e
after something such as:
use Encode qw/decode/;
$s = decode("UTF-8", $s);

$s =~ s/\x{201A}/,/g; # 0x201A Single low-9 quotation mark
$s =~ s/\x{201B}/`/g; # 0x201B Single high-reversed-9 quotation mark
$s =~ s/\x{201C}/"/g; # 0x201C Left double quotation mark
$s =~ s/\x{201D}/"/g; # 0x201D Right double quotation mark

Categories: UTF-8, Unicode, perl Tags:

Dictionary, word pairs code challenge

March 7th, 2010 dwright No comments

A few months ago, I came across a job posting with a code challenge to solve and submit with your resume.

I'm not in the job market but I generally like these sort of challenges and since the job is no longer listed, I will post what I came up with.

The Rules:
Given a dictionary, output all word pairs where both words
share all their letters except the last two, which are
distinct and reversed.

Notes:
- use a reasonable dictionary of your choice
- potential word pairs are: "ear, era" ; "revies, revise" ; "burglaries, burglarise"
- "shear, era" is not a valid pair because "she" != "e"
- "bell, bell" is not a valid pair because the last two letters are not distinct
- this will be benchmarked

Here is the final result I came up with, I don't have anything to compare it with, but I believe it's correct. (filter.pl)

# filter.pl
#to run:
#cat /usr/share/dict/words | perl filter.pl

use strict;
use warnings;
my $save_possible = {};

while (<>) {
   chomp $_;

   # lower case
   $_ = lc $_;

   # skip words smaller then 3 chars
   next unless length $_ >= 3;

   my $word_minus_2 = substr($_, 0, -2);
   my $last_2_chars_from_word = substr($_, length($_)-2, length($_));

   $save_possible->{$_} = $_;

   my $tmp = $_;
   # the last two chars of word, reversed
   my $last_2_chars_from_word_rev = chop($tmp);
   $last_2_chars_from_word_rev .= chop($tmp);

   if ($save_possible->{$word_minus_2.$last_2_chars_from_word_rev} #exists
       and ($_ ne $word_minus_2.$last_2_chars_from_word_rev)){#skip same word
      my $w2_rev=$save_possible->{$word_minus_2.$last_2_chars_from_word_rev};
      print "$_ -> $w2_rev\n";
     $save_possible->{$_} = undef; # skip dups
   }
}
real	0m0.992s
user	0m0.897s
sys	0m0.047s

Results verified with:

$ wc -l  /usr/share/dict/words
234936
$ time cat /usr/share/dict/words | perl filter.pl | sed -e 's/ ->.*//g' > OUT
$ wc -l OUT
565 OUT
$ grep -ciwf OUT  /usr/share/dict/words
565
=~ .2% of the words in this dict fit our criteria




Ok, now that I've posted that I will illustrate how the wrong assumptions can lead you down the wrong path, here was my original approach:

My assumption was that the cases would be consecutive.
i.e. (would exist lexicographically, in succession)
aab
aba

abel
able

ala
aal
...

So, my (as I thought, 'slick') idea was to use the Perl's native sort function, providing my own sorting routine, that should be very fast and I don't have to do much, great!

Here is my first draft in Perl: (yes, it's wrong - and ugly!)

#!/usr/bin/perl -w

#on my mac os x system one run's this as:
#$ cat /usr/share/dict/words | perl filter.pl

use strict;
use warnings;

my %possible = ();

sub _same {
   chomp $a;
   chomp $b;
   return -1 unless length $a > 1 && length $b > 2;

   my $last_two_a   = substr($a, length($a)-2, length($a));
   my $last_two_b_r = substr($b, length($b)-1, length($b));
   $last_two_b_r .= substr($b, length($b)-2, 1);

   if (((substr($a, 0, length($a)-2)
        cmp
         substr($b, 0, length($b)-2)) == 0)
        &&
       (($last_two_a cmp $last_two_b_r) == 0)){

         $possible{lc $a} = lc $b;
         return 0;
   }

   # we dont care about the returned sort, it's a byproduct
   return 1;
}

my @len_sort = sort _same <>;

foreach my $w (keys %possible){
  print "$w => $possible{$w}\n";
}
real    0m0.715s
user    0m0.660s
sys     0m0.053s

Great, results! and great performance, Well, that settles it, it must be working (haha).

One night a few weeks later, this popped into my head again (especially, since I had a feeling my approach was not accurate), so I revisited it.

Well, of course my 'ass'-umption was incredibly WRONG!

This approach does find some cases, I never bothered to really verify it, the quick eyeball said, 'geez, I would think there would be more, but ok,...'. (Ok, this is what I get for working past midnight all week)

After other attempts of trying to utilize Perl's native sort function. (I'll save you the from the example code, since at this point it's messy and complex, which alone is a red flag, since this should be simple!)

Basically by this point, it became clear that the sort method is not providing any benefit, is the wrong approach and is helping me to obfuscating the code in this specific situation.

That is when I came up with the approach I posted at the beginning. (I avoided doing this the first time because I thought, "I just need to compare consecutive words and how would I save the previous word, to compare with the next via an input stream" (i.e. without using global storage, saving all of them; and since I assumed they were consecutive, I would just provide a method to sort, which provides much for free. lazy! in a bad way). Had I realised, of course, that they are not consecutive I would have abandoned the sort method right off the bat.

Categories: perl Tags:

Perl XML well formed check

February 3rd, 2010 dwright No comments

Problem space:

Using Perl, you want to see if XML is 'valid'.

(it's actually just a 'well formed' check. To check validity, you would need to use the XML documents DTD, if one exists)

my $parser = XML::Parser->new( ErrorContext => 2 );
eval { $parser->parse( $data ); };
# report any error that stopped parsing
if( $@ ) {
    Carp::croak "ERROR: NOT WELL FORMED ELEMENT: $@\n";
}

Additionally, if you have a *NIX box available, xmllint should be installed, is available from the command line.

Categories: perl Tags:

Perl, Spreadsheet::ParseExcel, Parsing an XLS file with mixed encodings

February 3rd, 2010 dwright No comments

We are using the Perl module, Spreadsheet::ParseExcel->new to parse XLS files and insert the results into a UTF8 MySQL database.

I came across an interesting situation with a Microsoft Excel file, where only 3 fields of the 12 are encoded as UTF-16BE, the rest are latin1.

Parsing a file with mixed encoding returns unexpected results, since the data in the spreadsheet looks correct, but parsing in it one receives incorrect characters.

Here is some code using Encode::Guess which will resolve the mixed encoding issue.

my $enc = guess_encoding($text); # use Encode::Guess;
if (ref $enc) {
    $descr = decode($enc->name, $text);
}
else { # convert to utf8, does not turn on utf8 flag
  from_to($text, "iso-8859-1", "utf8");
}

or, if you are already expecting UTF8 data, or more specifically, data with the utf8 flag turned on (perl 5.8.* - see perldoc Encode)

use Encode qw/decode from_to/;
use Encode::Guess;
if (! utf8::is_utf8($val)) { # ok, its not utf8, what is it?
    my $enc = guess_encoding($val); # use Encode::Guess;
    if (ref $enc) {
        $val = decode($enc->name, $val); #turns on utf8 flag
    }
}

(you would have to use these on a field-by-field/val-by-val basis obviously)

CentOS 5.4 (Final) Weak references are not implemented in the version of perl

November 12th, 2009 dwright No comments

I just ran a sudo yum update on my development CentOS 5.3 box. (upgrade to 5.4)

All went well, except two issues.

1. I had to rollback perl-Apache-DBI.noarch 1.07-1.el5.rf
(luckily, I knew about this one, and had an old package available, and there are no deps issues - whew)

The issue manifests itself as:
[Tue Nov 10 10:43:02 2009] [error] Global $r object is not available. Set:\n\tPerlOptions +GlobalRequest\nin httpd.conf at /usr/lib/perl5/vendor_perl/5.8.8/Apache/DBI.pm line 144.\nCompilation failed in require at (eval 2) line 1.\n
[Tue Nov 10 10:43:02 2009] [error] Can't load Perl file: perl_load.pl for server , exiting...

To rollback:
$ sudo yum remove perl-Apache-DBI
$ sudo rpm -ivh perl-Apache-DBI-1.06-1.el5.rf.noarch.rpm

2. synopsis: "Weak references are not implemented in the version of perl "

The issue manifests itself as:
Weak references are not implemented in the version of perl at /usr/lib/perl5/vendor_perl/5.8.8/SOAP/Lite.pm line 2447\nBEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.8.8/SOAP/Lite.pm line 2447.\nCompilation failed in require,...
...
...
[Wed Nov 11 16:50:01 2009] [error] Can't load Perl file: perl_load.pl for server, exiting...

which seemed me to (incorrectly) initially look in SOAP::Lite. (it's a Scalar::Util warning) Luckily I had another box I had just updated and knew that the updated did in fact work.

here is a command line test, to check with:
% perl -e"use Scalar::Util qw( weaken );"
%

should return nothing, on my box it returned:
% perl -e"use Scalar::Util qw( weaken );"
Weak references are not implemented in the version of perl at -e line 1
BEGIN failed--compilation aborted at -e line 1.

I couldn't find Scalar::Util installed!?
$ yum list|grep perl |grep -i scalar
perl-Scalar-Defer.noarch 0.20-1.el5.rf rpmforge
perl-Scalar-List-Utils.x86_64 1.21-1.el5.rf rpmforge
perl-Scalar-Properties.noarch 0.13-1.el5.rf rpmforge
perl-Set-Scalar.noarch 1.24-1.el5.rf rpmforge
perl-Tie-Scalar-Timeout.noarch 1.33-1.el5.rf rpmforge

After some searching, it appears that it's part or perl core now.

Eventually, I stumbled onto:
% locate Util.pm
/usr/lib/perl5/5.8.8/CGI/Util.pm
/usr/lib/perl5/5.8.8/Hash/Util.pm
/usr/lib/perl5/5.8.8/List/Util.pm
/usr/lib/perl5/5.8.8/Scalar/Util.pm
/usr/lib/perl5/vendor_perl/5.8.8/CGI/Simple/Util.pm
/usr/lib/perl5/vendor_perl/5.8.8/Crypt/DSA/Util.pm
/usr/lib/perl5/vendor_perl/5.8.8/HTTP/Headers/Util.pm
/usr/lib/perl5/vendor_perl/5.8.8/Mail/Util.pm
/usr/lib/perl5/vendor_perl/5.8.8/Mail/SpamAssassin/Util.pm
/usr/lib/perl5/vendor_perl/5.8.8/Net/SFTP/Util.pm
/usr/lib/perl5/vendor_perl/5.8.8/Net/SSH/Perl/Util.pm
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/List/Util.pm
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/Scalar/Util.pm
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/APR/Util.pm
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/Apache/TestUtil.pm
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/Apache2/ConnectionUtil.pm
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/Apache2/RequestUtil.pm
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/Apache2/ServerUtil.pm
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/Apache2/Util.pm
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/ModPerl/MapUtil.pm
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/ModPerl/Util.pm
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/Params/Util.pm

The only difference in the results of this and the box that works,
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/List/Util.pm
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/Scalar/Util.pm
are not in the list, of the working box.

(I have no idea how those got there, or where they are from, except for what the obviously look like, rhel5/centos packages,...)

perl INC is loading those libs first too.

$ perl -e 'print join "\n", @INC'
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.8
/usr/lib/perl5/site_perl
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.8
/usr/lib/perl5/vendor_perl
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi
/usr/lib/perl5/5.8.8

so to 'resolve' the issue, I did this.

% sudo mv /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/List/Util.pm /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/List/Util.pm-1.19-broken-with-5.8.8-27
% sudo mv /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/Scalar/Util.pm /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/Scalar/Util.pm-1.19-broken-with-5.8.8-27

This worked for me, although is probably not the 'correct way'. I'm still not sure exactly what happened to get into this state, or I'd file a bug somewhere.

ref: (1)http://www.gossamer-threads.com/lists/modperl/modperl/98113

(1)http://rt.cpan.org/Public/Bug/Display.html?id=36346

(2) http://www.perlmonks.org/?node_id=424737 A different approach to the situation. (in general, we try to do everything through yum/rpm packages)
(2)http://bugs.centos.org/view.php?id=3022 an old bug, which made me realize I had to figure it out :)
(2) http://www.bluequartz.us/phpBB2/viewtopic.php?t=106577&sid=17481305416e016b01a061acb4ac3f45

Load quote content into ACME::QuoteDB

October 1st, 2009 dwright No comments

This is a example for how to load quotes content into the perl module, ACME::QuoteDB

(NOTE: use 0.1.2 or newer: http://search.cpan.org/~dvwright/ACME-QuoteDB-0.1.2/lib/ACME/QuoteDB.pm)
(There is a debian (squeeze/sid) package here: http://www.dwright.us/misc/libacme-quotedb-perl-0.1.2.tar.gz)

For this example we will be loading some quotes via crawling a website, Alan J. Perlis (no pun intended) quotes, available at http://www.cs.yale.edu/quotes.html

file: load_quotes.pl

#!perl

# subclass ACME::QuoteDB::LoadDB and override dbload to do our html parsing
package LoadQuoteDBFromHTML;
use base 'ACME::QuoteDB::LoadDB';
use strict;
use warnings;
use Carp qw/croak/;
use Data::Dumper qw/Dumper/;
use HTML::TokeParser;

sub dbload {
  my ($self, $file) = @_;

  my $p = HTML::TokeParser->new($file) || croak $!;

  while (my $token = $p->get_tag("p")) {
      my $data = $p->get_trimmed_text("p");
      # see $self->set_record in ACME::QuoteDB::LoadDB for attribute fields
      # to populate
      if ($data){
         # axe our beginning line nums: '110. "quote content",,..';
         $data =~ s/\A\d+\.\s+//xms;

         next if $data =~ m/EPIGRAMS IN PROGRAMMING/;
         next if $data =~ m/\AFrom ACM's SIGPLAN publication/;

         $self->set_record(quote => $data);
         $self->set_record(name => q{Alan J. Perlis});

         #$self->debug_record;
         $self->write_record;
      }
  }
}

package main;
use strict;
use warnings;

use ACME::QuoteDB::LoadDB;
use ACME::QuoteDB;
use Carp qw/croak/;
use LWP::Simple;

#my $source = q{Alan J. Perlis - "Epigrams in Programming"};
my $source = q{From ACM's SIGPLAN publication, (September, 1982), };
$source .= q{Article "Epigrams in Programming", };
$source .= q{ by Alan J. Perlis of Yale University};
my $file = '/tmp/q.html';
if (! -e $file){
  if (!is_success(getstore('http://www.cs.yale.edu/quotes.html', $file))){
    croak 'could not retrieve html page';
  }
}

my $load_db = LoadQuoteDBFromHTML->new({
                            file => $file,
                            file_format => 'html',
                            # overwrite all data, this is our first addition
                            create_db   => 1, # first run, create the db
                            # provide a category for all
                            category => [qw(Epigrams Programming)],
                            attr_source => $source,
                            # provide a general quality rating for all quotes
                            rating => 8,
                            # dont actually do anything,
                            #dry_run => 1,
                            #verbose => 1,
                        });

$load_db->data_to_db;

if (!$load_db->success){croak 'Something went wrong'}

my $sq = ACME::QuoteDB->new;
print $sq->get_quote();
#print join "\n", @{$sq->get_quotes({AttrName => 'Perlis', Limit => 3})};
print "\n\n";

if (120 ==  scalar @{$sq->get_quotes({AttrName => 'Perlis'})}){
    print "load complete\n";
}
else {
    print "unexpected amount of quotes loaded\n";
}

Which produces:
[dwright@debian perl]$ perl load_quotes.pl
If a listener nods his head when you're explaining your program, wake him up.
-- Alan J. Perlis

load complete
[dwright@debian perl]$ perl -MACME::QuoteDB -le 'print quote()'
There will always be things we wish to say in our programs that in all known languages can only be said poorly.
-- Alan J. Perlis

And of course my favorite Perlis quote:

#!perl

use strict;
use warnings;

use ACME::QuoteDB;

my $sq = ACME::QuoteDB->new;

print @{$sq->get_quotes_contain({ Contain =>  'parameters', })};
print "\n";

which produces:
If you have a procedure with ten parameters, you probably missed some.
-- Alan J. Perlis

Categories: perl Tags: , , , ,

debian .deb package for perl module criticism

September 14th, 2009 dwright No comments

I'm a big fan of The Perl Best Practices line of thinking stared by Damian Conway.

I am also a fan of the Perl::Critic school of modules available on the CPAN.

So in the tradition of contributing to open source, here is a debian/ubuntu package I just built for criticism

The one non-standard task I did was to build it without the Perl::Critic dependancy, as the docs state. See the bug I filed for details.

It's unsigned signed, and built on debian squeeze/sid, it contains some dependancies that were not covered by the perl modules versions in lenny.

there are a few lintian warnings. I should clean them up, and submit it to debian perl group proper. no lintian warnings

9/29/09: I filed an ITP bug http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=548991

use dh-make-perl to create a debian package from perl module.

September 12th, 2009 dwright No comments

Task: create a debian package for the CPAN perl module ACME-QuoteDB.

From the Docs:

ACME::QuoteDB − API implements CRUD for a Collection of Quotes (adages/proverbs/sayings/epigrams, etc)

This module provides an easy to use programmitic interface to a database (sqlite3 or mysql) of ’quotes’. (any content really, that can fit into our "defined format")

For simplicty you can think of it as a modern fancy perl version of fortune (with a management interface, remote database connection support, plus additional features and some not (yet) supported)

Supported actions include: (CRUD)

> Create
* Adding quote(s)
* ’Batch’ Loading quotes from a file (stream, other database, etc)

> Read
* Displaying a single quote, random or based on some criteria
* Displaying multiple quotes, based on some criteria
* Displaying a specific number of quotes, based on some search criteria

> Update
* Update an existing quote

> Delete
* Remove an existing quote

yeah, ok, whatever, that may be useful,...
so, let's get on with it.

$ wget http://search.cpan.org/CPAN/authors/id/D/DV/DVWRIGHT/ACME-QuoteDB-0.1.0.tar.gz
$ tar zxvf ACME-QuoteDB-0.1.0.tar.gz
$ tar zcvf libacme-quotedb-perl_0.1.0.orig.tar.gz ACME-QuoteDB
$ mv ACME-QuoteDB libacme-quotedb-perl-0.1.0
$ cd libacme-quotedb-perl-0.1.0
$ dh-make-perl

[ this failed first time becuase it could not find a Makefile.PL
the perl module author used Module::Build, which uses a Build.PL,
in place of a Makefile.PL and they did not include a Makefile.PL.
You can get around this by running:
$ perl Build.PL
$./Build dist (creates a Makefile.PL)
]
$ dh-make-perl (ran successfully now)

now edit files:
debian/changelog
debian/control
debian/copyright

$ debuild -us -uc
dpkg-buildpackage -rfakeroot -D -us -uc
dpkg-buildpackage: set CFLAGS to default value: -g -O2
dpkg-buildpackage: set CPPFLAGS to default value:
dpkg-buildpackage: set LDFLAGS to default value:
dpkg-buildpackage: set FFLAGS to default value: -g -O2
dpkg-buildpackage: set CXXFLAGS to default value: -g -O2
dpkg-buildpackage: source package libacme-quotedb-perl
dpkg-buildpackage: source version 0.1.0-1
dpkg-buildpackage: source changed by david wright
dpkg-buildpackage: host architecture i386
fakeroot debian/rules clean
dh_testdir
dh_testroot
dh_clean build-stamp install-stamp
# Add commands to clean up after the build process here
[ ! -f Build ] || /usr/bin/perl Build --allow_mb_mismatch 1 distclean
No such file: lib/ACME/QuoteDB/DB/quotes.db
Not in MANIFEST: debian/changelog
Not in MANIFEST: debian/compat
Not in MANIFEST: debian/control
Not in MANIFEST: debian/copyright
Not in MANIFEST: debian/rules
Not in MANIFEST: debian/watch
Not in MANIFEST: lib/ACME/QuoteDB/DB/quotes.db-ok
Not in MANIFEST: Makefile.PL
Not in MANIFEST: t/data/simpsons_quotes.tsv.csv-with-rating
MANIFEST appears to be out of sync with the distribution
Deleting blib
Deleting _build
Deleting Build
dpkg-source -b libacme-quotedb-perl-0.1.0
dpkg-source: info: using source format `1.0'
dpkg-source: info: building libacme-quotedb-perl using existing libacme-quotedb-perl_0.1.0.orig.tar.gz
dpkg-source: info: building libacme-quotedb-perl in libacme-quotedb-perl_0.1.0-1.diff.gz
dpkg-source: warning: ignoring deletion of file ACME-QuoteDB.pidaproject
dpkg-source: warning: ignoring deletion of file lib/ACME/QuoteDB/DB/quotes.db
dpkg-source: info: building libacme-quotedb-perl in libacme-quotedb-perl_0.1.0-1.dsc
debian/rules build
dh_testdir
# Add commands to compile the package here
/usr/bin/perl Build.PL installdirs=vendor
Creating custom builder _build/lib/MyModuleBuilder.pm in _build/lib
WARNING: the following files are missing in your kit:
lib/ACME/QuoteDB/DB/quotes.db
Please inform the author.

Checking whether your kit is complete...
Checking prerequisites...
Looks good

Creating new 'Build' script for 'ACME-QuoteDB' version '0.1.0'
/usr/bin/perl Build
Copying lib/ACME/QuoteDB.pm -> blib/lib/ACME/QuoteDB.pm
Copying lib/ACME/QuoteDB/LoadDB.pm -> blib/lib/ACME/QuoteDB/LoadDB.pm
Copying lib/ACME/QuoteDB/DB/Category.pm -> blib/lib/ACME/QuoteDB/DB/Category.pm
Copying lib/ACME/QuoteDB/DB/Attribution.pm -> blib/lib/ACME/QuoteDB/DB/Attribution.pm
Copying lib/ACME/QuoteDB/DB/QuoteCatg.pm -> blib/lib/ACME/QuoteDB/DB/QuoteCatg.pm
Copying lib/ACME/QuoteDB/DB/DBI.pm -> blib/lib/ACME/QuoteDB/DB/DBI.pm
Copying lib/ACME/QuoteDB/DB/Quote.pm -> blib/lib/ACME/QuoteDB/DB/Quote.pm
Manifying blib/lib/ACME/QuoteDB/LoadDB.pm -> blib/libdoc/ACME::QuoteDB::LoadDB.3pm
Manifying blib/lib/ACME/QuoteDB/DB/QuoteCatg.pm -> blib/libdoc/ACME::QuoteDB::DB::QuoteCatg.3pm
Manifying blib/lib/ACME/QuoteDB.pm -> blib/libdoc/ACME::QuoteDB.3pm
Manifying blib/lib/ACME/QuoteDB/DB/Attribution.pm -> blib/libdoc/ACME::QuoteDB::DB::Attribution.3pm
Manifying blib/lib/ACME/QuoteDB/DB/Category.pm -> blib/libdoc/ACME::QuoteDB::DB::Category.3pm
Manifying blib/lib/ACME/QuoteDB/DB/Quote.pm -> blib/libdoc/ACME::QuoteDB::DB::Quote.3pm
Manifying blib/lib/ACME/QuoteDB/DB/DBI.pm -> blib/libdoc/ACME::QuoteDB::DB::DBI.3pm
/usr/bin/perl Build test
t/00-load...................# Testing ACME::QuoteDB 0.1.0, Perl 5.010000, /usr/bin/perl
ok
t/01-load_quotes............ok
t/02-get_quotes.............ok
t/03-load_quotes_env........ok
t/04-get_quotes_more........ok
t/04-load_get_quote_utf8....ok
t/05-load_quotes_remote.....ok
t/boilerplate...............ok
t/pod-coverage..............ok
t/pod.......................ok
All tests successful.
Files=10, Tests=128, 22 wallclock secs ( 9.49 cusr + 0.76 csys = 10.25 CPU)
touch build-stamp
fakeroot debian/rules binary
dh_testdir
dh_testroot
dh_clean -k
# Add commands to install the package into /tmp/t/libacme-quotedb-perl-0.1.0/debian/libacme-quotedb-perl here
/usr/bin/perl Build install destdir=/tmp/t/libacme-quotedb-perl-0.1.0/debian/libacme-quotedb-perl create_packlist=0
Installing /tmp/t/libacme-quotedb-perl-0.1.0/debian/libacme-quotedb-perl/usr/share/perl5/ACME/QuoteDB.pm
Installing /tmp/t/libacme-quotedb-perl-0.1.0/debian/libacme-quotedb-perl/usr/share/perl5/ACME/QuoteDB/LoadDB.pm
Installing /tmp/t/libacme-quotedb-perl-0.1.0/debian/libacme-quotedb-perl/usr/share/perl5/ACME/QuoteDB/DB/Quote.pm
Installing /tmp/t/libacme-quotedb-perl-0.1.0/debian/libacme-quotedb-perl/usr/share/perl5/ACME/QuoteDB/DB/Category.pm
Installing /tmp/t/libacme-quotedb-perl-0.1.0/debian/libacme-quotedb-perl/usr/share/perl5/ACME/QuoteDB/DB/DBI.pm
Installing /tmp/t/libacme-quotedb-perl-0.1.0/debian/libacme-quotedb-perl/usr/share/perl5/ACME/QuoteDB/DB/QuoteCatg.pm
Installing /tmp/t/libacme-quotedb-perl-0.1.0/debian/libacme-quotedb-perl/usr/share/perl5/ACME/QuoteDB/DB/Attribution.pm
Installing /tmp/t/libacme-quotedb-perl-0.1.0/debian/libacme-quotedb-perl/usr/share/perl5/ACME/QuoteDB/DB/quotes.db
Installing /tmp/t/libacme-quotedb-perl-0.1.0/debian/libacme-quotedb-perl/usr/share/man/man3/ACME::QuoteDB::DB::Category.3pm
Installing /tmp/t/libacme-quotedb-perl-0.1.0/debian/libacme-quotedb-perl/usr/share/man/man3/ACME::QuoteDB::LoadDB.3pm
Installing /tmp/t/libacme-quotedb-perl-0.1.0/debian/libacme-quotedb-perl/usr/share/man/man3/ACME::QuoteDB::DB::QuoteCatg.3pm
Installing /tmp/t/libacme-quotedb-perl-0.1.0/debian/libacme-quotedb-perl/usr/share/man/man3/ACME::QuoteDB::DB::Attribution.3pm
Installing /tmp/t/libacme-quotedb-perl-0.1.0/debian/libacme-quotedb-perl/usr/share/man/man3/ACME::QuoteDB::DB::DBI.3pm
Installing /tmp/t/libacme-quotedb-perl-0.1.0/debian/libacme-quotedb-perl/usr/share/man/man3/ACME::QuoteDB::DB::Quote.3pm
Installing /tmp/t/libacme-quotedb-perl-0.1.0/debian/libacme-quotedb-perl/usr/share/man/man3/ACME::QuoteDB.3pm
touch install-stamp
dh_testdir
dh_testroot
dh_installdocs README
dh_installchangelogs Changes
dh_perl
dh_compress
dh_fixperms
dh_installdeb
dh_gencontrol
dpkg-gencontrol: warning: unknown substitution variable ${misc:Depends}
dh_md5sums
dh_builddeb
dpkg-deb: building package `libacme-quotedb-perl' in `../libacme-quotedb-perl_0.1.0-1_all.deb'.
dpkg-genchanges >../libacme-quotedb-perl_0.1.0-1_i386.changes
dpkg-genchanges: including full source code in upload
dpkg-buildpackage: full upload (original source is included)
Now running lintian...
W: libacme-quotedb-perl: manpage-has-errors-from-man usr/share/man/man3/ACME::QuoteDB.3pm.gz 950: warning [p 12, 5.7i]: can't break line
W: libacme-quotedb-perl: old-fsf-address-in-copyright-file
W: libacme-quotedb-perl: copyright-contains-dh-make-perl-boilerplate
E: libacme-quotedb-perl: description-too-long
W: libacme-quotedb-perl: spelling-error-in-description mysql MySQL
W: libacme-quotedb-perl: new-package-should-close-itp-bug
Finished running lintian.

Ok, now let's clean up those 'lintian' errors.

we are mainly concerned with fixing the lintian error 'E'
but we'll clear up all we can.

1. manpage-has-errors-from-man
skiping, cant control manpage now.

2.old-fsf-address-in-copyright-file
http://lintian.debian.org/tags/old-fsf-address-in-copyright-file.html

3. copyright-contains-dh-make-perl-boilerplate
self explanitory.

4. description-too-long
http://lintian.debian.org/tags/description-too-long.html
The first line of the "Description:" must not exceed 80 characters.

5. spelling-error-in-description mysql MySQL
self explanitory.

6. new-package-should-close-itp-bug
http://lintian.debian.org/tags/new-package-should-close-itp-bug.html
isn't relavent in this situation becuase there is no bug request for this package.

ok, cleared what we could up, re-run

$ debuild -us -uc
Now running lintian...
W: libacme-quotedb-perl: manpage-has-errors-from-man usr/share/man/man3/ACME::QuoteDB.3pm.gz 950: warning [p 12, 5.7i]: can't break line
W: libacme-quotedb-perl: new-package-should-close-itp-bug
Finished running lintian.

ok, better

built on
debian stable - lenny
cat /etc/debian_version
5.0.3

$ tar zcvf libacme-quotedb-perl-0.1.0.tar.gz libacme-quotedb-perl_0.1.0-1*
libacme-quotedb-perl_0.1.0-1_all.deb
libacme-quotedb-perl_0.1.0-1.diff.gz
libacme-quotedb-perl_0.1.0-1.dsc
libacme-quotedb-perl_0.1.0-1_i386.build
libacme-quotedb-perl_0.1.0-1_i386.changes

which you can find for download here:
http://www.dwright.us/misc/libacme-quotedb-perl-0.1.0.tar.gz

$ sudo dpkg -i libacme-quotedb-perl_0.1.0-1_all.deb

update Sat Sep 26 01:04:23 PDT 2009: the libacme-quotedb-perl_0.1.0-1_all.deb package was not properly created, when installed, one could not write to the database, that is corrected now with this version: (libacme-quotedb-perl_0.1.2)
http://www.dwright.us/misc/libacme-quotedb-perl-0.1.2.tar.gz

$ sudo dpkg -i libacme-quotedb-perl_0.1.2-1_all.deb
proof of the test db installed by default:
$ perl -MACME::QuoteDB -le 'print quote()'
나는 유리를 먹을 수 있어요. 그래도 아프지 않아요
-- I can eat grass (Korean)

this version is signed and was built on debian: squeeze/sid (which is currently testing; lenny is stable)

references:
http://www.debian-administration.org/articles/78
http://www.debian-administration.org/articles/78#comment_9
http://www.debian.org/doc/packaging-manuals/perl-policy/
http://www.debian.org/doc/maint-guide/

Ruby on rails seemless integration with legacy perl application

August 27th, 2009 dwright No comments

Comment:

This kind of set up should also work with an php, cgi (or any language, running on apache) site.

Overview:

I am (slowly) migrating a (large) legacy perl/cgi application to ruby on rails, the base requirement is that the (legacy) site continues to behave 'as expected' to the end user.

So, I need to run these two applications simultaneously from the same server and have it look like one application to clients. (i.e. transparent integration)

Existing links/sessions need to continue to work seamlessly.  I decided to go the route of keeping the legacy app functioning as before, so for instance http://example.com/admin/admin.pl will still work as expected. All new development (which is in rails) will be located at http://example.com/admin2/

The first thing I needed to do was handle all session/cookies info with rails (I will do a separate post on that another time, as it is involved, although since we use sessions via a database it wasn't that bad)

Setup:
We are using apache2, enterprise ruby and phusion passenger.

Conf:
NOTE: we will use example.com for our http host for demonstration purposes


Ruby on Rails vhost conf file:

# we are now handling *ALL* http requests with mod_rails
<VirtualHost *:80>

# set server name/admin
ServerAdmin you@example.com
ServerName example.com

# be sure to point to 'public'
DocumentRoot "/usr/local/httpd/
admin2/public"
ScriptAlias /admin2/ "/usr/local/httpd/admin2/public/"

RewriteEngine on
RewriteLog "/var/log/httpd/rewrite.log"
RewriteLogLevel 5

RewriteRule ^/$ /admin2/session [R,L]
RewriteRule ^/index.html$ /
admin2/session [R,L]

# pass all images to legacy perl/cgi
RewriteRule ^(.*)(\.)(gif|jpg|png)$ http://example.com:8080$1$2$3 [proxy]

# pass/reroute legacy css/js requests
RewriteRule ^/css/(.*) /stylesheets/$1 [R,L]
RewriteRule ^/js/(.*) /javascripts/$1 [R,L]

### send any /admin/ page request to perl/cgi on port 8080
# /admin/ requests have not yet been migrated to rails
#http://httpd.apache.org/docs/2.0/mod/mod_proxy.html#proxyrequests
ProxyRequests Off
ProxyPreserveHost On
#15 minutes
ProxyTimeout 900

<Proxy *>
Order deny,allow
Allow from all
<Proxy>

# all requests for /admin go to legacy app
ProxyPass /admin/ http://example.com:8080/admin/
#('hides' the port)
ProxyPassReverse /admin/ http://example.com:8080/admin/

SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1
</VirtualHost>


If you have some ajax requests that take 'awhile', you may need to add the:
ProxyTimeout 900
SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1

On some (long running) monthly report type queries we were getting:
Bad Gateway
The proxy server received an invalid response from an upstream server.

and this in apache logs:
[Thu Jun 11 10:58:08 2009] [error] (70014) End of file found: proxy: error reading status line from remote server


legacy perl/cgi vhost file:
# this would be the vhost file for the http://example.com:8080 host above
# we are now running perl/cgi on port 8080, mod_rails is handling
# all http requests
Listen 8080

<VirtualHost *:8080>
DocumentRoot /usr/local/httpd/htdocs

ServerName example.com
AddHandler cgi-script .pl

ScriptAlias /admin/ "/usr/local/httpd/htdocs/"
Alias /images/ "/usr/local/httpd/htdocs/images/"

RewriteEngine on

### handle with rails (/admin2/)
### 'any session page requests'
RewriteRule ^/$ http://%{HTTP_HOST}/admin2/session [L,R]
RewriteRule ^/index.html$ http://%{HTTP_HOST}/
admin2/session [L,R]
RewriteRule ^/admin/session.pl$ http://%{HTTP_HOST}/
admin2/session [L,R]

#### apps we have migrated to be handled with rails need a rule here
# passwd.pl is first
RewriteRule ^/admin/passwd.pl$ http://%{HTTP_HOST}/
admin2/passwd [L,R]
</VirtualHost>


To rollback (or for testing. to disable the rails site) we can just comment out the rails vhost file and update the legacy vhost file to:
#Listen 8080
#<VirtualHost *:80>
<VirtualHost *:80>

ref:
Integrating Ruby on Rails with existing PHP or Perl Based Sites
http://work.rowanhick.com/2008/04/10/rails-php-sharing-the-same-session/

perl command line fu (awk-ish)

August 6th, 2009 dwright No comments

Here's a neat trick I didn't know about to have perl behave in a way that is very 'awk-ish', nice addition  to boost your command line perl-fu:

$> cat file.txt
Name Age Number
xyz 22 123456
abc 21 234567

$> perl -F' ' -lane 'print $F[1]' file.txt
Age
22
21

$> perl -F' ' -lane 'print $F[0]' file.txt
Name
xyz
abc

$> perl -F' ' -lane 'print $F[2]' file.txt
Number
123456
234567

(The awk equivalent is awk '{print $2'} file.txt)

Categories: perl Tags: , ,