WWW::Mechanize::Firefox

Max Maischein

Frankfurt.pm

Overview

  • Why WWW::Mechanize::Firefox?

  • What is WWW::Mechanize::Firefox?

  • Applications

Who am I?

  • Max Maischein

  • DZ BANK Frankfurt

  • Deutsche Zentralgenossenschaftsbank

  • Information management

My Leitmotiv

Automation

  • If I can do it manually

  • ... then the computer can repeat it

  • ... correctly every time

My tools

  • Perl (of course)

  • WWW::Mechanize

  • WWW::Mechanize::Shell (GPW 2002)

  • Win32::OLE (GPW 2007)

  • ... not enough anymore

Web 2.0

  • Web applications are on the rise

  • Web 2.0 is problematic

  • Applications hold status in the client

  • ... use Javascript

  • Perl is not good at Javascript

Javascript

  • Javascript::SpiderMonkey by Mike Schili, Thomas Busch on CPAN

  • Mozilla Javascript Interpreter

  • Many external prerequisites

  • Only Javascript, no DOM

Javascript (II)

  • Javascript::Engine by Father Chrysostomos on CPAN

  • Pure Perl

  • Slooow

  • WWW::Mechanize::Javascript, WWW::Scripter

Firefox

What if I could use Firefox?

  • Accepted Platform

  • Compatible Platform

  • Interactive Platform

Communication Firefox/Perl

  • Send mouse clicks?

  • Send key presses?

  • not interactive

  • not robust if I'm using the machine at the same time

mozrepl to the rescue

Mozrepl opens a network interface into Firefox

Perl knows TCP

WWW::Mechanize::Firefox

 1:  my $mech = WWW::Mechanize::Firefox->new();
 2:  $mech->get('http://conferences.yapceurope.de/ye2010');

What is WWW::Mechanize::Firefox

WWW::Mechanize::Firefox provides

  • an extended API

  • of WWW::Mechanize

  • using Firefox as backend

Features

  • Normal WWW::Mechanize API

  • Javascript

  • CSS selectors (via HTML::Selector::XPath)

  • XPath selectors

  • Javascript events (onLoad)

  • Javascript error messages!

Uses of WWW::Mechanize::Firefox

  • Automation of websites

  • Integrated unit tests of Javascript

  • Input validation with JS code, even server-side

  • Crazy things

Live demo

Control Firefox

 1:  my $mech = WWW::Mechanize::Firefox->new();
 2:  $mech->get('http://conferences.yapceurope.org/ye2010');

Live demo

Control Firefox

01-open-local.pl

 1:  my $mech = WWW::Mechanize::Firefox->new();
 2:  $mech->get_local('file.html');

Live demo

Listen to Firefox events

02-urlbar.pl

Live demo

Listen to Firefox events

02-urlbar.pl

Live demo

Test usability of websites

03-dump-links.pl

 1:  my $mech = WWW::Mechanize::Firefox->new();
 2:  $mech->get_local('link.html');
 3:
 4:  sleep 5;
 5:  $mech->highlight_node(
 6:      $mech->selector('a.download'));
 7:  
 8:  print $_->{href}, " - ", $_->{innerHTML}, "\n"
 9:      for $mech->selector('a.download');

Live demo

Execute Javascript in a website

04-javascript.pl

 1:  my $mech = WWW::Mechanize::Firefox->new();
 2:  $mech->get_local('datei.html');
 3:  $mech->eval_in_page(<<'JS');
 4:      alert('Hello YAPC Europe');
 5:  JS

Live demo

Manipulate a website:

 1:  var secret = "12345"  + "secret";

Live demo

Manipulate a website:

 1:  var secret = "12345"  + "secret";

Read Javascript variables (and set them)

Live demo

05-manipulate-javascript.pl

 1:  my $mech = WWW::Mechanize::Firefox->new();
 2:  $mech->get_local('javascript.html');
 3:
 4:
 5:  my ($val,$type) = $mech->eval_in_page(<<'JS');
 6:      secret
 7:  JS
 8:
 9:
10:  print "The password ist $val";
11:
12:  # Set the value
13:  sleep 5;
14:  $mech->value('pass',$val);

Screenshots for documentation/logging

06-screenshot.pl

 1:  my $png = $mech->content_as_png()

Prerequisites of WWW::Mechanize::Firefox?

  • Firefox

  • mozrepl Add-On

  • MozRepl::RemoteObject (+ MozRepl + Net::Telnet)

  • WWW::Mechanize

What's missing?

  • (A)Synchronous event model

  • Asynchronous communication (AnyEvent)

  • More encapsulated Firefox API (MDC)

  • More WWW::Mechanize API

  • Firefox without a display ("headless")

  • DOM without Firefox (js_env), V8 Javascript Engine

Crazy things

  • Open browser windows or close them

  • Automate download dialogs

  • File conversion

  • HTML to PNG

  • SVG to PNG

  • JPG to PNG

More crazy things

  • Firefox as charting-engine using Flot(r)

  • Instead of Inkscape

  • Remotely control Firefox on other computers (TCP)

  • Firefox as media converter

  • Movie Resize / Video Element

  • Webcam in Perl?

Sample code

All samples are online on CPAN in

WWW::Mechanize::Firefox::Examples

Thank you

Questions?

Bonus - Using W:M:F right here!

Reauthenticate to the Wifi using W:M:F

Bonus - Using W:M:F right here!

Reauthenticate to the Wifi using W:M:F

  • Check whether http://google.it is reachable

  • if not

  • Switch off the SOCKS proxy

  • Connect to http://google.it

  • Enter credentials

  • Submit (does some Javascript)

  • Launch PuTTY for ssh tunneling

  • Re-enable SOCKS proxy

FF Settings

 1:  # Get Firefox settings
 2:  my $prefs = $mech->repl->expr(<<'JS');
 3:  Components.classes["@mozilla.org/preferences-service;1"]
 4:    .getService(Components.interfaces.nsIPrefBranch);
 5:  JS

FF Proxy Settings

 1:  # Get Firefox settings
 2:  my $prefs = $mech->repl->expr(<<'JS');
 3:  Components.classes["@mozilla.org/preferences-service;1"]
 4:    .getService(Components.interfaces.nsIPrefBranch);
 5:  JS
 6:
 7:  # Switch off the SOCKS proxy
 8:  if ($prefs->getIntPref('network.proxy.type') != 0) {
 9:      $prefs->setIntPref('network.proxy.type',0);
10:  };

Reauthenticate to DevItalia

 1:  $mech->get('http://google.it');
 2:  if ($mech->title =~ /\bdevitalia\b/i) {
 3:    # We need to log in
 4:    print "Reauthenticating";
 5:    $mech->field( User_tmp => $credentials->{user} );
 6:    $mech->field( Password => $credentials->{pass} );
 7:    $mech->click({ xpath => q{//input[@type="submit"]}, sync => 1 });
 8:  } else {
 9:    print "Already logged in"
10:  };

Switch proxy back on

 1:  $mech->get('http://google.it');
 2:  if ($mech->title =~ /\bdevitalia\b/i) {
 3:    # We need to log in
 4:    print "Reauthenticating";
 5:    $mech->field( User_tmp => $credentials->{user} );
 6:    $mech->field( Password => $credentials->{pass} );
 7:    $mech->click({ xpath => q{//input[@type="submit"]}, sync => 1 });
 8:  } else {
 9:    print "Already logged in"
10:  };
11:
12:  # Switch on the SOCKS proxy
13:  $prefs->setIntPref('network.proxy.type',1);