Building Web UIs with Perl + Mojolicious

in perl

Mojolicious is one of 3 leading web frameworks available in the perl ecosystem (along with Dancer and Catalyst) and by far my favorite.

Mojolicious aims to provide a complete web development experience. It thus has no hard dependencies, comes with a built-in development and production server and many other features one needs to build a web application. It's easy to install, has an applciation generator script and many plugins and extensions.

The following examples will take you from building a first hello world app in Mojolicious to building complex interactive interfaces using web sockets and limiting access to privileged users only. I think (hope) by the end of this post you too will begin to like Mojolicious.

Hello Mojolicious

Mojolicious is distributed via cpan so we can install it with:

cpan Mojolicious

As I write this post the most recent version is 7.46, and that's what I'll use for all the examples.

Mojolicious offers 2 flavors of web apps: The full applications and Mojo lite apps. Although many code snippets online show the lite version, I prefer to use the full application as it's easier to grow.

After installation we can create a new application with the following command (from cmd or shell):

mojo generate app HelloWorld

The result is a new directory tree with the following files:

.
├── hello_world.conf
├── lib
│   ├── HelloWorld
│   │   └── Controller
│   │       └── Example.pm
│   └── HelloWorld.pm
├── public
│   └── index.html
├── script
│   └── hello_world
├── t
│   └── basic.t
└── templates
    ├── example
    │   ├── welcome.html.ep
    └── layouts
        └── default.html.ep

Mojolicious is an MVC framework, but it's model agnostic meaning Mojolicious will only provide you with a view and a controller. Any plain old perl package can function as a model.

Before we dive into the file structure, let's see everything worked well and start the application using:

morbo script/hello_world

Then aim your web browser to http://localhost:3000 and you should see the following message:

Welcome to the Mojolicious real-time web framework!

This page was generated from the template "templates/example/welcome.html.ep" and the layout "templates/layouts/default.html.ep", click here to reload the page or here to move forward to a static page. To learn more, you can also browse through the documentation here.

Let's follow what happened so we can start to modify the code to our needs. When the browser accessed http://localhost:3000, the request reached Mojo's router. Routes are defined in the startup file lib/HelloWorld.pm. Observe the following simplified snippet from that file:


sub startup {
  my $self = shift;
   
  my $r = $self->routes;

  $r->get('/')->to('example#welcome');
}

The last line defines a route. A route is a mapping between a URL and perl code to execute when that URL is accessed. Each route may be handled by two Mojolicious components: The first is called a Controller and it's written in perl; The second is called a Template and it's written in a special Mojolicious template language called "ep".

The sample route definition means a GET request to '/' should lead to calling the subrouting welcome from the controller Example. The template may be specified in that subroutine, or by default be inferred from the handler name (in our case it'll be example/welcome.html.ep).

Let's continue to the controller in file lib/HelloWorld/Controller/Example.pm, and its welcome subroutine:

sub welcome {
  my $self = shift;

  # Render template "example/welcome.html.ep" with message
  $self->render(msg => 'Welcome to the Mojolicious real-time web framework!');
}

The function does very little. It takes $self and calls its render method passing a message from the controller. Since template file was not explicitly specified, it is inferred automatically from the route to be example/welcome.html.ep.

Let's move on to look at that template file. All Mojolicious templates are saved by default in templates directory, so we're looking at the file templates/example/welcome.html.ep:

% layout 'default';
% title 'Welcome';
<h2><%= $msg %></h2>
<p>
  This page was generated from the template "templates/example/welcome.html.ep"
  and the layout "templates/layouts/default.html.ep",
  <%= link_to 'click here' => url_for %> to reload the page or
  <%= link_to 'here' => '/index.html' %> to move forward to a static page.
  % if (config 'perldoc') {
    To learn more, you can also browse through the documentation
    <%= link_to 'here' => '/perldoc' %>.
  % }
</p>

This file is interesting because it combines 3 types of commands: HTML markup tags, inline embedded perl (ep) and full ep commands.

HTML markup tags can appear anywhere in the file and are just printed to the resulting HTML document the browser receives.

Inline ep is content of the form <%= ... %>. Inside the special <%= %> markup we can write any perl code and that'll be evaluated and the result printed to the HTML file. Note that any variable you use here is looked up in the stash, which is a special hash passed from the controller. You already know one way to pass data in the stash, and that's to pass it to the controller's render method:

$self->render(msg => 'Welcome to the Mojolicious real-time web framework!');

Alternatively we could have written the following and got the same effect:

$self->stash(msg => 'Welcome to the Mojolicious real-time web framework!');
$self->render;

Back to the template file, The final type of command is the full ep statement such as:

% layout 'default';

This calls a special helper method (in the above case it's layout, but there are plenty). Layout helper wraps the template inside another template (called the layout). And you can find documentation on many default helpers in Mojolicious default helper guide.

The other helpers used in this file are link_to which creates an anchor element, url_for which automatically generates a URL according to the routes configuration and config which takes a value from a configuration file.

Who's There

It's about time we modify the template code and add some of our own. I'd like to add a page which shows the output of who command. We'll use a pre HTML tag to show the data.

The addition includes the following changes:

  1. Add a new subroutine in the controller which calls who and puts its output on the stash.
  2. Create a new template which shows the output of a command in a pre tag.
  3. Add a new route to be able to access the new code.

Let's go over the above one by one.

I'll first add a subroutine in the controller, let's call it who:

sub who {
  my $self = shift;
  $self->render(who => [`who`]);
}

Now we can move to the template. As we didn't pass an explicit template file name to render, it is inferred to be example/who.html.ep so we're looking at the file templates/example/who.html.ep:

% layout 'default';
% title 'Welcome';
<h2>Connected Users</h2>
%= t pre => join('', @$who)

<p><%= link_to 'Back Home', '/' %></p>

The template looks very similiar to welcome.html.ep, only now we use a pre tag to show the content of $who stash variable. Note the use of the tag helper in the line:

%= t pre => join('', @$who)

This is an ep shorthand for:

<pre><%= join('', @$who) %></pre>

And you can use it to produce any HTML tag.

Finally we need to add the route to lib/HelloWorld.pm. Add the following line to the end of startup subroutine:

$r->get('/who')->to('example#who')->name('who');

Run the server and aim your browser to http://localhost:3000. If all goes well, you should see the output of who command on your machine.

The full source code for this demo is available on github at:
HelloWorld Sources.

Now would be a good time to take a break and read the following Mojolicious plugin docs:

  1. Mojolicious list of Default Helpers.
  2. Mojolicious list of Tag Helpers.

Query Parameters

It's easy to imagine one wants to use a different layout for the output of who, for example perhaps show it in a nice table layout. For that we can probably use a different template.

Let's create the new template file, called a variant, and name it templates/example/who.html+table.ep. Content is as follows:

% layout 'default';
% title 'Welcome';
<h2>Connected Users</h2>

<table>
% for (@$who) {
%   my @columns = split;
%=  tag tr => begin
%   for (@columns) {
%=    tag td => $_
%   }
%   end
% }
</table>

<p><%= link_to 'Back Home', url_for('home') %></p>

Note the use of begin to tell tr that it should include all nested content. The result will be a <tr> and inside multiple <td> elements.

Back to the controller, we'll need to modify who function to use the new variant:

sub who {
  my $self = shift;

  $self->render(who => [`who`], variant => 'table');
}

But now that we have 2 variants, we may want to allow the user to deceide which template variant she prefers.

One way to pass parameters from the web browser to the framework is called query parameters. These are the things we write in the URL after a question mark, which look like a list of key/value pairs. When we point our browser to http://localhost:3000?v=pre the browser passes a param named v with the value pre. Alternatively, when we point our browser to http://localhost:3000?v=table the browser passes the same v parameter but now with the value table.

In the controller code we can access all query parameters using $self->param('name'). The following modification will take the variant from a query parameter called v:

sub who {
  my $self = shift;

  $self->render(who => [`who`], variant => $self->param('v'));
}

Note that if a variant is not found, Mojolicious will automatically fall back to the non-variant version of the template. After changing the controller and adding the template, visiting http://localhost:3000/who?v=table will show the data as a table.

Mojolicious Forms: Creating SVG Calendars

The module SVG::Calendar creates an SVG calendar for a given month and year. With Mojolicious we can create a web user interface that allows a user to select a month and a year and get a calendar for that date.

To follow along, create a new Mojolicious project and install SVG::Calendar with:

mojo generate app MyCal
cpan SVG::Calendar

And write the following subroutine in lib/MyCal/Controller/Example.pm:

use SVG::Calendar;

sub cal {
  my ($c) = @_;
  my $svg = SVG::Calendar->new( page => 'A4' );
  my $mon = $c->param('mon') || '10';
  my $year = $c->param('year') || '2017';
  my $dt = "$year-$mon";
  $c->render(data => $svg->output_month( $dt ), format => 'svg');
}

Almost everything in that method should be familiar, except for the last line: For the first time we render something that is not a template but an SVG file. Mojolicious will let you render just about anything with the right params passed to render. In the above example, format is used to set content type. There are many built-in formats that you can find in Mojolicious Types Doc.

Second let's create a form to let the user select a month and a year. Still in the controller add:

sub cal_form {
  my ($c) = @_;
  my @months = qw/Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec/;
  my $idx = 0;
  my $all_months = [ map { [ $_ => ++$idx ] } @months ];
  my $mon = $c->param('mon') || 01;
  my $year = $c->param('year') || 2017;
  $c->stash('months_for_select', $all_months);
  $c->stash('cal_img', "/cal.svg?mon=$mon&year=$year");
  $c->render;
}

The code is a bit more complex than previous examples but presents very little new syntax. We create two variables: cal_img which contains a URL with mon and year parameters set to the selected month and year; and months_for_select which is a nested array that includes all the months and their indices. We'll use this one to create a select box in the HTML.

Moving to the template, and a new helper is available. The file is templates/example/cal_form.html.ep:

% layout 'default';
% title 'Welcome';
<h2>Calendar Generator</h2>
<p>
%= form_for cal => begin
%= label_for mon => 'Month'
%= select_field mon => $months_for_select
%= label_for year => 'Year'
%= text_field year => 2017
%= submit_button
%= end

</p>
<p>
%= image $cal_img
</p>

The helper form_for creates a form. The name 'cal' helps Mojolicious decide the form's action attribute (which is the URL that the form will be submitted to).

Following a sequence of helpers to create labels and fields from the data, which I think are self explanatory. You can find more information about them in the Tag Helpers Guide.

Finally the routes part in file lib/MyCal.pm requires only two lines:

$r->get('/cal.svg')->to('example#cal');
$r->get('/cal')->to('example#cal_form');

Full code example for this project is available on Github At:
MyCal Project Source.

Form POST Parameters

Both parameter examples we saw so far were GET routes by nature, that is a user could bookmark and return to the result page, or reload the result page and get the same result.

Not all pages or interfaces work this way. Some operations have side effects that if you run the same operation multiple times you'll get different answers. Take for example creating a new user on the machine. It's generally not something you'd want to bookmark, because it's an action that should only be performed once with the given inputs (i.e. user name and parameters).

HTTP POST requests represent actions that have side effects and that generally should not happen more than once with the same input data. Before reloading a page that was received from a POST request the browser presents a dialog asking the user if he's sure he wants to perform that same action twice. In addition, POST requests won't be sent by accessing a bookmarked page so user can't perform them again by mistake.

In Mojolicious we handle POST request by adding a POST route instead of GET using the post method:

$r->post('/run')->to('example#run');

With the form_for helper we use method parameter to set the form's method attribute to POST:

%= form_for run => (method => 'POST') => begin
  %= text_field 'cmd'
  %= submit_button
% end

Let's combine the two and create a new project that allows running commands on the server via the web interface. Start with:

mojo generate app RemoteCommander

In the controller create a single method run with the following code:

sub run {
  my $c = shift;
  my $cmd = $c->param('cmd');
  $c->stash(result => [qx/$cmd/]);
  $c->render;
}

This code reminds us of a previous example that executed who and displayed its output, but this time we allow user to run any command they want. Keep in mind having such a service may be a major security vulnerability as it allows anyone with access to the web interface to run arbitrary commands on the server.

Next the template templates/example/run.html.ep file:

% layout 'default';
% title 'Welcome';
<h2>Result:</h2>
<p>
%= t pre => join('', @$result)
%= link_to 'home', url_for('home')
</p>

And in the file lib/RemoteCommander.pm add the following route line:

$r->post('/run')->to('example#run');

Now that we have our POST handler in place, we need a form so users could submit run requests. For the form a template file will suffice, so create a new file templates/example/prepare.htm.ep:

% layout 'default';
% title 'Welcome';
<h2>Run App</h2>
<p>
%= form_for run => (method => 'POST') => begin
  %= text_field 'cmd'
  %= submit_button
% end
</p>

The word cmd which appears after text_field is the name of the parameter that will be passed to run. Specifying (method => 'POST') makes the form use POST instead of GET, so the parameter values passed in the request body instead of in the URL.

Finally the route handler for prepare. Add the following line to the route definitions in lib/RemoteCommander.pm:

$r->get('/')->to('example#prepare')->name('home');

Full source code for the project is available on Github at:
RemoteCommander Source.

Using Websockets To Keep Data Updated

How It Works

Now we know how to run commands on the server, there's one more trick Mojolicious has in stock for us - and it's called web sockets.

You see the problem with our previous solution was that when calling long running processes we needed to wait for the process to finish before receiving its output. And for some processes that could mean a long wait.

Web sockets are a relatively new technology which provides bidirectional TCP socket between web server and browsers. Modern browsers support it, and so does Mojolicious.

To get the output of a long running process we're going to have to restructure our program in a slightly different manner. First, the process itself would be written as a Task using a task queue called Minion. This would allow starting the process outside the normal request/response cycle Mojolicious operates in.

Second we'll need to open a web socket from the browser to a progress endpoint on the server. That endpoint operates using a web socket and every time new output is printed from the task it will be streamed to the socket.

Third we'll need JavaScript code on the client to read progress events arriving from the socket and modify a view with the new data.

Combined together the code for all 3 is less complex than it sounds. I hope.

Start with creating a new Mojolicious app using:

mojo generate app SocketsDemo

For the example site I'll use a short bash script that takes a while to complete:

#!/bin/bash

word="$1"
while true
do
  word=$(echo $word | tr a-z b-za)
  echo $word
  [[ $word == "$1" ]] && break
  sleep 0.5 
done

The script uses tr to repeatedly change a word until it cycles and returns to the original word. I saved it in a file called utils/process.sh.

Define Background Tasks

I'll call this script as a Task using Minion task queue. Minion is great because it helps to separate long running tasks from normal request processing. It does require a backend to save the queue in. The example uses SQLite as it's the easiest to install.

Install both using:

cpanm Minion
cpanm Minion::Backend::SQLite

A Minion task is just a function that receives a $job object automatically from the queue and some parameters from the controller that created it. We define the task in the startup file lib/SocketsDemo.pm:

sub tr_task {
  my ($job, $cmd) = @_;
  $job->app->log->debug('Start process.sh');
  my ($escaped) = $cmd =~ /([a-zA-Z]+)/;
  my $line = "bash utils/process.sh $escaped >> tmp/result.log";
  $job->app->log->debug("Calling: $line");
  system($line);
}

Basically the task just calls utils/process.sh with the given word and sends the output to a tmp/result.log log file. Piping the output to a file will allow the websocket to easily send back the output to the user.

Later in that same file, inside startup function we need to add the following 2 lines to initialize Minion and install our task:

sub startup {
    my $self = shift;
    # ...
    $self->plugin(Minion => { SQLite => 'sqlite:jobs.db' });
    $self->minion->add_task(tr => \&tr_task);
}

Calling the task from the controller is also straightforward. This is the controller's run subroutine which starts the task:

sub run {
  my $c = shift;
  my $cmd = $c->param('cmd');
  
  $c->minion->enqueue('tr', [$cmd]);
  $c->stash('result' => $cmd . "\n");
  $c->render;
}

The only new syntax here is $c->minion->enqueue which takes a task name and parameters and adds it to the queue. A worker will later dispatch the item from the queue and perform it.

To get some working code before adding a web socket monitoring you'll need add the following route definitions to lib/SocketsDemo.pm:

$r->get('/')->to('example#prepare');
$r->post('/run')->to('example#run');

And create the two templates. First is templates/example/prepare.html.ep:

% layout 'default';
% title 'Welcome';
<h2>Run Long Command Demo</h2>

<p>
%= form_for run => (method => 'POST') => begin
%= text_field 'cmd'
%= submit_button
% end
</p>

Then templates/example/run.html.ep for the result:

% layout 'default';
% title 'Welcome';
<h2>Result</h2>

%= t pre => (id => 'result') => $result

Running Background Workers

To run the example you'll now need two mojo processes: one for the server and another for the background workers. Open two terminals and in the first start the server:

morbo script/sockets_demo

In the second start the background worker:

./script/sockets_demo minion worker

Adding Web Sockets

To add web sockets we need to modify 3 files: Naturally a new route should be added to lib/SocketsDemo.pm:

$r->websocket('/progress')->to('example#progress');

Second we should add JavaScript code to the result page that would connect to our newly defined web socket and receive data from it. It's easiest to add this JavaScript code directly in the template as follows (in file templates/example/run.html.ep):

% layout 'default';
% title 'Welcome';
<h2>Result</h2>

%= t pre => (id => 'result') => $result

<script>
const progressSocket = new WebSocket("ws://localhost:3000/progress");
const panel = document.querySelector('#result');

progressSocket.onmessage = function(event) {
  panel.textContent += event.data;
};
</script>

A websocket's onmessage callback gets called whenever the server sends new data. In our implementation the data is appended to the display panel.

And now the last and most complex part which is the controller's progress subroutine:

sub progress {
  my $c = shift;
  my $pid = open my $log, '-|', 'tail', '-f', 'tmp/result.log';
  die "Could't spawn: $!" unless defined $pid;
  my $stream = Mojo::IOLoop::Stream->new($log);
  $stream->on(read  => sub { $c->send({text => pop}) });
  $stream->on(close => sub { kill KILL => $pid; close $log });

  my $sid = Mojo::IOLoop->stream($stream);
  $c->on(finish => sub { Mojo::IOLoop->remove($sid) });
}

Normally perl code runs on a line-after-line basis with blocking IO operations. This means a call to readline sends the program to sleep until data from the file handle is read.

Our file handle points to a program, tail -f, and this program never ends. But progress subroutine has to end if we should ever send data to the client.

Mojolicious solution is to use non-blocking IO inside request/response cycle. This means any subroutine that is used to handle request should finish its work quickly, but it can register event handlers and "wake up" to do other stuff after what we'd call finish in a blocking world.

Put it another way, progress' work is not done when the subroutine ends. The subroutine just sets up some event handlers and returns, but when the events we listen on do occur, the registered handlers will be called and we'll be back sending progress notifications through the socket.

Event handlers are registered with the on function. In the code above we can see it happens 3 times:

The line $stream->on(read => sub { $c->send({text => pop}) }); assigns an event handler to the read event of the program stream. This means any time new input is received from tail that data will be sent as text to the web socket.

The line $stream->on(close => sub { kill KILL => $pid; close $log }); sets up an event handler to the close event of the program stream. This means when for any reason the code will stop listening to the program it'll be killed and the stream closed.

The last event handler in the line $c->on(finish => sub { Mojo::IOLoop->remove($sid) }); sets up an event handler on the websocket's finish event, which happens when the client leaves the page, or connection is disrupted for any other reason. Removing the stream from the event loop will then lead to closing it which will stop tail.

The full source code for this example is also available on Github at:
SocketsDemo Source.

Managing Users

All user interfaces we built so far were open to the general public. It may be a good idea to limit some services to just a set of certified users, or to require user registration before proceeding.

A Mojolicious plugin named Mojolicious::Plugin::Authentication can be useful in limiting parts of the site to members only.

The plugin is configured with 2 functions: load_user and validate_user. The first's job is to find and load user object based on a user id, and the second receives username and password and should check if the user is valid and return its id.

This is really useful if you save your users data in a DB or a file, because then validate_user can check if the user is real and provided the correct password, and load_user will load a record from the DB with some basic user info.

To simplify this example I won't use a DB but rather an in memory hash. Here's the implementation for these 2 functions:

# file: lib/MembersOnly.pm
use Mojolicious::Plugin::Authentication;
use Mojo::Util qw(secure_compare);
use MembersOnly::Controller::Members;

my %user_password = (
  rob    => 'red',
  dean   => 'ninja',
  brenda => 'secret',
  rita   => 'banana',
);

sub load_user {
  my ($app, $uid) = @_;

  exists $user_password{$uid} ? $uid : undef;
}

sub validate_user {
  my ($app, $username, $password, $extradata) = @_;
  if (exists($user_password{$username}) && secure_compare($password, $user_password{$username})) {
    return $username;
  }
  return;
}

Later in that file we can find our old friend startup function:

sub startup {
  my $self = shift;

  # Load configuration from hash returned by "my_app.conf"
  my $config = $self->plugin('Config');

  $self->plugin('authentication' => {
      'load_user'     => \&load_user,
      'validate_user' => \&validate_user,
    });

  # Router
  my $r = $self->routes;

  # Normal route to controller
  $r->get('/')->to('example#new_session');
  $r->post('/login')->to('example#create_session');

  # Members Only Area
  my $auth = $r->route('/members')->over(authenticated => 1);
  $auth->get('/')->to('members#index');
  $auth->get('logout')->to('members#signout')->name('logout');
}

The first part loads the plugin and provides it with our previously defined load_user and validate_user subroutines.

The second defines the routes and incudes two new concepts. Turns out we can use the return value of route as a starting point when defining new routes. This way all previous route settings are included in the new route.

In addition the function over allows to embed conditions in the route matching algorithm. The name authenticated is defined by Mojolicious::Plugin::Authentication and represents being an authenticated user.

Now in the above routing table authenticated users will be able to visit /members, which will load members#index and /members/logout which will load members#signout. Guest users will only be able to visit / and /login.

Authentication itself happens in lib/MembersOnly/Controller/Example.pm using the plugin:

sub create_session {
  my $c = shift;
  my $username = $c->param('username');
  my $password = $c->param('password');
  if ($c->authenticate($username, $password)) {
    $c->redirect_to($c->url_for('members'));
  } else {
    $c->stash('error', 'Invalid User Or Password');
    $c->render(template => 'example/new_session');
  }
}

Note how our application code is decoupled from user loading or validating process. This example uses a hash, but we can start using a DB without any change in any of the controller's or template's code.

Full source code for the application is in the same Github repo:
MembersOnly Source.

Security Considerations

All code examples in this post should be used with caution and in a controlled environment. They provide a way for anyone to run any command on the server, which is in itself a bad idea.

Regarding XSS - almost all Mojolicious tags escape special signs in variables. I didn't check but I think most examples above, including the ones that reflect user input to the HTML, are not susceptible to XSS attacks. Do let me know if I missed anything in the comments below.

Finally many code examples presented here suffer from shell injection attack. This is not too bad considering they're already allowing anyone to run any command, but keep in mind when relying on the code here that you should quote any parameters passed to the shell.

Exercise Ideas

Using the above examples it should be possible (and fun) to write the following applications:

  1. Write a web form that uses Data::Faker to let a user generate fake data. A user selects fields and types in a number (n) and the system prints n fake records in a table.

  2. Write a web form to control xeyes process on the server. The system should show an indication if the process is running, and allow user to start and stop the process.

  3. Write a simplified messenger between multiple users:

    • Users can register for new account or sign in with selected credentials
    • Each user is presented with a list of incoming messages
    • Users can write new messages to other users

Comments