Automating interactive programs execution with Expect

To automate deployments and administration, we need non-interactive programs. This is because we want to be able to apply something like an Ansible playbook, or a Puppet manifesto, and see the changes propagating to any number of remote host. If we had to manually provide an input to each host, automation simply wouldn’t work.

Some programs are interactive by nature, but can still be executed in a non-interactive way. apt-get is an example: inside a script, we want to call it as apt-get install -y, so it won’t ask us to confirm. Another example is the mariadb client: it is a splendid interactive tool, but in the script we call it as mysql -e 'SELECT COUNT(*) FROM information_schema.SCHEMATA;' to run a single query.

But some programs are interactive and provide no way to call them non-interactively. In that case, we can make them non-interactive with the Expect language. It’s not always trivial, but it’s not Rocket science neither. This article shows how we I did that in a real-world case.

The theory

Expect is an interpreted language based on TCL, that aims to produce interactions with other processes. It can be a powerful tool for automation and testing. So you could have an Expect script that launches the mariadb client, runs some commands as if it were a human user, and optionally validates the output, or does something more complex with it.

While Expect claims to be simple, I hope I’ll never have to manually write some Expect code myself. The good news is that, normally, you don’t have to. Instead, you run autoexpect, you pass it the command it should run, autoexpect runs it, and you interact with the program. When the program terminates, autoexpect produces an Expect script that you can use to repeat the same exact operation.

The problem

We’re going to see how it works with a simple example. Spoiler: not everything will work as expected, but fixing problems is easy.

We created vagrant-immudb, a Vagrantfile to create a couple of Vagrant boxes (vettabase/immudb and vettabase/immudb-immugw) running immudb and related software, for the purpose of testing it.

immudb is a great technology, take a look at my article First impressions about immudb, the immutable database. But surely it wasn’t created with automation in mind. One of the consequences is that some programs are interactive, meaning that after launching them you are supposed to type something. This was a problem for our Vagrantfile: we needed to login, optionally change the superuser password, and optionally create databases and users configured in a YAML file.

With immudb, before running any administrative command (like creating databases or users), you need to login via immuadmin and it’s a good idea to change the default password:

vagrant@ubuntu-bionic:~$ immuadmin login immudb
Password:
logged in
SECURITY WARNING: immudb user has the default password: please change it to ensure proper security
Choose a password for immudb:
Confirm password:
immudb's password has been changed

As you can easily imagine, a Vagrantfile cannot login in that way. Unless it uses Expect.

The solution

Let’s start by launching autoexpect and doing the login manually:

vagrant@ubuntu-bionic:~$ autoexpect -f first_login.exp immuadmin login immudb
autoexpect started, file is first_login.exp
Password:
logged in
autoexpect done, file is first_login.exp

Great! Let’s see the file that autoexpect generated (comments removed):

#!/usr/bin/expect -f

set force_conservative 0  ;# set to 1 to force conservative mode even if
			  ;# script wasn't run conservatively originally
if {$force_conservative} {
	set send_slow {1 .1}
	proc send {ignore arg} {
		sleep .1
		exp_send -s -- $arg
	}
}

set timeout -1
spawn immuadmin login immudb
match_max 100000
expect -exact "Password:"
send -- "immudbr"
expect -exact "r
logged inr
[33mSECURITY WARNING: immudb user has the default password: please change it to ensure proper securityr
[0mChoose a password for immudb:"
send -- "POpo123.r"
expect -exact "r
Confirm password:"
send -- "POpo123.r"
expect eof

Now let’s port the file to a new virtual machine and let’s try to run it:

vagrant@ubuntu-bionic:~$ ./first_login.exp 
spawn immuadmin login immudb
Password:missing close-bracket
    while executing
"expect -exact "r
logged inr
[33mSECURITY WARNING: immudb user has the default password: please change it to ensure proper securityr
["
    (file "./first_login.exp" line 18)

Oh, no!

The problem is that the script expects “Password:”, then it sends the default password, and then… it expects another string, but the next output is a yellow text, and shell formatting seems to be unsupported by Expect, or trigger an Expect bug.

Let’s make the script simpler, let’s not say it what to expect. Let’s modify the file by removing an expect statement (this is only the last part of the script):

spawn immuadmin login immudb
match_max 100000
expect -exact "Password:"
send -- "immudbr"

# don't know what to expect?
# oh well, life is full of surprises...

send -- "$pwdr"
expect -exact "r
Confirm password:"
send -- "$pwdr"
expect eof

It works like magic! But don’t exceed, don’t remove all expect just because it looks easier or you may just break the script. For example removing expect eof makes the script hang forever.

I also wanted another version of the first login script, which doesn’t change the password. This is possible because, if you enter the old password instead of the new one, immuadmin stops with an error, but only after logging it. Here’s the new version (only the last part):

expect -exact "Password:"
send -- "immudbr"
send -- "immudbr"
expect eof

Finally, we may want to use a username and a password from environment variables. This is the case with the immu_create_user.exp script, that looks like this:

#!/usr/bin/expect -f

set usr [lindex $argv 0];
set pwd [lindex $argv 1];

set force_conservative 0  ;# set to 1 to force conservative mode even if
			  ;# script wasn't run conservatively originally
if {$force_conservative} {
	set send_slow {1 .1}
	proc send {ignore arg} {
		sleep .1
		exp_send -s -- $arg
	}
}

set timeout -1
spawn immuadmin user create $usr read defaultdb
match_max 100000
expect -exact "Choose a password for $usr:"
send -- "$pwdr"
expect -exact "r
Confirm password:"
send -- "$pwdr"
expect eof

Now you get the idea.

Expect can do much more than this, but currently I only use it to automate interactive CLI scripts.

If this article was useful for you, consider our Database Automation Service.

About the bug

If you are familiar with TCL, please take a look at the script generated by autoexpect. I suspect that fixing it just involves escaping some characters, but I don’t know how.

I wasn’t able to contact Don Libes, and there seems to be no other way to report an Expect bug.

Anyway, the point is: “expect” to encounter small problems, but also “expect” them to be easy to fix.

If you need expert help at automating your database infrastructure, consider our Database Automation services.

Federico Razzoli

Did you like this article?