diff --git a/applets/compiler_vm/README b/applets/compiler_vm/README deleted file mode 100644 index 5f3697db..00000000 --- a/applets/compiler_vm/README +++ /dev/null @@ -1,93 +0,0 @@ -Installation: - -If you want to run the compiler inside a virtual machine for more security, -these scripts are designed to work with qemu 0.11.1. This is required in -order to use PBot's trigger. - -In addition, you can use the provided 'cc' script to use your local compiler tools -without installing qemu. No PBot installation or configuration is required in this case. - -To use the local non-vm 'cc' script, you will need to have gcc, gdb and astyle installed locally. - -Be aware that you need to single-quote or escape the code if you use the local 'cc' within a shell, -e.g.: ./cc $'char s[] = "hello, world"; puts(s); if(s[0] == \'h\') puts("true");' - -WARNING: Using the local 'cc' script outside of the virtual machine will not use qemu at all; - it will affect local system -- compile "safe" code! - -Virtual machine installation: - -You will need to download qemu and set up a virtual machine containing a system -with a compiler and, optionally, sensible ulimits/fork-preventation/other security. - -1) copy compiler_vm_server.pl and compiler_watchdog.pl to the virtual machine. -2) then start up the compiler_vm_server.pl script inside the virtual machine -3) then connect to qemu's monitor and issue the 'savevm 1' command to save the virtual state - (After compiles, this state will be loaded via 'loadvm 1' to reset everything within the machine - for a clean and working environment for subsequent compiles.) - -Now the virtual machine state 1 is saved in a state where it is listening for incoming code. You can -go ahead and quit qemu without shutting down the guest operating sytem. - -Starting the virtual machine for PBot: - -Now that the virtual machine is configured and saved, you may launch the local server to listen for -code from PBot to send to the virtual machine's server. To do so, run the compiler_server.pl script. - -Dependencies: - -gcc (tested with 4.4.4) -gdb (tested with 7.2) -astyle (tested with 1.24 -- not working with astyle 2.0) - -Files: - -(Read within each file for configuration instructions.) - -- cc: Allows you to use the compiler locally with or without qemu installed. - Can be used within virtual machine for testing. - Requires only compiler_vm_client.pl, compiler_vm_server.pl and compiler_watchdog.pl. - Must have gcc, gdb and astyle installed locally if not used within virtual machine. - WARNING: If not used within virtual machine will not use qemu at all and will - affect local system -- compile "safe" code! - -- compiler_client.pl: Main entry point for compiling snippets. Sends over TCP to - compiler_server.pl. This file can be run be run from the - client machine or anywhere. - -- compiler_server.pl: Responsible for setting up a TCP server to listen for - incoming compile requests; and launching and resetting - the virtual machine. Sends to compiler_vm_client.pl. - Run this file on the server hosting the virtual machine. - -- compiler_vm_client.pl: Responsible for sending snippets to the virtual - machine. Also expands/translates and formats - snippets into compilable code (with main function and - headers), and handles "interactive-editing". - Sends over TCP to qemu serial port, waits for result, - then sends result back caller (compiler_server.pl). - Run this file on the server hosting the virtual machine. - -- compiler_vm_server.pl: Runs on the system inside the virtual machine. - This script listens for incoming code snippets over - the virtual machine's serial port. - Calls compiler_watchdog.pl to monitor its exit signal - or exit status, then returns result back over serial - port (to compiler_vm_client.pl). - -- compiler_watchdog.pl: Runs a program and watches its exit signals/status. - Run within the virtual machine. - -*** The following files are just auxiliary tools to start/connect to qemu, provided -for convenience only. Perhaps they will be useful during installation/testing: - -- monitor: Connects to qemu monitor (internal console) over TCP. - -- serial: Connects to qemu serial port over TCP. - -- runqemu: Launches qemu with a visible window, but without networking support. - -- runeqmu.net: Launches qemu with a visible window, and with networking support. - You may load a state previously saved with runqemu and reboot it - or otherwise reload its networking configuration to gain networking. - diff --git a/applets/compiler_vm/cc b/applets/compiler_vm/cc deleted file mode 100755 index 0715d08d..00000000 --- a/applets/compiler_vm/cc +++ /dev/null @@ -1,6 +0,0 @@ -#!/bin/sh -# This Source Code Form is subject to the terms of the Mozilla Public -# License, v. 2.0. If a copy of the MPL was not distributed with this -# file, You can obtain one at http://mozilla.org/MPL/2.0/. - -CC_LOCAL=1 ./compiler_vm_client.pl c99 compiler compiler "$@" diff --git a/applets/compiler_vm/compiler_client.pl b/applets/compiler_vm/compiler_client.pl deleted file mode 100755 index 76583cf3..00000000 --- a/applets/compiler_vm/compiler_client.pl +++ /dev/null @@ -1,44 +0,0 @@ -#!/usr/bin/perl - -# SPDX-FileCopyrightText: 2021 Pragmatic Software -# SPDX-License-Identifier: MIT - -# compiler_client.pl connects to compiler_server.pl hosted at PeerAddr/PeerPort below -# and sends a nick, language and code, then retreives and prints the compilation/execution output. -# -# this way we can run the compiler virtual machine on any remote server. - -use warnings; -use strict; - -use IO::Socket; -use JSON; - -my $sock = IO::Socket::INET->new( - PeerAddr => '127.0.0.1', - PeerPort => 9000, - Proto => 'tcp'); - -if(not defined $sock) { - print "Fatal error compiling: $!; try again later\n"; - die $!; -} - -my $json = join ' ', @ARGV; -my $h = decode_json $json; -my $lang = $h->{lang} // "c11"; - -if ($h->{code} =~ s/-lang=([^ ]+)//) { - $lang = lc $1; -} - -$h->{lang} = $lang; -$json = encode_json $h; - -print $sock "$json\n"; - -while(my $line = <$sock>) { - print "$line"; -} - -close $sock; diff --git a/applets/compiler_vm/compiler_server.pl b/applets/compiler_vm/compiler_server.pl deleted file mode 100755 index ccb4bc87..00000000 --- a/applets/compiler_vm/compiler_server.pl +++ /dev/null @@ -1,276 +0,0 @@ -#!/usr/bin/perl - -# SPDX-FileCopyrightText: 2021 Pragmatic Software -# SPDX-License-Identifier: MIT - -use warnings; -use strict; - -use IO::Socket; -use Net::hostent; -use IPC::Shareable; -use Time::HiRes qw/gettimeofday/; - -my $SERVER_PORT = 9000; -my $SERIAL_PORT = 3333; -my $HEARTBEAT_PORT = 3336; -my $DOMAIN_NAME = 'compiler'; - -my $COMPILE_TIMEOUT = 10; - -sub server_listen { - my $port = shift @_; - - my $server = IO::Socket::INET->new( - Proto => 'tcp', - LocalPort => $port, - Listen => SOMAXCONN, - Reuse => 1); - - die "can't setup server: $!" unless $server; - - print "[Server $0 accepting clients]\n"; - - return $server; -} - -sub vm_stop { - system("virsh shutdown $DOMAIN_NAME"); -} - -sub vm_start { - system("virsh start $DOMAIN_NAME"); -} - -sub vm_reset { - return if $ENV{NORESET}; - #system("virsh detach-disk $DOMAIN_NAME vdb"); - system("virsh snapshot-revert $DOMAIN_NAME 1"); - #system("virsh attach-disk $DOMAIN_NAME --source /var/lib/libvirt/images/factdata.qcow2 --target vdb"); - print "Reset vm\n"; -} - -sub execute { - my ($cmdline) = @_; - - print "execute($cmdline)\n"; - - my @list = split / /, $cmdline; - - my ($ret, $result); - - #$SIG{CHLD} = 'IGNORE'; - - my $child = fork; - - if($child == 0) { - ($ret, $result) = eval { - my $result = ''; - - my $pid = open(my $fh, '-|', @list); - - if (not defined $pid) { - print "Couldn't fork: $!\n"; - return (-13, "[Fatal error]"); - } - - local $SIG{ALRM} = sub { print "Time out\n"; kill 9, $pid; print "sent KILL to $pid\n"; die "Timed-out: $result\n"; }; - alarm($COMPILE_TIMEOUT); - - print "Reading...\n"; - while(my $line = <$fh>) { - print "read [$line]\n"; - $result .= $line; - } - - close $fh; - print "Done reading.\n"; - - my $ret = $? >> 8; - alarm 0; - - print "[$ret, $result]\n"; - return ($ret, $result); - }; - - alarm 0; - if($@ =~ /Timed-out: (.*)/) { - return (-13, "[Timed-out] $1"); - } - - return ($ret, $result); - } else { - waitpid($child, 0); - print "?: $?\n"; - my $result = $? >> 8; - print "child exited, parent continuing [result = $result]\n"; - return (undef, $result); - } -} - -sub compiler_server { - my ($server, $heartbeat_pid, $heartbeat_monitor); - - my $heartbeat; - my $running; - - tie $heartbeat, 'IPC::Shareable', 'dat1', { create => 1 }; - tie $running, 'IPC::Shareable', 'dat2', { create => 1 }; - - my $last_wait = 0; - - while(1) { - $running = 1; - $heartbeat = 0; - - vm_reset; - print "vm started\n"; - - $heartbeat_pid = fork; - die "Fork failed: $!" if not defined $heartbeat_pid; - - if($heartbeat_pid == 0) { - tie $heartbeat, 'IPC::Shareable', 'dat1', { create => 1 }; - tie $running, 'IPC::Shareable', 'dat2', { create => 1 }; - - $heartbeat_monitor = undef; - my $attempts = 0; - while((not $heartbeat_monitor) and $attempts < 5) { - print "Connecting to heartbeat ..."; - $heartbeat_monitor = IO::Socket::INET->new(PeerAddr => '127.0.0.1', PeerPort => $HEARTBEAT_PORT, Proto => 'tcp', Type => SOCK_STREAM); - if(not $heartbeat_monitor) { - print " failed.\n"; - ++$attempts; - sleep 2; - } else { - print " success!\n"; - } - } - - if ($attempts >= 5) { - print "heart not beating... restarting\n"; - $heartbeat = -1; - sleep 5; - next; - } - - print "child: running: $running\n"; - - while($running and <$heartbeat_monitor>) { - $heartbeat = 1; - #print "child: got heartbeat\n"; - } - - print "child no longer running\n"; - exit; - } else { - - while ($heartbeat <= 0) { - if ($heartbeat == -1) { - print "heartbeat died\n"; - last; - } - print "sleeping for heartbeat...\n"; - sleep 1; - } - - if ($heartbeat == -1) { - print "fucking dead, restarting\n"; - waitpid $heartbeat_pid, 0; - #vm_stop; - next; - } - - print "K, got heartbeat, here we go...\n"; - - if(not defined $server) { - print "Starting compiler server on port $SERVER_PORT\n"; - $server = server_listen($SERVER_PORT); - } else { - print "Compiler server already listening on port $SERVER_PORT\n"; - } - - print "parent: running: $running\n"; - - while ($running and my $client = $server->accept()) { - $client->autoflush(1); - my $hostinfo = gethostbyaddr($client->peeraddr); - print '-' x 20, "\n"; - printf "[Connect from %s at %s]\n", $client->peerhost, scalar localtime; - my $timed_out = 0; - my $killed = 0; - - eval { - local $SIG{ALRM} = sub { die 'Timed-out'; }; - alarm 5; - - while (my $line = <$client>) { - $line =~ s/[\r\n]+$//; - next if $line =~ m/^\s*$/; - alarm 5; - print "got: [$line]\n"; - - if($heartbeat <= 0) { - print "No heartbeat yet, ignoring compile attempt.\n"; - print $client "Recovering from previous snippet, please wait.\n" if gettimeofday - $last_wait > 60; - $last_wait = gettimeofday; - last; - } - - print "Attempting compile...\n"; - alarm 0; - - my ($ret, $result) = execute("perl compiler_vm_client.pl $line"); - - if(not defined $ret) { - #print "parent continued\n"; - print "parent continued [$result]\n"; - $timed_out = 1 if $result == 243 or $result == -13; # -13 == 243 - $killed = 1 if $result == 242 or $result == -14; # -14 = 242 - last; - } - - $result =~ s/\s+$//; - print "Ret: $ret; result: [$result]\n"; - - if($result =~ m/\[Killed\]$/) { - print "Process was killed\n"; - $killed = 1; - } - - print $client $result . "\n"; - close $client; - - $ret = -14 if $killed; - - # child exit - print "child exit\n"; - exit $ret; - } - - alarm 0; - }; - - alarm 0; - - close $client; - - print "timed out: $timed_out; killed: $killed\n"; - next unless ($timed_out or $killed); - - vm_reset; - next; - - print "stopping vm\n"; - #vm_stop; - $running = 0; - last; - } - print "Compiler server no longer running, restarting...\n"; - } - print "waiting on heartbeat pid?\n"; - waitpid($heartbeat_pid, 0); - } -} - -compiler_server; diff --git a/applets/compiler_vm/compiler_server_watchdog.pl b/applets/compiler_vm/compiler_server_watchdog.pl deleted file mode 100755 index 44706b1e..00000000 --- a/applets/compiler_vm/compiler_server_watchdog.pl +++ /dev/null @@ -1,62 +0,0 @@ -#!/usr/bin/env perl - -# SPDX-FileCopyrightText: 2021 Pragmatic Software -# SPDX-License-Identifier: MIT - -use warnings; -use strict; - -use Proc::ProcessTable; -use IO::Socket; - -my $SLEEP = 15; -my $MAX_PCTCPU = 25; -my $QEMU = 'qemu-system-x86'; -my $MONITOR_PORT = 3335; - -my $last_pctcpu = 0; - -sub reset_vm { - print "Resetting vm\n"; - - my $sock = IO::Socket::INET->new(PeerAddr => '127.0.0.1', PeerPort => $MONITOR_PORT, Prot => 'tcp'); - if(not defined $sock) { - print "[vm_reset] Unable to connect to monitor: $!\n"; - return; - } - - print $sock "loadvm 1\n"; - close $sock; - - print "Reset vm\n"; -} - -while (1) { - my $t = new Proc::ProcessTable(enable_ttys => 0); - - my ($pids, $p); - - foreach $p (@{$t->table}) { - $pids->{$p->pid} = { fname => $p->fname, ppid => $p->ppid }; - } - - foreach $p (keys %$pids) { - if ($pids->{$p}->{fname} eq $QEMU) { - my $ppid = $pids->{$p}->{ppid}; - if ($pids->{$ppid}->{fname} eq 'compiler_server') { - my $pctcpu = `top -b -n 1 -p $p | tail -n 1 | awk '{print \$9}'`; - $pctcpu =~ s/^\s+|\s+$//g; - print scalar localtime, " :: Got compiler qemu pid: $p; using $pctcpu cpu\n" if $pctcpu > 0; - - if ($pctcpu >= $last_pctcpu and $last_pctcpu >= $MAX_PCTCPU) { - reset_vm; - $last_pctcpu = 0; - } else { - $last_pctcpu = $pctcpu; - } - } - } - } - - sleep $SLEEP; -} diff --git a/applets/compiler_vm/compiler_vm_client.pl b/applets/compiler_vm/compiler_vm_client.pl deleted file mode 100755 index f733cd90..00000000 --- a/applets/compiler_vm/compiler_vm_client.pl +++ /dev/null @@ -1,73 +0,0 @@ -#!/usr/bin/env perl - -# SPDX-FileCopyrightText: 2021 Pragmatic Software -# SPDX-License-Identifier: MIT - -use warnings; -use strict; - -use File::Basename; -use JSON; - -use lib '.'; - -my $json = join ' ', @ARGV; -my $h = decode_json $json; - -my $language = lc $h->{lang}; - -eval { - use lib 'languages'; - require "$language.pm"; -} or do { - my @modules = glob 'languages/*.pm'; - my $found = 0; - my ($languages, $comma) = ('', ''); - - foreach my $module (sort @modules) { - $module = basename $module; - $module =~ s/.pm$//; - next if $module =~ m/^_/; - - require "$module.pm" or die $!; - my $mod = $module->new; - - - if (exists $mod->{name} and $mod->{name} eq $language) { - $language = $module; - $found = 1; - last; - } - - $module = $mod->{name} if exists $mod->{name}; - $languages .= "$comma$module"; - $comma = ', '; - } - - if (not $found) { - print "Language '$language' is not supported.\nSupported languages are: $languages\n"; - exit; - } -}; - -if (not length $h->{code}) { - if (exists $h->{usage}) { - print "$h->{usage}\n"; - } else { - print "Usage: cc [-lang=] [-info] [-paste] [-args \"command-line arguments\"] [compiler/language options] [-stdin ]\n"; - } - exit; -} - -my $lang = $language->new(%{$h}); - -$lang->{local} = $ENV{CC_LOCAL}; - -$lang->process_interactive_edit; -$lang->process_standard_options; -$lang->process_custom_options; -$lang->process_cmdline_options; -$lang->preprocess_code; -$lang->execute; -$lang->postprocess_output; -$lang->show_output; diff --git a/applets/compiler_vm/compiler_vm_server.pl b/applets/compiler_vm/compiler_vm_server.pl deleted file mode 100755 index fa6e9f0c..00000000 --- a/applets/compiler_vm/compiler_vm_server.pl +++ /dev/null @@ -1,212 +0,0 @@ -#!/usr/bin/env perl - -# SPDX-FileCopyrightText: 2021 Pragmatic Software -# SPDX-License-Identifier: MIT - -use warnings; -use strict; - -use English; -use File::Basename; -use JSON; - -my $USERNAME = 'compiler'; -my $USE_LOCAL = defined $ENV{'CC_LOCAL'}; - -use constant MOD_DIR => '/usr/local/share/compiler_vm/languages'; - -use lib MOD_DIR; - -my %languages; - -sub load_modules { - my @files = glob MOD_DIR . "/*.pm"; - foreach my $mod (@files){ - print "Loading module $mod\n"; - my $filename = basename($mod); - require $filename; - $filename =~ s/\.pm$//; - $languages{$filename} = 1; - } -} - -sub run_server { - my ($input, $output, $heartbeat); - - if(not defined $USE_LOCAL or $USE_LOCAL == 0) { - open($input, '<', "/dev/ttyS0") or die $!; - open($output, '>', "/dev/ttyS0") or die $!; - open($heartbeat, '>', "/dev/ttyS1") or die $!; - } else { - open($input, '<', "/dev/stdin") or die $!; - open($output, '>', "/dev/stdout") or die $!; - } - - my $date; - my $lang; - my $sourcefile; - my $execfile; - my $code; - my $cmdline; - my $user_input; - - my $pid = fork; - die "Fork failed: $!" if not defined $pid; - - if($pid == 0) { - my $buffer = ""; - my $length = 4096; - my $line; - my $total_read = 0; - - while (1) { - print "Waiting for input...\n"; - my $ret = sysread($input, my $buf, $length); - - if (not defined $ret) { - print "Error reading: $!\n"; - next; - } - - $total_read += $ret; - - if ($ret == 0) { - print "input ded?\n"; - print "got buffer [$buffer]\n"; - exit; - } - - chomp $buf; - print "read $ret bytes [$total_read so far] [$buf]\n"; - $buffer.= $buf; - - if ($buffer =~ s/\s*:end:\s*$//m) { - $line = $buffer; - $buffer = ""; - $total_read = 0; - } else { - next; - } - - chomp $line; - - print "-" x 40, "\n"; - print "Got [$line]\n"; - - my $compile_in = decode_json($line); - - $compile_in->{arguments} //= ''; - $compile_in->{input} //= ''; - - print "Attempting compile [$compile_in->{lang}] ...\n"; - - use Data::Dumper; - print Dumper $compile_in; - - my $pid = fork; - - if (not defined $pid) { - print "fork failed: $!\n"; - next; - } - - if ($pid == 0) { - my ($uid, $gid, $home) = (getpwnam $USERNAME)[2, 3, 7]; - if (not $uid and not $gid) { - print "Could not find user $USERNAME: $!\n"; - exit; - } - - if ($compile_in->{'persist-key'}) { - system ("rm -rf \"/home/compiler/$compile_in->{'persist-key'}\""); - system("mount /dev/vdb1 /root/factdata"); - system("mkdir -p \"/root/factdata/$compile_in->{'persist-key'}\""); - system("cp -R -p \"/root/factdata/$compile_in->{'persist-key'}\" \"/home/compiler/$compile_in->{'persist-key'}\""); - } - - system("chmod -R 755 /home/compiler"); - system("chown -R compiler /home/compiler"); - system("chgrp -R compiler /home/compiler"); - system("rm -rf /home/compiler/prog*"); - system("pkill -u compiler"); - - $ENV{USER} = $USERNAME; - $ENV{LOGNAME} = $USERNAME; - $ENV{HOME} = $home; - - $GID = $gid; - $EGID = "$gid $gid"; - $EUID = $UID = $uid; - - my $result = interpret(%$compile_in); - - my $compile_out = { result => $result }; - my $json = encode_json($compile_out); - - print "Done compiling; result: [$result] [$json]\n"; - print $output "result:$json\n"; - print $output "result:end\n"; - - $( = 0; - $< = 0; - - if ($compile_in->{'persist-key'}) { - system("id"); - system("cp -R -p \"/home/compiler/$compile_in->{'persist-key'}\" \"/root/factdata/$compile_in->{'persist-key'}\""); - system("umount /root/factdata"); - system ("rm -rf \"/home/compiler/$compile_in->{'persist-key'}\""); - } - - exit; - } else { - waitpid $pid, 0; - } - - if(not defined $USE_LOCAL or $USE_LOCAL == 0) { - print "=" x 40, "\n"; - print "input: "; - next; - } else { - exit; - } - } - } else { - while(1) { - print $heartbeat "\n"; - sleep 1; - } - } - - close $input; - close $output; - close $heartbeat; -} - -sub interpret { - my %h = @_; - - $h{lang} = '_default' if not exists $languages{$h{lang}}; - - chdir("/home/compiler"); - - my $mod = $h{lang}->new(%h); - - $mod->preprocess; - - $mod->postprocess if not $mod->{error} and not $mod->{done}; - - if (exists $mod->{no_output} or not length $mod->{output}) { - if ($h{factoid}) { - $mod->{output} = ""; - } else { - $mod->{output} .= "\n" if length $mod->{output}; - $mod->{output} .= "Success (no output).\n" if not $mod->{error}; - $mod->{output} .= "Success (exit code $mod->{error}).\n" if $mod->{error}; - } - } - - return $mod->{output}; -} - -load_modules; -run_server; diff --git a/applets/compiler_vm/compiler_watchdog.pl b/applets/compiler_vm/guest/bin/guest-gdb similarity index 99% rename from applets/compiler_vm/compiler_watchdog.pl rename to applets/compiler_vm/guest/bin/guest-gdb index e6e5ea3f..9b76c714 100755 --- a/applets/compiler_vm/compiler_watchdog.pl +++ b/applets/compiler_vm/guest/bin/guest-gdb @@ -3,9 +3,18 @@ # SPDX-FileCopyrightText: 2021 Pragmatic Software # SPDX-License-Identifier: MIT +# This script was thrown together quickly and sloppily. It will be +# rewritten "soon". + no warnings; use strict; +my $cmdlineargs = ''; +foreach my $arg (@ARGV) { + $arg =~ s/'/'"'"'/g; + $cmdlineargs .= "'$arg' "; +} + use IPC::Open2; my $debug = $ENV{DEBUG} // 0; @@ -26,12 +35,6 @@ my ($main_start, $main_end); sub flushall; sub gdb; -my $cmdlineargs = ''; -foreach my $arg (@ARGV) { - $arg =~ s/'/'"'"'/g; - $cmdlineargs .= "'$arg' "; -} - my ($out, $in); sub getlocals { diff --git a/applets/compiler_vm/guest/bin/setup-guest b/applets/compiler_vm/guest/bin/setup-guest new file mode 100755 index 00000000..4e927368 --- /dev/null +++ b/applets/compiler_vm/guest/bin/setup-guest @@ -0,0 +1,9 @@ +cp guest/bin/* /usr/local/bin +mkdir /usr/local/share/pbot-vm/ +cp -r guest/lib/Languages/ /usr/local/share/pbot-vm/ +cp guest/include/prelude.h /usr/include + +echo unset DEBUGINFOD_URLS >> /root/.bashrc +echo export ASAN_OPTIONS=detect_leaks=0 >> /root/.bashrc + +echo PBot Guest VM is now set up. For changes to take effect, run this command now: source /root/.bashrc diff --git a/applets/compiler_vm/guest/bin/start-guest b/applets/compiler_vm/guest/bin/start-guest new file mode 100755 index 00000000..53c10382 --- /dev/null +++ b/applets/compiler_vm/guest/bin/start-guest @@ -0,0 +1,226 @@ +#!/usr/bin/env perl + +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at http://mozilla.org/MPL/2.0/. + +use warnings; +use strict; + +use English; +use Encode; +use File::Basename; +use JSON::XS; + +my $USERNAME = 'vm'; # variable for easier string interpolation + +use constant MOD_DIR => '/usr/local/share/pbot-vm/Languages'; +use constant SERIAL => '/dev/ttyS1'; +use constant HEARTBEAT => '/dev/ttyS2'; +use constant STDIN => '/dev/stdin'; +use constant STDOUT => '/dev/stdout'; + +use lib MOD_DIR; + +my %languages; + +my $USE_LOCAL = $ENV{'CC_LOCAL'}; + +sub load_modules { + my @files = glob MOD_DIR . "/*.pm"; + foreach my $mod (@files){ + print "Loading module $mod\n"; + my $filename = basename($mod); + require $filename; + $filename =~ s/\.pm$//; + $languages{$filename} = 1; + } +} + +sub run_server { + my ($input, $output, $heartbeat); + + if(not defined $USE_LOCAL or $USE_LOCAL == 0) { + # set serial to 115200 baud instead of 9600 + system('stty -F ' . SERIAL . ' 115200'); + + open($input, '<', SERIAL) or die $!; + open($output, '>', SERIAL) or die $!; + open($heartbeat, '>', HEARTBEAT) or die $!; + } else { + open($input, '<', STDIN) or die $!; + open($output, '>', STDOUT) or die $!; + } + + my $date; + my $lang; + my $sourcefile; + my $execfile; + my $code; + my $cmdline; + my $user_input; + + my $pid = fork; + die "Fork failed: $!" if not defined $pid; + + if($pid == 0) { + my $buffer = ""; + my $length = 4096; + my $line; + my $total_read = 0; + + while (1) { + print "Waiting for input...\n"; + my $ret = sysread($input, my $buf, $length); + + if (not defined $ret) { + print "Error reading: $!\n"; + next; + } + + $total_read += $ret; + + if ($ret == 0) { + print "input ded?\n"; + print "got buffer [$buffer]\n"; + exit; + } + + chomp $buf; + print "read $ret bytes [$total_read so far] [$buf]\n"; + $buffer.= $buf; + + if ($buffer =~ s/\s*:end:\s*$//m) { + $line = $buffer; + $buffer = ""; + $total_read = 0; + } else { + next; + } + + chomp $line; + + print "-" x 40, "\n"; + print "Got [$line]\n"; + + $line = encode('UTF-8', $line); + my $compile_in = decode_json($line); + + $compile_in->{arguments} //= ''; + $compile_in->{input} //= ''; + + print "Attempting compile [$compile_in->{lang}] ...\n"; + + use Data::Dumper; + print Dumper $compile_in; + + my $pid = fork; + + if (not defined $pid) { + print "fork failed: $!\n"; + next; + } + + if ($pid == 0) { + my ($uid, $gid, $home) = (getpwnam $USERNAME)[2, 3, 7]; + if (not $uid and not $gid) { + print "Could not find user $USERNAME: $!\n"; + exit; + } + + if ($compile_in->{'persist-key'}) { + system ("rm -rf \"/home/$USERNAME/$compile_in->{'persist-key'}\""); + system("mount /dev/vdb1 /root/factdata"); + system("mkdir -p \"/root/factdata/$compile_in->{'persist-key'}\""); + system("cp -R -p \"/root/factdata/$compile_in->{'persist-key'}\" \"/home/$USERNAME/$compile_in->{'persist-key'}\""); + } + + system("chmod -R 755 /home/$USERNAME"); + system("chown -R $USERNAME /home/$USERNAME"); + system("chgrp -R $USERNAME /home/$USERNAME"); + system("rm -rf /home/$USERNAME/prog*"); + system("pkill -u $USERNAME"); + + $ENV{USER} = $USERNAME; + $ENV{LOGNAME} = $USERNAME; + $ENV{HOME} = $home; + + $GID = $gid; + $EGID = "$gid $gid"; + $EUID = $UID = $uid; + + my $result = interpret(%$compile_in); + + my $compile_out = { result => $result }; + my $json = encode_json($compile_out); + + print "Done compiling; result: [$result] [$json]\n"; + print $output "result:$json\n"; + print $output "result:end\n"; + + $( = 0; + $< = 0; + + if ($compile_in->{'persist-key'}) { + system("id"); + system("cp -R -p \"/home/$USERNAME/$compile_in->{'persist-key'}\" \"/root/factdata/$compile_in->{'persist-key'}\""); + system("umount /root/factdata"); + system ("rm -rf \"/home/$USERNAME/$compile_in->{'persist-key'}\""); + } + + exit; + } else { + waitpid $pid, 0; + } + + if(not defined $USE_LOCAL or $USE_LOCAL == 0) { + print "=" x 40, "\n"; + next; + } else { + exit; + } + } + } else { + while (1) { + print $heartbeat "\n"; + sleep 5; + } + } + + close $input; + close $output; + close $heartbeat; +} + +sub interpret { + my %h = @_; + + $h{lang} = '_default' if not exists $languages{$h{lang}}; + + chdir("/home/$USERNAME"); + + my $mod = $h{lang}->new(%h); + + $mod->preprocess; + + print "after preprocess: ", Dumper $mod, "\n"; + + $mod->postprocess if not $mod->{error} and not $mod->{done}; + + print "after postprocess: ", Dumper $mod, "\n"; + + if (exists $mod->{no_output} or not length $mod->{output}) { + if ($h{factoid}) { + $mod->{output} = ""; + } else { + $mod->{output} .= "\n" if length $mod->{output}; + $mod->{output} .= "Success (no output).\n" if not $mod->{error}; + $mod->{output} .= "Success (exit code $mod->{error}).\n" if $mod->{error}; + } + } + + return $mod->{output}; +} + +load_modules; +run_server; diff --git a/applets/compiler_vm/misc/prelude.h b/applets/compiler_vm/guest/include/prelude.h similarity index 100% rename from applets/compiler_vm/misc/prelude.h rename to applets/compiler_vm/guest/include/prelude.h diff --git a/applets/compiler_vm/languages/server/_c_base.pm b/applets/compiler_vm/guest/lib/Languages/_c_base.pm similarity index 99% rename from applets/compiler_vm/languages/server/_c_base.pm rename to applets/compiler_vm/guest/lib/Languages/_c_base.pm index c1c55549..54bfc9d9 100755 --- a/applets/compiler_vm/languages/server/_c_base.pm +++ b/applets/compiler_vm/guest/lib/Languages/_c_base.pm @@ -86,7 +86,7 @@ sub postprocess { # ASAN_OPTIONS=strict_string_checks=1:detect_stack_use_after_return=1:check_initialization_order=1:strict_init_order=1 ($exitval, $stdout, $stderr) = $self->execute(60, undef, './prog', @args); } else { - ($exitval, $stdout, $stderr) = $self->execute(60, undef, 'compiler_watchdog.pl', @args); + ($exitval, $stdout, $stderr) = $self->execute(60, undef, 'guest-gdb', @args); } my $result = $stderr; diff --git a/applets/compiler_vm/languages/server/_default.pm b/applets/compiler_vm/guest/lib/Languages/_default.pm similarity index 100% rename from applets/compiler_vm/languages/server/_default.pm rename to applets/compiler_vm/guest/lib/Languages/_default.pm diff --git a/applets/compiler_vm/languages/server/c11.pm b/applets/compiler_vm/guest/lib/Languages/c11.pm similarity index 100% rename from applets/compiler_vm/languages/server/c11.pm rename to applets/compiler_vm/guest/lib/Languages/c11.pm diff --git a/applets/compiler_vm/languages/server/c89.pm b/applets/compiler_vm/guest/lib/Languages/c89.pm similarity index 100% rename from applets/compiler_vm/languages/server/c89.pm rename to applets/compiler_vm/guest/lib/Languages/c89.pm diff --git a/applets/compiler_vm/languages/server/c99.pm b/applets/compiler_vm/guest/lib/Languages/c99.pm similarity index 100% rename from applets/compiler_vm/languages/server/c99.pm rename to applets/compiler_vm/guest/lib/Languages/c99.pm diff --git a/applets/compiler_vm/languages/server/clang.pm b/applets/compiler_vm/guest/lib/Languages/clang.pm similarity index 100% rename from applets/compiler_vm/languages/server/clang.pm rename to applets/compiler_vm/guest/lib/Languages/clang.pm diff --git a/applets/compiler_vm/languages/server/clang11.pm b/applets/compiler_vm/guest/lib/Languages/clang11.pm similarity index 100% rename from applets/compiler_vm/languages/server/clang11.pm rename to applets/compiler_vm/guest/lib/Languages/clang11.pm diff --git a/applets/compiler_vm/languages/server/clang89.pm b/applets/compiler_vm/guest/lib/Languages/clang89.pm similarity index 100% rename from applets/compiler_vm/languages/server/clang89.pm rename to applets/compiler_vm/guest/lib/Languages/clang89.pm diff --git a/applets/compiler_vm/languages/server/clang99.pm b/applets/compiler_vm/guest/lib/Languages/clang99.pm similarity index 100% rename from applets/compiler_vm/languages/server/clang99.pm rename to applets/compiler_vm/guest/lib/Languages/clang99.pm diff --git a/applets/compiler_vm/languages/server/clangpp.pm b/applets/compiler_vm/guest/lib/Languages/clangpp.pm similarity index 100% rename from applets/compiler_vm/languages/server/clangpp.pm rename to applets/compiler_vm/guest/lib/Languages/clangpp.pm diff --git a/applets/compiler_vm/languages/server/cpp.pm b/applets/compiler_vm/guest/lib/Languages/cpp.pm similarity index 100% rename from applets/compiler_vm/languages/server/cpp.pm rename to applets/compiler_vm/guest/lib/Languages/cpp.pm diff --git a/applets/compiler_vm/languages/server/freebasic.pm b/applets/compiler_vm/guest/lib/Languages/freebasic.pm similarity index 100% rename from applets/compiler_vm/languages/server/freebasic.pm rename to applets/compiler_vm/guest/lib/Languages/freebasic.pm diff --git a/applets/compiler_vm/languages/server/haskell.pm b/applets/compiler_vm/guest/lib/Languages/haskell.pm similarity index 100% rename from applets/compiler_vm/languages/server/haskell.pm rename to applets/compiler_vm/guest/lib/Languages/haskell.pm diff --git a/applets/compiler_vm/languages/server/java.pm b/applets/compiler_vm/guest/lib/Languages/java.pm similarity index 100% rename from applets/compiler_vm/languages/server/java.pm rename to applets/compiler_vm/guest/lib/Languages/java.pm diff --git a/applets/compiler_vm/languages/server/qbasic.pm b/applets/compiler_vm/guest/lib/Languages/qbasic.pm similarity index 100% rename from applets/compiler_vm/languages/server/qbasic.pm rename to applets/compiler_vm/guest/lib/Languages/qbasic.pm diff --git a/applets/compiler_vm/languages/server/tendra.pm b/applets/compiler_vm/guest/lib/Languages/tendra.pm similarity index 100% rename from applets/compiler_vm/languages/server/tendra.pm rename to applets/compiler_vm/guest/lib/Languages/tendra.pm diff --git a/applets/compiler_vm/host/bin/vm-client b/applets/compiler_vm/host/bin/vm-client new file mode 100755 index 00000000..146a8f77 --- /dev/null +++ b/applets/compiler_vm/host/bin/vm-client @@ -0,0 +1,51 @@ +#!/usr/bin/env perl + +# File: vm-client +# +# Purpose: Interfaces with the PBot virtual machine server hosted at +# PeerAddr/PeerPort defined below. This allows us to host instances of +# virtual machines on remote servers. +# +# This script is intended to be installed to PBot's applets directory +# and attached to a PBot command such as `cc`. + +# SPDX-FileCopyrightText: 2021 Pragmatic Software +# SPDX-License-Identifier: MIT + +# TODO: extend to take a list of server/ports to cycle for load-balancing + +use warnings; +use strict; + +use IO::Socket; + +my $sock = IO::Socket::INET->new( + PeerAddr => '127.0.0.1', + PeerPort => 9000, + Proto => 'tcp', +); + +if (not defined $sock) { + print "Fatal error compiling: $!; try again later\n"; + die $!; +} + +my $nick = shift @ARGV; +my $channel = shift @ARGV; +my $code = join '', @ARGV; + +my $lang = "c11"; + +if ($code =~ s/-lang=([^ ]+)//) { + $lang = lc $1; +} + +print $sock "compile:$nick:$channel:$lang\n"; +print $sock "$code\n"; +print $sock "compile:end\n"; + +while (my $line = <$sock>) { + print "$line"; +} + +close $sock; diff --git a/applets/compiler_vm/host/bin/vm-exec b/applets/compiler_vm/host/bin/vm-exec new file mode 100755 index 00000000..e97413c4 --- /dev/null +++ b/applets/compiler_vm/host/bin/vm-exec @@ -0,0 +1,103 @@ +#!/usr/bin/env perl + +# File: vm-exec +# +# Purpose: Process and send commands to the PBot virtual machine on the +# default TCP port (5555). Use the PBOT_VM_PORT environment variable to +# override the port. E.g., to use port 6666 instead: +# +# $ PBOT_VM_PORT=6666 vm-exec -lang=sh echo test + +# SPDX-FileCopyrightText: 2021 Pragmatic Software +# SPDX-License-Identifier: MIT + +use warnings; +use strict; + +use File::Basename; +use JSON::XS; + +use FindBin qw($RealBin); +use lib "$RealBin/../lib"; + +my $json = join ' ', @ARGV; + +my $args = eval { decode_json $json }; + +if ($@) { + # wasn't JSON; make structure manually + if ($json =~ s/^-lang=([^ ]+)//) { + $args = { lang => $1, code => $json }; + } else { + $args = { code => $json }; + } +} + +if (not exists $args->{code}) { + die "Missing `code` field. Usage: $0 {'code':''}\n"; +} + +# set any missing fields to default values +$args->{nick} //= 'vm'; +$args->{channel} //= 'vm'; +$args->{lang} //= 'c11'; + +# override vm-port with environment variable +if ($ENV{PBOT_VM_PORT}) { + $args->{'vm-port'} = $ENV{PBOT_VM_PORT}; +} + +my $language = lc $args->{lang}; + +eval { + require "Languages/$language.pm"; +} or do { + my $found = 0; + my ($languages, $comma) = ('', ''); + + foreach my $module (sort glob "$RealBin/../lib/Languages/*.pm") { + $module = basename $module; + $module =~ s/.pm$//; + next if $module =~ m/^_/; + + require "Languages/$module.pm" or die $!; + my $mod = "Languages::$module"->new; + + if (exists $mod->{name} and $mod->{name} eq $language) { + $language = $module; + $found = 1; + last; + } + + $module = $mod->{name} if exists $mod->{name}; + $languages .= "$comma$module"; + $comma = ', '; + } + + if (not $found) { + print "Language '$language' is not supported.\nSupported languages are: $languages\n"; + exit; + } +}; + +if (not length $args->{code}) { + if (exists $args->{usage}) { + print "$args->{usage}\n"; + } else { + print "Usage: cc [-lang=] [-info] [-paste] [-args \"command-line arguments\"] [compiler/language options] [-stdin ]\n"; + } + exit; +} + +my $lang = "Languages::$language"->new(%{$args}); + +$lang->{local} = $ENV{CC_LOCAL}; + +$lang->process_interactive_edit; +$lang->process_standard_options; +$lang->process_custom_options; +$lang->process_cmdline_options; +$lang->preprocess_code; +$lang->execute; +$lang->postprocess_output; +$lang->show_output; diff --git a/applets/compiler_vm/host/bin/vm-host-watchdog b/applets/compiler_vm/host/bin/vm-host-watchdog new file mode 100755 index 00000000..194fe4de --- /dev/null +++ b/applets/compiler_vm/host/bin/vm-host-watchdog @@ -0,0 +1,44 @@ +#!/usr/bin/env perl + +# SPDX-FileCopyrightText: 2021 Pragmatic Software +# SPDX-License-Identifier: MIT + +use warnings; +use strict; + +use Proc::ProcessTable; +use IO::Socket; + +my $SLEEP = 15; +my $MAX_PCTCPU = 25; +my $QEMU = 'qemu-system-x86'; +my $DOMAIN = 'pbot-vm'; + +my $last_pctcpu = 0; + +sub reset_vm { + print "Resetting vm\n"; + system("virsh snapshot-revert $DOMAIN 1"); + print "Reset vm\n"; +} + +while (1) { + my $t = Proc::ProcessTable->new(enable_ttys => 0); + + foreach my $p (@{$t->table}) { + if ($p->fname eq $QEMU and $p->cmndline =~ m/guest=\Q$DOMAIN\E/) { + # $p->pctcpu never updates? so we use top instead. + my $pctcpu = `top -b -n 1 -p $p->{pid} | tail -n 1 | awk '{print \$9}'`; + $pctcpu =~ s/^\s+|\s+$//g; + print scalar localtime, " :: Got $DOMAIN qemu pid: $p; using $pctcpu cpu\n" if $pctcpu > 0; + + if ($pctcpu >= $last_pctcpu and $last_pctcpu >= $MAX_PCTCPU) { + reset_vm; + $last_pctcpu = 0; + } else { + $last_pctcpu = $pctcpu; + } + } + } + sleep $SLEEP; +} diff --git a/applets/compiler_vm/host/bin/vm-server b/applets/compiler_vm/host/bin/vm-server new file mode 100755 index 00000000..0507a1d4 --- /dev/null +++ b/applets/compiler_vm/host/bin/vm-server @@ -0,0 +1,277 @@ +#!/usr/bin/env perl + +# File: compiler_server.pl +# +# Purpose: The compiler server manages the guest virtual machine state and +# listens for incoming compile requests. This server can be run on any remote +# machine. There can be multiple servers using different ports on the same machine. +# +# SPDX-FileCopyrightText: 2021 Pragmatic Software +# SPDX-License-Identifier: MIT + +use warnings; +use strict; + +use IO::Socket; +use Net::hostent; +use IPC::Shareable; +use Time::HiRes qw/gettimeofday/; +use Encode; + +my $SERVER_PORT = 9000; +my $SERIAL_PORT = 5555; +my $HEARTBEAT_PORT = 5556; +my $DOMAIN_NAME = 'pbot-vm'; + +my $COMPILE_TIMEOUT = 15; + +sub vm_stop { + system("virsh shutdown $DOMAIN_NAME"); +} + +sub vm_start { + system("virsh start $DOMAIN_NAME"); +} + +sub vm_reset { + return if $ENV{NORESET}; + #system("virsh detach-disk $DOMAIN_NAME vdb"); + system("virsh snapshot-revert $DOMAIN_NAME 1"); + #system("virsh attach-disk $DOMAIN_NAME --source /var/lib/libvirt/images/factdata.qcow2 --target vdb"); + print "Reset vm\n"; +} + +sub execute { + my ($cmdline) = @_; + + print "execute($cmdline)\n"; + + my @list = split / /, $cmdline; + + my ($ret, $result); + + my $child = fork; + + if($child == 0) { + ($ret, $result) = eval { + my $result = ''; + + my $pid = open(my $fh, '-|', @list); + + if (not defined $pid) { + print "Couldn't fork: $!\n"; + return (-13, "[Fatal error]"); + } + + local $SIG{ALRM} = sub { kill 9, $pid; die "Timed-out: $result\n"; }; + alarm($COMPILE_TIMEOUT); + + print "Reading result...\n"; + while (my $line = <$fh>) { + print "read result [$line]\n"; + $result .= $line; + } + + close $fh; + print "Done reading result.\n"; + + my $ret = $? >> 8; + + print "[$ret, $result]\n"; + return ($ret, $result); + }; + + alarm 0; + if ($@ =~ /Timed-out: (.*)/) { + return (-13, "[Timed-out] $1"); + } + + return ($ret, $result); + } else { + waitpid($child, 0); + print "?: $?\n"; + my $result = $? >> 8; + print "child exited, parent continuing [result = $result]\n"; + return (undef, $result); + } +} + +sub connect_to_heartbeat { + my $heartbeat; + my $attempts = 15; + + while (!$heartbeat && $attempts > 0) { + print "Connecting to heartbeat ... "; + + $heartbeat = IO::Socket::INET->new ( + PeerAddr => '127.0.0.1', + PeerPort => $HEARTBEAT_PORT, + Proto => 'tcp', + Type => SOCK_STREAM, + ); + + if (!$heartbeat) { + print "failed.\n"; + --$attempts; + print "Trying again in 2 seconds ($attempts attempts remaining) ...\n" if $attempts > 0; + sleep 2; + } else { + print "success!\n"; + } + } + + return $heartbeat; +} + +sub server_listen { + my $port = shift @_; + my $server = IO::Socket::INET->new ( + Proto => 'tcp', + LocalPort => $port, + Listen => SOMAXCONN, + ReuseAddr => 1, + Reuse => 1, + ); + die "Can't setup server: $!" unless $server; + print "[Server $0 accepting clients at :$port]\n"; + return $server; +} + +sub vm_server { + my ($server, $heartbeat_pid, $heartbeat_monitor); + + my $heartbeat; + my $running; + + tie $heartbeat, 'IPC::Shareable', 'dat1', { create => 1 }; + tie $running, 'IPC::Shareable', 'dat2', { create => 1 }; + + my $last_wait = 0; + + $running = 1; + $heartbeat = 0; + + vm_reset; + print "vm started\n"; + + $heartbeat_pid = fork; + die "Fork failed: $!" if not defined $heartbeat_pid; + + if ($heartbeat_pid == 0) { + # heartbeat + + tie $heartbeat, 'IPC::Shareable', 'dat1', { create => 1 }; + tie $running, 'IPC::Shareable', 'dat2', { create => 1 }; + + if (!($heartbeat_monitor = connect_to_heartbeat)) { + die "Could not start heartbeat.\n"; + } + + print "child: running: $running\n"; + + while($running and <$heartbeat_monitor>) { + $heartbeat = 1; + #print "child: got heartbeat\n"; + } + + $heartbeat = -1; + print "child no longer running\n"; + + exit; + + } else { + # server + + if (not defined $server) { + print "Starting compiler server on port $SERVER_PORT\n"; + $server = server_listen($SERVER_PORT); + } else { + print "Compiler server already listening on port $SERVER_PORT\n"; + } + + print "parent: running: $running\n"; + + while ($running and my $client = $server->accept) { + $client->autoflush(1); + my $hostinfo = gethostbyaddr($client->peeraddr); + + print '-' x 20, "\n"; + printf "[Connect from %s at %s]\n", $client->peerhost, scalar localtime; + + my ($timed_out, $killed); + + eval { + local $SIG{ALRM} = sub { die 'Timed-out'; }; + alarm 5; + + while (my $line = <$client>) { + $line =~ s/[\r\n]+$//; + next if $line =~ m/^\s*$/; + alarm 5; + print "got: [$line]\n"; + + if($heartbeat <= 0) { + print "No heartbeat yet, ignoring compile attempt.\n"; + print $client "Recovering from previous snippet, please wait.\n" if gettimeofday - $last_wait > 60; + $last_wait = gettimeofday; + last; + } + + print "Attempting compile...\n"; + alarm 0; + + my ($ret, $result) = execute("perl bin/compiler_run.pl $line"); + + if(not defined $ret) { + #print "parent continued\n"; + print "parent continued [$result]\n"; + $timed_out = 1 if $result == 243 or $result == -13; # -13 == 243 + $killed = 1 if $result == 242 or $result == -14; # -14 = 242 + last; + } + + $result =~ s/\s+$//; + print "Ret: $ret; result: [$result]\n"; + + if($result =~ m/\[Killed\]$/) { + print "Process was killed\n"; + $killed = 1; + } + + print $client $result . "\n"; + close $client; + + $ret = -14 if $killed; + + # child exit + print "child exit\n"; + exit $ret; + } + + alarm 0; + }; + + alarm 0; + + close $client; + + print "timed out: $timed_out; killed: $killed\n"; + next unless ($timed_out or $killed); + + vm_reset; + next; + + print "stopping vm\n"; + #vm_stop; + $running = 0; + last; + } + + print "Compiler server no longer running, restarting...\n"; + } + + print "waiting on heartbeat pid\n"; + waitpid($heartbeat_pid, 0); +} + +vm_server; diff --git a/applets/compiler_vm/host/devices/add-serials b/applets/compiler_vm/host/devices/add-serials new file mode 100755 index 00000000..b321cf58 --- /dev/null +++ b/applets/compiler_vm/host/devices/add-serials @@ -0,0 +1,2 @@ +virsh attach-device --config pbot-vm serial-2.xml +virsh attach-device --config pbot-vm serial-3.xml diff --git a/applets/compiler_vm/host/devices/serial-2.xml b/applets/compiler_vm/host/devices/serial-2.xml new file mode 100644 index 00000000..45ec745f --- /dev/null +++ b/applets/compiler_vm/host/devices/serial-2.xml @@ -0,0 +1,5 @@ + + + + + diff --git a/applets/compiler_vm/host/devices/serial-3.xml b/applets/compiler_vm/host/devices/serial-3.xml new file mode 100644 index 00000000..229be48d --- /dev/null +++ b/applets/compiler_vm/host/devices/serial-3.xml @@ -0,0 +1,5 @@ + + + + + diff --git a/applets/compiler_vm/history/.gitignore b/applets/compiler_vm/host/history/.gitignore similarity index 100% rename from applets/compiler_vm/history/.gitignore rename to applets/compiler_vm/host/history/.gitignore diff --git a/applets/compiler_vm/Diff.pm b/applets/compiler_vm/host/lib/Diff.pm similarity index 100% rename from applets/compiler_vm/Diff.pm rename to applets/compiler_vm/host/lib/Diff.pm diff --git a/applets/compiler_vm/languages/_c_base.pm b/applets/compiler_vm/host/lib/Languages/_c_base.pm similarity index 99% rename from applets/compiler_vm/languages/_c_base.pm rename to applets/compiler_vm/host/lib/Languages/_c_base.pm index bc5e4991..094248fc 100755 --- a/applets/compiler_vm/languages/_c_base.pm +++ b/applets/compiler_vm/host/lib/Languages/_c_base.pm @@ -12,8 +12,8 @@ use feature 'unicode_strings'; no if $] >= 5.018, warnings => "experimental::smartmatch"; -package _c_base; -use parent '_default'; +package Languages::_c_base; +use parent 'Languages::_default'; use Text::Balanced qw/extract_bracketed/; diff --git a/applets/compiler_vm/languages/_default.pm b/applets/compiler_vm/host/lib/Languages/_default.pm similarity index 94% rename from applets/compiler_vm/languages/_default.pm rename to applets/compiler_vm/host/lib/Languages/_default.pm index c57330d1..788590d3 100755 --- a/applets/compiler_vm/languages/_default.pm +++ b/applets/compiler_vm/host/lib/Languages/_default.pm @@ -10,7 +10,7 @@ use feature 'unicode_strings'; no if $] >= 5.018, warnings => "experimental::smartmatch"; -package _default; +package Languages::_default; use IPC::Open2; use IO::Socket; @@ -21,21 +21,24 @@ use JSON; use Getopt::Long qw/GetOptionsFromArray :config pass_through no_ignore_case no_auto_abbrev/; use Encode; +use FindBin qw($RealBin); + my $EXECUTE_PORT = '3333'; sub new { my ($class, %conf) = @_; my $self = bless {}, $class; - $self->{debug} = $conf{debug} // 0; - $self->{nick} = $conf{nick}; - $self->{channel} = $conf{channel}; - $self->{lang} = $conf{lang}; - $self->{code} = $conf{code}; - $self->{max_history} = $conf{max_history} // 10000; - $self->{arguments} = $conf{arguments}; - $self->{factoid} = $conf{factoid}; + $self->{debug} = $conf{debug} // 0; + $self->{nick} = $conf{nick}; + $self->{channel} = $conf{channel}; + $self->{lang} = $conf{lang}; + $self->{code} = $conf{code}; + $self->{max_history} = $conf{max_history} // 10000; + $self->{arguments} = $conf{arguments}; + $self->{factoid} = $conf{factoid}; $self->{'persist-key'} = $conf{'persist-key'}; + $self->{'vm-port'} = $conf{'vm-port'} // $EXECUTE_PORT; $self->{default_options} = ''; $self->{cmdline} = 'echo Hello, world!'; @@ -68,7 +71,7 @@ sub preprocess_code { } unless($self->{got_run} and $self->{copy_code}) { - open FILE, ">> log.txt"; + open FILE, ">> $RealBin/../log.txt"; print FILE localtime() . "\n"; print FILE "$self->{nick} $self->{channel}: [" . $self->{arguments} . "] " . $self->{cmdline_options} . "$self->{code}\n"; close FILE; @@ -140,7 +143,7 @@ sub postprocess_output { my $self = shift; unless($self->{got_run} and $self->{copy_code}) { - open FILE, ">> log.txt"; + open FILE, ">> $RealBin/../log.txt"; print FILE "--------------------------post processing----------------------------------------------\n"; print FILE localtime() . "\n"; print FILE "$self->{output}\n"; @@ -173,7 +176,7 @@ sub show_output { my $output = $self->{output}; unless ($self->{got_run} and $self->{copy_code}) { - open FILE, ">> log.txt"; + open FILE, ">> $RealBin/../log.txt"; print FILE "------------------------show output------------------------------------------------\n"; print FILE localtime() . "\n"; print FILE "$output\n"; @@ -237,7 +240,7 @@ sub show_output { exit 0; } - if($self->{channel} =~ m/^#/ and length $output > 22 and open FILE, "< history/$self->{channel}-$self->{lang}.last-output") { + if($self->{channel} =~ m/^#/ and length $output > 22 and open FILE, "< $RealBin/../history/$self->{channel}-$self->{lang}.last-output") { my $last_output; my $time = ; @@ -258,7 +261,7 @@ sub show_output { print "$output\n"; - open FILE, "> history/$self->{channel}-$self->{lang}.last-output" or die "Couldn't open $self->{channel}-$self->{lang}.last-output: $!"; + open FILE, "> $RealBin/../history/$self->{channel}-$self->{lang}.last-output" or die "Couldn't open $self->{channel}-$self->{lang}.last-output: $!"; my $now = gettimeofday; print FILE "$now\n"; print FILE "$output"; @@ -319,17 +322,18 @@ sub paste_0x0 { sub execute { my ($self) = @_; - my ($compiler, $compiler_output, $pid); + my ($vm, $vm_output, $pid); delete $self->{local}; if(exists $self->{local} and $self->{local} != 0) { - print "Using local compiler instead of virtual machine\n"; - $pid = open2($compiler_output, $compiler, './compiler_vm_server.pl') || die "repl failed: $@\n"; - print "Started compiler, pid: $pid\n"; + print "Using local machine instead of virtual machine\n"; + $pid = open2($vm_output, $vm, './compiler_vm_server.pl') || die "repl failed: $@\n"; # XXX + print "Started fake-vm, pid: $pid\n"; } else { - $compiler = IO::Socket::INET->new(PeerAddr => '127.0.0.1', PeerPort => $EXECUTE_PORT, Proto => 'tcp', Type => SOCK_STREAM); - die "Could not create socket: $!" unless $compiler; - $compiler_output = $compiler; + print STDERR "Connecting to remote VM port $self->{'vm-port'}\n"; + $vm = IO::Socket::INET->new(PeerAddr => '127.0.0.1', PeerPort => $self->{'vm-port'}, Proto => 'tcp', Type => SOCK_STREAM); + die "Could not create connection to VM: $!" unless $vm; + $vm_output = $vm; } my $date = time; @@ -383,7 +387,7 @@ sub execute { $cmdline =~ s/\$options\s+//; } - open FILE, ">> log.txt"; + open FILE, ">> $RealBin/../log.txt"; print FILE "---------------------executing---------------------------------------------------\n"; print FILE localtime() . "\n"; print FILE "$cmdline\n$stdin\n$pretty_code\n"; @@ -422,7 +426,7 @@ sub execute { #print FILE "Sending chunk [$chunk]\n"; $chunks_sent += length $chunk; - my $ret = syswrite($compiler, $chunk); + my $ret = syswrite($vm, $chunk); if (not defined $ret) { print FILE "Error sending: $!\n"; @@ -445,7 +449,7 @@ sub execute { my $result = ""; my $got_result = 0; - while(my $line = <$compiler_output>) { + while(my $line = <$vm_output>) { utf8::decode($line); print STDERR "Read from vm [$line]\n"; @@ -465,7 +469,7 @@ sub execute { } } - close $compiler; + close $vm; waitpid($pid, 0) if defined $pid; $self->{output} = $result; @@ -486,7 +490,7 @@ sub add_option { sub process_standard_options { my $self = shift; - my @opt_args = $self->split_line($self->{code}, preserve_escapes => 1, keep_spaces => 1); + my @opt_args = $self->split_line($self->{code}, preserve_escapes => 1, keep_spaces => 0); use Data::Dumper; print STDERR "code:\n$self->{code}\n"; @@ -569,7 +573,7 @@ sub process_interactive_edit { if($subcode =~ s/^\s*copy\s+(\S+)\s*//) { my $copy = $1; - if(open FILE, "< history/$copy-$self->{lang}.hist") { + if(open FILE, "< $RealBin/../history/$copy-$self->{lang}.hist") { $copy_code = ; close FILE; goto COPY_ERROR if not $copy_code;; @@ -594,7 +598,7 @@ sub process_interactive_edit { $self->{channel} = $1; } - if(open FILE, "< history/$self->{channel}-$self->{lang}.hist") { + if(open FILE, "< $RealBin/../history/$self->{channel}-$self->{lang}.hist") { while(my $line = ) { chomp $line; push @last_code, $line; @@ -1018,7 +1022,7 @@ sub process_interactive_edit { unshift @last_code, $code; } - open FILE, "> history/$self->{channel}-$self->{lang}.hist"; + open FILE, "> $RealBin/../history/$self->{channel}-$self->{lang}.hist"; my $i = 0; foreach my $line (@last_code) { diff --git a/applets/compiler_vm/languages/bash.pm b/applets/compiler_vm/host/lib/Languages/bash.pm similarity index 91% rename from applets/compiler_vm/languages/bash.pm rename to applets/compiler_vm/host/lib/Languages/bash.pm index 4eb1e0f0..c5a4b989 100755 --- a/applets/compiler_vm/languages/bash.pm +++ b/applets/compiler_vm/host/lib/Languages/bash.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package bash; -use parent '_default'; +package Languages::bash; +use parent 'Languages::_default'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/bc.pm b/applets/compiler_vm/host/lib/Languages/bc.pm similarity index 89% rename from applets/compiler_vm/languages/bc.pm rename to applets/compiler_vm/host/lib/Languages/bc.pm index cbb5610e..974d622d 100755 --- a/applets/compiler_vm/languages/bc.pm +++ b/applets/compiler_vm/host/lib/Languages/bc.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package bc; -use parent '_default'; +package Languages::bc; +use parent 'Languages::_default'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/bf.pm b/applets/compiler_vm/host/lib/Languages/bf.pm similarity index 86% rename from applets/compiler_vm/languages/bf.pm rename to applets/compiler_vm/host/lib/Languages/bf.pm index 3050ebfb..f6d07cb2 100755 --- a/applets/compiler_vm/languages/bf.pm +++ b/applets/compiler_vm/host/lib/Languages/bf.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package bf; -use parent '_default'; +package Languages::bf; +use parent 'Languages::_default'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/c11.pm b/applets/compiler_vm/host/lib/Languages/c11.pm similarity index 95% rename from applets/compiler_vm/languages/c11.pm rename to applets/compiler_vm/host/lib/Languages/c11.pm index 6792493f..650d043a 100755 --- a/applets/compiler_vm/languages/c11.pm +++ b/applets/compiler_vm/host/lib/Languages/c11.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package c11; -use parent '_c_base'; +package Languages::c11; +use parent 'Languages::_c_base'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/c89.pm b/applets/compiler_vm/host/lib/Languages/c89.pm similarity index 94% rename from applets/compiler_vm/languages/c89.pm rename to applets/compiler_vm/host/lib/Languages/c89.pm index fd08fe89..28fad8fe 100755 --- a/applets/compiler_vm/languages/c89.pm +++ b/applets/compiler_vm/host/lib/Languages/c89.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package c89; -use parent '_c_base'; +package Languages::c89; +use parent 'Languages::_c_base'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/c99.pm b/applets/compiler_vm/host/lib/Languages/c99.pm similarity index 95% rename from applets/compiler_vm/languages/c99.pm rename to applets/compiler_vm/host/lib/Languages/c99.pm index ead33054..47761396 100755 --- a/applets/compiler_vm/languages/c99.pm +++ b/applets/compiler_vm/host/lib/Languages/c99.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package c99; -use parent '_c_base'; +package Languages::c99; +use parent 'Languages::_c_base'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/clang.pm b/applets/compiler_vm/host/lib/Languages/clang.pm similarity index 71% rename from applets/compiler_vm/languages/clang.pm rename to applets/compiler_vm/host/lib/Languages/clang.pm index 5e299187..4ef6eba6 100755 --- a/applets/compiler_vm/languages/clang.pm +++ b/applets/compiler_vm/host/lib/Languages/clang.pm @@ -6,7 +6,7 @@ use warnings; use strict; -package clang; -use parent 'clang11'; +package Languages::clang; +use parent 'Languages::clang11'; 1; diff --git a/applets/compiler_vm/languages/clang11.pm b/applets/compiler_vm/host/lib/Languages/clang11.pm similarity index 85% rename from applets/compiler_vm/languages/clang11.pm rename to applets/compiler_vm/host/lib/Languages/clang11.pm index 5d9d92f1..5c0a41e8 100755 --- a/applets/compiler_vm/languages/clang11.pm +++ b/applets/compiler_vm/host/lib/Languages/clang11.pm @@ -1,20 +1,17 @@ #!/usr/bin/perl -# SPDX-FileCopyrightText: 2021 Pragmatic Software -# SPDX-License-Identifier: MIT - use warnings; use strict; -package clang11; -use parent '_c_base'; +package Languages::clang11; +use parent 'Languages::_c_base'; sub initialize { my ($self, %conf) = @_; $self->{sourcefile} = 'prog.c'; $self->{execfile} = 'prog'; - $self->{default_options} = '-Wextra -Wall -Wno-unused -Wno-unused-parameter -pedantic -Wfloat-equal -Wshadow -std=c11 -lm -Wfatal-errors -fsanitize=integer,undefined,alignment'; + $self->{default_options} = '-Wextra -Wall -Wno-unused -Wno-unused-parameter -pedantic -Wfloat-equal -Wshadow -std=c11 -lm -Wfatal-errors -fsanitize=integer,undefined,alignment -fsanitize-address-use-after-scope -fno-omit-frame-pointer'; $self->{options_paste} = '-fcaret-diagnostics'; $self->{options_nopaste} = '-fno-caret-diagnostics'; $self->{cmdline} = 'clang -ggdb -g3 $sourcefile $options -o $execfile'; diff --git a/applets/compiler_vm/languages/clang89.pm b/applets/compiler_vm/host/lib/Languages/clang89.pm similarity index 94% rename from applets/compiler_vm/languages/clang89.pm rename to applets/compiler_vm/host/lib/Languages/clang89.pm index 86972cff..98c2c8bb 100755 --- a/applets/compiler_vm/languages/clang89.pm +++ b/applets/compiler_vm/host/lib/Languages/clang89.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package clang89; -use parent '_c_base'; +package Languages::clang89; +use parent 'Languages::_c_base'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/clang99.pm b/applets/compiler_vm/host/lib/Languages/clang99.pm similarity index 95% rename from applets/compiler_vm/languages/clang99.pm rename to applets/compiler_vm/host/lib/Languages/clang99.pm index e5e1b9c9..c52ecd39 100755 --- a/applets/compiler_vm/languages/clang99.pm +++ b/applets/compiler_vm/host/lib/Languages/clang99.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package clang99; -use parent '_c_base'; +package Languages::clang99; +use parent 'Languages::_c_base'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/clangpp.pm b/applets/compiler_vm/host/lib/Languages/clangpp.pm similarity index 97% rename from applets/compiler_vm/languages/clangpp.pm rename to applets/compiler_vm/host/lib/Languages/clangpp.pm index 4b794e28..96701956 100755 --- a/applets/compiler_vm/languages/clangpp.pm +++ b/applets/compiler_vm/host/lib/Languages/clangpp.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package clangpp; -use parent '_c_base'; +package Languages::clangpp; +use parent 'Languages::_c_base'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/clisp.pm b/applets/compiler_vm/host/lib/Languages/clisp.pm similarity index 94% rename from applets/compiler_vm/languages/clisp.pm rename to applets/compiler_vm/host/lib/Languages/clisp.pm index fac247cd..982c82f0 100755 --- a/applets/compiler_vm/languages/clisp.pm +++ b/applets/compiler_vm/host/lib/Languages/clisp.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package clisp; -use parent '_default'; +package Languages::clisp; +use parent 'Languages::_default'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/cpp.pm b/applets/compiler_vm/host/lib/Languages/cpp.pm similarity index 97% rename from applets/compiler_vm/languages/cpp.pm rename to applets/compiler_vm/host/lib/Languages/cpp.pm index 979159f8..f4fafb23 100755 --- a/applets/compiler_vm/languages/cpp.pm +++ b/applets/compiler_vm/host/lib/Languages/cpp.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package cpp; -use parent '_c_base'; +package Languages::cpp; +use parent 'Languages::_c_base'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/freebasic.pm b/applets/compiler_vm/host/lib/Languages/freebasic.pm similarity index 92% rename from applets/compiler_vm/languages/freebasic.pm rename to applets/compiler_vm/host/lib/Languages/freebasic.pm index 1c3acc3d..88b7d52d 100755 --- a/applets/compiler_vm/languages/freebasic.pm +++ b/applets/compiler_vm/host/lib/Languages/freebasic.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package freebasic; -use parent '_default'; +package Languages::freebasic; +use parent 'Languages::_default'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/go.pm b/applets/compiler_vm/host/lib/Languages/go.pm similarity index 86% rename from applets/compiler_vm/languages/go.pm rename to applets/compiler_vm/host/lib/Languages/go.pm index 73282b9d..acc95137 100755 --- a/applets/compiler_vm/languages/go.pm +++ b/applets/compiler_vm/host/lib/Languages/go.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package go; -use parent '_default'; +package Languages::go; +use parent 'Languages::_default'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/haskell.pm b/applets/compiler_vm/host/lib/Languages/haskell.pm similarity index 92% rename from applets/compiler_vm/languages/haskell.pm rename to applets/compiler_vm/host/lib/Languages/haskell.pm index c807cb66..225b6f43 100755 --- a/applets/compiler_vm/languages/haskell.pm +++ b/applets/compiler_vm/host/lib/Languages/haskell.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package haskell; -use parent '_default'; +package Languages::haskell; +use parent 'Languages::_default'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/java.pm b/applets/compiler_vm/host/lib/Languages/java.pm similarity index 99% rename from applets/compiler_vm/languages/java.pm rename to applets/compiler_vm/host/lib/Languages/java.pm index 8dd7e8fd..04fcaecc 100755 --- a/applets/compiler_vm/languages/java.pm +++ b/applets/compiler_vm/host/lib/Languages/java.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package java; -use parent '_default'; +package Languages::java; +use parent 'Languages::_default'; use Text::Balanced qw/extract_bracketed/; diff --git a/applets/compiler_vm/languages/javascript.pm b/applets/compiler_vm/host/lib/Languages/javascript.pm similarity index 91% rename from applets/compiler_vm/languages/javascript.pm rename to applets/compiler_vm/host/lib/Languages/javascript.pm index 8a3d77ec..9f0c46b2 100755 --- a/applets/compiler_vm/languages/javascript.pm +++ b/applets/compiler_vm/host/lib/Languages/javascript.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package javascript; -use parent '_default'; +package Languages::javascript; +use parent 'Languages::_default'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/ksh.pm b/applets/compiler_vm/host/lib/Languages/ksh.pm similarity index 91% rename from applets/compiler_vm/languages/ksh.pm rename to applets/compiler_vm/host/lib/Languages/ksh.pm index c3ce2572..5f8d1b61 100755 --- a/applets/compiler_vm/languages/ksh.pm +++ b/applets/compiler_vm/host/lib/Languages/ksh.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package ksh; -use parent '_default'; +package Languages::ksh; +use parent 'Languages::_default'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/lua.pm b/applets/compiler_vm/host/lib/Languages/lua.pm similarity index 92% rename from applets/compiler_vm/languages/lua.pm rename to applets/compiler_vm/host/lib/Languages/lua.pm index 12595277..21355907 100755 --- a/applets/compiler_vm/languages/lua.pm +++ b/applets/compiler_vm/host/lib/Languages/lua.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package lua; -use parent '_default'; +package Languages::lua; +use parent 'Languages::_default'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/perl.pm b/applets/compiler_vm/host/lib/Languages/perl.pm similarity index 94% rename from applets/compiler_vm/languages/perl.pm rename to applets/compiler_vm/host/lib/Languages/perl.pm index 99e9c8cd..9eb5eeb3 100755 --- a/applets/compiler_vm/languages/perl.pm +++ b/applets/compiler_vm/host/lib/Languages/perl.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package perl; -use parent '_default'; +package Languages::perl; +use parent 'Languages::_default'; use Text::ParseWords qw(shellwords); diff --git a/applets/compiler_vm/languages/php.pm b/applets/compiler_vm/host/lib/Languages/php.pm similarity index 86% rename from applets/compiler_vm/languages/php.pm rename to applets/compiler_vm/host/lib/Languages/php.pm index 4993a39f..db502b26 100755 --- a/applets/compiler_vm/languages/php.pm +++ b/applets/compiler_vm/host/lib/Languages/php.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package php; -use parent '_default'; +package Languages::php; +use parent 'Languages::_default'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/python.pm b/applets/compiler_vm/host/lib/Languages/python.pm similarity index 92% rename from applets/compiler_vm/languages/python.pm rename to applets/compiler_vm/host/lib/Languages/python.pm index 8047233e..f7b53c50 100755 --- a/applets/compiler_vm/languages/python.pm +++ b/applets/compiler_vm/host/lib/Languages/python.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package python; -use parent '_default'; +package Languages::python; +use parent 'Languages::_default'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/python3.pm b/applets/compiler_vm/host/lib/Languages/python3.pm similarity index 92% rename from applets/compiler_vm/languages/python3.pm rename to applets/compiler_vm/host/lib/Languages/python3.pm index 14d981b3..d935b15f 100755 --- a/applets/compiler_vm/languages/python3.pm +++ b/applets/compiler_vm/host/lib/Languages/python3.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package python3; -use parent '_default'; +package Languages::python3; +use parent 'Languages::_default'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/qbasic.pm b/applets/compiler_vm/host/lib/Languages/qbasic.pm similarity index 92% rename from applets/compiler_vm/languages/qbasic.pm rename to applets/compiler_vm/host/lib/Languages/qbasic.pm index 25585ac5..398a3606 100755 --- a/applets/compiler_vm/languages/qbasic.pm +++ b/applets/compiler_vm/host/lib/Languages/qbasic.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package qbasic; -use parent '_default'; +package Languages::qbasic; +use parent 'Languages::_default'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/ruby.pm b/applets/compiler_vm/host/lib/Languages/ruby.pm similarity index 86% rename from applets/compiler_vm/languages/ruby.pm rename to applets/compiler_vm/host/lib/Languages/ruby.pm index 0d16d82e..f41c027c 100755 --- a/applets/compiler_vm/languages/ruby.pm +++ b/applets/compiler_vm/host/lib/Languages/ruby.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package ruby; -use parent '_default'; +package Languages::ruby; +use parent 'Languages::_default'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/scheme.pm b/applets/compiler_vm/host/lib/Languages/scheme.pm similarity index 93% rename from applets/compiler_vm/languages/scheme.pm rename to applets/compiler_vm/host/lib/Languages/scheme.pm index 7f90230b..333f6c13 100755 --- a/applets/compiler_vm/languages/scheme.pm +++ b/applets/compiler_vm/host/lib/Languages/scheme.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package scheme; -use parent '_default'; +package Languages::scheme; +use parent 'Languages::_default'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/sh.pm b/applets/compiler_vm/host/lib/Languages/sh.pm similarity index 91% rename from applets/compiler_vm/languages/sh.pm rename to applets/compiler_vm/host/lib/Languages/sh.pm index 7c5c9c9c..5872c0a9 100755 --- a/applets/compiler_vm/languages/sh.pm +++ b/applets/compiler_vm/host/lib/Languages/sh.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package sh; -use parent '_default'; +package Languages::sh; +use parent 'Languages::_default'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/tcl.pm b/applets/compiler_vm/host/lib/Languages/tcl.pm similarity index 90% rename from applets/compiler_vm/languages/tcl.pm rename to applets/compiler_vm/host/lib/Languages/tcl.pm index 0d85e1d4..33089c1b 100755 --- a/applets/compiler_vm/languages/tcl.pm +++ b/applets/compiler_vm/host/lib/Languages/tcl.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package tcl; -use parent '_default'; +package Languages::tcl; +use parent 'Languages::_default'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/tendra.pm b/applets/compiler_vm/host/lib/Languages/tendra.pm similarity index 96% rename from applets/compiler_vm/languages/tendra.pm rename to applets/compiler_vm/host/lib/Languages/tendra.pm index a08e2539..4e2eac5e 100755 --- a/applets/compiler_vm/languages/tendra.pm +++ b/applets/compiler_vm/host/lib/Languages/tendra.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package tendra; -use parent '_c_base'; +package Languages::tendra; +use parent 'Languages::_c_base'; sub initialize { my ($self, %conf) = @_; diff --git a/applets/compiler_vm/languages/zsh.pm b/applets/compiler_vm/host/lib/Languages/zsh.pm similarity index 91% rename from applets/compiler_vm/languages/zsh.pm rename to applets/compiler_vm/host/lib/Languages/zsh.pm index 1428d927..3152b86d 100755 --- a/applets/compiler_vm/languages/zsh.pm +++ b/applets/compiler_vm/host/lib/Languages/zsh.pm @@ -6,8 +6,8 @@ use warnings; use strict; -package zsh; -use parent '_default'; +package Languages::zsh; +use parent 'Languages::_default'; sub initialize { my ($self, %conf) = @_; diff --git a/doc/README.md b/doc/README.md index 960cb0bd..687ea77d 100644 --- a/doc/README.md +++ b/doc/README.md @@ -317,11 +317,27 @@ * [Virtual Machine](VirtualMachine.md#virtual-machine) * [About](VirtualMachine.md#about) - * [Creating a new virtual machine](VirtualMachine.md#creating-a-new-virtual-machine) - * [Configuring the virtual machine](VirtualMachine.md#configuring-the-virtual-machine) - * [Installing Linux in the virtual machine](VirtualMachine.md#installing-linux-in-the-virtual-machine) - * [Configuring Linux for PBot Communication](VirtualMachine.md#configuring-linux-for-pbot-communication) - * [Hardening the PBot virtual machine](VirtualMachine.md#hardening-the-pbot-virtual-machine) + * [Initial virtual machine set-up](VirtualMachine.md#initial-virtual-machine-set-up) + * [Prerequisites](VirtualMachine.md#prerequisites) + * [CPU Virtualization Technology](VirtualMachine.md#cpu-virtualization-technology) + * [KVM](VirtualMachine.md#kvm) + * [libvirt](VirtualMachine.md#libvirt) + * [Make a pbot-vm user or directory](VirtualMachine.md#make-a-pbot-vm-user-or-directory) + * [Add libvirt group to your user](VirtualMachine.md#add-libvirt-group-to-your-user) + * [Download Linux ISO](VirtualMachine.md#download-linux-iso) + * [Creating a new virtual machine](VirtualMachine.md#creating-a-new-virtual-machine) + * [Installing Linux in the virtual machine](VirtualMachine.md#installing-linux-in-the-virtual-machine) + * [Configuring virtual machine for PBot](VirtualMachine.md#configuring-virtual-machine-for-pbot) + * [Set up serial ports](VirtualMachine.md#set-up-serial-ports) + * [Install Perl](VirtualMachine.md#install-perl) + * [Install PBot VM Guest](VirtualMachine.md#install-pbot-vm-guest) + * [Install software](VirtualMachine.md#install-software) + * [Start PBot VM Guest](VirtualMachine.md#start-pbot-vm-guest) + * [Test PBot VM Guest](VirtualMachine.md#test-pbot-vm-guest) + * [Save initial state](VirtualMachine.md#save-initial-state) + * [Initial virtual machine set-up complete](VirtualMachine.md#initial-virtual-machine-set-up-complete) + * [Start PBot VM Host](VirtualMachine.md#start-pbot-vm-host) + * [Test PBot](VirtualMachine.md#test-pbot) * [Frequently Asked Questions](FAQ.md#frequently-asked-questions) diff --git a/doc/VirtualMachine.md b/doc/VirtualMachine.md index e2bb7e4e..b410b305 100644 --- a/doc/VirtualMachine.md +++ b/doc/VirtualMachine.md @@ -1,5 +1,30 @@ # Virtual Machine + +* [About](#about) +* [Initial virtual machine set-up](#initial-virtual-machine-set-up) + * [Prerequisites](#prerequisites) + * [CPU Virtualization Technology](#cpu-virtualization-technology) + * [KVM](#kvm) + * [libvirt](#libvirt) + * [Make a pbot-vm user or directory](#make-a-pbot-vm-user-or-directory) + * [Add libvirt group to your user](#add-libvirt-group-to-your-user) + * [Download Linux ISO](#download-linux-iso) + * [Creating a new virtual machine](#creating-a-new-virtual-machine) + * [Installing Linux in the virtual machine](#installing-linux-in-the-virtual-machine) + * [Configuring virtual machine for PBot](#configuring-virtual-machine-for-pbot) + * [Set up serial ports](#set-up-serial-ports) + * [Install Perl](#install-perl) + * [Install PBot VM Guest](#install-pbot-vm-guest) + * [Install software](#install-software) + * [Start PBot VM Guest](#start-pbot-vm-guest) + * [Test PBot VM Guest](#test-pbot-vm-guest) + * [Save initial state](#save-initial-state) + * [Initial virtual machine set-up complete](#initial-virtual-machine-set-up-complete) +* [Start PBot VM Host](#start-pbot-vm-host) + * [Test PBot](#test-pbot) + + ## About PBot can interact with a virtual machine to safely execute arbitrary user-submitted @@ -9,29 +34,228 @@ This document will guide you through installing and configuring a virtual machin by using the widely available [libvirt](https://libvirt.org) project tools, such as `virt-install`, `virsh`, `virt-manager`, `virt-viewer`, etc. -Though there are many, many tutorials and walk-throughs available for these tools, -this guide will demonstrate the necessary `virt-install` and `virsh` commands to -configure the virtual machine. +If you're more comfortable working with QEMU directly instead, feel free to do that. +I hope this guide will answer everything you need to know to set that up. If not, +open an GitHub issue or /msg me on IRC. -You may install a guest Linux distribution of your choice. Any of the recent popular -Linux distributions should suffice. This guide will use Fedora Rawhide because -playing around with the latest bleeding-edge software is fun! +Some quick terminology: -Then we will show you the necessary commands to configure the Linux guest system -to be able to communicate with PBot. Commands and code snippets are sent over a -virtual serial cable. We'll show you how to set that up. + * host: your physical Linux system hosting the virtual machine + * guest: the Linux system installed inside the virtual machine -We'll also show a few tips and tricks to help secure the virtual machine against -malicious user-submitted commands. +## Initial virtual machine set-up +These steps need to be done only once during the first-time set-up. -Let's get started. +### Prerequisites +#### CPU Virtualization Technology +Ensure CPU Virtualization Technology is enabled in your motherboard BIOS. -## Creating a new virtual machine + $ egrep '(vmx|svm)' /proc/cpuinfo -## Configuring the virtual machine +If you see your CPUs listed with `vmx` or `svm` flags, you're good to go. +Otherwise, consult your motherboard manual to see how to enable VT. -## Installing Linux in the virtual machine +#### KVM +Ensure KVM is set up and loaded. -## Configuring Linux for PBot Communication + $ kvm-ok + INFO: /dev/kvm exists + KVM acceleration can be used -## Hardening the PBot virtual machine +If you see the above, everything's set up. Otherwise, consult your operating +system manual or KVM manual to install and load KVM. + +#### libvirt +Ensure libvirt is installed and ready. + + $ virsh version --daemon + Compiled against library: libvirt 7.6.0 + Using library: libvirt 7.6.0 + Using API: QEMU 7.6.0 + Running hypervisor: QEMU 6.0.0 + Running against daemon: 7.6.0 + +If there's anything missing, please consult your operating system manual to +install the libvirt and QEMU packages. + +On Ubuntu: `sudo apt install qemu-kvm libvirt-daemon-system` + +#### Make a pbot-vm user or directory +You can either make a new user account or make a new directory in your current user account. +In either case, name it `pbot-vm` so we'll have one place for the install ISO file and the +virtual machine disk and snapshot files. + +#### Add libvirt group to your user +Add your user or the `pbot-vm` user to the `libvirt` group. + + $ sudo adduser $USER libvirt + +Log out and then log back in for the new group to take effect on your user. + +#### Download Linux ISO +Download a preferred Linux ISO. For this guide, we'll use Fedora. Why? I'm +using Fedora Rawhide for my PBot VM because I want convenient and reliable +access to the latest bleeding-edge versions of software. + +I recommend using the Fedora Stable net-installer for this guide unless you +are more comfortable in another Linux distribution. Make sure you choose +the minimal install option without a graphical desktop. + +https://download.fedoraproject.org/pub/fedora/linux/releases/35/Server/x86_64/iso/Fedora-Server-netinst-x86_64-35-1.2.iso +is the Fedora Stable net-installer ISO used in this guide. + +### Creating a new virtual machine +To create a new virtual machine we'll use the `virt-install` command. First, ensure you are +the `pbot-vm` user or that you have changed your current working directory to `pbot-vm`. + + $ virt-install --name=pbot-vm --disk=size=12,cache=none,driver.io=native,snapshot=external,path=vm.qcow2 --cpu=host --os-variant=fedora34 --graphics=spice,gl.enable=yes --video=virtio --location=Fedora-Server-netinst-x86_64-35-1.2.iso + +If you are installing over an X-forwarded SSH session, strip the `,gl.enable=yes` +part. Note that `disk=size=12` will create a 12 GB sparse file. Sparse means the file +won't actually take up 12 GB. It will start at 0 bytes and grow as needed. You can +use the `du` command to verify this. After a minimal Fedora install, the size will be +approximately 1.7 GB. It will grow to about 2.5 GB with most PBot features installed. + +For further information about `virt-install`, read its manual page. While the above command should +give sufficient performance and compatability, there are a great many options worth investigating +if you want to fine-tune your virtual machine. + +#### Installing Linux in the virtual machine +After executing the `virt-install` command above, you should now see a window +showing Linux booting up and launching an installer. For this guide, we'll walk +through the Fedora 35 installer. You can adapt these steps for your own distribution +of choice. + + * Click `Partition disks`. Don't change anything. Click `Done`. + * Click `Root account`. Click `Enable root account`. Set a password. Click `Done`. + * Click `User creation`. Create a new user. Skip Fullname and set Username to `vm`. Untick `Add to wheel` or `Set as administrator`. Untick `Require password`. + * Wait until `Software selection` is done processing and is no longer greyed out. Click it. Change install from `Server` to `Minimal`. Click `Done`. + * Click `Begin installation`. + +Installation will need to download about 328 RPMs consisting of about 425 MB. It'll take 5 minutes to an hour or longer +depending on your hardware and network configuration. + +#### Configuring virtual machine for PBot +Once the install finishes, click the `Reboot` button in the Fedora installer in the virtual machine window. + +#### Set up serial ports +Now, while the virtual machine is rebooting, switch to a terminal on your host system. Go into the +`pbot-vm/host/devices` directory and run the `add-serials` script. Feel free to look inside. It will add +the `serial-2.xml` and `serial-3.xml` files to the configuration for the `pbot-vm` libvirt machine. + +This will enable and connect the `/dev/ttyS1` and `/dev/ttyS2` serial ports inside the virtual machine +to TCP connections on `127.0.0.1:5555` and `127.0.0.1:5556`, respectively. `ttyS1/5555` is the data +channel used to send commands or code to the virtual machine and to read back output. `ttyS2/5556` is +simply a newline sent every 5 seconds, representing a heartbeat, used to ensure that the PBot communication +channel is healthy. + +Once that's done, switch back to the virtual machine window. Once the virtual machine has rebooted, +log in as `root`. Now go ahead and shut the virtual machine down with `shutdown now -h`. We need to +restart the virtual machine itself so it loads the new serial device configuration. Once the machine +has shutdown, bring it right back up with the following commands on the host system in the terminal +used for `virt-install`: + + $ virsh start pbot-vm + +Now the virtual machine will start back up in the background. + + $ virt-viewer pbot-vm + +Now you should see the virtual machine window after a few seconds. Log in as `root` once the login +prompt appears. + +#### Install Perl +Now we need to install Perl inside the virtual machine. This allows us to run the PBot VM Guest +script. + + $ dnf install perl-interpreter perl-lib perl-IPC-Run perl-JSON-XS perl-English + +That installs the minium packages for the Perl interpreter (note we used `perl-interpreter` instead of `perl`), +the package for the Perl `lib`, `IPC::Run`, `JSON::XS` and `English` modules. + +#### Install PBot VM Guest +Next we install the PBot VM Guest script that fosters communication between the virtual machine guest +and the physical host system. We'll do this inside the virtual machine guest system. + +The `rsync` command isn't installed in a Fedora minimal install, but `scp` is available. Replace +`192.168.100.42` below with your own local IP address and `user` with the user account that has the +PBot directory and `pbot` with the path to the directory. + + $ scp -r user@192.168.100.42:~/pbot/applets/pbot-vm/guest . + +Once that's done, run the following command: + + $ ./guest/bin/setup-guest + +Feel free to take a look inside to see what it does. It's very short. After running +the `setup-guest` script make sure you run `source /root/.bashrc` so the environment +changes take effect. + +#### Install software +Now you can install any languages you want to use. + +Python3 is already installed. + +For the C programming language you will need at least these: + + $ dnf install libubsan libasan gdb gcc clang + +I'll list all the packages for the others soon. You can use `dnf search ` to find packages +in Fedora. + +#### Start PBot VM Guest +We're ready to start the PBot VM Guest. + + $ start-guest + +This starts up a server to listen for incoming commands or code and to handle them. We'll leave +this running. + +#### Test PBot VM Guest +Let's make sure everything's working up to this point. There should be two open TCP ports on +`5555` and `5556`. + + $ nc -zv 127.0.0.1 5555-5556 + +If it says anything other than `Connection succeeded` then make sure you have completed the steps +under [Set up serial ports](#set-up-serial-ports) and that your network configuration is allowing +access. + +Let's make sure the PBot VM Guest is listening for and can execute commands. The `vm-exec` command +in the `applets/pbot-vm/host/bin` directory allows you to send commands from the shell. + + $ vm-exec -lang=sh echo hello world + +This should output some logging noise followed by "hello world". You can test other language modules +by changing the `-lang=` option. I recommend testing and verifying that all of your desired language +modules are configured before going on to the next step. + +If you have multiple PBot VM Guests, or if you used a different TCP port, you can specify the +`PBOT_VM_PORT` environment variable when executing the `vm-exec` command: + + $ PBOT_VM_PORT=6666 vm-exec -lang=sh echo test + +#### Save initial state +Switch back to an available terminal on the physical host machine. Enter the following command +to save a snapshot of the virtual machine waiting for incoming commands. + + $ virsh snapshot-create-as pbot-vm 1 + +This will create a snapshot file `vm.1` next to the `vm.qcow2` disk file. If the virtual machine +ever times-out or its heartbeat stops responding, PBot will reset the virtual machine to this +saved snapshot. + +### Initial virtual machine set-up complete +This concludes the initial one-time set-up. You can close the `virt-viewer` window. The +virtual machine will continue running in the background until it is manually shutdown (via +`shutdown now -h` inside the VM or via `virsh shutdown pbot-vm` on the host). + +## Start PBot VM Host +To start the PBot VM Host server, execute the `vm-server` script in the +`applets/pbot_vm/host/bin` directory on the host. + +This will start a TCP server on port `9000`. It will listen for incoming commands and +pass them along to the virtual machine's TCP serial port. + +### Test PBot +All done. Everything is set up now. In your instance of PBot, the `sh echo hello` command should output `hello`.