QEMU GameZIP Server: Difference between revisions

From Flashpoint Datahub
Jump to navigation Jump to search
No edit summary
m (Nosamu moved page GameZIP Server to QEMU GameZIP Server: Disambiguating with the new Flashpoint Game Server)
 
(9 intermediate revisions by 3 users not shown)
Line 14: Line 14:
The GameZIP implementation still uses Apache but it runs on Alpine Linux instead. This allows the use of FUSE file systems to achieve a simulated view of "merged htdocs" without the extraction of GameZIPs.
The GameZIP implementation still uses Apache but it runs on Alpine Linux instead. This allows the use of FUSE file systems to achieve a simulated view of "merged htdocs" without the extraction of GameZIPs.
This instance of Alpine Linux is emulated under QEMU. QEMU was chosen because it requires no driver installation or special privileges and Alpine Linux because of its low file size and overhead.
This instance of Alpine Linux is emulated under QEMU. QEMU was chosen because it requires no driver installation or special privileges and Alpine Linux because of its low file size and overhead.
The Linux guest cannot directly access files on the host file system, so an FTP server runs on the host to share needed files.
The Linux guest cannot directly access files on the host file system, instead, when a file lookup fails, the request is proxied to a fallback server to allow loading files for Legacy games. This is the Legacy Apache server in the case of Flashpoint Ultimate or the PHP router in the case of Flashpoint Infinity.


=== Why emulate Linux? ===
=== Why emulate Linux? ===
Line 22: Line 22:


== How does it work? ==
== How does it work? ==
Here is a high-level overview of how files are served:
[[File:Qemu.png|800px]]
And here is a more detailed look at how the GameZip server works:


[[File:QEMU-FP.png]]
[[File:QEMU-FP.png]]
Line 28: Line 34:
This path is the target of a union mount, which merges the following paths:
This path is the target of a union mount, which merges the following paths:
* /root/base: Contains the PHP scripts required for mounting game ZIPs and the <code>.htaccess</code> that rewrites the HTTP Host header into a subfolder.
* /root/base: Contains the PHP scripts required for mounting game ZIPs and the <code>.htaccess</code> that rewrites the HTTP Host header into a subfolder.
* /tmp/htdocs: (Initially unmounted) Contains the contents of the existing merged htdocs, for non-GameZIP games.
* /tmp/<code>[device]</code>: (Initially unmounted) Contains the contents of a single GameZIP.
* /tmp/_<code>[game]</code>.zip: (Initially unmounted) Contains the contents of a single GameZIP.


When the emulated Alpine Linux instance starts, only <code>/root/base</code> is part of the union.
When the emulated Alpine Linux instance starts, only <code>/root/base</code> is part of the union.
Line 37: Line 42:
The scripts found in <code>/root/base</code> make up the API used for controlling the union mount.
The scripts found in <code>/root/base</code> make up the API used for controlling the union mount.


* [GET] <code>http://127.0.0.1:22500/wake.php</code>
* [GET] <code>http://127.0.0.1:22500/mount.php?file=[device_serial_code]</code>


This endpoint must receive a request at least once before games in the old htdocs folder can be played. The launcher should send a request to it on startup.
This endpoint allows mounting a game ZIP from an attached block device into the virtual htdocs.
 
This performs the following actions:
 
1. Mounts <code>Server\htdocs</code> from Flashpoint into <code>/mnt/htdocs</code> via FTP (using curlftpfs)
 
2. Mounts <code>Games</code> from Flashpoint into <code>/mnt/games</code> via FTP (using curlftpfs)
 
3. Mounts <code>/mnt/htdocs</code> into <code>/tmp/htdocs</code> as a case-insensitive mirror (using fuzzyfs)
 
4. Adds <code>/tmp/htdocs</code> to the union mount.
 
* [GET] <code>http://127.0.0.1:22500/mount.php?file=[filename]</code>
 
This endpoint allows mounting a game ZIP file into the virtual htdocs. The specified file must be located in the Games folder under the Flashpoint directory.
It has a few possible outcomes which it will print in plain text:
It has a few possible outcomes which it will print in plain text:
  “OK” (200): The file was mounted successfully.
  “OK” (200): The file was mounted successfully.
Line 67: Line 58:
This performs the following actions:
This performs the following actions:


1. If <code>wake.php</code> has not run yet. It will run first.
1. Finds the attached block device with the specified serial.
 
If no device was found with this serial, it checks again once every 100 ms.
If no matching device was found after 10 seconds, "NO_SUCH_FILE" is returned.
 
2. A symbolic link is created from <code>/dev/[device]</code> to <code>/tmp/[device].zip</code>
 
This is done in order to trick AVFS into thinking that the block device is a regular ZIP file.
AVFS can transparently access ZIP files as a folder by appending a pound sign after the filename.
An AVFS mount of the root directory exists in <code>/root/.avfs</code>
 
3. Mounts <code>/root/.avfs/tmp/[device].zip#</code> to <code>/tmp/[device]</code> as a case-insensitive mirror using fuzzyfs


2. Mounts <code>/mnt/games/[filename]</code> to <code>/tmp/[filename]</code> as a ZIP file (using fuse-zip)
fuzzyfs is a read-only filesystem that allows case-insensitive access to files.


3. Mounts <code>/tmp/[filename]</code> to <code>/tmp/_[filename]</code> as a case-insensitive mirror (using fuzzyfs)
4. Adds <code>/tmp/[device]/content</code> to the union mount.


4. Adds <code>/tmp/_[filename]</code> to the union mount.
This is done by unmounting <code>/var/www/localhost/htdocs</code> then re-mounting with a new set of settings.


=== Shell access ===
=== Shell access ===
Line 100: Line 102:


=== Installing Alpine Linux ===
=== Installing Alpine Linux ===
'''[https://drive.google.com/file/d/1pnbA9l4E0vi0EhmeO9Qi64ifvSbHShjF/view?usp=sharing Download Fresh Alpine Linux image (Updated: 2020-09-21)]'''


* Download the latest "Virtual" Alpine Linux release for x86 processors [https://www.alpinelinux.org/downloads/]
* Download the latest "Virtual" Alpine Linux release for x86 processors [https://www.alpinelinux.org/downloads/]
Line 118: Line 122:
=== Setting up the Flashpoint toolchain ===
=== Setting up the Flashpoint toolchain ===


* Copy the PHP scripts/.htaccess that will be placed in <code>/root/base</code> to <code>Server\htdocs\base</code>
* Run Flashpoint Launcher (this will start the FTP server which allows access to the PHP scripts)
* Boot the machine
* Boot the machine
  qemu-system-i386 -m 512 -net nic,model=e1000 -net user,hostfwd=tcp::22500-:80 -hda alpine.qcow2
  qemu-system-i386 -m 512 -net nic,model=e1000 -net user -hda alpine.qcow2
* Run the setup script (on the guest)
* Run the setup script (on the guest)
  cd /tmp
  cd /tmp
  wget https://tinyurl.com/setfpup -O setup
  wget https://go.nul.sh/fpultimate -O setup
  chmod +x setup
  chmod +x setup
  ./setup
  ./setup
For Infinity use: https://go.nul.sh/fpinfinity
* Clean up <code>/tmp</code> and power off the machine (on the guest)
* Clean up <code>/tmp</code> and power off the machine (on the guest)
  rm -r /tmp/*
  rm -r /tmp/*
Line 138: Line 141:
Finally, we need to set up our special save state to resume execution from.
Finally, we need to set up our special save state to resume execution from.
* Boot the machine
* Boot the machine
  qemu-system-i386 -m 512 -net nic,model=e1000 -net user,hostfwd=tcp::22500-:80 -hda alpine.qcow2
  qemu-system-i386 -m 512 -net nic,model=e1000 -net user -hda alpine.qcow2
* Run the following commands (on the guest)
* Run the following commands (on the guest)
modprobe fuse
unionfs /root/base /var/www/localhost/htdocs -o allow_other
  clear
  clear
  tail -f /var/log/apache2/access.log >/dev/ttyS0
  tail -f /var/log/apache2/access.log >/dev/ttyS0
Line 147: Line 148:


Your <code>alpine.qcow2</code> image is ready!
Your <code>alpine.qcow2</code> image is ready!
<noinclude>[[Category:Technologies]]</noinclude>

Latest revision as of 12:10, 20 October 2023

The server allows accessing zipped game contents, also known as GameZIPs, as well as merged htdocs. This page aims to cover the details of the Flashpoint implementation.

Glossary

  • GameZIP: A zipped curation content folder, according to the GameZIP specification. Read More
  • merged htdocs: The root folder where web content is served from, contains matching file structure for hostnames used by games in Flashpoint.
  • FUSE: Filesystem in Userspace
  • union mount: A union mount is a mount which can appear to merge the contents of several directories, while keeping their physical content separate. [1]

Introduction

In the past, Flashpoint's server was an Apache httpd installation running on Windows. The GameZIP implementation still uses Apache but it runs on Alpine Linux instead. This allows the use of FUSE file systems to achieve a simulated view of "merged htdocs" without the extraction of GameZIPs. This instance of Alpine Linux is emulated under QEMU. QEMU was chosen because it requires no driver installation or special privileges and Alpine Linux because of its low file size and overhead. The Linux guest cannot directly access files on the host file system, instead, when a file lookup fails, the request is proxied to a fallback server to allow loading files for Legacy games. This is the Legacy Apache server in the case of Flashpoint Ultimate or the PHP router in the case of Flashpoint Infinity.

Why emulate Linux?

For our purposes, Apache is limited to a single web root and cannot serve files from ZIPs. Custom file systems require custom Windows drivers and are less mature or performant than FUSE. In addition, the use of Linux allows for the use of the entire GNU/Linux toolkit which is very flexible. It also provides a security advantage, as it sandboxes potentially untrusted PHP scripts.

How does it work?

Here is a high-level overview of how files are served:

Qemu.png

And here is a more detailed look at how the GameZip server works:

QEMU-FP.png

The Apache installation has a single web root, found at /var/www/localhost/htdocs. This path is the target of a union mount, which merges the following paths:

  • /root/base: Contains the PHP scripts required for mounting game ZIPs and the .htaccess that rewrites the HTTP Host header into a subfolder.
  • /tmp/[device]: (Initially unmounted) Contains the contents of a single GameZIP.

When the emulated Alpine Linux instance starts, only /root/base is part of the union.

Mount API

The scripts found in /root/base make up the API used for controlling the union mount.

This endpoint allows mounting a game ZIP from an attached block device into the virtual htdocs. It has a few possible outcomes which it will print in plain text: 

“OK” (200): The file was mounted successfully.
“ALREADY_MOUNTED” (200):
The file was already previously mounted and remains available.
“NO_SUCH_FILE” (400):
The specified file was not found under the Games folder.
“NO_CONTENT_FOLDER” (400):
The specified file was not mounted because it did not contain a content folder.
“BAD_ZIP” (400):
The specified file was not mounted because it was not a valid ZIP file.

This performs the following actions:

1. Finds the attached block device with the specified serial.

If no device was found with this serial, it checks again once every 100 ms. If no matching device was found after 10 seconds, "NO_SUCH_FILE" is returned.

2. A symbolic link is created from /dev/[device] to /tmp/[device].zip

This is done in order to trick AVFS into thinking that the block device is a regular ZIP file. AVFS can transparently access ZIP files as a folder by appending a pound sign after the filename. An AVFS mount of the root directory exists in /root/.avfs

3. Mounts /root/.avfs/tmp/[device].zip# to /tmp/[device] as a case-insensitive mirror using fuzzyfs

fuzzyfs is a read-only filesystem that allows case-insensitive access to files.

4. Adds /tmp/[device]/content to the union mount.

This is done by unmounting /var/www/localhost/htdocs then re-mounting with a new set of settings.

Shell access

Open Data\services.json with a text editor. Remove "-display", "none" from the arguments array in Apache Webserver and save. The next time you run Flashpoint, you'll get a terminal to Alpine Linux. The root user has a blank password.

QEMU.png

Press Ctrl+C to terminate the tail command, this is used to pipe the contents of the Apache access log to the Flashpoint Launcher.

To avoid the Linux boot time when starting the launcher, a QEMU savestate is used.

  • Press Ctrl+Alt+2 to access the QEMU console. Type loadvm quick to reset the state of the machine.
  • Press Ctrl+Alt+1 to return to the Alpine Linux shell.

Building the Alpine Linux image

Prerequisites

Flashpoint contains a subset of the QEMU binaries to cut down of space. This means qemu-img is missing.

  • Download the latest QEMU binaries [2]
  • Extract them to the Server folder in Flashpoint

Installing Alpine Linux

Download Fresh Alpine Linux image (Updated: 2020-09-21)

  • Download the latest "Virtual" Alpine Linux release for x86 processors [3]
  • Create a virtual disk image
qemu-img create -f qcow2 alpine.qcow2 2G

This will create a virtual disk image which will grow as needed up to a size of 2 GB.

  • Boot the machine
qemu-system-i386 -m 512 -net nic,model=e1000 -net user -hda alpine.qcow2 -boot d -cdrom alpine-virt-3.12.0-x86.iso

This will boot the machine with maximum 512 MB of RAM and a bridged networking card.

Go through the installation process. Once done, power off the machine using poweroff.

It is recommended to back up the qcow2 image at this point.

Setting up the Flashpoint toolchain

  • Boot the machine
qemu-system-i386 -m 512 -net nic,model=e1000 -net user -hda alpine.qcow2
  • Run the setup script (on the guest)
cd /tmp
wget https://go.nul.sh/fpultimate -O setup
chmod +x setup
./setup

For Infinity use: https://go.nul.sh/fpinfinity

  • Clean up /tmp and power off the machine (on the guest)
rm -r /tmp/*
poweroff
  • Shrink your qcow2 image (Recommended)
qemu-img convert -O qcow2 alpine.qcow2 alpine_setup.qcow2
copy /Y alpine_setup.qcow2 alpine.qcow2

Finalizing the image

Finally, we need to set up our special save state to resume execution from.

  • Boot the machine
qemu-system-i386 -m 512 -net nic,model=e1000 -net user -hda alpine.qcow2
  • Run the following commands (on the guest)
clear
tail -f /var/log/apache2/access.log >/dev/ttyS0
  • Press Ctrl+Alt+2. Type savevm quick and press Enter. Wait, then quit QEMU.

Your alpine.qcow2 image is ready!

Retrieved from "https://flashpointarchive.org/w/index.php?title=QEMU_GameZIP_Server&oldid=9607"