A real world example

You might think that this is not such a big issue since you most likely won't have a file structure as I described above. But I had to deal with this a lot more than I should've in the past. If you've ever worked on setting up Minecraft servers, you probably know what I'm talking about. See, Minecraft worlds are structured almost exactly how I described it above. They're made up of a lot of small files which, when you add them up, are not big at all. … And it still takes FTP multiple minutes to delete them, aaargh. And it's even worse if you want to upload a new world later.

Same thing happens with larger codebases consisting of hundreds of files. Of course, if you still have any sanity left in you, you would use Git or some other VCS to move around your code. But sadly that's not always the case, which I got to learn the hard way. So FTP to the rescue I guess.

If you're wondering why I'm complaining about a couple of minutes, go and read Software Disenchantment. It's a really entertaining blog post by Niki Tonsky .

BuT Antony, SfTp ExIsTs

Yes, SFTP does exist. And it is better than FTP… sometimes. SFTP is a lot more secure, thanks to one of the best protocols out there SSH. But additional security, doesn't eliminate the speed issues. I've experienced SFTP to be slower than FTP in the real world, but for some reason, it showed to be a bit faster while I was doing testing for this post.

RSYNC Man Wiki

This is probably one of the most versitile utilities for transfering files. While not being a protocol itself, it does work over SSH, and we all love SSH. It's fast, it's simple and it works. Hell, you can even copy files while keeping their permissions, groups, links, modification times etc.

Performing a basic file copy is as simple as:

rsync -avP /source username@host:/destination

You can also add an optional --delete flag to delete extra files from destination

SCP Man Wiki

Secure copy protocol is also a very nice thing if you want to perform a very easy and basic copy to a remote location. It also works via SSH. It's a bit more easy to use than RSYNC but not nearly as versitile.

A file copy can be done with:

scp -r /source username@host:/destination

Good ol' HTTPS

Well, HTTPS, as you probably know, is not a protocol for transfering files. But it can do it. And from my usage in the real world I saw it performed amazingly. Pterodactyl has an absolutely amazing file manager working over just HTTP(S) and it has been nothing but joy to work with.

In my testing I just set up a very simple Express app that is just grabbing files and writing them to disk.

Here's the relevant code:

app.post("/upload", multer().any(), async (req, res) => {
    const files = req.files;
    if(!files || !files.length || !Array.isArray(files))
        return res.status(400);

    for(const file of files) {
        const destination = `./upload/${file.originalname}`;
        const dir = dirname(destination);
        fs.mkdir(dir, { recursive: true });
        fs.writeFile(destination, file.buffer);
    }
    
    logger.info("Uploaded " + files.length + " files (" + files.reduce((acc, curr) => acc + curr.size, 0) + " bytes)")
    res.status(201).end();
});

And to upload, I made a small shell script to collect all the files and make a request to the server

#!/bin/bash

FILES=$(find world/ -type f -exec printf " -F files[]=@{}" ;)

curl $FILES http://host:port/upload