My backup procedure - Kulverstukas's blog

Oh hello again… man, lately I have no time to keep my blog updated, but I’m glad I can write a post occasionally at least.
This time I’d like to write about how I do backups and a way I automated half of it. It might not be the most efficient way or “the right way”, but as with anything – it works for me, so shut up! :P (j/k, you can bitch in the comments).
Anyway, I have several places to grab backups from and one place to store them – I don’t use mainstream cloud services (maybe I should…?), instead I store those onto my external drive. Google drive gives free 15GB to store stuff, in the future I might use that instead of my external, but the place doesn’t matter, it’s HOW I store those backups that matters.

Like I said before, I have several places to backup, which include a couple of websites with insufficient storage space to keep backups on the server (you should keep them off site anyway eh? :)) and some of important computer files and their system images.

With computers it’s basic – clean, defrag, make system clone, transfer to external hdd.
I also have a raspberry home media server (wrote about it here), I make an SD card clone of that too. Some say I should use RSync instead, so I don’t have to power it off, but tbh it seems less hassle to simply make a clone off SD card and store it.
After making a clone, I use a software like FBackup to sync my important ~~porn~~ files.

After that I would go ahead and order website backups through DirectAdmin, download those and then transfer them, but this part was so tedious that I would often forget about it. So I thought of automating that by writing up a simple python backup script.
The automated process is made up of 2 parts – PHP scripts that send requests to DA on servers gets run via cronjob at midnight, backups get made in few minutes. Then the Python script running at 1AM on my RPI grabs these backup archives via FTP from those websites and stores them on NAS.
Simple as that.

PHP files are available publicly from DirectAdmin here and my Python backup script is available here.

Hopefully it helps someone to save time as it does for me :)

Also for a quick overview:

```
#!/usr/local/bin/python
```
```
'''
```
```
    Date: 2015.08.02
```
```
    Author: Kulverstukas
```
```
    Website: 9v.lt; evilzone.org
```
```
    Description:
```

        Downloads backup files from /backups in set hosts

        and downloads whole website content from defined website.

```
'''
```
```
 
```
```
import os
```
```
import sys
```
```
import time
```
```
import shutil
```
```
import zipfile
```
```
import ftputil
```
```
from ftplib import FTP
```
```
 
```

#-------------------------------------------------------

```
# where to put everything
```
```
rootFolder = 'webhost_backups'
```
```
 
```

# configuration of as many webshosts as you have

```
webhosts = [
```

    {'folder' : 'somesite.lt', # where to store your files

     'address' : 'somesite.lt', # URL of your website

     'username' : 'lorem', # FTP username

     'password' : 'ipsum', # FTP password

     'backup_path' : '/backups', # usually this is where backups are stored on the server

     'mirror' : True}, # should mirror the whole website starting from root

```
 
```

     {'folder' : 'subdomain.domain.com',

     'address' : 'subdomain.domain.com',

```
     'username' : 'lorem',
```
```
     'password' : 'ipsum',
```
```
     'backup_path' : '/backups',
```
```
     'mirror' : False}
```
```
]
```

#-------------------------------------------------------

# create folders if they don't exist already

if not os.path.exists(rootFolder) or not os.path.isdir(rootFolder):

```
    os.mkdir(rootFolder)
```

#-------------------------------------------------------

# replace symbols with encoded equivalents

```
def makeFriendlyFilename(input):
```
```
    badSymbols = {'<' : '%3C',
```
```
                  '>' : '%3E',
```
```
                  ':' : '%3A',
```
```
                  '"' : '%22',
```
```
                  '|' : '%7C',
```
```
                  '?' : '%3F',
```
```
                  '*' : '%2A'}
```
```
    i = ''
```
```
    for i in badSymbols.keys():
```

        input = input.replace(i, badSymbols[i]);

```
    return input
```

#-------------------------------------------------------

```
def zipdir(path, zipname):
```

    zipf = zipfile.ZipFile(zipname, 'w')

    for root, dirs, files in os.walk(path):

```
        for dir in dirs:
```

            zipf.write(os.path.join(root, dir))

```
        for file in files:
```

            zipf.write(os.path.join(root, file))

```
    zipf.close()
```

#-------------------------------------------------------

```
# download all of the files from FTP
```
```
def mirrorFtp():
```
```
    for config in webhosts:
```
```
        if config['mirror']:
```

            today = time.strftime("%Y-%m-%d")

            mirrorName = config['folder']+'_mirror_'+today

            localMirrorRoot = os.path.join(rootFolder, config['folder'], mirrorName)

            if not os.path.exists(localMirrorRoot) or not os.path.isdir(localMirrorRoot):

                os.makedirs(localMirrorRoot)

```
 
```

            ftp = ftputil.FTPHost(config['address'], config['username'], config['password'])

            for root, dirs, files in ftp.walk('/'):

```
                for dir in dirs:
```
```
                    try:
```

                        os.mkdir(os.path.join(localMirrorRoot, root[1:], dir))

```
                    except:
```
```
                        pass
```
```
                for file in files:
```
```
                    try:
```

                        ftp.download(root+'/'+file, os.path.join(localMirrorRoot, root[1:], makeFriendlyFilename(file)))

```
                    except:
```

                        print root+'/'+file

            zipdir(localMirrorRoot, localMirrorRoot+'.zip')

            shutil.rmtree(localMirrorRoot)

#-------------------------------------------------------

# download backup files made by DirectAdmin

```
def getBackupFiles():
```
```
    ftp = FTP()
```
```
    for config in webhosts:
```

        if not os.path.exists(rootFolder+'/'+config['folder']) or not os.path.isdir(rootFolder+'/'+config['folder']):

            os.mkdir(rootFolder+'/'+config['folder'])

```
        try:
```

            ftp.connect(config['address'])

            ftp.login(config['username'], config['password'])

            ftp.cwd(config['backup_path'])

```
            filesToDl = ftp.nlst();
```
```
            for file in filesToDl:
```

                if (file not in ['.', '..']):

                    ftp.retrbinary('RETR '+file, open(rootFolder+'/'+config['folder']+'/'+file, 'wb').write)

```
                    try:
```

                        # comment this out if you want to keep backup archives on the server

                        ftp.delete(file)

```
                    except:
```
```
                        pass
```
```
            ftp.quit()
```
```
        except:
```
```
            pass
```

#-------------------------------------------------------

```
 
```
```
getBackupFiles()
```
```
mirrorFtp()
```
```
 
```

#-------------------------------------------------------