My backup procedure

Oh hello again… man, lately I have no time to keep my blog updated, but I’m glad I can write a post occasionally at least.
This time I’d like to write about how I do backups and a way I automated half of it. It might not be the most efficient way or “the right way”, but as with anything – it works for me, so shut up! :P (j/k, you can bitch in the comments).
Anyway, I have several places to grab backups from and one place to store them – I don’t use mainstream cloud services (maybe I should…?), instead I store those onto my external drive. Google drive gives free 15GB to store stuff, in the future I might use that instead of my external, but the place doesn’t matter, it’s HOW I store those backups that matters.

Like I said before, I have several places to backup, which include a couple of websites with insufficient storage space to keep backups on the server (you should keep them off site anyway eh? :)) and some of important computer files and their system images.

With computers it’s basic – clean, defrag, make system clone, transfer to external hdd.
I also have a raspberry home media server (wrote about it here), I make an SD card clone of that too. Some say I should use RSync instead, so I don’t have to power it off, but tbh it seems less hassle to simply make a clone off SD card and store it.
After making a clone, I use a software like FBackup to sync my important porn files.

After that I would go ahead and order website backups through DirectAdmin, download those and then transfer them, but this part was so tedious that I would often forget about it. So I thought of automating that by writing up a simple python backup script.
The automated process is made up of 2 parts – PHP scripts that send requests to DA on servers gets run via cronjob at midnight, backups get made in few minutes. Then the Python script running at 1AM on my RPI grabs these backup archives via FTP from those websites and stores them on NAS.
Simple as that.

PHP files are available publicly from DirectAdmin here and my Python backup script is available here.

Hopefully it helps someone to save time as it does for me :)

Also for a quick overview:

  1. #!/usr/local/bin/python
  2. '''
  3.     Date: 2015.08.02
  4.     Author: Kulverstukas
  5.     Website: 9v.lt; evilzone.org
  6.     Description:
  7.         Downloads backup files from /backups in set hosts
  8.         and downloads whole website content from defined website.
  9. '''
  10.  
  11. import os
  12. import sys
  13. import time
  14. import shutil
  15. import zipfile
  16. import ftputil
  17. from ftplib import FTP
  18.  
  19. #-------------------------------------------------------
  20. # where to put everything
  21. rootFolder = 'webhost_backups'
  22.  
  23. # configuration of as many webshosts as you have
  24. webhosts = [
  25.     {'folder' : 'somesite.lt', # where to store your files
  26.      'address' : 'somesite.lt', # URL of your website
  27.      'username' : 'lorem', # FTP username
  28.      'password' : 'ipsum', # FTP password
  29.      'backup_path' : '/backups', # usually this is where backups are stored on the server
  30.      'mirror' : True}, # should mirror the whole website starting from root
  31.  
  32.      {'folder' : 'subdomain.domain.com',
  33.      'address' : 'subdomain.domain.com',
  34.      'username' : 'lorem',
  35.      'password' : 'ipsum',
  36.      'backup_path' : '/backups',
  37.      'mirror' : False}
  38. ]
  39. #-------------------------------------------------------
  40. # create folders if they don't exist already
  41. if not os.path.exists(rootFolder) or not os.path.isdir(rootFolder):
  42.     os.mkdir(rootFolder)
  43. #-------------------------------------------------------
  44. # replace symbols with encoded equivalents
  45. def makeFriendlyFilename(input):
  46.     badSymbols = {'<' : '%3C',
  47.                   '>' : '%3E',
  48.                   ':' : '%3A',
  49.                   '"' : '%22',
  50.                   '|' : '%7C',
  51.                   '?' : '%3F',
  52.                   '*' : '%2A'}
  53.     i = ''
  54.     for i in badSymbols.keys():
  55.         input = input.replace(i, badSymbols[i]);
  56.     return input
  57. #-------------------------------------------------------
  58. def zipdir(path, zipname):
  59.     zipf = zipfile.ZipFile(zipname, 'w')
  60.     for root, dirs, files in os.walk(path):
  61.         for dir in dirs:
  62.             zipf.write(os.path.join(root, dir))
  63.         for file in files:
  64.             zipf.write(os.path.join(root, file))
  65.     zipf.close()
  66. #-------------------------------------------------------
  67. # download all of the files from FTP
  68. def mirrorFtp():
  69.     for config in webhosts:
  70.         if config['mirror']:
  71.             today = time.strftime("%Y-%m-%d")
  72.             mirrorName = config['folder']+'_mirror_'+today
  73.             localMirrorRoot = os.path.join(rootFolder, config['folder'], mirrorName)
  74.             if not os.path.exists(localMirrorRoot) or not os.path.isdir(localMirrorRoot):
  75.                 os.makedirs(localMirrorRoot)
  76.  
  77.             ftp = ftputil.FTPHost(config['address'], config['username'], config['password'])
  78.             for root, dirs, files in ftp.walk('/'):
  79.                 for dir in dirs:
  80.                     try:
  81.                         os.mkdir(os.path.join(localMirrorRoot, root[1:], dir))
  82.                     except:
  83.                         pass
  84.                 for file in files:
  85.                     try:
  86.                         ftp.download(root+'/'+file, os.path.join(localMirrorRoot, root[1:], makeFriendlyFilename(file)))
  87.                     except:
  88.                         print root+'/'+file
  89.             zipdir(localMirrorRoot, localMirrorRoot+'.zip')
  90.             shutil.rmtree(localMirrorRoot)
  91. #-------------------------------------------------------
  92. # download backup files made by DirectAdmin
  93. def getBackupFiles():
  94.     ftp = FTP()
  95.     for config in webhosts:
  96.         if not os.path.exists(rootFolder+'/'+config['folder']) or not os.path.isdir(rootFolder+'/'+config['folder']):
  97.             os.mkdir(rootFolder+'/'+config['folder'])
  98.         try:
  99.             ftp.connect(config['address'])
  100.             ftp.login(config['username'], config['password'])
  101.             ftp.cwd(config['backup_path'])
  102.             filesToDl = ftp.nlst();
  103.             for file in filesToDl:
  104.                 if (file not in ['.', '..']):
  105.                     ftp.retrbinary('RETR '+file, open(rootFolder+'/'+config['folder']+'/'+file, 'wb').write)
  106.                     try:
  107.                         # comment this out if you want to keep backup archives on the server
  108.                         ftp.delete(file)
  109.                     except:
  110.                         pass
  111.             ftp.quit()
  112.         except:
  113.             pass
  114. #-------------------------------------------------------
  115.  
  116. getBackupFiles()
  117. mirrorFtp()
  118.  
  119. #-------------------------------------------------------

Leave a Reply

Your email address will not be published. Required fields are marked *