VMware vCSA 6.5 Scheduled Backup

The new vCenter 6.5 Server Appliance comes with a backup function, but it is not possible to schedule the backup out of the box.

But there are several ways that you can do this yourself.

Powershell

The first option is to schedule a powershell script on a Windows host, or maybe on your vRealize Orchestrator.

Brian Graf has done an article on the script, and how it works here: http://www.vtagion.com/vsphere-6-5-automate-vcsa-backup/

Crontab

I however found that scheduling stuff on windows is a bit cumbersome, and I think if is much better to run it locally on vCenter.

My co-worker Allan Kjær brought to my attention that it is possible to schedule this using crontab directly on the vCSA appliance, and directed me to this VMware example: http://pubs.vmware.com/vsphere-6-5/index.jsp?topic=%2Fcom.vmware.vsphere.vcsapg-rest.doc%2FGUID-222400F3-678E-4028-874F-1F83036D2E85.html

I altered the code slightly, so that it would use FTPS instead of plain FTP.

Save the code to a file on vCSA using your favorite editor. I recommend putting the file in /usr/local/bin/

Since passwords will be saved to this file we will remove access for non-root users.

# Commands:
vi /usr/local/bin/vCSA-Backup.sh

# Make the file executable
chmod u+x /usr/local/bin/vCSA-Backup.sh

# Make it only accessible by root
chmod g-rxw /usr/local/bin/vCSA-Backup.sh
chmod o-rxw /usr/local/bin/vCSA-Backup.sh

You have to replace usernames and passwords for vCSA and FTPs server. You can also replace FTPS with SCP, FTP, PATH, HTTP or HTTPS in the following line.

There have been some requests to use SCP, and that is easy to setup. Just change the location-type to SCP, and ind the location field change “ftp” to “scp” to it says “scp://$FTP_ADDRESS……”

"location_type":"FTPS",

vCSA-Backup.sh File contents:

#!/bin/bash
 ##### EDITABLE BY USER to specify vCenter Server instance and backup destination. #####
 VC_ADDRESS=vcenter.domain.local
 VC_USER=administrator@vsphere.local
 VC_PASSWORD=password
 FTP_ADDRESS=ftp-server.domain.local
 FTP_USER=ftp-user
 FTP_PASSWORD=ftp-password
 BACKUP_FOLDER=vCSA-Backup
 ############################

 # Authenticate with basic credentials.
 curl -u "$VC_USER:$VC_PASSWORD" \
    -X POST \
    -k --cookie-jar cookies.txt \
    "https://$VC_ADDRESS/rest/com/vmware/cis/session"

 # Create a message body for the backup request.
 TIME=$(date +%Y-%m-%d-%H-%M-%S)
 cat << EOF >task.json
 { "piece":
      {
          "location_type":"FTPS",
          //"location_type":"SCP",
          "comment":"Automatic backup",
          "parts":["seat"],
          "location":"ftp://$FTP_ADDRESS/$BACKUP_FOLDER/$TIME",
          //"location":"scp://$FTP_ADDRESS/$BACKUP_FOLDER/$TIME",
          "location_user":"$FTP_USER",
          "location_password":"$FTP_PASSWORD"
      }
 }
EOF

 # Issue a request to start the backup operation.
 echo Starting backup $TIME >>backup.log
 curl -k --cookie cookies.txt \
    -H 'Accept:application/json' \
    -H 'Content-Type:application/json' \
    -X POST \
    --data @task.json 2>>backup.log >response.txt \
    "https://$VC_ADDRESS/rest/appliance/recovery/backup/job"
 cat response.txt >>backup.log
 echo '' >>backup.log

 # Parse the response to locate the unique identifier of the backup operation.
 ID=$(awk '{if (match($0,/"id":"\w+-\w+-\w+"/)) \
           print substr($0, RSTART+6, RLENGTH-7);}' \
          response.txt)
 echo 'Backup job id: '$ID

 # Monitor progress of the operation until it is complete.
 PROGRESS=INPROGRESS
 until [ "$PROGRESS" != "INPROGRESS" ]
 do
      sleep 10s
      curl -k --cookie cookies.txt \
        -H 'Accept:application/json' \
        --globoff \
        "https://$VC_ADDRESS/rest/appliance/recovery/backup/job/$ID" \
        >response.txt
      cat response.txt >>backup.log
      echo ''  >>backup.log
      PROGRESS=$(awk '{if (match($0,/"state":"\w+"/)) \
                      print substr($0, RSTART+9, RLENGTH-10);}' \
                     response.txt)
      echo 'Backup job state: '$PROGRESS
 done

 # Report job completion and clean up temporary files.
 echo ''
 echo "Backup job completion status: $PROGRESS"
 rm -f task.json
 rm -f response.txt
 rm -f cookies.txt
 echo ''  >>backup.log

Test the script to see if it is working by running the command on your vCSA server:

# Command:
/usr/local/bin/vCSA-Backup.sh

Now it is time to schedule the script using crontab. I am going to make it run every night af 2 am. You can schedule it however you want. You can find out more about how to schedule stuff using crontab here: https://en.wikipedia.org/wiki/Cron

Open your crontab.

#Command:
crontab -e

# Press 'i' to goto insert mode
# Insert you task into crontab
0 2 * * * /usr/local/bin/vCSA-Backup.sh

# Press ':wq'
# Press ENTER

Your task will now run a 2 am every day.

Remember to do some maintenance on your FTP server since it will keep adding data to it.

Troubleshooting

It seems that there is an issue with cron jobs not running in recent versions of vCSA. There is a short description of how to fix this here: https://www.drewgreen.net/wordpress/2017/04/19/fix-for-cron-failing-on-vmware-vcenter-server-appliance-vcsa-6-5/

68 thoughts on “VMware vCSA 6.5 Scheduled Backup”

  1. Fix for Cron Failing on VMware vCenter Server Appliance (VCSA) 6.5

    When trying to enable scheduled jobs via cron on VMware VCSA 6.5 I kept seeing the errors below, and my job would not run.

    2017-04-19T09:56:01.996673-04:00 VCSA crond[104661]: PAM _pam_load_conf_file: unable to open config for password-auth
    2017-04-19T09:56:01.996797-04:00 VCSA crond[104661]: PAM _pam_load_conf_file: unable to open config for password-auth
    2017-04-19T09:56:01.996907-04:00 VCSA crond[104661]: PAM _pam_load_conf_file: unable to open config for password-auth
    2017-04-19T09:56:01.997010-04:00 VCSA crond[104661]: (root) PAM ERROR (Permission denied)
    2017-04-19T09:56:01.997116-04:00 VCSA crond[104661]: (root) FAILED to authorize user with PAM (Permission denied)

    The contents of /etc/pam.d/crond had 3 references to “password-auth”, however there was no file in /etc/pam.d called “password-auth”. I changed “password-auth” to “system-auth” in /etc/pam.d/crond, as seen below, and everything worked.

    account required pam_access.so
    account include system-auth
    session required pam_loginuid.so
    session include system-auth
    auth include system-auth

    https://www.drewgreen.net/wordpress/2017/04/19/fix-for-cron-failing-on-vmware-vcenter-server-appliance-vcsa-6-5/

    1. You could also link the “ln -s system-auth password-auth”

      That way you keep the aligned at all times.

  2. Hi Guys,

    it seems i have trouble to run this script (miss typo)

    could any one please send me a template of working script (jospehlozian@gmail.com) so i can edit it directly

    thank
    jospe

    1. actually a thing the script is running but i am facing a trouble, the output is :

      {“type”:”com.vmware.vapi.std.errors.unauthenticated”,”value”:{“messages”:[{“args”:[],”default_message”:”Authentication required.”,”id”:”com.vmware.vapi.endpoint.method.authentication.required”}]}}Backup job id:

      do u have any idea what does is it mean !!!!!!!

      Thanks

      1. Seems link you have a problem with you credentials: com.vmware.vapi.std.errors.unauthenticated

    2. You can get it directly from the VMware documentation link in the article. It might be a copy paste issue. Sometimes I notices that quotations marks get corrupted by browser language/font codes.

  3. Your link to the documentation is no longer valid as VMware moved their 6.5 documentation to the new documentation platform.

    I could not find the one you referenced in the documentation. could you maybe update your link?

    1. You cannot do single signon to rest using your root account.

      But you can restrict other users than root from reading the script using something like chmod o-rwx and chmod g-rwx.

      If they are already on your vCenter with the root account, you have lost the war.

  4. Hello,

    when i start the script it’s start, but it finished after some second without any errors and no backup.

    root@******** [ ~ ]# /usr/local/bin/vCSA-Backup.sh
    {“value”:”ac6682e8faf416402a685dbf9729355f”}Backup job id:
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 173 0 173 0 0 2692 0 –:–:– –:–:– –:–:– 2703
    Backup job state:

    Backup job completion status:

    Do you have any idea or where i can look for logs?

    Thank you.

    1. Hi Florian,
      I am getting the same output and when I look at the backup.log file (cat backup.log), I see the following: {“name”:”com.vmware.vapi.rest.badRequest”,”localizableMessages”:[{“defaultMessage”:”Bad Request”,”id”:”com.vmware.vapi.rest.badRequest”}],”majorErrorCode”:400}
      {“value”:[“20170802-155734-5705665″,”20170802-151121-5705665”]}

      As suggested by Brian, I am going to try to break the script down and run it in pieces but I was wondering if you found the solution to this?

      1. The issue was that the script did not pass the FTP_USER and FTP_PASSWORD I had set since it contained special characters. I had to add single quotes and two backslashes (since one backslash ignores the next character).
        So I changed it to:
        FTP_USER=’domain\\user’
        FTP_PASSWORD=’!!!fFA41ga31hj31!!’

  5. Hello,

    how can i remove old backups automatic which are older than 30 days for example ? Best with your script.

    1. FORFILES /p E:\vCenterBackup\vCenterBackup -D -30 -C “CMD /C IF @ISDIR==TRUE ECHO rd /q /s @FILE&rd /q /s @FILE”

      Task schedule from batch file

  6. Hello,

    I have trouble that VCSA6.5 backup often fails because tar command often returns not [0] but [1] with error “file changed as we read it” on archiving “/storage/sso/***.log”.
    Can I avoid processing tar “/storage/sso/***.log” in backup by choosing not to back up Statistics, Events, and Tasks ?
    If it is so, how do I write instead of statement: “parts”:[“seat”], in the Bash Example of Backing Up the vCenter Server Instance ?

    1. I suspect that VMware should fix this in their backup rutine on VCSA, but until then you could check the output and retry a number of times.

  7. Hi Brian

    When I try to run the script I get the following messages in the backup log:
    {“type”:”com.vmware.vapi.std.errors.unauthorized”,”value”:{“messages”:[{“args”:[],”default_message”:”Unable to authorize user”,”id”:”vapi.security.authorization.invalid”}]}}
    I’m using administrator@vsphere.local to connect to vcenter. Any ideas what could be wrong

    1. Hi Michael,

      It sound like there is something wrong with your password or username then. Could it be that you changed the SSO domain, or maybe your password has some symbols that REST og CURL does not like. You can test you credentials on the rest api using a browser.

      Go to your VCSA: https://[vCenter]/apiexplorer and press login in the top right corner.

  8. Hey Brian, great article…. Thanks for taking the time to share.

    -Where in the code are you telling it what to backup? Comparing it to the appliance backup, your code gets a lot more.

    -What’s your process for restore?

    1. Thank you Doug.

      This article is just about scheduling what VMware already built into vCSA.

      In the script you can see a line that says: “parts”:[“seat”]
      This is the selection, and that means everything in the database, but you can select less according to the documentation:

      “The request specifies several attributes, especially the backup location, the protocol used to communicate with the storage server, the necessary authorization, and which optional parts of the database you want to back up. The core inventory data and Alarms are always backed up, but you can choose whether or not to back up Statistics, Events, and Tasks. Collectively, this optional part of the backup is referred to as seat.”

      If you follow the VMware article linked to in this article you will also find some instructions on how to restore. These are however not that simple. I will consider making an, easy to understand, article about restoring.

  9. So I restarted VCSA
    I verified that I could logon using SSO credentials /apiexplorer

    If I go to vcenter/rest/com/vmware/cis/session
    Get error:
    {“name”:”com.vmware.vapi.rest.httpNotFound”,”localizableMessages”:[{“defaultMessage”:”Not found.”,”id”:”com.vmware.vapi.rest.httpNotFound”}],”majorErrorCode”:404}

    Backup error shows this
    {“type”:”com.vmware.vapi.std.errors.error”,”value”:{“messages”:[{“args”:[],”default_message”:”Access to the remote server is denied. Check your credentials and permissions.”,”id”:”com.vmware.applmgmt.err_access_denied”}]}}
    {“value”:[“20170809-164458-5973321”]}

  10. Disregard my question.
    Figured out.

    updated script file as follows and it works

    VC_ADDRESS=Vcenter.domain.local
    VC_USER=administrator@Vcenter.domain.local
    VC_PASSWORD=’SSOPassword’
    FTP_ADDRESS=FTPSERVERFQN
    FTP_USER=’FTPSERVERFQDN|ftp_user’
    FTP_PASSWORD=’Password’
    BACKUP_FOLDER=/VCSA/Vcenterserver/

  11. Dear All,
    I am also running the VCSA 6.5
    My FTP server is FileZilla on a Windwos Server 2016.
    How my backup script can not work for override the files.
    It’s just keeping told me the destination folder is not empty and stop the process.

    1. You can’t backup to a folder with files. You should backup to a new timestamped folder each time

  12. Hi Guys
    create job.
    But I have another question. In this szenario you are speak only of vCenter Backup! Who can I automate the backup of PSC? If there is external.
    Br Tobias

  13. I’ve made a small windows software to do vCSA 6.5 config backup as easy as it could be. If you are interested, visit my website

    It has a built-in FTP, can manage retentions and is able to save multiple vCenters.

  14. I have to use FTP and not FTPS. Is there a way to add the encryption option to this script?

    Also, for retention, I am wondering how to change the script for a specific folder (e.g. MON, TUE, WED, ETC…) and have it overwrite a previous backup. I would make 5 different scripts and run each weekly.

  15. I figured out how to add encryption. I also figured out how to change the folder, but the backup job fails saying the location is not empty.

    Here is the link to the available parameters: http://pubs.vmware.com/vsphere-6-5/index.jsp?topic=%2Fcom.vmware.dcli.cmdref.doc%2Fcom%2Fvmware%2Fappliance%2Frecovery%2FBackup.html

    Here is what you change:

    BACKUP_PASSWORD=’put password here’
    ############################

    # Create a message body for the backup request.
    TIME=$(date +%Y-%m-%d-%H-%M-%S)
    cat <task.json
    { “piece”:
    {
    “location_type”:”FTP”,
    “comment”:”Automatic backup”,
    “parts”:[“seat”],
    “location”:”ftp://$FTP_ADDRESS/$BACKUP_FOLDER/MON”,
    “location_user”:”$FTP_USER”,
    “location_password”:”$FTP_PASSWORD”,
    “backup-password”:”$BACKUP_PASSWORD”
    }
    }
    EOF

    Also, I put single quotes around the passwords because I have special characters that killed the scripts. The single quotes let the characters pass through.

  16. Dang, correction: the hyphen doesn’t work. You have to put an underscore in the “Create a message body for the backup request” section.

    backup_password not backup-password

    Sorry

  17. Hello,
    When I try to run the script manually with enabled proxy settings I get the following error:
    curl: (56) Received HTTP code 403 from proxy after CONNECT. In the backup.lof file it is nothing
    (Starting backup 2017-10-16-11-21-44
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    0 0 0 0 0 0 0 0 –:–:– 0:00:03 –:–:– 0
    curl: (56) Received HTTP code 403 from proxy after CONNECT)
    But when I disable proxy settings everything is ok.
    How can I run the script whith enabled proxy settings?
    Thanks!

    1. 1) Are your sure that you can access your vCenter through your proxy server?
      2) Why would you want to access it through your proxy when you do not need to?

      1. 1) Yes, we have access to vCenter through our proxy server.
        It’s open to internal network adresses and has access through proxy server. I added an exception to the field “NO_PROXY” of ftp-server ip adress in the /etc/sysconfig/proxy. But it doesn’t work.
        2) We use proxy to get patches and appliance updates.
        Could you suggest how we can use the script with enabled the proxy server settings?
        Thank you.

  18. having a heck of a time getting the script to accept my IIS FTP location on my new 6.5 update 1a VCSA, anyone else come across this error?

    Starting backup 2017-10-26-12-18-28
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 468 0 178 100 290 447 728 –:–:– –:–:– –:–:– 728
    {“type”:”com.vmware.vapi.std.errors.error”,”value”:{“messages”:[{“args”:[“FTP”],”default_message”:”FTP location is invalid.”,”id”:”com.vmware.applmgmt.err_invalid_remote_loc”}]}}

    also, when the job fails like this is it leaving orphaned files somewhere that need cleaned up?

    thanks for any insight

    1. Could you paste your script vars to pastebin.com, and reference it here so we can have a look.

      Don’t forget to remove your password 🙂

      Also you could try all the steps manually at see where it goes wrong.

      1. thanks, Brian. the script vars look pretty cut and dry
        https://pastebin.com/V6fyH6Zn

        Also noticed I get “FTP location is invalid” in the VCSA web console admin site as well. Perhaps I have something misconfigured in IIS FTP, need to find better documentation on how to set that up but seems ok.

          1. certainly does, I opened VMware support ticket and will update if they find out what i’m doing wrong

          2. the problem was exemptions in etc/sysconfig/proxy such that VCSA did not like use of a wildcard.
            *.domain.com did not work but ‘ftpserver.domain.com’ did

            perhaps just our specific environment but all is well

    1. You have sendmail available on the vCenter Appliance, but by default it uses the internal dns name as email domain, so it is not that easy. You will have to make a new sendmail.cf configuration to masquerade your from address.

      If you get that working it should be pretty streight forward to send the report using the sendmail command.

  19. great tutorial, thank you very much. Is there already a custom script that deletes the old backup files?

  20. I’m getting

    “curl: (56) Received HTTP code 504 from proxy after CONNECT”

    when running the script against vCSA-FQDN. I’ve configured my vCSA to use proxy for HTTP and HTTPS since I have to use proxy to get updates. But nothing for FTP.

    If i try script against localhost i don’t get the curl error; it seems to do something, but I’m not getting a job id, but another value (different each time):
    {“value”:”f7a68bf4923c8a3daf7822cf09e02051″}
    Also the status shows some numbers instead of only zeros:
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 173 0 173 0 0 4552 0 –:–:– –:–:– –:–:– 4675

    Any suggestions?

    1. Are you trying to send the backup to FTP?
      If so, is it a possibility to access the FTP server through your proxy?

      1. FTPS originally, but failed back to FTP for troubleshooting.

        I’ve added the FTP server to NO_PROXY and that got me further.
        It worked backing up manually via VAMI, but not via script. So I thought I’d try what Ben did; a reboot worked magic.

        Unfortunatly that was probably 4 hours of my life waisted today, and almost a full day last week.

        Like IT Crowd keeps saying: https://www.youtube.com/watch?v=5UT8RkSmN4k

  21. I’m using scp and ssh keys to authenticate between the backup server and the vCenter. I keep getting the error, “defaultMessage”:”Bad Request”,”id”:”com.vmware.vapi.rest.badRequest”}],”majorErrorCode”:400} and believe it is because the json file does not have a username and password to authenticate. The ssh keys are working. How can I work around this so that the script does not need to authenticate to its backup server?

    1. I do not think it is supported to use SSH Keys, but you could try to put the username with a blank password, since SSH will not ask for the password when you are using SSH Keys.

      But why bother, I don’t see how that would increase security in this scenario?

  22. Thought I would leave my experience here.
    I was first experiencing this error,
    {“type”:”com.vmware.vapi.std.errors.unauthorized”,”value”:{“messages”:[{“args”:[],”default_message”:”Unable to authorize user”,”id”:”vapi.security.authorization.invalid”}]}}
    And as suggested above a reboot of the vCenter fixed this.

    I then got the following error,
    {“type”:”com.vmware.vapi.std.errors.error”,”value”:{“messages”:[{“args”:[“FTP”],”default_message”:”FTP location is invalid.”,”id”:”com.vmware.applmgmt.err_invalid_remote_loc”}]}}

    and again as suggested above this was caused by using a proxy server in the appliance.

    If you use a proxy server you must modify the /etc/sysconfig/proxy file to put your ftp server ip address in the bypass list.

    Wow that was a waste of 2 hours of my life.

    Thanks to everyone who has shared their experiences above as I probably wouldnt have worked this out without the responses.

  23. Hi I have thi response whe i try ti start backup job
    {“name”:”com.vmware.vapi.rest.badRequest”,”localizableMessages”:[{“defaultMessage”:”Bad Request”,”id”:”com.vmware.vapi.rest.badRequest”}],”majorErrorCode”:400}[

  24. Hi Brian,
    thanks for your excelent backup script.
    Could you please explane (all steps) what I need to do to enable crontab job on VCSA6.5. (Photon OS)
    All working fine if i start the script manualy. But If I locked out the cronjob will not start the “vCSAB_Backup.sh”.
    Many Thanks for your help.

  25. Hello!

    I also got the error:
    {“name”:”com.vmware.vapi.rest.badRequest”,”localizableMessages”:[{“defaultMessage”:”Bad Request”,”id”:”com.vmware.vapi.rest.badRequest”}],”majorErrorCode”:400}[

    It is the json part. In the script two lines are commented out using # (SCP part, which I though used but instead I commented out the FTP with the same #) but as this part of the script actually creates a json file the comment shuld be behind // or easiest way just delete the lines that you are not using.

    1. Thank you. You are absolutely right. I added that recently, and did not notice that it is of course in the json part.

Leave a Reply

Your email address will not be published. Required fields are marked *