automated_zfs_snapshots_for_omv5

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
Last revisionBoth sides next revision
automated_zfs_snapshots_for_omv5 [2021/05/05 23:24] – [Individual filesystem, folder, or file recovery] crashtestautomated_zfs_snapshots_for_omv5 [2022/07/26 01:35] – crashtest
Line 1: Line 1:
-{{ :underconstruction.jpg?400 |}} +<html><center><strong>Automated ZFS Snapshots for Openmediavault 5 and 6</strong></center></html>
- +
-<html><center><strong>Automated ZFS Snapshots for openmediavault 5</strong></center></html>+
  
 {{ :zfs1.jpg?400 |}} {{ :zfs1.jpg?400 |}}
  
-====== Automated ZFS Snapshots for openmediavault 5 ======+====== Automated ZFS Snapshots for Openmediavault 5 and 6 ======
 \\ \\
 <html><center><strong>How To setup automated, Self  Rotating and Purging ZFS snapshots</strong></center></html> <html><center><strong>How To setup automated, Self  Rotating and Purging ZFS snapshots</strong></center></html>
Line 22: Line 20:
 \\ \\
 **For ZFS users:**\\ **For ZFS users:**\\
-This guide will show how to setup and take advantage of one of ZFS' most valuable features for restoration, SNAPSHOTS.  **zfs-auto-snapshot**, automatically creates, prunes, and purges periodic snapshots.  Other than ZFS utilities which are installed with the ZFS filesystem, and cron which is part of most Linux installations, there are no specific dependencies.+This guide will show how to setup and take advantage of one of ZFS' most valuable features for restoration, SNAPSHOTS.\\   
 +**zfs-auto-snapshot**, automatically creates, prunes, and purges periodic snapshots.  Other than ZFS utilities, which are installed with the ZFS filesystem and cron, which is part of most Linux installations, there are no specific dependencies.
  
 While intended primarily for openmendiavault, this How-To and the package zfs-auto-snapshot will work with other Linux distributions with ZFS on Linux (ZOL). While intended primarily for openmendiavault, this How-To and the package zfs-auto-snapshot will work with other Linux distributions with ZFS on Linux (ZOL).
 ---- ----
  
-**Tested – December 31rst, 2020:  With openmediavault 5 and Debian 10 (Buster)**\\+**Tested – December 31rst, 2020:  With openmediavault 5 and Debian 10 (Buster).**\\
 Tested, prior, with openmediavault 4 and Debian 9. Tested, prior, with openmediavault 4 and Debian 9.
  
 ---- ----
-\\+
 ==== General ==== ==== General ====
-\\+
 Given the design and function of a CoW (copy on write) filesystem, ZFS gives users the ability to “capture” the state of their file system at a given moment in time and preserve it using snapshots.  Since only changes to ZFS filesystems are maintained between current filesystem states and historic snapshots, snapshoting is very efficient.  For largely static pools, maintaining the cumulative differences between snapshots, shouldn't require more than an extra 25% disk space.   Given the design and function of a CoW (copy on write) filesystem, ZFS gives users the ability to “capture” the state of their file system at a given moment in time and preserve it using snapshots.  Since only changes to ZFS filesystems are maintained between current filesystem states and historic snapshots, snapshoting is very efficient.  For largely static pools, maintaining the cumulative differences between snapshots, shouldn't require more than an extra 25% disk space.  
 Having the ability to “roll back” the pool, individual filesystems in the pool, or retrieve individual files from previous snapshots has obvious advantages.  These abilities would allow an entire pool to be restored to a time before a virus infection, a ransomware infection, and allow the retrieval of  deleted files or previous versions of files.\\ Having the ability to “roll back” the pool, individual filesystems in the pool, or retrieve individual files from previous snapshots has obvious advantages.  These abilities would allow an entire pool to be restored to a time before a virus infection, a ransomware infection, and allow the retrieval of  deleted files or previous versions of files.\\
Line 43: Line 42:
  
  
-{{ ::zfs2.jpg?600 |}}+{{ ::zfs2.jpg?700 |}}
  
 ---- ----
Line 83: Line 82:
 \\ \\
 **In the interests of clarity:**\\ **In the interests of clarity:**\\
-A ZFS “filesystem” is interchangeable with a standard Linux folder at the root of the pool and is navigable, on the command line, in the same manner.  The difference is a standard Linux folder at the root of the pool, is part of the parent pool.  A child “filesystem” is created by ZFS and is assigned editable ZFS properties.  A “filesystem snapshot” is taken independent of the parent pool and is independently restoreable. \\+A ZFS “filesystem” is interchangeable with a standard Linux folder at the root of the parent pool and is navigable, on the command line, in the same manner.  The difference is a standard Linux folder at the root of the pool, is part of the parent pool.  A child “filesystem” is created by ZFS and is assigned editable ZFS properties.  A “filesystem snapshot” is taken independent of the parent pool and is independently restoreable. \\
 \\ \\
 ---- ----
-==== Customizing Snapshot jobs ====+===== Customizing Snapshot jobs =====
  
 By default all snapshot time intervals are set to “**true**”.  Changes are required only if a specific time interval is not desired. By default all snapshot time intervals are set to “**true**”.  Changes are required only if a specific time interval is not desired.
Line 96: Line 95:
 \\ \\
 zfs set com.sun:auto-snapshot=**true** Rocky\\ zfs set com.sun:auto-snapshot=**true** Rocky\\
 +\\
 zfs set com.sun:auto-snapshot:frequent=true Rocky\\ zfs set com.sun:auto-snapshot:frequent=true Rocky\\
 zfs set com.sun:auto-snapshot:hourly=true Rocky\\ zfs set com.sun:auto-snapshot:hourly=true Rocky\\
Line 145: Line 145:
 While rolling back a file system or the parent pool is relatively easy, if going significantly back in time (beyond the most recent snapshot) the roll back feature will not work in the OMV GUI.  However, a roll back can be done with the following command line: While rolling back a file system or the parent pool is relatively easy, if going significantly back in time (beyond the most recent snapshot) the roll back feature will not work in the OMV GUI.  However, a roll back can be done with the following command line:
  
-(In this example, the roll back would be done to the entire parent pool “Rocky”.)+(In this example, the roll back would be done to the parent pool “Rocky”.)
  
 ''zfs rollback Rocky@zfs-auto-snap_hourly-2018-01-14-1617 -r''    ''zfs rollback Rocky@zfs-auto-snap_hourly-2018-01-14-1617 -r''   
Line 167: Line 167:
 \\ \\
 \\ \\
-When the hidden directory is made visible, with the set visible command above, it will appear in the associated SMB share. +When the hidden directory is made visible, with the set visible command above, it will appear in the associated SMB share.\\ 
 +----
 <html><center>The following example is of a ZFS filesystem named <strong>Documents</strong>.  The folder <strong>.zfs</strong> is now revealed.</center></html> <html><center>The following example is of a ZFS filesystem named <strong>Documents</strong>.  The folder <strong>.zfs</strong> is now revealed.</center></html>
  
-{{ ::zfs3.jpg?400 |}}+{{ ::zfs3.jpg?600 |}}
 ---- ----
 \\ \\
-Under **.zfs**, the **snapshot** directory is found.  Within the snapshot folder are past snapshots for the filesystem, in folders named by the snapshot interval, with date and time. +Under **.zfs**, the **snapshot** directory is found.  Within the snapshot folder are past snapshots for the **Documents** filesystem, in folders named by the snapshot interval, with date and time.
- +
-{{ ::zfs4.jpg?400 |}}+
  
 +{{ ::zfs4.jpg?600 |}}
 +\\
 +\\
 Inside of each snapshot folders exists the exact state of the filesystem, as it existed on the date shown. Inside of each snapshot folders exists the exact state of the filesystem, as it existed on the date shown.
 +
 The contents of any snapshot can be copied from the snapshot to replace or overwrite any part of, or all of, the current top level file system.  (*Provided that the client has “write” access to the share.*) The contents of any snapshot can be copied from the snapshot to replace or overwrite any part of, or all of, the current top level file system.  (*Provided that the client has “write” access to the share.*)
  
-For businesses and users, going back to a time before malware existed on a share; this feature can be utilized as a ransomware or virus “Killer”.  In such a scenario, be sure to work from the server using a package like Midnight Commander (apt-get mc) or use a known clean client.+For businesses and users, going back to a time before malware existed on a share; this feature can be utilized as a ransomware or virus “Killer”.  In such a scenario, be sure to work from the server using a package like Midnight Commander (''apt-get mc'') or use a known clean mal-ware free client.
  
-*Caution*:\\+**Caution**:\\
 The above will make all of the current filesystem's files AND all past versions within snapshots look like nested but individual (separate) files and folders, complete with a unique directory structure (the snapshot names).  The above will make all of the current filesystem's files AND all past versions within snapshots look like nested but individual (separate) files and folders, complete with a unique directory structure (the snapshot names). 
-If the snapshot directory is visible and an rsync job or any other backup type runs pulling files from the share, the destination will be **OVER RUN** with multiple versions of the same files and folders.  It is **IMPORTANT** to turn off any type of server share backup job before using this command.  +If the snapshot directory is visible and an rsync job or any other backup type runs, pulling files from the share, the destination will be **OVER RUN** with multiple versions of the same files and folders.  It is **IMPORTANT** to turn off any type of server share backup job before using this command.  
 Equally important is the need for any copy, paste or replace operations, from past snapshots to the current file system, to be “//**surgical**//”. Equally important is the need for any copy, paste or replace operations, from past snapshots to the current file system, to be “//**surgical**//”.
  
Line 193: Line 195:
  
 ---- ----
-Notes: +===== Notes: =====
- +
-1. For maximum snapshot flexibility, creating child filesystems on the parent pool is essential.+
  
 +1. For maximum snapshot flexibility, creating child filesystems on the parent pool is essential.\\
 +\\
 2. Think of the roll back feature as going back in time, in the ZFS “time line”. When rolling back to a specific date/time snapshot, ZFS destroys all snapshots (all file changes, additions and deletions) between the present state and the past state of the filesystem.  This is necessary, using a time travel analogy, to avoid a “Paradox”.  (Where two different time lines of the file system would exist.)  Rolling back is permanent.  It's not possible to undo a roll back or “spring forward”. 2. Think of the roll back feature as going back in time, in the ZFS “time line”. When rolling back to a specific date/time snapshot, ZFS destroys all snapshots (all file changes, additions and deletions) between the present state and the past state of the filesystem.  This is necessary, using a time travel analogy, to avoid a “Paradox”.  (Where two different time lines of the file system would exist.)  Rolling back is permanent.  It's not possible to undo a roll back or “spring forward”.
- +\\ 
-The practical implications are:  +The practical implications are:\\  
--  In order to not lose file changes, deletions, etc., it's better to roll back the shortest interval possible.  +  * In order to not lose file changes, deletions, etc., it's better to roll back the shortest interval possible.  
-- By extension, it's better to roll back a single filesystem (the ZFS equivalent of a root folder), than it is to roll back an entire pool without child filesystems. +  * By extension, it's better to roll back a single filesystem (the ZFS equivalent of a root folder), than it is to roll back an entire pool without child filesystems. 
-- Due to the minimal overall impact, to prevent the loss of file versions, etc; selective restoration(s) of files and folders is preferred and is “best practice”. +  * Due to the minimal overall impact, to prevent the loss of file versions, etc; selective restoration(s) of files and folders is preferred and is “best practice”. 
-- Roll backs are a disaster recovery option that, if applied, should but be as narrow as possible (a single filesystem) and limited to the shortest time interval possible. +  * Roll backs are a disaster recovery option that, if applied, should but be as narrow as possible (a single filesystem) and limited to the shortest time interval possible. 
- +\\ 
-3. If a particular file system experiences a high rate of  file turnover or file versioning (databases, etc.), snapshot retention periods may need to be shortened to prevent excessive use of disk space.  Given this consideration, a dedicated filesystem could be used to isolate frequently changing, transactional data sets, from relatively static file storage. +3. If a particular file system experiences a high rate of file turnover or file versioning (databases, etc.), snapshot retention periods may need to be shortened to prevent excessive use of disk space.  Given this consideration, a dedicated filesystem could be used to isolate frequently changing, transactional data sets, from relatively static file storage.\\ 
-4. While it is an advanced technique; it is possible to create a file system “clone” from a past snapshot, without rolling back.  The clone could be used as a source of folders and files as they existed when the snapshot was taken, without sacrificing the snapshots in between.  (See references for further details.) +\\ 
-5. While zfs-auto-snapshot creates considerable flexibility in file, folder, filesystem, and pool restorations, it is NOT BACKUP.  If a ZFS pool is lost, local snapshots are lost as well. +4. While it is an advanced technique; it is possible to create a file system “clone” from a past snapshot, without rolling back.  The clone could be used as a source of folders and files as they existed when the snapshot was taken, without sacrificing the snapshots in between.  (See references for further details.)\\ 
-6. If more advanced features are needed, such as offloading snapshots to an external host, ZnapZend is a more appropriate solution. +\\ 
- +5. While zfs-auto-snapshot creates considerable flexibility in file, folder, filesystem, and pool restorations, it is NOT BACKUP.  If a ZFS pool is lost, local snapshots are lost as well.\\ 
-Additional Information:+\\ 
 +6. If more advanced features are needed, such as offloading snapshots to an external host, [[http://www.znapzend.org/|ZnapZend]] is a more appropriate solution.\\     
 +\\ 
 +==== Additional Information ====
    
-zfs-auto-snapshot source: +==== zfs-auto-snapshot source ==== 
-https://github.com/zfsonlinux/zfs-auto-snapshot/ +https://github.com/zfsonlinux/zfs-auto-snapshot/ \\ 
-A Comprehensive and well written ZFS reference:  +\\ 
-https://pthree.org/2012/12/19/zfs-administration-part-xii-snapshots-and-clones/ +A Comprehensive and well written ZFS reference: \\ 
- +https://pthree.org/2012/12/19/zfs-administration-part-xii-snapshots-and-clones/ \\ 
-ZFS Video Tutorial: +\\ 
-Part 1: https://sysadmincasts.com/episodes/35-zfs-on-linux-part-1-of-2 +==== ZFS Video Tutorial ==== 
-Part 2: https://sysadmincasts.com/episodes/37-zfs-on-linux-part-2-of-2 +Part 1: https://sysadmincasts.com/episodes/35-zfs-on-linux-part-1-of-2  \\ 
-** Due to a difference in repositories, do not install ZFS and use a pool created in Tutorial, with OMV.  It's best to install ZFS using OMV's GUI.   +Part 2: https://sysadmincasts.com/episodes/37-zfs-on-linux-part-2-of-2  \\ 
-** While this tutorial is informative, it would be best done using a newly pool that can be destroyed when the tutorial is complete.  Doing any roll back operation on an existing ZFS pool/filesystem comes with a risk of “fat finger” mistakes.+* Due to a difference in repositories, do not install ZFS and use a pool created in Tutorial, with OMV.  It's best to install ZFS using the plugin, in OMV's GUI. \\  
 +* While this tutorial is informative, it would be best done using a newly created pool that can be destroyed when the tutorial is complete.  Doing any roll back operation on an existing ZFS pool/filesystem comes with a risk of “fat finger” mistakes.  \\