Show pageOld revisionsBacklinksBack to top This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong. {{indexmenu_n>6}} <html><center><span style="color:#000066;font-size:300%;">Utilities, Maintenance and Backup</span></center></html> {{ :omv7ug_-_top.jpg?700 |}} <html><center><span style="color:#000066;font-size:300%;">For Openmediavault 7</span></center></html> \\ <html><center><strong>January 23rd, 2024</strong></center></html> \\ \\ Version History: \\ January 23rd, 2024 - (Draft):\\ {{ :divider2.png?600 |}} ===== Introduction to Server Maintenance ===== So you now have a working Openmediavault installation. That's a great start! \\ For casual home users that may seem like all that's required and for a computer hobbyist, a working Linux server may be enough. Adding to that, a working Linux server installation that's not tampered with may outlast the hardware it's installed on. \\ \\ On the other hand, PC hardware and hard drives fail faster than one might think. Some users may want to change their hardware, add new hardware, expand openmediavault with server add-on packages, etc. As users discover the value added packages available to expand capabilities, Linux servers tend to "evolve". Lastly, on a rare occasion, software that is pushed down from upstream, in a security or package update, may not be compatible with existing server packages. \\ \\ All of the above possibilities compound the risk of creating an unresolvable problem. The end result of these possible events leaves an open question; "How do I recover?" ==== Purpose ==== The purpose and intent of this document is to provide home users and new Linux admin's with basic utilities and simple backup processes needed to support and successfully maintain a server. While openmediavault is driven by an easy to use web page GUI, server maintenance is greatly facilitated if users know how to get on the command line. When asking for advice on the forum, forum supporters may ask for "the output of a command". A running assumption is that Openmediavault users will know how to get on the command line when necessary. This document was written to introduce new users to useful utilities and to provide examples of maintenance concepts and relatively simple procedures. \\ {{ :divider2.png?800 |----}} ===== Utilities to Help With Openmediavault Management ===== As previously noted, being able to work from the command line would be very useful to users, who may need to gather detailed information on the OS and platform hardware, for troubleshooting and for an occasional edit to a configuration file. Much can be learned with the following utilities that allow users to look at Openmediavault “under the hood”. \\ ==== WinSCP ==== WinSCP allows users, beginners and experienced alike, to visualize the Linux file structure in a manner similar to Windows Explorer. WinSCP installs on a Windows Client and connects to Linux servers, allowing users to work with their server remotely. \\ \\ One of the more useful features of WinSCP is that it gives users the ability to edit Linux configuration files with a familiar editor like Notepad. For experienced Linux Desktop users who would like to use WinSCP, it will run from WINE (in Linux Mint, Ubuntu and others). WinSCP can be downloaded here -> [[https://winscp.net/eng/download.php|WinSCP]]. ---- === Installing WinSCP === During the installation process, if prompted, select the **Explorer Interface**. This display shows the remote file system only. If the Explorer Interface is not offered it can be selected after the installation, under View, **Preferences**, **Environment**, **Interface**. ---- == Open WINSCP == <html><center> Click on <b>Session</b>, <b>New Session</b>:</center></html> \\ {{ :omv6-maint-01.jpg?400 |}} ---- \\ <html><center>Enter the following into the <b>LOGIN</b>, <b>New Site</b>, dialog box.</center></html> \\ - In **Host name**: Enter the **IP address** of the OMV server.\\ - In **User name**: Enter **root**\\ - In **Password**: Enter the **root password**.\\ \\ - Optional: Clicking on **Save** will store the above data so that it's not required to enter the same information when logging in again.\\ \\ {{ :omv6-maint-02.jpg?400 |}} \\ <html> <body> <table width="100%" border="0"> <tr> <td colspan="2" style="background-color:#69A5FF;height:30px;"> <strong><span style="color:#FFFFFF;font-size:110%;">  Note</span></strong> </td> </tr> <tr> <td style="background-color:#E6FEFF;height:25px;width:380px;"> R-PI users would enter the user <code>pi</code> and the pi <code>password</code> or a previously added user with admin privileges. Due to the restrictions of a non-root “sudo” environment, WinSCP will be restricted from root functions. This restriction can be mitigated, but it’s beyond the scope of this guide. </tr> </table> </body> </html> \\ ---- \\ <html><center>The following is normal for the first SSH connection to any client or server. Click <b>Yes</b></center></html> \\ {{ :omv6-maint-03.jpg?400 |}} \\ ---- \\ WinSCP opens with a two pane window. Selections are made in the left pane; operations are done on the right.\\ The folder **srv** was selected on the left. **dev-disk-by-uuid-xxxx** was highlighted on the right. A right click of the mouse brings up an operations menu. **Properties** was selected. In this particular popup, permissions could be changed. (Without backup, changing permissions is **NOT** a recommended action for beginners. [[https://wiki.omv-extras.org/doku.php?id=omv6:utilities_maint_backup#backups_and_backup-strategy|Backup]] is covered later.)\\ \\ {{ :omv6-maint-04.jpg?600 |}} \\ In a similar manner, a configuration file can be highlighted in the right pane. A right click of the mouse, on a file or folder, brings up the menu. Select **EDIT**, from the menu and Windows notepad, or the internal editor can be used for editing configuration files. Beginners will find either choice to be easier than using nano or vi on the Linux command line. **File and Folder Operations:** While they can be done in WinSCP, very large file copies, moves, or deletes are best done using [[https://wiki.omv-extras.org/doku.php?id=docs_in_draft:utilities_bckup_maint#mc_midnight_commander|Midnight Commander]]. <html> <body> <table width="100%" border="0"> <tr> <td colspan="2" style="background-color:#69A5FF;height:30px;"> <strong><span style="color:#FFFFFF;font-size:110%;">  New User Note:</span></strong> </td> </tr> <tr> <td style="background-color:#E6FEFF;height:25px;width:380px;"> When using the standard Linux file systems available in OMV, data drives are <b/>mounted</b> under the <b>srv</b> folder. </tr> </table> </body> </html> ---- ==== PuTTY ==== PuTTY is similar to a Window’s command prompt, but it allows users to work on openmediavault’s command line from a remote PC. If PuTTY was not installed as part of your installation process, install it on a Windows PC. It’s available here. → [[https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html|PuTTY]] Using PuTTY is as simple as typing in the server’s IP address in the **Host Name** field and clicking on open. There will be a warning for a first time connection – click **OK**. Then, login on the command line. {{ ::omv6-maint-05.jpg?600 |}} \\ Login as **root** or, in the case of a Raspberry PI, as the admin user (typically **pi**). ---- ==== MC (Midnight Commander) ==== Midnight Commander is a command line file utility that utilizes a very cleverly created graphical interface. It’s very useful for navigating through openmediavault’s directory structure. It excels in efficient copying, moving, and deleting folders and files.\\ \\ The installation process:\\ \\ * Use [[https://wiki.omv-extras.org/doku.php?id=omv6:utilities_maint_backup#putty|PuTTY]] to get to openmediavault’s command line. * Log in as ''root''. * On the command line type the following; ''apt-get install mc'' * When prompted continue with “''Y''” \\ (R-PI users will log in as ''pi'' and use ''sudo apt-get install mc'')\\ \\ When the installation finishes, on the command line, type ''mc''\\ \\ Midnight Command is a two pane window where the source is the left pane and the destination is the right pane. Copies and moves are done, left to right. Since it’s possible to navigate to any location on the openmediavault host, in either pane, the source and destination can be set for any location.\\ \\ A mouse works in MC. Click on the various menu items at the top and bottom, to select them. Similarly, files or folders can be selected by clicking on them. To level up, click on the ''/..'' at the top left of either window.\\ \\ {{ ::omv6-maint-06.jpg?600 |}} \\ <html> <body> <table width="100%" border="0"> <tr> <td colspan="2" style="background-color:#FFB663;height:30px;"> <strong><span style="color:#FFFFFF;font-size:110%;">  Warning </span></strong> </td> </tr> <tr> <td style="background-color:#FFE4A6;height:25px;width:380px;"> <b>Beginners</b>: Midnight Commander is powerful and potentially dangerous. MC does not have “<b>Undo</b>”. A careless operation on the boot drive, such as accidental file “Move” or “Delete”, can ruin your installation. </tr> </table> </body> </html> \\ Work with MC carefully and before doing anything extensive with it, the appropriate backups are recommended. [[https://wiki.omv-extras.org/doku.php?id=omv6:utilities_maint_backup#operating_system_backup|Operating System Backup]] – [[https://wiki.omv-extras.org/doku.php?id=omv6:utilities_maint_backup#backing_up_data|Data Backup]]. ---- ==== USBImager ==== [[https://bztsrc.gitlab.io/usbimager/|USBImager]] is a utility that’s designed to write raw image files to SD-cards and USB drives. What makes it stand out from similar utilities is that it can “read” a flash drive and create an image file from the contents of the device. Further USBImager is capable of compressing image files it creates, from flash media, which reduces the size of the image file. If users decide to use an SD-card or a USB thumb-drive as a boot drive; the ability to read flash media devices makes USBimager useful for cloning flash boot drives. Details for using USBimager are found in Operating System Backup , under [[https://wiki.omv-extras.org/doku.php?id=omv6:utilities_maint_backup#cloning_flash_media|Cloning Flash Media]]. ---- ==== Virtual Box ==== Virtual Box is a cross platform virtualization platform that will work with both servers and clients. For learning about openmediavault, there simply is no better tool than working with an openmediavault Virtual Machine (VM). An openmediavault VM can be built, configured, and put on the local network complete with shares, in the same manner as real hardware. VM’s can be created, cloned, used for test beds, and destroyed without consequence. Many advanced openmediavault users fully test upgrades, Docker’s, plugin’s, server add-ons and changes in configuration, in openmediavault VM’s before upgrading or reconfiguring their real-world servers. If users have a Windows client with at least 6GB RAM and plenty of hard disk space, installing Virtual Box on the client to host test VM’s of openmediavault is highly recommended. → [[https://www.virtualbox.org/|Virtual Box]] {{ :divider2.png?400 |}} ===== Backups and Backup-strategy ===== It’s important to understand the concept of backup and why backup is important. In understanding the concept of backup, an automotive analogy may be helpful.\\ \\ If one has a car and that car has a spare tire, is the “car” backed up? The answer is “No”. There are a great number of things that can happen to a car that can disable it, until parts are replaced or the car is otherwise repaired. These items would include the battery, alternator, any component of the ignition system, the transmission, the cooling system, etc., etc. To backup the car, **a second car is needed**. This is why using RAID of any type is **NOT backup**. At best, RAID could be thought of as a “spare tire” for a PC.\\ \\ Where the automotive analogy fails, generally speaking, is when a car fails it can be repaired. In computing, if a user’s personal data is lost without backup, it’s __permanently lost__. There are many possible events where data may be corrupted beyond recovery (viruses, ransomware) or is completely lost due to drive failures, a failing drive controller or other hardware failures. This is why real data back up is far more important than the computing equivalent of a spare tire (RAID).\\ \\ ---- ==== Backing Up Data ==== \\ {{ :omv6-maint-07.jpg?500 |}} \\ The scenario depicted in this graphic represents true backup. There are **two full copies of data**. With two separate copies, this backup strategy is superior to traditional RAID1 for home or small business use cases for a couple reasons. \\ * Rsync can be used with most USB connected hard drives where RAID1, when used with USB connected drives, is notably unreliable. * If there’s a drive error, an accidental deletion, a virus, or other data related issue; in RAID1 the effects are //instantly// replicated to the second drive. With Rsync, both drives are independent and, in most cases, the second disk will be available after the source disk fails. In any case, the Rsync replication interval allows time for admin intervention before the second disk is affected.\\ \\ {{ :divider2.png?400 |}} \\ ===== Full Disk Mirroring / Backup with Rsync ===== While individual shared folders can be replicated using Services, Rsync, a more efficient approach is using an Rsync Command line, in a scheduled job, under System, Scheduled Jobs to mirror a drive. This method allows for replicating the file and folder contents of an entire data drive, to an external drive or a second internal drive of adequate size. * To implement something similar to the following example; it’s necessary to add and mount a destination drive, in accordance with the section labeled A Basic Data Drive. * When formatted, the hard drives used in this example were labeled to indicate their function. This is a good practice that will help new users to easily identify drives and avoid admin mistakes. * Dissimilar sized drives can be used, provided that the destination drive is large enough to hold the source drive’s data. ---- The following Rsync command line is an example of how a data drive can be mirrored onto a second drive.\\ (**Note**: The following line is a single command line. **UUID**'s (**U**niversal **U**nique **ID**entifiers) create a long line.)\\ \\ <html><tt><span style="color:#fa502c;font-size:90%;">rsync -av --delete /srv/dev-disk-by-uuid-3bdb3de5-218b-4930-bb61-05cda64f8c6b/ /srv/dev-disk-by-uuid-dfa2df17-9764-4a47-8e9d-9a36ff74ca37/ </span></tt></html>\\ \\ The source drive is on the left and the destination is on the right. In this example, the entire contents of **dev-disk-by-uuid-3bdb3de5-218b-4930-bb61-05cda64f8c6b**, on the left, would be copied to **dev-disk-by-uuid-dfa2df17-9764-4a47-8e9d-9a36ff74ca37**, on the right.\\ \\ The switches are:\\ \\ <html><tt><span style="color:#fa502c">-a </span></tt></html>**Archive Mode.** Archive mode adds an array of options to an Rsync command. <html><tt><span style="color:#fa502c">-a </span></tt></html>is the equivalent of switches <html><tt><span style="color:#fa502c">-r -l -p -t -g -o </span></tt></html> and <html><tt><span style="color:#fa502c">-D </span></tt></html> which copies files and folders recursively, copies links and devices, preserves permissions, groups, owners and file time stamps.\\ \\ <html><tt><span style="color:#fa502c">-v </span></tt></html> **Increase Verbosity**. This can be useful when examining Rsync command output or log files.\\ \\ <html><tt><span style="color:#fa502c">--delete </span></tt></html> **Deletes files in the destination drive that are not in the source**. If accidental deletion protection is desired, this switch could be left out of the command line. However, from time to time, it may be necessary to be temporarily re-added the <html><tt><span style="color:#fa502c">--delete </span></tt></html> switch to purge previously deleted and unwanted files from the destination drive.\\ ---- \\ <html><center>To find the appropriate Rsync command line entries for the user’s server, under <b>Storage</b>, <b>File Systems</b>, on the far right, click on the <b>table</b> icon. In the pop down menu, check the <b>Mount Point</b> box.</center></html> \\ {{ ::omv6-maint-08.jpg?800 |}} \\ ---- \\ <html><center>In the popup menu, check the <b>Mount Point</b> Box</center></html> \\ {{ ::omv6-maint-09.jpg?200 |}} \\ ---- \\ Under the **Mount Point** column are the full paths, by drive UUID, needed for the Rsync command line. Also note the "**copy**" icon under each drive UUID. This icon copies the complete drive path to the clipboard, making it much easier to construct the Rsync command line in exact detail. In this example drive **/dev/sdb1** is the source drive and **/dev/sdc1** is the destination drive.\\ \\ {{ ::omv6-maint-10.jpg?800 |}} \\ ---- \\ To organize the construction of the Rsync command line, users should consider using a text editor like Windows Note Pad. Copy the source drive on the left and the destination on the right. Add in the command and switches. Take care to end each drive with a "**/**".\\ The following is an example of a completed command line that will copy the entire contents of the source drive (**/dev/sdb1**) to the destination drive (**/dev/sdc1**). {{ ::omv6-maint-11.jpg?1000 |}} ---- <html> <body> <table width="100%" border="0"> <tr> <td colspan="2" style="background-color:#FFB663;height:30px;"> <strong><span style="color:#FFFFFF;font-size:110%;">  Warning </span></strong> </td> </tr> <tr> <td style="background-color:#FFE4A6;height:25px;width:380px;"> <b>Beginners Warning, Note and Sanity Check</b> </tr> </table> </body> </html> * Getting the **source** (**left**) and **destination** (**right**) in the correct order, in the command line, is **CRUCIAL**. If they’re accidentally **//reversed//**, with the <html><tt><span style="color:#fa502c">--delete</span></tt></html> switch in place, **the empty source drive will delete all data on the destination drive**. * The safest option would be to leave the switch <html><tt><span style="color:#fa502c">--delete</span></tt></html> out of the command line until it is confirmed that two full data copies exist. \\ ---- As previously mentioned, this Rsync operation can be manually run or automated under: **System**, **Scheduled Tasks**. From the line created in notepad, copy and paste the Rsync command line into the **Command** box and select scheduling parameters as desired. As shown in this example: This Scheduled Task will run the Rsync command will run every Sunday at 0110AM. {{ ::omv6-maint-12.jpg?800 |}} ---- === User Options for Rsync Backup === **Automated:**\\ As configured above, and if **ENABLED** box is checked, this Scheduled Job will run the Rsync command line once a week, on Sunday, at 01:10AM. After the first run of the command, which may take an extended period to complete, a week or even more would be a good backup interval. Generally speaking, the backup interval should be long enough to allow for the discovery of a data disaster (drive failure, a virus, accidentally deleted files, etc.), with some time to intervene //before// the next automated backup replicates the problem to the 2nd drive.\\ \\ This is also a drawback of using automation; if data loss or corruption is not noticed by the user, those problems may be replicated to the back up drive during the next Rsync event. Longer automated backup intervals, such as two weeks or even a month, allow more time to discover issues and disable replication.\\ \\ **Manual Run:**\\ If the job is **disabled** (the **ENABLED** box is not checked), the job won’t run automatically. However, the job can be run manually, at any time, by highlighting the job under **System**, **Scheduled tasks**, and clicking the **Run** (right arrow) button. This may be the best option for users who do not check their server regularly.\\ \\ **Delete Protection:**\\ Removing the <html><tt><span style="color:#fa502c">--delete</span></tt></html> switch from the command line adds delete protection, and may allow the retrieval of files accidentally deleted from the source drive. As previously noted, to clean up the destination drive of intentionally deleted and unwanted files, the --delete switch could be manually entered into the command line, from time to time, as may be deemed necessary.\\ \\ **Keep in mind:**\\ In the event of a failing or failed source data drive, if the job is automated, it is **crucial** that the drive-to-drive Rsync job, in **Scheduled Task**s, is turned **OFF**. Similarly, after noting a problem, DO NOT run the rsync job manually. \\ \\ **The Bottom Line:**\\ The additional cost for full data backup using Rsync is the cost of an external drive, or an additional internal drive, of adequate size. For the insurance provided, the additional cost is very reasonable.\\ \\ <html> <body> <table width="100%" border="0"> <tr> <td colspan="2" style="background-color:#69A5FF;height:30px;"> <strong><span style="color:#FFFFFF;font-size:110%;">  Note</span></strong> </td> </tr> <tr> <td style="background-color:#E6FEFF;height:25px;width:380px;"> If errors occur when running the command line, see the following; </tr> </table> </body> </html> → [[https://wiki.omv-extras.org/doku.php?id=docs_in_draft:utilities_bckup_maint#rsync_drive_copy_errors|Rsync Drive Copy Errors]] \\ ---- ==== Recovery from a Data Drive Failure - Using an Rsync’ed Backup Drive ==== === General: === Again, as a reminder, when the NAS primary drive is failing or has failed, it’s **CRUCIAL** to turn **OFF** an automated drive-to-drive Rsync command line in **Scheduled Tasks**.\\ \\ === Restoration to the Backup Drive: === The backup Rsync’ed “destination” disk can become the data source for network shares. This involves repointing existing shared folders, from the old drive location, to the backup drive. All additional services layered on top of **Shared Folders**, to include **SMB/CIF** shares and other shared folder services, will follow the shared folder to the new location on the back up drive.\\ \\ === Repointing a Shared Folder: === In the following example, the data drive has failed and it’s been determined that it’s not repairable. Under **Storage**, **File Systems** we have a missing source drive. Note that the drive is "**referenced**" and **Missing**.\\ \\ {{ ::omv6-maint-13.jpg?600 |}} \\ <html> <body> <table width="100%" border="0"> <tr> <td colspan="2" style="background-color:#69A5FF;height:30px;"> <strong><span style="color:#FFFFFF;font-size:110%;">  Note</span></strong> </td> </tr> <tr> <td style="background-color:#E6FEFF;height:25px;width:380px;"> There may be ERROR dialog boxes regarding the failed mount of existing shared folders. With a missing but referenced drive, this is to be expected. When all shares are redirected, these error messages will stop. </tr> </table> </body> </html> ---- \\ The actual references to the failed source drive are the Shared Folders assigned to the drive. In this example, the Shared Folders are named **Documents** and **Music** shown below:\\ \\ {{ ::omv6-maint-14.jpg?600 |}} \\ Since the source drive no longer exists and there’s an //**exact duplicate**// of all folders and files on the backup drive, we’ll repoint the shared folder named Documents to the unreferenced backup drive.\\ \\ Click on the **Documents** line in **Shared Folders**, above, and then click on the **Edit** (pencil) button. \\ ---- \\ <html><center>In the <b>File System</b> field below, click on the pop down arrow, on the far right.</center></html> <html><center>Select the Rsync backup drive.</center></html> \\ {{ :docs_in_draft:omv6-maint-15.jpg?600 |}} \\ <html> <body> <table width="100%" border="0"> <tr> <td colspan="2" style="background-color:#69A5FF;height:30px;"> <strong><span style="color:#FFFFFF;font-size:110%;">  Note</span></strong> </td> </tr> <tr> <td style="background-color:#E6FEFF;height:25px;width:380px;"> <p>When physical drives are removed, the <b>device names</b> of the remaining drives <b>may change</b>. Drive <b>UUID's</b>, however, <b>do NOT change</b>.</p> Accordingly, insure that the drive selected in the <b>File System</b> field is the rsync backup drive. If necessary, check the destination drive <b>UUID</b> in the <b>Scheduled Task</b> rsync command line against the <b>UUID</b> in <b>Storage</b>, <b>File Systems</b>, to insure that the right device name is selected in the <b>File System</b> field above (in this example the device name is <b>/dev/sdb1</b>).</p> <p>Further </tr> </table> </body> </html> \\ <html><center>When the correct drive is selected <b>Save</b> and confirm the change.</center></html> \\ (Remember that //all// contents of the now missing source drive and the destination drive were //identical// as of the last backup. Further changes are not necessary. Repointing the shared folder is just a matter of selecting **the backup drive**.)\\ \\ Do the same process for all remaining Shared Folders. (In this example, Music was repointed as well, but not shown.) Again, error messages may appear during the process. Acknowledge them (with **OK**) but do not revert, or back out of change confirmations. When all Shared Folders are redirected to the backup drive and saved, error messages will end. ---- **In the final result:**\\ \\ With one operation per shared folder, all shared folders can be redirected to the rsync'ed backup drive.\\ {{ ::omv6-maint-16.jpg?600 |}} \\ ---- In this case, SMB network shares were layered on top of the Shared Folders above. A check of the network reveals that server network shares are up and running from the backup drive, with the same content, accesses and permissions as of the last rsync copy event.\\ \\ {{ ::omv6-maint-17.jpg?600 |}} \\ ---- A check of **Storage**, **File Systems** reveals that the destination disk is now referenced and the original source drive is no longer referenced. \\ {{ :docs_in_draft:omv6-maint-18.jpg?600 |}} \\ At this point, all shares in this example have been successfully redirected to the backup drive and the server is fully functional again. ---- One last operation is needed to completely remove the failed DATA drive. Go to **Storage**, **File Systems** and note that missing drive DATA is no longer referenced. When clicking on the failed drive, the **unmount** button is now active. **Unmount** the drive and confirm the choice. {{ ::omv6-maint-19.jpg?600 |}} ---- **When physically replacing the failed disk:** Note that the original rsync command line will no longer valid. The new disk will need to be physically added, wiped, formatted and **mounted** to be able to view the **mount point**, under **Storage**, **File Systems**. Finally, the rsync command line will need to be updated to reflect the new source and new destination drives. {{ ::omv6-maint-20.jpg?800 |}} ---- ===== Second Level Backup – Replication to a Second Host ===== {{ :docs_in_draft:omv6-maint-21.jpg?600 |}} The first item to note, is that this scenario can be accomplished using a LAN client, as the second host, and it could be a Windows client. The additional cost would be the price of a second drive of sufficient size (internal or external) to house the second copy of data, attached to a remote host. The Remote Mount Plugin can mount a Windows network share (a user name and password with write access is required) and Rsync can be configured to replicate NAS data to the Windows share.\\ \\ As illustrated above, the second host could be a low cost SBC. This scenario can be designed with a number of desirable features.\\ \\ * First, if backing-up to a second server platform, two fully independent copies of data are possible. * When using an SBC with openmediavault installed: \\ If the primary server fails completely, the second platform can be configured to take over as a backup file server. With all data backed up and resident on the SBC, this data can be made available to the network with SMB shares. \\ * Other than re-homing clients to the shares on the backup device, there’s no recovery time and no “crisis” involved in getting data back on-line. It’s already there. \\ The costs for this level of backup are very reasonable, with the cost of a hard drive of adequate size and an SBC. Good performing SBC’s are a low cost option. For home and small business use cases, older PC platforms or laptops could be configured as a backup server as well.\\ \\ The Practical details for setting up Primary Server to Backup Server share replication, using Remote Mount and Local Rsync Jobs are covered here → [[https://wiki.omv-extras.org/doku.php?id=omv5:omv5_plugins:remote_mount|Remote Mount]] \\ ---- While replication to an independent host is an excellent method of avoiding data loss catastrophes, there are other potential events which can threaten irreplaceable data. Fires, roof or plumbing leaks and other unforeseen events can result in the loss of data, even on two independent hosts. For these reasons, backup professionals and experienced server administrators recommend an off-site copy. While this may seem extreme, it’s actually fairly easy to accomplish. It can be done with an SBC or an old laptop, connected wirelessly, and housed in a utility shed with AC power. Some users set up a backup host in a family members’ house, and replicate changed data over the internet through VPN's.\\ \\ In the bottom line, if users want to keep their irreplaceable data, an absolute minimum of two full copies is recommended, with the 2nd or 3rd copy off-site being preferred. As previously noted, effective backup strategies do not have to be expensive and are relatively easy to set up.\\ \\ For further information on Backup concepts and best practices, an excellent explanation of Backup is provided by [[https://www.backblaze.com/blog/the-3-2-1-backup-strategy/|Backblaze.com]].\\ \\ ---- ===== Operating System Backup ===== By design, the OMV/Debian operating system installs on its own partition, segregated from data. This makes copying or cloning the openmediavault boot/OS drive an easy process. So, one might ask, why is a clone or a copy of the operating system important?\\ \\ Building openmediavault, from scratch, using the installer ISO is a 15-minute proposition, give or take. While it takes longer, roughly 30 to 45 minutes, the actual hands-on portion of an SBC build is even less.\\ \\ As users configure their servers, add services, reconfigure shares, move their data around, tweak access controls, etc., servers tend to become “works in progress”. Configuring a server to the user’s preferences can be an evolution that may take weeks or even months. If a complete server rebuild is required, the customization, add-ons, and the collection of various user tweaks may take several hours to recreate. It is this time and effort that Operating System Backup will preserve.\\ \\ There are several ways to duplicate an operating system boot drive but many can be technically involved, requiring network access to remote servers, bootable utilities and somewhat complex processes.\\ \\ Given the low cost of flash media and with sockets mounted on the outside of a PC case, SD-cards and USB thumb-drives lend themselves to cloning and very quick recovery.\\ ---- ==== The Benefits of Maintaining Operating System Backup ==== In accordance with [[https://en.wikipedia.org/wiki/Murphy's_law|“Murphy’s Law”]], users may encounter issues where things go wrong. As examples, users may test software on their active server or try new settings. On occasion, installing an add-on may have unintended consequences. Trying new settings or working on the command line, may break openmediavault in a way that might not be recoverable. In other cases, there may be instances where a software update goes south – the source repository may go off-line in the middle of an update resulting in broken packages.\\ \\ In all of these cases, having a confirmed working clone of the boot drive will allow users to “drop back” to a known good state. The “FIX” would be as simple as shutting down and booting the server on a known working clone.\\ \\ The advantages of maintaining operating system backup are obvious. Beginners, with very little knowledge of Linux, can work with their servers without fear, which facilitates learning. If a Linux update causes ill effects, it’s possible to drop back and selectively install packages to isolate the exact cause of the problem. If an add-on update doesn’t work (direct installed software, a plugin, Docker, etc.), the user can gracefully back out of the update and leave the older (but working) software package in place.\\ \\ It’s the easiest, quickest, and most effective fix, for resolving problems with openmediavault and the underlying Debian Operating System.\\ \\ **The practical issues of maintaining boot drive clones – when to update and rotate?**\\ \\ 1. Linux Operating System package and security updates are unlikely to have a direct impact on the operation of the server. While updating the boot drive is not strictly required after a package or security update, it would make sense to wait a week or two (more?) to insure that all is working. Remember, the cloned drive is "the fall back". Insure that there are no ill effects with an OS update, before updating the clone. \\ \\ 2. Add-on packages that are direct installed on the boot drive, Dockers (Plex, Urbackup, Pi-Hole, etc.), are another matter. If add-ons are updated, where the update may change the way server add-on's interact with the data they generate, updating the cloned boot drive may be a good idea.\\ \\ 3. If a network share is added, deleted, or any aspect of the NAS is reconfigured that changes the operation of the NAS, the backup would need to be updated. Otherwise, if needed for recovery, the configuration of the cloned boot drive would not mesh with the configuration and contents of data storage drives.\\ \\ 4. If a cloning mistake is made (let’s respect Murphy’s Law), a 3rd clone could become a “fallback of last resort”. Given that Linux package upgrades and openmediavault sub-version upgrades have little to no effect on network shares or the high level configuration of the NAS, a 3rd clone can be maintained that is updated //only// when the NAS configuration is changed.\\ \\ === A Last Important Note About Backing Up your OS === Just as it is in the commercial world, where support for a product may be discontinued, the open source community is constantly moving forward as well.\\ \\ Users may believe that an ISO file, or image, contains all the software needed for a build. In some current build cases, that assumption would be incorrect. Linux distro’s, during the initial build and to finalize the installation, may depend on on-line software repositories. After the installation is complete, patches and updates may be applied which rely on on-line repositories as well.\\ \\ Can it be assumed that those same software repositories and resources will be available on some future date, exactly as they were at the time of a current build? The answer is “**No**”. Distributions of a specific Linux version, complete with specific applications, fully patched and updated, can be built for a **limited time**.\\ \\ Therefore, if users have extensively configured builds, are using specialty hardware (such as SBC’s) or are using openmediavault to serve a critical function; it would be wise to backup the boot drive to an image file, or Clone the fully configured working installation to separate media and save one or more copies for future use.\\ \\ ==== Cloning Flash Media ==== To avoid issues that can result from dissimilar sizes, it’s best to clone images from/to identical SD-cards or USB thumb-drives. Otherwise, it’s easier to clone if a new drive is slightly larger than the working drive.\\ \\ (And while it’s an intermediate level technique, [[https://gparted.org/livecd.php|Gparted]] can be used to slightly shrink flash drive partitions, to fit on the smaller of the two flash drives.)\\ \\ **The Cloning Process for USB thumbdrives and SD-Cards**\\ \\ * Install [[https://bztsrc.gitlab.io/usbimager/|USBImager]] on a Windows PC. * Format a new SD-Card or USB thumb-drive with [[https://www.sdcard.org/downloads/formatter/|SDFormatter]] * Test the new card or USB drive with [[http://www.heise.de/ct/Redaktion/bo/downloads/h2testw_1.4.zip|h2testw1.4]]. One verify test is enough. (Endless verify is unnecessary.) \\ If the device registers errors, or if the capacity is significantly different from what is that’s marked on the label (a fake), return it for a refund or throw it away.\\ \\ **At this point users might consider marking the working SD-card or thumb-drive to insure that it doesn’t get mixed up with the blank card. Otherwise, it is possible to read a “blank card” and use the blank image to “overwrite” the working card.**\\ \\ Insert the **working** card and start USBimager.\\ \\ <html> <body> <table width="100%" border="0"> <tr> <td colspan="2" style="background-color:#FFB663;height:30px;"> <strong><span style="color:#FFFFFF;font-size:110%;">  Warning </span></strong> </td> </tr> <tr> <td style="background-color:#FFE4A6;height:25px;width:380px;"> <b>SANITY Check</b>; at this point, make sure you inserted your <b>working</b> SD-card or USB thumb-drive . </tr> </table> </body> </html> \\ **Note:** Windows will not be able to read the format of the partitions on the working boot drive and offer to format it for you. **DO NOT** format the drive. Close the dialog box with the **X**.\\ \\ * In most instances, USBImager will detect USB thumb-drives and SD-cards, and set the Device drive letter. However, it would be prudent to check the letter Windows assigns to the drive with Windows Explorer. * First click on the **down arrow** (under Write and Read). The formatted card or USB drive should be visible. **Select it**. * If blank, check the boxes for "**Verify**" and "**Compress**" (Compress greatly reduces the image file size.) * Click **Read**. \\ {{ ::omv6-maint-22.jpg?400 |}} If USBImager.exe is opened on the Desktop, the image file will be deposited on the Desktop. The file name will appear in the following format: **usbimager-20221209T1455.dd.zst**\\ This format provides the year, month, date and time that the image is taken.\\ If storing the image for future use, renaming it to something like **OMV6-Server1-2022-1209.dd.zst** may be helpful.\\ \\ <html> <body> <table width="100%" border="0"> <tr> <td colspan="2" style="background-color:#69A5FF;height:30px;"> <strong><span style="color:#FFFFFF;font-size:110%;">  Note</span></strong> </td> </tr> <tr> <td style="background-color:#E6FEFF;height:25px;width:380px;"> <p><b>In consideration of the time required to "write" a clone:</b></p> If the user/admin is running a business or is in another time sensitive scenario, where the NAS server can not be out of service for an extended period; the server can be booted on the source drive while the clone is being written. Thereafter, the drive swap could be accomplished after-hours or during a low use period. </tr> </table> </body> </html> \\ ---- The last process uses USBImager to burn the image file and verify it in one pass. * Insert the new flash drive and start USBImager. * Select the image file previously created, select the destination flash media drive, and click the **Write** button. {{ :docs_in_draft:omv6-maint-23.jpg?400 |}} <html> <body> <table width="100%" border="0"> <tr> <td colspan="2" style="background-color:#69A5FF;height:30px;"> <strong><span style="color:#FFFFFF;font-size:110%;">  Note</span></strong> </td> </tr> <tr> <td style="background-color:#E6FEFF;height:25px;width:380px;"> Depending on the size and speed of the flash media being used, the speed of USB port and other factors, the write operation may take from 15 minutes up to an hour or more. </tr> </table> </body> </html> To be certain of success, test the newly created clone by shutting down the server and booting on the cloned media. With two (or more) working boot drives in hand, user/admins will be well prepared for boot drive failures or mishaps. ---- ===== Add-on’s – Adding Value to Your OMV server ===== ==== General ==== The [[https://forum.openmediavault.org/|openmediavault Forum]] has an extensive [[https://forum.openmediavault.org/index.php?board/29-guides/|Guides section]]. Whether a user’s preference is videos or printed text, there’s something for everyone among the numerous “How-To’s”. Beginners and Advanced users alike should take a few minutes to familiarize themselves with the content in the Guides section of the Forum. ==== Openmediavault’s Plugins ==== Openmediavault has numerous plugin’s. Some are integrated into the base package by openmediavault’s developer Volker Theile. Examples are iSCSItarget, usbbackup, among others.\\ \\ Still more were created by openmediavault plugin developers, such as Remote Mount, the flash-memory plug-in, backup plugins, and more.\\ \\ Many plugins are integrations of third party packages such as [[http://www.snapraid.it/|SNAPRAID]], [[https://github.com/trapexit/mergerfs#name|MergerFS]], etc. While questions or issues regarding the **integration of plugin’s**, into openmediavault, are of interest to openmediavault’s developers, questions regarding the **operation** of the base software package are best directed to the external application’s supporting web site.\\ ==== Dockers - General ==== While Dockers are an avenue toward adding //extensive// functionality to openmediavault, they are an advanced topic that may prove to be frustrating for beginners. To get started, after installing OMV-Extras, beginners should consider installing the Compose Plugin and reviewing the [[https://wiki.omv-extras.org/doku.php?id=omv6:docker_in_omv|Compose Plugin document]].\\ \\ While it’s command line oriented, this [[https://docker-curriculum.com/|Docker Tutorial]] is very helpful for understanding basic concepts. User authored Docker - How To’s can be found on the openmediavault forum in the [[https://forum.openmediavault.org/index.php?board/29-guides/|Guides section]]. In addition, there’s a forum section dedicated to [[https://forum.openmediavault.org/index.php?board/39-docker/|Docker issues and questions]]. === So, What is a “Docker”? === Dockers are a type of Virtual Machine (VM) that share the Linux kernel and memory spaces with the host. A Docker is spawned from a Docker image. The resultant VM equivalent, that’s built from a Docker image, is referred to as a “container”. A container is fully self-sufficient, bare-bones, Linux operating system. The idea behind a Docker image is to create a Linux installation, that is as small and as lean as possible, that includes all necessary dependencies required to run the Docker application and nothing more. Since these containers tend to be very small, they can be constructed and destroyed rapidly. (After downloading, usually, in a matter of seconds.)\\ \\ Dockers are more resource efficient when compared to running a full VM in a hypervisor, due to direct allocation of hardware resources. Typically, VM hypervisors provision fixed blocks of memory and may require access to dedicated hard disk space or block device partitions. Whether these dedicated resources are used by the VM or not, they’re no longer available to the Host operating system or other VM’s. A Docker, on the other hand, uses the needed memory space to run its processes and the host’s hard drive for storage, without wasted resources. Resource management is lean and tight, allowing more Docker containers to run concurrently with much greater efficiency.\\ \\ ---- === Installing Docker === Effective May 30th, 2023 Docker is supported with the -> [[https://wiki.omv-extras.org/doku.php?id=omv6:omv6_plugins:docker_compose|Docker Compose Plugin]] A guide for installing Dockers is covered in -> [[https://wiki.omv-extras.org/doku.php?id=omv6:docker_in_omv|Docker in OMV]]. ---- ---- \\ === Dockers - It’s about choices === While there are 100,000+ Dockers, available on the [[https://hub.docker.com/search?image_filter=official&q=&type=image&operating_system=linux|Docker Hub]], all are not created equal. The offerings, from Docker authors, range from one-off experiments with no documentation (users are on their own) to organizations like [[https://www.linuxserver.io/|Linuxserver.io]] that specialize in building first-rate Docker images. Linuxserver.io offers Dockers that have been thoroughly tested, they support multiple architectures, they provide detailed container setup instructions, their offerings are “Tagged” and they retain inventories of their older images.\\ \\ ---- === Selecting a Docker - Primary Considerations === **First:**\\ \\ Potential Docker users must use Dockers that support their architecture. The three primary architectures supported by openmediavault are **ARMHF** or **ARM64**, **i386(32 bit)**, and **amd64(64 bit)**. In most cases, 32bit Dockers will run on 64bit hardware. While there may be exceptions, i386 and amd64 Docker images may not run on ARM platforms. “**Multi-arch**” (multiple architecture) Docker images are more platform flexible.\\ \\ **Second:**\\ \\ To increase the chance of success, when attempting to install a Docker without a guide, look for the more popular Dockers with the highest number of “pulls” on the Docker Hub. There are good reasons why these Dockers are broadly popular – they tend to work. -> [[https://hub.docker.com/|hub.docker.com]]\\ (Note: It is not necessary to sign in or to create an account. Simply use the search bar.)\\ \\ **Third:**\\ \\ In the vast majority of cases, Dockers that fail to work won’t have anything to do with openmediavault or Portainer. Their issues tend to originate from selecting the wrong architecture, selecting the wrong network mode (host, bridged, macvlan) for the application, other configuration issues (such as port 80 OMV/Docker conflicts), permissions problems or the Dockers themselves.\\ \\ Since most Dockers share Network ports with the host (openmediavault), it’s important to use ports that are __not__ currently in use. To get a better understanding of network ports and for commands that will reveal ports that are in use, refer to this forum post for more information:\\ \\ [[https://forum.openmediavault.org/index.php?thread/28506-how-to-define-exposed-ports-in-docker-which-do-not-interfere-with-other-services/|[How-To] Define exposed ports in Docker which do not interfere with other services/applications]]\\ \\ {{ :divider2.png?600 |----}} ===== When things go wrong ===== First take note of any error dialog boxes. On most Windows and Linux machines it’s possible to copy and paste the text out of a dialog box by holding down the left mouse button and dragging the mouse pointer over text, to highlight it. Then use the keys with Ctrl+c (to copy), then click in a Notepad document and use Ctrl+v (to paste). This basic information will be helpful, in searching out the details related to the problem.\\ \\ ==== The First Resource – The Internet ==== Users should search the internet first. The solutions for many generic problems can be found with [[https://www.google.com/|google]] , [[https://duckduckgo.com/|duckduckgo]], and other search engines. When searching on key words that match error messages or other problems users may be having, in some cases, answers can be found in real time. This is the fastest and often the best way to learn how to fix server problems. Since openmediavault is based on “Debian”, it may be a useful search term.\\ \\ While the search function of the openmediavault forum site will produce “hits” on search criteria, it is by no means all inclusive. If OMV is included in search terms, a Google search may generate more result hits on information found on the openmediavault forum, than the forum’s integrated search function.\\ \\ With information from searches, users should make an effort to address their own issues. This approach tends to be the path to the fastest answers and greatly facilitates the learning process.\\ \\ ==== The Openmediavault Forum ==== → [[https://forum.openmediavault.org/|Forum]]\\ \\ **When coming to the forum for help:**\\ \\ First search the forum. In many cases, user problems can be resolved with a few searches and a bit of reading. Second, look at the dates of posts and the version of openmediavault referenced. Posts that are 3 or more years old may not apply to the current openmediavault version.\\ \\ If posting a problem on the forum, start at the [[https://forum.openmediavault.org/|forum index]], and look for the category that looks to be appropriate for the post. Along with an explanation of the the issue, the openmediavault version, the appropriate logs and command line output, if known, and the hardware platform in use are the absolute minimums required. Realize that, without information, even the most experienced users, Moderators, and / or Developers will not be able to provide assistance.\\ \\ * Ask the right questions. For beginners, this can be deceptively difficult. There’s some “straight forward” guidance on this topic here → [[http://www.catb.org/~esr/faqs/smart-questions.html#idm368|Ask the right questions]].\\ * While openmediavault’s forum is known for responsiveness, it’s unrealistic to expect answers in real time. It may be a matter of days before a forum member, who is familiar with the described problem, will read and respond to a post.\\ * When looking at answers, try to focus on the information presented, not the perceived tone. Remember that support is provided “**gratis**”, so act accordingly.\\ * Be open-minded. The reason why users post on the forum should be because they couldn’t solve a problem on their own. With that in mind, when an experienced forum user replies, taking the time to make a suggestion or requesting more information, forum users should follow up and post the result. Whether the issue is fixed or not, user posts help other users with the same or a similar problem.\\ * If a forum post or a “How To” fixes your problem, or gets you through a configuration issue, consider giving the author a “Like” or “Thanks”. The gesture is free and it’s an indicator to other users who may have the same problem. In essence, you’d be saying “I agree” or “this worked for me”.\\ * When users are experiencing problems with their data store (a file system issue, a hard drive, array, etc.) the working assumption on the part of experienced forum users and moderators will be that **users have full data backup**. Accordingly, recommendations for correcting filesystems, hard drives, and RAID array issues may result in the loss of data. **Keep this in mind**.\\ \\ ---- ===== Solutions to Common Problems ===== Follow this link to the maintained list on the forum. → [[https://forum.openmediavault.org/index.php?thread/21269-solutions-to-common-problems/|Solutions to Common Problems]]\\ \\ ==== USB RAID ==== \\ **Problem:** I have USB connected hard drives that I want to configure as a RAID array.\\ \\ **N/A:** There is no support for creating USB RAID, in the GUI. There are many reasons for this. While a USB RAID array can be initialized from the command line, it's strongly discouraged.\\ \\ ==== Rsync Drive Copy Errors ==== \\ **Problem:** Rsync shows errors “Operation not permitted (1)” or “renaming” with regard to the files <html><tt><span style="color:#fa502c">aquota.user</span></tt></html> and <html><tt><span style="color:#fa502c">aquota.group</span></tt></html> These files are found at the root of data drives.\\ \\ In a small number of instances, the quota service may interfere with an Rsync drive-to-drive copy.\\ \\ **Solution 1:**\\ \\ Turn the quota service off.\\ \\ <html><tt><span style="color:#fa502c">sudo /etc/init.d/quota stop</span></tt></html>\\ \\ (In the following examples, substitute the appropriate UUID's for the source and destination drives in the Rsycn command line.)\\ \\ <html><tt><span style="color:#fa502c">sudo quotaoff --user --group /srv/dev-disk-by-uuid-3bdb3de5-218b-4930-bb61-05cda64f8c6b</span></tt></html> ''''\\ <html><tt><span style="color:#fa502c">sudo quotaoff --user --group /srv/dev-disk-by-uuid-dfa2df17-9764-4a47-8e9d-9a36ff74ca37</span></tt></html> ''''\\ \\ Optionally, delete the files <html><tt><span style="color:#fa502c">aquota.group</span></tt></html> and <html><tt><span style="color:#fa502c">aquota.user</span></tt></html> from the source and destination drives.\\ \\ **Solution 2:**\\ \\ Add the following exclude statements to the rsync command line: <html><tt><span style="color:#fa502c">--exclude='aquota.user' --exclude='aquota.group'</span></tt></html> A full command line example: <html><tt><span style="color:#fa502c">rsync -av --delete --exclude='aquota.user' --exclude='aquota.group' /source-drive/ /destination-drive/</span></tt></html> ---- ==== USB Power - A Common Raspberry PI problem ==== **General:**\\ \\ Many problems with R-PI’s, in versions up to the R-PI3.X, are related to under-powering. While the R-PI4 is much improved, depending on the power requirements of connected peripherals, it is not exempt from power problems. The issues caused by under-powering can range from bizarre behavior to data corruption on storage devices.\\ \\ **Do I have a problem?**\\ \\ With all peripherals attached that are normally used – use the command ''dmesg'' on the CLI and scroll through the output. If an **undervoltage** situation exists, it will be noted in the output.\\ \\ **What is the problem?**\\ \\ Beyond using a power supply with the appropriate current rating for the R-PI model, it should be noted that a USB power source must meet certain voltage specifications “at the socket”. In essence, the output voltage of a USB power supply can’t be increased to compensate for external voltage losses typical when using a long USB cable with small gauge wire.\\ \\ Making matters worse is, models prior to the R-PI4 use a micro USB plug as the power connection. The tiny contacts of a micro USB connection, combined with cables for micro USB that have small gauge wires, drop power supply voltage significantly.\\ \\ ---- \\ **Consider the following chart of voltage losses, versus wire length and gauge**\\ \\ (Note that voltage drops increase as current draw requirements rise.)\\ \\ {{ ::omv6-maint-27.jpg?600 |}} \\ **Potential Remedies:**\\ \\ * Use a power supply that meets at least the minimum recommended current rating for the R-PI model being used. * Use the shortest possible USB cable. Cables that are 1 foot / 30cm or less, made of thick gauge wire are preferred. If a short USB cable is not long enough to place an R-PI in a convenient location, use an AC extension cord rather than a long USB cable. * Avoid using direct connected USB powered hard drives. The additional current load will drop voltage and may stress a weak power supply. A self powered USB hub or a drive dock is preferred. * Avoid leaving peripherals attached, such as a monitor, keyboard or a mouse. Even when they’re not in use, they consume power. {{ :divider2.png?400 |}} ===== A Closing Note ===== We, who support the openmediavault project, hope you’ve found this guide to be useful and that you’ll find your openmediavault server to be efficient, easy to use, and enjoyable.\\ \\ If you found this guide to be helpful, please consider a modest donation to support the hosting costs of this server (OMV-Extras) and the host project (Openmediavault). \\ \\ **OMV-Extras.org** \\ <html> <form action="https://www.paypal.com/cgi-bin/webscr" method="post" target="_top"> <input type="hidden" name="cmd" value="_s-xclick"> <input type="hidden" name="hosted_button_id" value="2BQNGSC8HQJME"> <input type="image" src="https://www.paypalobjects.com/en_US/i/btn/btn_donate_SM.gif" name="submit" title="PayPal - The safer, easier way to pay online!" alt="Donate with PayPal button" border="0"> <img alt="" src="https://www.paypal.com/en_US/i/scr/pixel.gif" width="1" height="1" border="0"> </form> </html> \\ \\ **www.openmediavault.org** \\ <html> <form action="https://www.paypal.com/cgi-bin/webscr" method="post" target="_blank"> <input name="cmd" type="hidden" value="_s-xclick" /> <input name="hosted_button_id" type="hidden" value="95MF5UQ66PW2E" /> <input style="border: 0px; padding: 0px;" alt="PayPal - The safer, easier way to pay online!" name="submit" src="https://www.paypalobjects.com/en_US/i/btn/btn_donate_LG.gif" type="image" /> <img loading="lazy" src="https://www.paypalobjects.com/de_DE/i/scr/pixel.gif" alt="" width="1" height="1" border="0" /> </form> </html> \\ \\ \\ omv7/utilities_maint_backup.txt Last modified: 2024/04/22 04:05by crashtest