Confluence Docs 3.0 : Hierarchical File System Attachment Storage This page last changed on Jun 08, 2009 by pcurren. # Introduction For Confluence version 3.0, the structure of attachments stored on the filesystem was changed. In versions of Confluence prior to 3.0, attachments were stored in directories corresponding to the id of the content to which they belong. The more content in Confluence with attachments, the more directories you would have immediately beneath your configured attachments directory. This directory structure has been changed in Confluence 3.0 and since the default configuration of Confluence is to store attachments in the filesystem, this change is likely to have relevance to administrators of most existing Confluence installations. If you are installing Confluence for the first time, there will be no consequences as a result of this change. If you are upgrading from a previous version of Confluence, the migration to this new filesystem structure should happen automatically during the upgrade. The reason for introducing this change was to address the issue CONF-13004. Certain file systems have a limit on the number of files that can be stored in a directory and large Confluence installations were reaching this limit. In addition, storing too many files at a single directory level can cause performance degradation in some circumstances. This new attachment storage strategy ensures this will no longer be the case.
| Backup Confluence Home Before upgrading to Confluence 3.0, as with any upgrade you must ensure you have a backup of your Confluence home directory before you proceed.
# The New Directory Layout The attachment storage layout was chosen to fulfil the following main requirements: Limit the number of entries at any single level in a directory structure. Partition attachments per space making it possible for a system admin to selectively back up attachments from particular spaces (see the JIRA issue for more details). An attachment in Confluence can be thought of as having a number of identifying attributes: id, space id and content id. That is to say, the attachment logically belongs to a piece of content which logically belongs in a space (not all content belongs to a space). For attachments within a space in Confluence, the directory structure is typically 8 levels, with the name of each directory level based on the following algorithm:
level | Derived From
--- | ---
1 (top) | Always 'ver003' indicating the Confluence version 3 storage format
2 | The least significant 3 digits of the space id, modulo 250
3 | The next 3 least significant digits of the space id, modulo 250
4 | The full space id
5 | The least significant 3 digits of the content id, modulo 250
6 | The next 3 least significant digits of the content id, modulo 250
7 | The full content id
8 | The full attachment id
Within the 8th level will be a file for each version of that attachment, named to match the version number e.g. 1 An example: To find the directory where attachments for a particular space are stored, you can use the JSP findspaceattachments.jsp at the location `<confluence url>/admin/findspaceattachments.jsp`. This JSP requires a space key and returns the directory on the file system where attachments for that space are stored. Attachment D in the above diagram is stored in a slightly different structure. Attachments that are not conceptually within a space replace the level 2 - 4 directories with a single directory called 'nonspaced'. Examples of such attachments are the global site logo and also attachments on draft content. # Upgrading to the new attachment storage structure As mentioned previously, this upgrade is only necessary if you have Confluence configured to store attachments on the file system. If migration is not necessary due to a different storage configuration (for example, because attachments are stored in the database), then no migration will occur during upgrade and the Confluence log will simply show the following messages -
    INFO [main] [AbstractUpgradeManager] upgradeStarted Starting automatic upgrade of Confluence
    INFO [main] [UpgradeTask] isUpgradeNeeded The configured attachmentDataDao does not store
        attachment data on the file system so the HierarchicalFileSystemAttachmentUpgradeTask is not necessary.
    INFO [main] [AbstractUpgradeManager] upgradeFinished Upgrade completed successfully
Should migration be required, it will occur automatically during upgrade and the log will show output similar to this -
    INFO [main] [UpgradeTask] doUpgrade Beginning HierarchicalFileSystemAttachmentUpgradeTask. Depending on the size of the
        attachment data this may take some time.
    INFO [main] [UpgradeTask] run 4023 pages may have attachments to be moved to a new hierarchical structure.
    INFO [main] [UpgradeTask] run 0 of 4023 pages have had their attachments moved to the new structure
    INFO [main] [UpgradeTask] run 500 of 4023 pages have had their attachments moved to the new structure
    INFO [main] [UpgradeTask] run 1000 of 4023 pages have had their attachments moved to the new structure
    INFO [main] [UpgradeTask] run 1500 of 4023 pages have had their attachments moved to the new structure
    INFO [main] [UpgradeTask] run 2000 of 4023 pages have had their attachments moved to the new structure
    INFO [main] [UpgradeTask] run 2500 of 4023 pages have had their attachments moved to the new structure
    INFO [main] [UpgradeTask] run 3000 of 4023 pages have had their attachments moved to the new structure
    INFO [main] [UpgradeTask] run 3500 of 4023 pages have had their attachments moved to the new structure
    INFO [main] [UpgradeTask] run 4000 of 4023 pages have had their attachments moved to the new structure
    INFO [main] [UpgradeTask] run Successfully moved the attachments for all 4023 pages to the new hierarchical structure.
    INFO [main] [UpgradeTask] doUpgrade Completed HierarchicalFileSystemAttachmentUpgradeTask.
    INFO [main] [AbstractUpgradeManager] upgradeFinished Upgrade completed successfully
| It should be noted that for most implementations of Java, the migration to the new data structure involves moving the files (not copying them). Hence, there should not be a need to have additional disk space available. It also means that the migration should be relatively fast.
# Have you previously applied the CONF-8298 patch? The patch or workaround on the CONF-8298 issue changed the structure of attachment storage but not to the most efficient possible structure. So during the Confluence 3.0 upgrade process this intermediate (CONF-8298) structure will be detected and automatically upgraded. # Troubleshooting the upgrade It should be noted that in the event of a failure, your attachment directory may be in an inconsistent state and your first step in troubleshooting should be to restore the backup of your home directory. There are a number of reasons the migration could fail. This will be shown in the log with a message similar to "`Failed to move the attachments for all pages to the new hierarchical structure.`". Immediately preceding this message in the log will be entries for each page whose attachments could not be moved. The following table shows examples of these messages and offers some possible explanations.
Example Message | Description
--- | ---
The configured attachment directory `<directory name>` could not be found or was not a directory. | The configured Confluence attachment directory is not accessible. Check confluence home for the attachment directory and ensure the permissions are correct to allow reading and writing for this directory.
It is not possible to migrate the attachments to the new structure since files already exist which the attachment process may need to create. | Your attachments directory contains files or directories which the upgrade task wants to create. That is, a top level directory called ver003 containing directories or files with names containing up to 3 digits (e.g. 1, 213). This could be due to a previous failed attempt to migrate the attachments. You should restore a previous good copy of your attachments directory and remove any files or directories with this naming pattern before retrying.
Couldn't find current Confluence content for the id `<content Id>`. The attachment is a non-spaced attachment (e.g. global logo, draft attachment, etc) and will be migrated to the nonspaced directory. | This is a normal message indicating that the attachment being migrated does not belong to a space e.g. global logo.
Problem while accessing the database for content id `content Id` so its attachments will not be migrated. | It was not possible to access the database at this point during the migration. You will need restore your Confluence attachment directory from the backup and attempt the upgrade again, once the database is accessible again.
Could not create the new attachment directory `directory`. | The upgrade task could not create the new directory to contain the attachment being moved. Does the server user have sufficient permission to perform this operation in the indicated directory? Is there sufficient disk space?
Failed to move the current attachment directory `<some path>` to the new location of `<some other path>`. | The upgrade task could not move the directory. Does the server user have sufficient permission to perform this operation in the indicated directory?
Attachments: attachmentstorage.png (image/png) attachmentstorage.png (image/png) attachmentstorage.png (image/png) attachmentstorage.png (image/png)

Document generated by Confluence on Nov 05, 2009 23:26