Back to Blog Posts

File Asset Storage with Amazon S3

By Matt / March 22nd, 2019


In WHMCS 7.7, we introduced new functionality designed to give you more control over where and how file assets are stored.

The most visible change from this work was the introduction of a new Storage Settings configuration page within the WHMCS admin area that enables you to view and manage file storage settings via the admin user interface without having to edit configuration files directly on your web server.

Before we take a deep dive into the new functionality and how it can benefit you, let's first cover some of the key concepts.
What are File Assets?
File assets are any user uploaded content. The most obvious and common example is attachments uploaded as part of support tickets, but file assets can also include things such as client profile attachments, downloads you make available to customers, email attachments and project management files.
What is Amazon S3?
Amazon Simple Storage Service (or Amazon S3 for short) is an object storage service offered by Amazon Web Services (AWS) that provides access to an infinitely scalable cloud-based data store. Amazon S3 uses the same highly scalable, reliable, fast and inexpensive data storage infrastructure that Amazon uses to run its own global e-commerce network and gives all of us access to highly available and highly durable data storage at a cost-effective price.
What have we added in WHMCS 7.7?
As part of a broader undertaking of work to make WHMCS easier to work within highly available architectures, we've introduced support for leveraging storage providers other than the local file system. Specifically, we've introduced support for the Amazon S3 object storage protocol.

Using a service such as Amazon S3 for file storage has many possible benefits:
  • Scalable: Amazon S3 offers an infinitely scalable data store that grows with you.
  • Redundancy: Amazon S3 stores data on multiple devices across a minimum of three Availability Zones (AZs) in an Amazon S3 Region, allowing amazing to offer 10 9's of durability.
  • High Availability: Amazon S3 offers high levels of uptime guarantees ensuring that files are always accessible, and coupled with the redundancy gives you protection against power outages and natural disasters.
  • Security: Data stored in Amazon S3 is stored separately from your web server and files, allowing you to benefit from better separation of responsibilities, as well as offering you the ability to store file assets encrypted at rest.
  • Cost Savings: Amazon S3 gives you the ability to store a large amount of data at very low cost. With Amazon S3 you also only pay for your what you use while disk and storage space on your own server has to be predicted and has a fixed cost whether you use it or not.
  • Centralised Storage: Allows you to centralise the storage of files which allows multiple deployments of WHMCS such as those found in a highly available architecture to access and use the assets concurrently.

As the leading pioneer in the object storage space, AWS S3 has set the standard for cloud-based object storage and now many other cloud-based data storage providers now also offer S3 compatible services. As a result, this means that by introducing support for the S3 object storage protocol to WHMCS, you can choose to store your files not only with Amazon S3, but also with any other S3 compatible service including the likes of Google Cloud, DigitalOcean Spaces, and even self-hosted solutions such as Minio.
How do I get started?
The new Storage Settings setup interface can be accessed via the Setup menu. To get started, navigate to Setup > Storage Settings.

Upon navigating to this UI for the first time, you will be presented with a list of file asset types and their currently configured storage location. By default, all file assets are stored in Local Storage, which means they are stored on the local file system.

Selecting the Configurations tab allows you to configure additional storage configurations, both local and remote.

To add a new local storage configuration, simply click the plus icon next to the local storage dropdown and then enter the path to the local directory in the field where requested. Note that the directory must be writeable by the web server as this will be validated prior to saving the new storage configuration.
To add a new S3 storage configuration, select S3 from the "Add New Configuration" dropdown menu and then click the plus icon. The following modal window will be displayed. Enter your Amazon S3 credentials as requested and then hit Save to validation the configuration. If successful, the configuration will be saved and the modal will close.
We have an excellent walkthrough tutorial guide that walks you through the process of setting up storage with Amazon S3 here. For further details, please visit Configuring Amazon S3 Storage Provider within WHMCS

Once you've configured your desired storage locations, return to the Settings tab where you can configure the desired storage locations for each file asset type.
What about my existing file assets?
We knew when we started work on this feature that it was vitally important to provide an easy migration path to allow users to migrate from one storage location to another. And our team has done an excellent job providing fully bi-directional migration support that can safely and automatically transfer files from an existing local storage location to a remote S3 compatible storage location, and back again.

Any time you make a change to the storage location configured for a file asset type, you will be prompted to choose if you wish to start the automated migration process. Choose Migrate to begin, or Switch if you plan to migrate the existing file assets manually.
Once started, the migration process will run in the background via cron until it has completed. When migrations are performed in this way, we do not remove files from the existing storage location and therefore there is no risk of losing data.
Questions?
We hope you'll find this new functionality useful. If you have any questions about the new storage configuration options or the migration process to S3, I invite you to ask those questions in the comments below.

Thanks for reading!

Liked this article? Share it