Uploading assets

Upload | QBank | HTML5

It is hard to upload files over the web! Specifically it is hard to upload many and/or large files. The main problems lies in the limited resources of time and space, but we also have to handle browser differences and make the process easy for the user.

Time

We use PHP as our back-end and one thing that every programmer who uses PHP soon learns is that every process is time limited. The limit is usually in the range of 20-60 seconds. When that limit is up, the execution is shut down, no matter what. This means that we have to complete the upload within the allotted time or we are left with an incomplete file, usually thrown away, and the user is shown an error. The time limit is a setting and can be extended, but there is of course a downside, a longer time limit makes the server more sensitive to errors. The time limit exists to restart processes that somehow are stuck and not doing any good. The reasoning is that by using a relative short time limit the server will never be unresponsive for a long time as stuck processes are restarted quickly.

The solution is to keep processing of the uploaded file to a minimum and offload heavy work onto processes specifically designed for it. We make sure to only upload in short bursts and just store the file before returning. This way we keep the the probability that we would ever hit the time limit very low. When an upload is complete messages are sent out to the processes which does the time consuming work. These processes are not time limited and are not exposed to the user, which means that if they get stuck for a longer time it is not a big deal.

Space

Files occupy space, whether it is in memory or storage. When a process is receiving a file during upload, the file resides in the process memory. Until recently the maximum allowed upload size for a PHP process was 2GB regardless of how much memory was available. A large file also takes a long time to transfer which may cause troubles described previously. Our solution is to use a technique known as chunking which means that we only send a small part of the file at a time, minimizing its space requirement. We then reassemble the pieces on the server and once they are all transferred we store the complete original file. Typically this is only done if the file being transferred is larger then a certain threshold as small files would be less then one "chunk" anyway.

Browsers

All browsers are different and all implement the standards differently. While competition is great for technical progress, it can be a nightmare to flatten out the bumps to ensure that our code is run the same. To maximise the abilities we like to utilize HTML5, but being such a new technology not all browsers have implemented all the APIs. Some browsers are more behind then others and some are harder to keep updated as well, *cough* Internet Explorer *cough*.

This meant that we have to use fallbacks so that those stuck with older browsers are not left behind. We do this by using libraries that emulate the behaviour should it have implemented the standard fully, effectively filling in the gaps. These libraries are commonly known as polyfills. By using the polyfills we fully support IE9+.

User experience

When the user uploads many and/or large files, he/she is usually left to wait for the upload to finish before resuming work. This is not ideal as it breaks the workflow. QBank 3 allows the user to continue working by letting the user index the files currently uploading. It is also possible to hide the dialog and let the upload finish in the background. The interface is nicely presented with large and easily understood progress bars displaying the status.

We believe that we have created one of the best uploading experiences out there, and if there is a need to automate the uploading there are transfer agents.

Subscribe now

Don't miss the latest from QBank Blog.

Start your journey today

Book a demo with our sales team