File Serving with a Twist

A Credera client recently asked us about hosting static PDFs and images. But they needed to support very granular permissions (think private documents you’d only want to give one or two people access to view). This is something that comes up fairly often, so I thought I’d share the various approaches we’ve seen work (and not work).

The typical approaches are:

• Write something custom to stream bytes over the Internet.

• Use a cloud based solution that supports permissions/expiring links (such as this solution).

• Use a static web server (Apache, nginix, etc.) and use some set of plugins:

  • Native authentication providers such as Basic Auth, LDAP, etc.

  • Custom authentication providers

Client usually recoil in horror at the first one (stream bytes). But this solution actually works the best in many cases. It’s not that hard these days—especially as most frameworks provide all the necessary building blocks: IO can be done with non-blocking methods, files can be cached on the server to reduce disk IO, you can gzip the byte stream to reduce bandwidth, etc.

The other alternatives have some serious drawbacks:

• A CDN/cache doesn’t buy you much—cache hits are pretty much limited to the two to three parties involved.

• Expiring/signed links aren’t foolproof, can have varying user experiences, and require a lot of ongoing maintenance.

• Most authentication providers for web servers don’t support this type of granular permissions.

• Writing custom authentication providers is difficult and error prone (native code).

You also have the problem of the browser downloading the entire document and maybe rendering correctly versus prompting for a download (i.e., the user has Adobe installed, the browser plugins are configured, correctly, etc.). A couple common workarounds:

• Save image and convert to some video format on the back end—stream like that and include paging controls, etc.

• Use some type of universal JS media viewer (like jQuery Media Plugin).

Obviously, you want to use caching proxies, CDNs, etc., if you don’t have these granular authorization requirements. But if you do, we’d recommend:

• Write a simple download server that checks authentication and streams bytes:

  • Read/write from some type of shared disk somewhere.

  • Cache if necessary to save IO (but you probably won’t save much I/O).

  • Gzip if necessary (but you probably won’t save much bandwidth).

• Have the front end use something like jQuery Media Plugin to control the display.

• Monitor that solution and adjust if bottlenecks arise.

Have you run into other edge cases when serving static files?  We’d love to hear about them.

• Caching
• Cloud Services
• Streams
• CDNs
• File Serving