The Evolution of the Cloud
I’ve been really (really) excited about a recent project, not unlike my excitement at this moment in time https://t.co/zN3VKEC5Pc How’s that for a tease for now? :)
— timmmmyboy (@timmmmyboy) May 5, 2020
We have some really exciting things that we're working on right now that we'll be sharing in the coming days and weeks ahead. But I think it's worth first taking a look back at my personal and Reclaim Hosting's history in hosting to understand why I think this next step is such an important one. This will be a long post, but anyone who has been watching or has been a part of this history will likely appreciate the context for how these ideas have evolved (I know I personally do, and after all that's what this blog is for).
Shortly after I joined DTLT at the University of Mary Washington in the summer of 2011 one of my first projects was to start DTLT Today, a semi-regular video podcast that everyone in the group to some capacity participated in. We streamed it live which provided a really cool synchronous element and I still go back and watch the shows today to get a sense of where our heads were at around particular topics at the time. You can find all the episodes on my YouTube channel if you scroll back far enough. And that streaming part was an interesting one. I had experience through ds106.tv with third party tools like Justin.tv (now defunct because they became Twitch) and it was before the days of YouTube offering free streaming. I got it in my head to play around with Amazon Web Services to quickly create a streaming server of our own running streaming software called Wowza. The software would have been quite expensive and I surely had no server of our own on campus I could run it on, but for just an hour of live streaming we could do it very cheap (pennies per day).
That was my first time experiencing what "the cloud" could really do for web-based projects. The distributed plug and play nature of the tools got me hooked because there was a very low barrier to entry. If something wasn't working right, destroy it and start a new one. There was a marketplace of applications I could rely on for prebuilt stuff and even back in 2011 AWS had a lot of different services (now exponentially so).
By 2014 we had started to turn our eyes to AWS once again as a possible alternative hosting environment for UMW Blogs. We had suffered performance issues on that platform for quite awhile as we tried to handle growth and this seemed an opportunity for us to have a more flexible infrastructure that could grow with that project. We had planned to just move the database to an RDS instance and maybe the uploads to S3. But we ran into such latency that rather than scrap everything we added an EC2 server in and moved the whole application. And it was so fast!
In the Fall of 2014 I had also begun playing with a piece of software I was starting to hear about called Docker. Something about servers being the old guard and "containers" being the new hotness. But what was a container? On the face of it they felt a lot like virtual servers. There was a hub I could search and find all kinds of different applications just like at AWS. But these were much faster than spinning up an EC2 instance because containers didn't have to start an entire operating system and make use of all the hardware of the computer they were running on, they were much smaller virtualized instances of just the application they were running with the ability to share all the common OS-level stuff across multiple containers.
Keep in mind Reclaim Hosting had been a thing for a year at this point and Domain of One's Own at UMW for 2 years. Add to that Hippie Hosting and I had about 3 years of server experience under my belt but it was all in the context of shared hosting with Plesk and cPanel. Because Docker allowed me to run applications I could never run previously in LAMP servers I was immediately hooked. I figured out a way to share multiple Ghost containers on a single server so we started offering that through Reclaim Hosting. Similar with Discourse forum hosting. I even had a Federated Wiki instance up and running for a time that was running on Docker containers (remember that?!). It's been a long running dream for us to offer an elegant solution to allow people to play with this stuff. But there was still a lot of manual work creating and removing these containers when users would sign up, and a lack of a real solution meant slow adoption and difficulty supporting it.
Fast forward to the end of 2014 and I've given notice that I'm going to do Reclaim Hosting full time, quite a leap of faith given the security of working at a public institution. But this shit was fun, people were trusting us, and the community was growing. In December of that year Jim and I had the thought to bring Kin Lane to Fredericksburg to talk to us about APIs and think through how we might build an architecture for that at Reclaim Hosting. My head was still in the clouds with all the Docker stuff and we couldn't resist thinking about it in terms of containers, all different endpoints and applications as various containers with their own unique hostnames. It was a solid few days of brainstorming that pushed the very boundaries of my understanding at the time and I still treasure not just these scribbled notes on a whiteboard but also the generosity of Kin's time with us.
In early 2015 I learned of a project called Sandstorm that allowed for container-based hosting of applications. It was open source and I set a server up to start playing. It checked a lot of boxes in making it easy to start containers running Ghost, Etherpad, and other apps without the overhead of needing to be a developer, but the drawbacks were that it was not using Docker and building apps to run in the environment was very difficult with their model. It also seemed pretty heavily to favor collaboration applications with less of a focus on publishing so oddly content management systems were more difficult to work with in their model. The group developing Sandstorm got hired by Cloudflare and the project is mostly abandoned at this point, but I know OpenETC has made quite a bit of use of the platform.
The Coming Sandstorm.io
By August of 2015 I return back to AWS again, this time to experiment with moving Reclaim Hosting's main site to a distributed stack there. Not just a single EC2 instance with files in S3, but rather a load balanced stack with multiple app servers in different regions, a CDN on top, and deployment through Git and a staging server to boot. This was amazing but it also ended up being quite expensive and a lot of overhead to manage. I didn't find Amazon's tools very user-friendly (there are much more developer-focused of course) and I couldn't justify the high costs of multiple EC2 instances for a site that while mission critical also wasn't a massive load at the time even on a shared server.
3 years ago in Fall of 2017 I learned of a project called Cloudron and started playing with it. Similar to Sandstorm you could install it on your own server, it was (key word was) open source and this system used Docker containers at its core. Even more it made it fairly easy to map custom domains, provision SSL certs, and run your software on the web with ease and the list of support apps had a lot of amazing stuff like Gitlab, Ghost, Etherpad, and more. We piloted a program with Bates College in which we integrated a Cloudron server with their Domain of One's Own program and we even considered at one point whether or not Cloudron could be the backend of some kind of future "Domains 2.0" environment where folks weren't limited by the LAMP stack. Unfortunately what we found was that the focus on Cloudron as a personal server made it less of a great fit for a multi-tenant model. There was a basic user management built in, but most models reflected a single or multiple admins handling all the installation rather than a true platform as a service style offering that we were hoping for. And the business model of Cloudron has changed a lot over the years with open source now being nothing more than a label (turns out the ability to install applications is not open source and requires subscription fees paid to them so there's a lot of lock-in there) and development of our own applications was proving to be challenging because they supported Docker but you had to build with the container as a base and setup volumes in a very particular way for them to work. So back to the drawing board.
The common themes here are that this stuff is not new (nothing ever is) but that it has traditionally been very complex and framed for developer audiences. And in lot of ways that's where Reclaim Hosting found themselves with the state of web hosting in 2013. It wasn't that cPanel or LAMP stacks were some brave new world we were exploring, but we saw an opportunity to build a community around providing easy hosting for educators in a context where you could get real support and build out a presence on the web. We found our niche in that and have continued to double down on that with our Domain of One's Own program and Managed Hosting services. The platform plays a big role, but so do the people.
Domain of One’s Own: Notes from the Trailing Edge
Maybe you're building your Next Generation Digital Learning Environment and wanting to use open source tools like Mattermost, Etherpad, Jitsi Meet, and WordPress to create a framework for your courses. Maybe you're like me in 2011 and wanted to play with some complex piece of software without investing in the high server costs and overhead of buying a VPS (ever tried to run your own install of Canvas?). Maybe you're curious to try running your blog on Ghost like this one here. And maybe you want to start a project small with potential for it to grow in a big way over time and you don't want to have to continually adjust hosting environments for whatever level will account for new traffic you are experiencing (we see that a lot with WP Multisites on campuses where year 1 looks very different from year 5 of the project, or hell even a Sunday evening when everyone's homework is due). Is there a DH project that you've been eyeing but not sure how to install and run it? Are you trying to get support for newer scholarly publishing platforms at your university but you aren't sure where to start?
Our efforts and experiments throughout all of this have been to answer the question of "what's next?" and look towards the future of web hosting with an evolution of the past. It needs to have a user-friendly interface. It needs to be affordable. It needs to be without limitations. It needs to scale.
Today starts that journey, and Reclaim Cloud is coming very soon. In the coming weeks we will be talking more and more about what this platform is and what it can provide. If any of this post hits on struggles you've had or interests please do add your email to the list to be updated and we'll be working to add more people to the beta as we ramp up towards a public launch.