As it turns out, running a chrome instance on my little VM absolutely destroys it. CPU goes to 100%, RAM usage goes to 100%, things don't do so well after that. Clearly I should have a bunch of worker nodes that pull sites in parallel so I don't put the load on the backend interface itself. I'd probably like to have at least a four core VM, or maybe a bunch of smaller VMs, unsure exactly what would be best. The config I have in mind would probably run me over $100/month. There's just a lot of optimization to be done here that I don't have time for.

I think for this one I may have to release the code and document it nicely here rather than deploying the site. It's too bad since I put so much time into getting the deployment stuff set up but probably best to keep things moving.