Otee

Making Short Links Shorter: Using NGINX + Custom Domain

January 13, 2022

In this post, I explain how to shorten the short links generated by Twirl. For example, from oteetwirl.herokuapp.com/l/L5w2 to twirl.otee.dev/l/L5w2.

Problem Statement

  1. Twirl is a per-user URL shortening app. To learn more on how Twirl was built and designed, read this, this and this. Here’s the GitHub repository hosting Twirl: https://github.com/oitee/twirl.

  2. Twirl is deployed on Heroku, as a free app. It is available at: oteetwirl.herokuapp.com.

  3. Custom domains with HTTPS support is not available as a free service on Heroku. This causes the resultant short links generated by Twirl to be not so short, after all.

  4. Solution: we keep running Twirl on Heroku but use a shorter custom domain which will shorten the short links generated by the app. For example, instead of generating oteetwirl.herokuapp.com/l/L5w2, the app should generate twirl.otee.dev/l/L5w2

  5. In order to expose this new short link, Twirl should also display the shorter version of short links on its UI.

Technical Considerations

There are, largely, three ways to go about this:

  1. Approach I: Migrate Entirely to IaaS: Run the whole app on a self-owned server, like Google Cloud Compute. This will allow us to choose our domain name and also run the system. But if we do this, we will need to manage the database, and have the server run all the time (unlike Heroku’s auto-sleep option). The bigger concern here is that the world already knows about oteetwirl.herokuapp.com. In order to not break user experience, we will not only have to copy the data from the Heroku database to the GCP server but also continuously serve the older short URLs on this app (i.e., have Heroku redirect requests for the current short links to GCP).

  2. Approach II: Have custom redirection to PaaS: Provide a redirection mechanism from shorter links (using a custom short domain name) to the main system, which continues to run on Heroku. This has the advantage of cost-efficiency, since we already have a GCP server, which was set up for another project

  3. Approach III: Hybrid of I & II: In Approach I, instead of the lift-and-shift way of migration, we could, in theory, keep the database running on Heroku, but have the app run on GCP. But this will still not be as cost-efficient as Approach II, because the HTTP server on GCP will have to run continously.

So, our preferred approach is the second approach:

The above C4 diagram is generated using this

Step 1: Creation of DNS Entry

We add the DNS entry for the GCP VM which we had already spawned another project:

So now, twirl.otee.dev points to my GCP VM.

Step 2: Setting up NGINX

We will loosely follow the steps for rewriting URLs, mentioned in the official docs:

  1. We will use the return directive

  2. We will redirect to the client using HTTP status 301 (moved permanently)

  3. We will construct the redirection URL by keeping everything the same, as before, except the hostname.

  4. We will set up HTTPS using LetsEncrypt

To do steps 1, 2, 3, we create a new file here: /etc/nginx/sites-available/twirl.otee.dev with :

server {
    listen 80;
    listen 443 ssl;
    server_name twirl.otee.dev;
    return 301 $scheme://oteetwirl.herokuapp.com$request_uri;
}

Next, we enable this site, by creating a symlink:

sudo ln -s /etc/nginx/sites-available/twirl.otee.dev /etc/nginx/sites-enabled/twirl.otee.dev

Next, we should restart NGINX:

sudo systemctl status nginx
sudo systemctl restart nginx

At this point, if we go to twirl.otee.dev it should take us to Twirl, which runs on Heroku. But, since we are using a .dev domain, it is necessary that we create an SSL certificate for twirl.otee.dev so that HTTPS is supported.

If certbot is installed we can simply use this command : sudo certbot --nginx. If certbot is not installed, it needs to be installed (see documentation).

Now, the redirection to twirl.otee.dev should work. But if it does not, we may have to flush the DNS caches. This can be done for Google Domains here: https://dns.google/cache.

At this point, we have reached our goal of using a short domain name for our short links. Example: the following link should now work: twirl.otee.dev/l/L5w2.

While we have enabled redirection of short links with the domain twirl.otee.dev, the Twirl application will still display the longer URL, using Heroku’s default domain name.

Instead of displaying oteetwirl.herokuapp.com/l/L5w2 we want Twirl to display twirl.otee.dev/l/L5w2. Currently, Twirl is deisgned in such a way that the server generates the unique URL path and the final link is constructed by client-side JavaScript by appending the current domain:

function shortLinkHref(link) {
  let finalLink = window.location.protocol + "//" + window.location.host + link;
  return `<a href='${link}' target='_blank'> ${finalLink}<a/>`;
}

Instead of this approach, we can create the short link at server-side itself. To do this, we should first expose the intended domain name from the environment on the server. This is required because the server does not know the domain name on which it is serving requests. It just knows that it is serving requests on a local port. Even if it was aware of the IP address of the machine (which is sometimes the case), it would still not know which domain names (there can be multiple) map to that IP address because this part (i.e., domain name to IP address conversion) is necessarily done by the client by making DNS queries. Therefore, the intended domain name should be provided to the server as an additional information. This is purely decorative and the server will blindly generate the short URL, by appending the short URL path to the given domain name.

To enable this, the server was exposed to a new environment variable, namely CUSTOM_DOMAIN_NAME which contains our shorter domain name, namely twirl.otee.dev. Now, everytime a request for shortening a link is received, the server sends back the complete short URL (as opposed to just sending the URL path, as was the case earlier).

According to this design, the front-end does not operate on short-link received from the server. This causes a slight hindrance, as the client-side JavaScript needs to generate requests for enabling and disabling existing short links, when the user clicks on the toggle button on the home page. Earlier, the client-side JavaScript would generate these requests in the following manner:

async function updateStatus(link, currentStatus) {
  // 'link' is the URL path of a short link, 
  // 'currentStatus' indicates whether a short link is enabled or disabled at the moment
  link = link.substring(3);
  if (currentStatus) {
    await fetch(`/l/disable/${link}`, { method: "POST" });
  } else {
    await fetch(`/l/enable/${link}`, { method: "POST" });
  }
  await analyticsGenerator();
}

To allow the client-side JavaScript to generate the above requests, the server, in addition to sending the complete short URL, also sends the short URL path (without the first three common characters /l/). This allows the above function to operate without much changes:

async function updateStatus(linkPath, currentStatus) {
  if (currentStatus) {
    await fetch(`/l/disable/${linkPath}`, { method: "POST" });
  } else {
    await fetch(`/l/enable/${linkPath}`, { method: "POST" });
  }
  await analyticsGenerator();
}

Twirl, now, displays shorter short links on its home page 🎉