Jekyll2026-07-07T14:49:50+00:00https://otee.dev/feed.xmlOitihjya SenBackend engineer and former lawyer writing about distributed systems, legaltech, and Indian policy.Don’t move back to the Writers’2026-05-06T00:00:00+00:002026-05-06T00:00:00+00:00https://otee.dev/2026/05/06/why-not-writersThe reported plan of the new BJP government in West Bengal to shift the state secretariat back to the Writers’ Building in BBD Bagh, from Nabanna, may be presented as a pragmatic step and a return to heritage. But it is actually a failure of imagination.
What the building was meant for
The Writers’ Building was built in 1777 by Thomas Lyon, on behalf of the British East India Company. It was Calcutta’s first three-storeyed building — a 150-metre Greco-Roman structure designed to project authority.
It stood as a grand symbol of who held power, who dispensed it, and who was expected to receive it in silence. The building sits at the heart of what was then called the White Town, deliberately separated from the Black Town where the native population lived. It was not built for Indians. It was built to administer them.
More than just a secretariat
On December 8, 1930, three young revolutionaries — Benoy Basu, Badal Gupta, and Dinesh Gupta — walked into this building and shot dead Colonel N.S. Simpson, the Inspector General of Prisons, notorious for his brutality toward political prisoners. Badal took cyanide on the spot. Benoy died in hospital five days later. Dinesh was hanged in 1931.
The statues of Benoy, Badal and Dinesh proudly stand in front of the building to this day. The square is named after them. This place has real, consequential history. In fact, the Writers’ Building is the most important building in BBD Bagh. It is the kind of place that should be a museum, or a public cultural space. A place where Calcuttans can actually walk in and experience their own history. Turning it back into a working secretariat, with thousands of employees, filing cabinets, and departmental paperwork, is the least imaginative thing you can do with it.
Weight of history
This history did not end with independence: the Writers’ Building has been the seat of every government that presided over West Bengal’s long, steady decline. The Left Front governed from it for 34 years. The Congress before them. The Trinamool Congress after (albeit briefly). Whatever their individual failures or achievements, the aggregate result is stark: capital flight, industrial decay, and a metropolis in decline. All of this happened under administrations that governed from this very building.
If the new government genuinely wants to signal a break from the past, the symbolism matters. Coming in and immediately moving into the same building that every previous government occupied is not a signal of change. It is a signal of continuity with the very past you are trying to distinguish yourself from.
Colonial Mindset
The BJP has made much of moving away from the colonial hangover in Indian governance. The Central Vista project was explicitly framed as a break from Lutyens’ Delhi, from the North and South Blocks that the British built to administer their empire. The logic, as articulated by the government, was that independent India should govern itself from spaces it built for itself, not from spaces built to serve colonial masters.
The Writers’ Building was built to serve colonial masters. If the BJP truly believes in Viksit Bharat — a developed India free of colonial baggage — it should start its new innings in Kolkata by not returning to a building commissioned by the British East India Company.
A new Secretariat for New Bengal
The new government should operate from a new secretariat that represents the wishes and aspirations of the 21st Century Bengali — a space that reflects what Bengal should become, not what it has been.
And free the Writers’ Building. Convert it into what it should be: a public space for Calcuttans to walk into and experience. Return the building to BBD Bagh, not the bureaucracy.
]]>Personal Project, Public Outage: A Lesson in Incident Management2025-07-08T00:00:00+00:002025-07-08T00:00:00+00:00https://otee.dev/2025/07/08/personal-project-public-outageRecently, I experienced a minor outage on a personal VM instance hosted on Google Cloud Compute. While the stakes were low, the process of diagnosing and resolving the issue, and then ensuring it doesn’t happen again, was a learning experience that I’d like to share.
The incident began, as many do, with a simple observation: a new, work-in-progress application I had deployed was not responding. This application was built using Spring Boot and was deployed as a containerised service (using docker). The fact that my new application was not working was not alarming; new deployments often have teething problems - and I had consciously deployed a rough-around-the-edges application to test it on different devices.
However, my routine check on other, stable services running on the same VM (twirl.otee.dev and remind.otee.dev) revealed a more serious issue – they were down as well.
My first instinct was to SSH into the virtual machine to inspect the logs. However, to my shock, my attempts to connect, both from my local machine and through the GCP browser-based utility, timed out. This was a significant red flag; an unreachable SSH daemon indicated a fundamental problem with the VM itself.
A glance at the GCP dashboard for my instance revealed stressed CPU and disk I/O graphs from sometime earlier, confirming that the system had been under duress.
Disk operations during the outage
CPU utilisation during the outage
This was a classic case of a “silent failure” – an issue that could have persisted for days had I not chanced upon it. In any software engineering company, this would have been an incident, in the rest of the post, I will treat it as such in three steps: mitigation, root cause analysis, and prevention.
The Mitigation: A Step-by-Step Recovery
The primary goal was to bring the stable services back up and running. With direct access to the instance unavailable, I was constrained to take a more forceful step: a “hard” reboot. I initiated a stop and start of the VM instance directly from the GCP dashboard.
This brought one of the services, remind.otee.dev, back online. However, twirl.otee.dev was now serving a 502 Bad Gateway error, (meaning, nginx was working but the application server was not).
Thankfully, the reboot had restored SSH access. Once inside the VM, a quick check of the running services confirmed my suspicion. The twirl.service was not running. I inspected the list of enabled systemd services using:
As it turned out, twirl.service was not on this list, meaning it wouldn’t automatically start on a system reboot. I enabled it with:
sudo systemctl enable twirl.service
And then started it manually:
sudo systemctl start twirl.service
A final check of the service’s logs showed no errors, and twirl.otee.dev was back online.
sudo journalctl -u twirl.service -f
Finding the root cause
With the immediate fires extinguished, my focus shifted to understanding the root cause of the initial outage. The swipe service, my new work-in-progress application, was the prime suspect.
An analysis of the serial port logs from the GCP dashboard provided the smoking gun:
Jul 5 09:08:13 calculus kernel: [94239163.698111] Out of memory: Killed process 4494 (java) total-vm:1870112kB, anon-rss:170216kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:604kB oom_score_adj:0
This meant that a Java process had become so memory-intensive that the Linux kernel’s “OOM Killer” had to step in and kill it.
The swipe service was the only Java process and the only containerized service running on the instance. It was clear that my new application was using more memory than what the VM could provide, causing the OOM Killer to step in.
Upon checking the Dockerfile it emerged that there were no resource limits specified (like the -Xmx flag for heap size). The JVM, by default, can be greedy with memory, consuming more memory than the instance could afford.
FROM eclipse-temurin:24-jdk-alpine
COPY target/swipe-0.0.1-SNAPSHOT.jar swipe.jar
EXPOSE 3001
ENTRYPOINT ["java","-jar","swipe.jar","--server.port=3001"]
Prevention of future outages
To prevent a similar silent outage from happening again, I implemented the following:
Proactive Alerting: The first and most obvious fix. I have now configured alerts in Google Cloud Monitoring to send me an email if CPU utilization or memory usage exceeds 80% for more than five minutes. A silent failure should never be silent again.
JVM memory limits: Limiting the max heap size available to swipe to 128MB, while starting the application via Dockerfile:
# Set max heap size to 128MB
ENTRYPOINT ["java","-Xmx128m","-jar","swipe.jar","--server.port=3001"]
Enforce Container Resource Limits: In addition to limiting memory usage at the application-level, updating the docker-compose.yml file to limit the total memory available to the containers:
services:app:# ... other configsmem_limit:256mmem_reservation:128mpostgres:# ... other configsmem_limit:256m
Add memory limits on docker process: Additionally, limiting the total memory that the docker process can access, by adding MemoryLimit and MemoryAccounting in swipe.service in systemd:
[Service]...MemoryAccounting=trueMemoryLimit=512M
Takeaway: A lesson in devops
This incident was a stark reminder that our work doesn’t end with writing code. The infrastructure that supports our applications is equally critical.
It brought to light the importance of solid software engineering fundamentals, like resource management and proactive monitoring.
But perhaps the most important lesson was in the incident response itself. In the heat of the moment, the priority should always be to restore service first - diagnosing the root case can be done after services are restored.
]]>Ping-pong: Using multiple Kafka consumers to process events2023-06-19T00:00:00+00:002023-06-19T00:00:00+00:00https://otee.dev/2023/06/19/ping-pongIn this post, I write about ping-pong - a project that aims at simulating how messages in a live chatroom (with multiple concurrent users) may get processed, using a Kafka messaging system.
This project makes use of a Kafka producer to push user messages to a topic and uses two independent Kafka consumers to process them. Here’s the github repository for this project: https://github.com/oitee/ping-pong
Largely, the system relies on a Kafka producer to push each new user message and on two independent Kafka consumers to process each new message:
Kafka producer: It pushes new user-generated messages to a common Kafka topic. These messages are meant to be consumed by different consumers.
Messages consumer: This consumer reads each event (containing user-generated messages) from the Kafka topic and prints out the contents of each user message
Active users consumer: This consumer consumes each Kafka event and periodically publishes a list of all active users.
Redis: The active users consumer stores each active user, in a Redis store, and periodically fetches the list of all active users from this store.
The interaction of the above components are demonstrated in the following sequence diagram:
Demo
Before going ahead with further details, here is a demo of the system:
Generating user-messages
User-messages have to be generated before pushing them to the Kafka producer. To simulate a chat-room:
A list of users is hard-coded in the project (present in ping-pong.utils name-space). As an avid fan of The Office (US), the users are all the characters of The Office(US)
Using the list of user-names, a one-time list of emails for each user is generated by randomly using a list of common email domains and the user-names of each user. Each user entity, thus, is represented as follows:
The list of user-entities is stored in an atom called users in the ping-pong.users namespace. This namespace exposes a function called get-user that randomly returns a user-object from this atom.
The ping-pong.producer periodically fetches a user-entity from this function and sends two types of Kafka events:
Messages: Every second, after fetching a user entity, it makes a HTTP get request to an open API which returns random quotes. This forms the message body of the respective user. Finally, it sends the Kafka event. Here’s a snapshot of how this is done (the entire namespace can be found here):
Hearbeats: These events are sent at random intervals (between 1 and 7 seconds) and are meant to represent a user is active, even though they are not typing a new message. Unlike a message event, a heartbeat event does not contain any message key in the payload(the entire namespace can be found here):
Here’s a sequence diagram representing how the producer generates kafka events:
Processing User Messages
This project uses two different Kafka consumers to process each Kafka event:
Messages Consumer: This consumer consumes each Kafka event and prints out the message payload
Active Users Consumer: This consumer consumes each Kafka event and stores each user in a Redis store. Every three seconds, it prints out the current list of active users by querying the Redis store.
Messages Consumer
This consumer does three things:
Consumes each event
Checks if the activity key to determine if it is a send-message event.
If yes, it prints out the message along with the user name.
Here’s a code-snapshot of this (the entire code can be found here):
Broadly, this consumer consumes each Kafka event, and stores the user in the Redis store along with the timestamp mentioned in the event payload. The Redis store keeps a timestamp-to-user mapping inside a sorted set.
Every three seconds, the Redis store is queried for fetching the current set of active users. An active user is defined as a user who is seen at least once in the past 30 seconds.
Here’s a snapshot of how this is done (the entire code can be found here)
(defnstart-active-users-consumer[sys](let[continue-fn(fn[_]@continue?)consuming-fn(fn[value](let[{:keys[tsuseremail]}(utils/keywordise-payloadvalue)](add-usersysuseremailts)))](ping-pong.utils/start-consumingconsumer-configutils/topiccontinue-fnconsuming-fn)))(defnprint-active-users[](while@continue?(let[active-users(get-active-users)](if(empty?active-users)(println"No Active Users...")(do(println"Active Users:")(run!#(print(str%", "))active-users)(println"\n---"))))(Thread/sleep3000))(println"----xx---"))
Here’s a sequence diagram of the above:
Tracking Active users
The system relies on Redis to store active users. Specifically, a sorted set is used for storing users. In a sorted set, entities are sorted on the basis of their respective scores. In the present case case, the event time-stamp is the score.
Thus, every time we need to fetch active users, we can query the sorted set in Redis, by using zrangeByScore where the min score is the cutoff timestamp (i.e., 30 seconds prior to the current time-stamp) and the max score is the current timestamp.
Importantly, while fetching the list of active users, it is important to clean up redundant data, i.e., all users with a score less than the cutoff timestamp.
(defnget-active-users[](let[curr-ts(double(System/currentTimeMillis))cutoff-ts(double(-curr-ts30000))active-users(.zrangeByScorejedisstorecutoff-tscurr-ts)];; Remove users who were active before the cutoff time-stamp(.zremrangeByScorejedisstoreDouble/NEGATIVE_INFINITY(deccutoff-ts))(map#(first(cs/split%#"::"))active-users)))
Importance of passing event timestamp
Notably, we are using the time-stamp value recorded in the Kafka event (i.e. event timestamp), while pushing a user in the redis store. This may seem redundant: we can use the current time-stamp at the time of consuming the events to do this (i.e. processing timestamp). This will reduce the payload of the kafka event.
However, relying on processing timestamp may lead to inconsistencies if the consumer dies in the middle of consuming events. In such an event, there will be a gap before consumption can resume. When the consumer resumes comsumption, it will start with consuming the pending events. Many of these events would be stale events. Without the event timestamp recorded in the event payload, all these events will be considered as “new” events. However, by relying on the event time-stamp, the consumer can easily detect stale events and reject them as inactive.
]]>Understanding: Log-Structured Merge Trees2023-04-17T00:00:00+00:002023-04-17T00:00:00+00:00https://otee.dev/2023/04/17/log-structured-merge-treeIn this post, I am listing down my understanding of Log-Structured Merge Trees (LSM Trees, in short), based on my reading of the first part of Chapter 3 of ‘Designing Data-Intensive Applications’ by Martin Kleppmann.
An LSM tree-backed storage engine comprises of an in-memory data-structure, called a mem-table. Every write operation is done on this mem-table. This data-structure ensures that keys are stored in a sorted manner, irrespective of the write pattern.
Once the size of this data-structure reaches a given threshold, its contents are written to a file on disk (called a log). As the mem-table stores data in a sorted manner, the log file also contains keys in a sorted manner. Note that log files are immutable - each time you write to a file you either append or write to a new log file.
In the background, multiple log files are merged and compacted (i.e. removal of duplicate keys) from time to time.
Read path: Reads are done in the following manner: search for a key in the mem-table, then on the latest log file, all the way up to the oldest log-file.
Write path: Directly update the mem-table. This is one of the reasons LSM trees are so fast.
In addition to the memtable, we also keep an in-memory hash-index of keys, for each log, where each key maps to the on-disk location of the key. Because keys are always sorted, we keep a sparse index.
Detailed Notes
Simple k-v store: The Log
The simplest form of a key-value datastore would involve a text file (often called a ‘log’), where each line consists of a key-value pair (separated by a comma)
For writing to this data store, i.e., creating a new pair or updating an existing pair, we simply append the key-value pair at the end of that log
For reading, we need to scan the entire log, by looking for the given key. If two keys are found, the latest key-value pair will be considered. This is because we never overwrite an existing entry in the log: we simply append at the end of the log.
This kind of data-store has very fast writes - O(1) as appending to a file is a very cheap operation. Conversely, look-ups are very painful - O(n), where n is the number of records in the log.
Important: we never write existing data present in the log file.
Using an in-memory hash-map to improve performance
To improve the look-up operations, we can make use of indexes, which store additional meta-data about the records and act as a “signpost”.
Essentially, while doing writes, we store this additional metadata in addition to the appending operation. Thus, adding any kind of index will slow down write operations, but will also speed up read operations. Because the effectiveness of an index is dependent on the patterns of read queries, an application developer needs to manually add indices - they don’t come set by default.
One common index is to use an in-memory hash-map, which contains all the keys of the actual data-store, and these keys map to the on-disk location of that key in the data store. This makes look-ups far more efficient: given a key, we can use the in-memory data-structure to find the location of the key on the disk (as opposed to scanning the whole log file).
One limitation of this in-memory hashmap approach is that the total number of keys is limited by the finite size of RAM - since all keys of the data-store needs to be stored in the in-memory data-structure. Given this limitation, this kind of data-store is very useful in cases where the number of keys are finite but the values are updated frequently.
We also need to consider the situation where the log file becomes so large that its size becomes comparable to that of the disk. To overcome this, we can break the log file into segments, such that, the moment the log reaches a given size, we make subsequent writes on a new segment file.
Merger and compaction of logs
We can use compaction on existing segment files, ie, removing duplicates between two or more existing segments. This can reduce the overall size of the data-store as we get rid of duplicate keys (which are really just redundant data).
Source: Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems
Since compaction can make segments smaller, we can merge two or more segments into one new segment.
This compaction and merging of frozen segments can happen asynchronously (in the background), while new writes take place on a new segment and reads use existing segments.
Source: Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems
We keep separate in-memory hash-map indices for each segment. When we need to look up a given key, we start with the hash-map of the latest segment and go all the way back to the oldest segment, stopping the moment we find the key we are looking for. Deleting operations need to be dealt with separately for this kind of data-store: when a key is deleted, a special value should be stored against it on the log file.
If the database crashes, we will need to re-construct the in-memory hash-maps of each segment from scratch. This limitation can be overcome, if we keep a copy of the in-memory data-stores on a separate log of operations on the disk.
Another limitation of this append-only data-store is that it is not efficient for making range queries.
SSTables and LSM-Tree
In an append-only data-store, writes are not made in any particular order: they are made sequentially.
But we can also store the key-value pairs in a sorted manner (i.e. sorted by keys), while storing in the segment logs. This format is called Sorted String Tables.
For a given segment log, a single key appears only once (this can be easily achieved by compaction, as seen above).
SSTables have several advantages over simple append-only databases.
First, merging segments is possible, even when the size of the segment files exceed the available memory. While merging multiple segments, we start with the first key of each segment file, and write the key with the lowest value, and so on (similar to mergesort algorithm). If two or more segments contain the same file, we pick from the segment which is more recent (as that’s how segments are created).
Second, we no longer need to maintain the file location of each key of the data-store. If we know the location of “handbag” and “hardwork”, we will know that the location of “handsome” will be in between these two locations. So we can jump to the offset of “handbag” and scan till we come across “handsome”. Thus, we no longer need to keep an in-memory map of all the keys of the data-store.
Source: Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems
How to store keys in a sorted manner, in the first place? We can use an appropriate in-memory tree data-structure (such as a red-black tree) which allows us to write data in any order, but which stores them in a sorted manner. Thus, when a write operation is made, the key-value pair is stored in an in-memory data-structure which ensures this sorting property.
Once this data-structure passes a given threshold, we write the key-value pairs to a segment file in a sorted manner, thereby creating SSTables.
For a read operation, we first search for the key in the memtable, then to the most recent SSTable segment file, and so on.
One downside: in case of a crash, the most recent keys in the memtable will be lost. To fix this, we take a backup of the memtable in a separate file - which can simply be an append-only log file. In effect, each write operation makes two operations, it writes to the memtable and to the back-up append-only log file. The latter is used only for re-constructing the mem-table in case of a server crash.
Storage engines which are based on this principle of merging and compacting sorted files are often called LSM storage engines, or, Log-structured Merge-Tree storage engines.
In LSM storage engines, look-up operations for non-existent keys are slow: to confirm a key is absent, we need to traverse from the memtable all the way to the oldest segment file. We can use bloom filters, to optimise this, as it can help us in knowing if a key is absent or not.
Like the append-only storage engine, we keep in-memory hash maps for each segment file, which stores keys sparsely (ie not every key, but every n keys).
References
Martin Kleppmann, Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems (O’Reilly Media. 1st Edn.), Chapter 3
]]>Transitioning from Heroku Free Tier to GCP + YugabyteDB2023-01-08T00:00:00+00:002023-01-08T00:00:00+00:00https://otee.dev/2023/01/08/migrating-heroku-to-gcp-and-yugabyteThis post captures the list of steps I followed to migrate a HTTP service from a Heroku dyno to a Google Cloud Platform instance powered by YugabyteDB cluster as the database.
To deploy Twirl, my URL shortening service, I was using Heroku’s free tier plans (more on how I built this app here, here, and here). But now that it is no longer free and (more importantly) there is no plausible way to pay for their premium plans from India (whether via Indian cards or any other payment methods), I was forced to shift my app to a more traditional cloud provider.
Since I have an existing setup on Google Cloud Platform (GCP), I decided to migrate my setup to my GCP instance. Additionally, I chose to use YugabyteDB instead of PostgreSQL as my database.
Here are the steps I followed to migrate out of Heroku, for my URL shortening service: twirl.otee.dev
nginx: this is running on GCP and is used for custom domain mapping to a free Heroku tier dyno, without having the default herokuapp domain in the URL.
PostgreSQL: the database that stores the users, short links and other associated data.
New Architecture
In the first phase, we want to independently run the system on GCP. In this lift-and-shift phase, each of the components of the system will be running outside of Heroku:
Existing component
Change
nginx
Redirect to local port instead of Heroku dyno
Heroku HTTP server
GCP Compute Instance
PostgreSQL
Managed YugabyteDB
In the second phase, we want to move the data that is already present in the PostgreSQL database on Heroku, to the new YugabyteDB Managed instance. The reason for choosing YugabyteDB was not lightly made. Most importantly, it supports PostgreSQL operations so it fits our use-case. There are many other advantages as well which I will illustrate in another post.
In the last phase, we will decommission Heroku dyno.
Phase 1: Setting up a parallel service on GCP
Step 1.1: Create YugabyteDB cluster
Setting up a YugabyteDB Managed instance is quite straight-forward. After completing the sign-up process on https://cloud.yugabyte.com/signup, the cluster looks like this:
Once set up, we need to connect to this database to set up our schema for the URL shortening app, as per the ReadMe of the project. To do this, we have to use the connection string as per the documentation on YugabyteDB:
Note that to connect to the YugabyteDB cluster, we need to download a digital certificate - this can be done from the dashboard itself, by clicking on the connect button on the top left corner, and choosing the Connect to your Application option. We need to add the path to this certificate to the connection string. This certificate is used for verify the identity of clusters, as YugabyteDB uses TLS protocol for communicating with its clusters.
Also, to connect to the YugabyteDB cluster, we need to white-list the client IP address. This also can be done from the dashboard itself (which provides a convenient option to add your current IP address to the IP Allow list). At this point, we must add the IP address of our local machine. Ultimately, the IP Allow list should contain the public IP address of our GCP instance only.
Step 1.2: Set up the Database
To connect to the YugabyteDB, we can use the command line utility psql which is compatible with PostgreSQL and like databases.
We will use the connection string mentioned above to connect:
At this point, psql should connect to the default yugabyte database. We can now create a new database for our app, twirl
yugabyte=> DROP DATABASE IF EXISTS twirl;
NOTICE: database "twirl" does not exist, skipping
DROP DATABASE
yugabyte=> CREATE DATABASE twirl;
CREATE DATABASE
yugabyte=>yugabyte=>\connect twirl;
psql (12.12 (Ubuntu 12.12-0ubuntu0.20.04.1), server 11.2-YB-2.15.3.2-b0)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
You are now connected to database "twirl" as user "admin".twirl=>
After this, the requisite schema for this app can be set up, following the queries mentioned ont the ReadMe.
Step 1.3: Local sanity testing
Now that the schema has been set up correctly, we need to make sure our app can use this. For this, we can run our app locally and use this connection string as the db endpoint. This is for local sanity testing that the app functions normally before we try anything infrastructure specific.
To run the app locally, the following command needs to be run. This command is specific to the app at hand, which expects a few environment variables, like the db connection string:
# Note that we need to use the new database `twirl`# and not the default `yugabyte` in the connection stringPG_SSL_CONNECTION=true CUSTOM_DOMAIN_NAME=localhost:4001 RECAPTCHA_SECRET=EXEMPTED PORT=4001 COOKIE_SECRET=<eiffeltower...> PG_CONNECTION_STRING="postgresql://<DB USER>:<DB PWD>@us-west-2....aws.ybdb.io:5433/twirl?ssl=true&sslmode=verify-full&sslrootcert=/home/otee/Downloads/root.crt" node app.js
Twirl listenning on 4001...
At this point, the app works locally! 🎇
Step 1.4: Deploy HTTP server on GCP
Now, we need to setup the app on our GCP instance. For this, the following steps need to be followed:
After ssh-ing to GCP instance, clone the repo:
Copy the root certificate file (downloaded from YugabyteDB dashboard) to the GCP instance (ssh alias already set up as calculus locally).
Once we see the system startup, we have no way of connecting to this port from the browser because this port is not exposed. We will do this in the next step.
Note: When we run a service on an instance like this, one of the housekeeping items to ensure is that if the instance reboots (which it will eventually), we have to make sure that our HTTP server is automatically started. This was already being taken care of by Heroku, but here we need to do this ourselves (this will be done later). But even when we do this, it is not guaranteed that the instance would not be irrevocably lost; in which case we will need to set it up manually again.
Step 1.5: Setting up nginx config
For our parallel (trial) service, we need to create a simple nginx config that maps requests to a test sub-domain twirl-test.otee.dev to this new HTTP service.
Next, we need to add a new DNS entry pointing the trial sub-domain twirl-test.otee.dev to the GCP instance.
Unfortunately, .dev domains only support https connections. So, we have to install a certificate for this sub-domain. We can do this with certbot, (following the official documentation)
sudo certbot --nginx
After cerbot finishes adding the certificate, the earlier nginx configuration will be updated with the new certificate enforcing HTTPS:
server {
server_name twirl-test.otee.dev;
location / {
proxy_pass http://127.0.0.1:4122;}
listen [::]:443 ssl;# managed by Certbot
listen 443 ssl;# managed by Certbot
ssl_certificate /etc/letsencrypt/live/twirl-test.otee.dev/fullchain.pem;# managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/twirl-test.otee.dev/privkey.pem;# managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf;# managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;# managed by Certbot}
server {if($host= twirl-test.otee.dev){return 301 https://$host$request_uri;}# managed by Certbot
listen 80;
listen [::]:80;
server_name twirl-test.otee.dev;return 404;# managed by Certbot}
Step 1.6: Add GCP instance IP address to Yugabyte allow-list
The GCP IP address needs to be white-listed on the YugabyteDB dashboard, because random connections to the database are not allowed. Note that we need to add the permanent public IP address (and not the internal ephemeral IP address) of the GCP instance.
Now, the system should work with the YugabyteDB as the database!
Note: we have thus far set-up a parallel service using a temporary sub-domain; we will still need to point twirl.otee.dev (the original sub-domain) to this GCP instance. This will be done at a later step.
Phase 2: Bring the GCP service upto speed
Step 2.1: Migrate existing data from Heroku PosgreSQL to YugabyteDB Managed
Now that the HTTP server is working, we need to migrate the existing data from the postgreSQL database on Heroku to the newly set-up YugabyteDB database.
For this, we can use pg_dump which comes installed with psql. pg_dump is a back-up utility for a PostgreSQL database. It spits out SQL commands which can be used to reconstruct the database with all its the data present at the time of the backup.
For this, we need to use the same PostgreSQL connection string that the Heroku app uses.
Note: We could not use pg_dump from the terminal of our local machine directly, because the Heroku instance was using postgres version 13 and the corresponding postgres-client version was not readily available. So, we used this docker file and used pg_dump inside that docker container:
# Going into the containersudo docker exec-it twirl_postgres_1 bash
root@3012d2b15617:/# pg_dump -d"<psql connection string>"-f sql_queries.txt
##Leave the Docker container and copy the filesudo docker cp twirl_postgres_1:/sql_queries.txt /tmp/sql_queries.txt
Now that the backup is ready, we need to recreate the twirl database on YugabyteDB. For this, we need to connect to the default database on YugabyteDB called yugabyte and create the twirl database afresh.
yugabyte=> DROP DATABASE IF EXISTS twirl;
DROP DATABASE
yugabyte=> CREATE DATABASE twirl;
CREATE DATABASE
yugabyte=>
Next, we need to restore the data from the file created by pg_dump. Since this file contains just SQL commands, we can just run psql on the new database taking the SQL commands from the file instead of the prompt:
# Connection string to yugabyte
psql "postgresql://....ybdb.io:5433/twirl?ssl=true&sslmode=verify-full&sslrootcert=/home/otee/Downloads/root.crt"-f /tmp/sql_queries.txt
Note that some of the statements may fail - this can be due to commands related to ownership access and other settings but the data should not be missing. Once the back-up is complete, we can do a simple sanity test by checking the number of rows in each table in both the databases.
At this point, our parallel service is upto speed with the existing service. In other words, twirl-test.otee.dev and twirl.otee.dev should have the same data, i.e., user credentials, short-links, access counts etc. So, we have successfully migrated the data from Heroku to YugabyteDB and therefore most of the work is now done. 🥂
Step 2.2: Install Twirl as a service on GCP
The node process running in the background can crash due to unforeseen exceptions, out-of-memory errors or some other reason like restarting of the VM itself. When this happens (as it inevitably will), our node process will terminate and our app will be unavailable. To solve this, we can use systemd. It is a “system and service manager” for Linux Operating Systems.
On the GCP instance, we need to create a new systemd service:
Note that we added the PORT environment variable which was not required on Heroku because it was setup by the setup of Heroku.
Step 2.3: Divert traffic from Heroku service to the GCP service
So far we have been making changes on a new sub-domain twirl-test.otee.dev. Now that we are confident that the GCP service is working as expected, we can stop the traffic to the Heroku server via twirl.otee.dev and instead serve it from the GCP instance through twirl.otee.dev. Once we achieve this, twirl-test.otee.dev will no longer be required.
We have to change the nginx configuration for twirl.otee.dev to point to the local port instead of Heroku. Current configuration looks like:
server {
listen 443 ssl;
server_name twirl.otee.dev;return 301 $scheme://oteetwirl.herokuapp.com$request_uri;
ssl_certificate /etc/letsencrypt/live/twirl.otee.dev/fullchain.pem;# managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/twirl.otee.dev/privkey.pem;# managed by Certbot}
server {if($host= twirl.otee.dev){return 301 https://$host$request_uri;}# managed by Certbot
listen 80;
server_name twirl.otee.dev;return 404;# managed by Certbot}
Note the 301 redirect to oteetwirl.herokuapp.com. We need to replace this line mainly, by moving away from the return directive to the location directive using a proxy_pass:
server {
listen 443 ssl;
server_name twirl.otee.dev;
location / {
proxy_pass http://127.0.0.1:4122;}
ssl_certificate /etc/letsencrypt/live/twirl.otee.dev/fullchain.pem;# managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/twirl.otee.dev/privkey.pem;# managed by Certbot}
server {if($host= twirl.otee.dev){return 301 https://$host$request_uri;}# managed by Certbot
listen 80;
server_name twirl.otee.dev;return 404;# managed by Certbot}
Phase 3: Shut down Heroku server
We can now stop the Heroku dyno.
Now twirl.otee.dev should still work and be served from GCP instance powered by the YugabyteDB Managed cluster.
Future Improvements
Permanent Redirections
Note that because we used permanent redirection via HTTP status 301 browsers which previously got redirected to Heroku domain might see the following error:
A simple solution to this is to disable cache on the browser and reload:
Alternatively, we can write a simple Heroku app that just redirects back to GCP in order to undo the permanent caching without having our users’ browsers. We are going to leave this aside for now.
Backups
Our free Yugabyte instance does not have data-backups. This means that we are at a risk of losing all our data if the Yugabyte cluster goes down. A simple solution to this would be to run the pg_dump command above periodically (crontab) and uploading it to a cloud storage (like s3).
For now, we will run this command manually and store it.
]]>Understanding Tail Recursion2022-04-03T00:00:00+00:002022-04-03T00:00:00+00:00https://otee.dev/2022/04/03/understanding-tail-recursionIn this post, I try to explain how tail call optimization can be used to make recursive functions (of certain kind) more efficient.
Understanding Recursion
A recursive process can be divided in two parts:
A base case(s), which defines a simple case (such as the first item in a sequence)
A recursive step, where new cases are defined in terms of previous cases.
When a recursive process is called with the base case, it simply returns the result. If the process is called with a more complex case, it would break down the problem into two parts: a part it knows how to evaluate and a part which it does not. The latter part will resemble the original case, except that it will be a slightly smaller or simpler version of it. It would then call a fresh instance of itself on this latter part to work on the simpler case.
What is essential for a proper use of recursion is that the objects can be expressed in
terms of simpler objects, where “simpler” means closer to the basis of the recursion. [Source]
To put it in another way, a recursive algorithm can be defined as solving a problem by solving a smaller version of the same problem (a sub-problem) and then using that solution to solve the original problem. To solve a sub-problem, we need to solve a still smaller version of that sub-problem and so on. To prevent an infinite series of sub-problems, we need the series of sub-problems to eventually terminate at a base case.
Examples of recursive functions
We can use recursion to find the factorial of a positive natural number:
In the above example, line (1) represents the base case and line (2) represents the recursive call.
Keeping Track of Recursive Calls
Other than the base case, a recursive function cannot compute the final result (of a given input) without first knowing the result of the same function with a smaller input. For example, in the case of the factorial function, for any value of n (where n != 1), we need to first know the value of factorial(n - 1), and to calculate the value of factorial(n - 1) we need to compute factorial(n - 2) and so on, till n == 1 is true.
This means that for every step of this recursive ladder, we must wait for return value of the recursive call. Till this is done, we must pause our execution. Once we have this result, we can complete our evaluation by carrying out our operation(s) on it and return the result so computed.
So, to find the factorial of 4, we need to suspend our execution till we know the result of the factorial of 3 (and so on).
Also, note that at each step of the recursive process, we do not carry out the same operation with the result we get from the recursive call. Operations at each recursive step involves variables (not constants) relevant to that recursive step. In the case of factorial, for each step, we need to multiply the current value of n with the value we get from factorial(n - 1).
The answer to the new factorial subproblem is not the answer to the original problem. The value obtained for (n - 1)! must be multiplied by n to get the final answer. [Source]
From the above discussion we know that for a recursive operation-
we need a series of invocations of the same function with different values starting from a given value to the base-case value;
to compute the value at a given recursive step, we need to wait till all the function calls from the base case leading up to it are completed;
while waiting, we need to store the state of the values at that recursive step.
To illustrate this, let’s take the example of factorial(n), where n = 4:
Since there is no ex ante way of knowing the number of recursive calls required to reach the base case for a given value, we need to be able to store the states of a dynamic number of recursive steps. Also, as we can see above, the order in which the states of each recursive step is accessed is one of Last-In-First-Out, i.e., the values of each recursive step are accessed in the reverse order of their insertion. Because of this property of recursive function calls, we can use stacks to store the state of each recursive step.
Understanding Call Stack
Whenever a function is called, a memory block (known as ‘stack frame’ or ‘activation record’) representing its state (mainly, its parameters, local variables, its return address) is pushed to a call stack that is maintained during the run-time of the program. When the execution of the function is complete, it’s corresponding memory block is popped from the call stack. At any point in time, the call stack only contains memory blocks representing functions that have been called, but not yet been executed. (As an aside, most programming languages have the functionality to display the call stack at any particular point of time during run-time. This is called stack trace and it is a useful tool for debugging run-time errors)
In the case of recursive functions, a function that is already part of the call stack, makes another call of the same function (with a smaller sub-problem). The original function cannot be executed till this newly called function returns its value. So, a new memory block representing the newly called function gets pushed on to the call stack. The process repeats itself till we reach the base case. Once we have reached the base case, we will have reached the top of the recursive call-stack. As soon as the base-case function call returns its value, it’s associated memory block gets popped from the stack. The function that made the base-case function call, gets popped next which returns its value to the next function that had called it. This goes on till we reach the bottom of the stack. The bottom-most block on the stack represents the original function call. By the time we reach there, we will have all the values needed to complete this evaluation.
Here’s a visualization of a recursive call stack for the factorial of 5:
Computing Space Complexity for Recursive Calls
The space complexity for a recursive function depends on the maximum size of the call stack at any point of time during its execution. For a linear recursive function, i.e., a function that makes a single call to itself during each recursive step, the maximum size of the call stack will be the sum of the number of recursive calls made by each recursive step. Thus, for the factorial function, the maximum size of the call stack will be n (as shown above).
For non-linear recursive functions, if we visualise the pattern of recursive calls as a recursive tree, where each node represents a recursive call, the maximum size of the call stack will be the height of the tree. Thus, for the following function generating nth Fibonacci number, the maximum size of the call stack will be n
In the above example, even though we are making two recursive calls at each step, we will still only require a call stack of size n. This is because at each node of the recursive tree, we need to wait for it’s children nodes to return their values, and the same goes for their children nodes as well. We essentially do a depth-first traversal from each node in the recursive tree, thereby requiring only a max stack size of the height (or depth) of the recursive tree.
Stack Overflow
The size of call stacks are finite and limited (their exact size depend on multiple factors, including the programming language and the run time environment). This means that there are only so many recursive calls we can make for a given call stack. When we exceed this size, we get a ‘stack overflow’ error. Let’s take the following function that recursively finds the sum of the first n natural numbers:
function sumOfNaturalNumbers(n){if(n == 1)return 1
return n + sumOfNaturalNumbers(n - 1)}
If we pass a relatively large number to the above function, say 100000, we get the following error
RangeError: Maximum call stack size exceeded
at sumOfNaturalNumbers (/home/otee/projects/recursive_factorial.js:13:5)
at sumOfNaturalNumbers (/home/otee/projects/recursive_factorial.js:15:16)
at sumOfNaturalNumbers (/home/otee/projects/recursive_factorial.js:15:16)
at sumOfNaturalNumbers (/home/otee/projects/recursive_factorial.js:15:16)
at sumOfNaturalNumbers (/home/otee/projects/recursive_factorial.js:15:16)
at sumOfNaturalNumbers (/home/otee/projects/recursive_factorial.js:15:16)
at sumOfNaturalNumbers (/home/otee/projects/recursive_factorial.js:15:16)
at sumOfNaturalNumbers (/home/otee/projects/recursive_factorial.js:15:16)
at sumOfNaturalNumbers (/home/otee/projects/recursive_factorial.js:15:16)
at sumOfNaturalNumbers (/home/otee/projects/recursive_factorial.js:15:16)
However, if we write the same function using a for loop and pass the same parameter (100000), we do not get any errors.
function sumOfNaturalNumbersIteration(n){sum= 0;for(let i = 0; i <= n; i++){sum += i;}return sum;}
The reason why we the above prograat ism does not break is because it takes constant space to run a for loop, irrespective of the number of iterations being executed. To run the above loop, all we need to keep track of are the current values of n, i and sum.
Tail recursion
Tail recursion is a special kind of recursion where the recursive call is the last operation carried out in the recursive case and the result of the recursive call is not manipulated by the caller.
A call from procedure f ( ) to procedure g( ) is a tail call if the only thing f ( ) does, after g( ) returns to it, is itself return. The call is tail-recursive if f ( ) and g( ) are the same procedure (Steven S. Muchnick, Advanced Compiler Design and Implementation, p 461)
The most important aspect of a tail call, is that a new frame does not need to be added to the call stack for every function call. For tail calls, there is no need to store the state of the function making the tail call (as all the operations inside that function are executed by the time the tail call is made and all that is left to do is to pass on the value so returned, to its original caller). Instead, the tail-called function returns the value directly to the original caller. As a result, tail recursion takes constant space to be executed. In fact, a tail-call is essentially a goto statement:
In this way, if the last thing a procedure does is call another (external) procedure, that call can be compiled as a GOTO.Such a call is called tail-recursive, because the call appears to be recursive, but is not, since it appears at the tail end of the caller. [Source]
The above implementation of the sum of natural numbers,sumOfNaturalNumbers, is not written in a tail-recursive style, as we do one operation after receiving the return value from the recursive call (namely, adding the current value of n to the returned value). Here’s a tail-recursive implementation of that function:
However, this implementation, too, fails when we pass 100000 to it:
RangeError: Maximum call stack size exceeded
at sumOfNaturalNumbersTailRecursive (/home/otee/projects/recursive_factorial.js:34:42)
at sumOfNaturalNumbersTailRecursive (/home/otee/projects/recursive_factorial.js:37:12)
at sumOfNaturalNumbersTailRecursive (/home/otee/projects/recursive_factorial.js:37:12)
at sumOfNaturalNumbersTailRecursive (/home/otee/projects/recursive_factorial.js:37:12)
at sumOfNaturalNumbersTailRecursive (/home/otee/projects/recursive_factorial.js:37:12)
at sumOfNaturalNumbersTailRecursive (/home/otee/projects/recursive_factorial.js:37:12)
at sumOfNaturalNumbersTailRecursive (/home/otee/projects/recursive_factorial.js:37:12)
at sumOfNaturalNumbersTailRecursive (/home/otee/projects/recursive_factorial.js:37:12)
at sumOfNaturalNumbersTailRecursive (/home/otee/projects/recursive_factorial.js:37:12)
at sumOfNaturalNumbersTailRecursive (/home/otee/projects/recursive_factorial.js:37:12)
The reason why this implementation also results in stack overflow is because JavaScript (in the NodeJs runtime environment) does not support tail call optimization. In fact, many other programming languages, including Java, do not support tail call optimization:
Not all programming languages require tail-call elimination. However, in functional programming languages, tail-call elimination is often guaranteed by the language standard, allowing tail recursion to use a similar amount of memory as an equivalent loop. [Wikipedia]
Clojure, being a functional programming language, supports tail call optimization.
Tail Recursion In Clojure
Unlike compilers of some other functional languages, Clojure’s compiler will not automatically detect a recursive tail-call and optimise it accordingly. We need to make the tail call using a special form, recur, to explicitly utilise tail recursion optimisation.
Many such languages guarantee that function calls made in tail position do not consume stack space, and thus recursive loops utilize constant space. Since Clojure uses the Java calling conventions, it cannot, and does not, make the same tail call optimization guarantees. Instead, it provides the recur special operator, which does constant-space recursive looping by rebinding and jumping to the nearest enclosing loop or function frame. While not as general as tail-call-optimization, it allows most of the same elegant constructs, and offers the advantage of checking that calls to recur can only happen in a tail position.[Source]
Here’s a non-tail recursive implementation of the sum of first n natural numbers:
The above tail recursion examples involve a common pattern:
For each recursive call, we pass an accumulator parameter, in addition to the main argument.
The initial value of this accumulator represents the base case
At each recursive step, we modify the main argument as per the definition of the problem (in the case of sum-of-natural-numbers-tail-recursive, decrementing the value of n) and update the current value of the accumulator. We pass both these parameters along with the recursive call.
The operations we were doing after the return of the non-tail recursive function, are now done before calling the tail recursive function and this is passed as the accumulator
The value of the accumulator in the base-case, is the final result, which is returned by the base case.
Takeaways
Representing a problem in a recursive manner, involves breaking down the problem into a series of simpler sub-problems and providing a result for the simplest version of the problem
Although recursion is intuitive and easy to reason about while solving computational problems, they take significantly more space than iterative processes
A special kind of recursion, namely tail recursion, solves the problem of stack overflow, when we can express our recursive calls as tail calls.
We can transform (ordinary) recursive functions to tail recursive functions, with the help of an additional accumulator parameter.
]]>Preventing Phantom Meetings Using Transactions and Serializable Isolation2022-02-10T00:00:00+00:002022-02-10T00:00:00+00:00https://otee.dev/2022/02/10/transactions-and-phantom-readsIn this post, I discuss how time-slot collisions in a meeting scheduling application can be resolved. First, I discuss the business logic of determining a ‘time-slot conflict’. Second, I explain why we need transactions to prevent scheduling of concurrent conflicting meetings. Finally, I do a deep-dive on the different levels of isolation provided by database systems, to better understand why simply using transactions (with their default set-up) does not guarantee against conflicting meetings.
Project Scope and Design Goals
This project supports the following features:
Add new users and rooms: The system allows for the addition of new users and rooms (they are treated as ‘entities’ in the data model). Once added, users and rooms can be included in future meetings.
Scheduling of meetings: The system allows for scheduling of new meetings involving one or more users at a given room.
Detect conflicts: While setting up a meeting, the system will check if there are any time-slot conflicts (explained below) involving any of the proposed participants or the proposed room.
The system interacts with the user on the Command Line.
The system guarantees that a meeting should be permitted to be scheduled only if every participating user and the respective room have no time-slot conflict. Even if one entity has a conflict, the system will reject the request for setting up the meeting and prompt the user to try again.
For the purposes of time-slot conflict-resolution, the system treats users and rooms alike.
Resolving Time-Slot Conflicts
What constitutes a time-slot conflict? Simply put, it refers to a situation where a single user or room is assigned to more than one meeting at any given point of time.
With respect to a meeting for a given time-slot A (with a start-point ‘s’ and an end-point ‘e’), there can be five types of time-slot conflicts, as shown below.
Case 1: A meeting which started before ‘s’, but is scheduled to complete after the ‘s’ (but before ‘e’).
Case 2: A meeting which started before ‘s’, and is scheduled to complete after ‘e’.
Case 3: A meeting which has the same start and end points as time-slot A.
Case 4: A meeting which started and ended between points ‘s’ and ‘e’.
Case 5: A meeting which started after ‘s’ but before ‘e’.
In each of these cases, there is a time-slot conflict, i.e., at least one point where there are two simultaneous meetings.
How to Resolve Time-Slot Conflicts?
Obviously, if the parties involved in two meetings with conflicting time-slots are distinct and separate, there is no issue. The system will allow both the meetings to continue.(Note here that when we use ‘participants’, we include both users and rooms).
Thus, only if there is at least one common participant between the two conflicting time-slots, do we need to be careful. Thus, at the time of scheduling each meeting, we need to check if there is any potential conflict with respect to any of the participants of the meeting. This can be done using the following SQL query:
In the above example, (entity1, entity2... entityN) represents the list of all the participating entities of a proposed meeting and start_point and end_point represent the two end-points of the time-slot of the proposed meeting.
At the time of creating a new meeting, we run this query on our database. If this returns a non-empty response, it will signify a conflict and the system will prevent the creation of the meeting
Need for Transactions
In an ideal world, we follow a two-step process while creating a new meeting:
First, check for conflicts
Next, insert the new meeting.
This approach has one downside: if there is more than one system trying to write to the database simultaneously, the database may change its state between step one and step two above. For example, let’s say there are two systems attempting to simultaneously schedule the same meeting with the same time-slot and entities. It is possible, that read-write sequence interleaves in the following manner:
System 1 reads the database ... (realizes that there is no conflict)
System 2 reads the database ... (realizes that there is no conflict)
System 1 writes the database ...(creates the meeting)
System 2 writes the database ... (creates the same conflicting meeting)
Thus, we need to take an all-or-nothing approach while reading and writing. This can be achieved by using transactions.
What is a Transaction?
Simply put, a transaction represents a single or atomic unit of work performed by a database management system. A transaction is typically used to group multiple reads and writes into one logical unit. Because of their atomic nature, transactions cannot be broken down into its constituent actions: if, in the middle of a transaction, an error or failure takes place which prevents the transaction from being successfully completed, the database will rollback all the intermediate operations of that transaction.
This is very useful for our use-case, as we can use transactions to ensure an all-or-nothing approach while scheduling meetings: if our reading and writing operations form part of a single transaction, we can potentially prevent partial failures, like the one discussed above.
ACID Properties
Every database transaction has four key properties: atomicity, consistency, isolation and durability (commonly referred to as ‘ACID’). These ACID properties guarantee data validity even in the events of failures, errors and other mishaps.
Atomicity: Every operation in the transaction should either all succeed (also called ‘committed’) or all fail. Partial failure or partial success is disallowed. Without the atomicity guarantee, if an error or a failure takes place during a transaction, it can get very difficult to reason about which operations were successful and which need to be tried again.
Consistency: If the database is consistent before execution of a transaction, it should remain consistent after the transaction has been committed. In other words, a transaction should take the database from one valid state to another. If there are any rules or invariants enforced on the data, they should continue to be respected after a transaction is completed. (In fact, consistency is a property of the application layer instead of the database system itself, as the latter cannot prevent the violation of invariants if the application feeds improper or erroneous data. For this reason, it is said that “the letter C doesn’t really belong in ACID” [1]).
Isolation: Often, a database needs to execute multiple transactions concurrently. This property provides a guarantee that concurrent transactions will be executed as if they were sequentially or serially executed. In other words, the goal of isolation is to ensure that simultaneous transactions making writes on the same set of objects (rows) of a database should not step onto each other’s toes.
Durability: Once a transaction is committed, it should persist on the database, even in the wake of a system failure. In the case of single-node databases, this is usually achieved by storing results of transactions on non-volatile memory(disk). In the case of replicated databases, this is achieved by copying the data written during a transaction to a certain number of nodes of that database.
Writing Transactions in PostgreSQL
To fold multiple queries into one transaction, we should place them between BEGIN and COMMIT commands. During the middle of a transaction, if our application layer needs to withdraw a transaction,we should use ROLLBACK instead of COMMIT.
Here’s how the SQL expressions for scheduling a meeting can be written as a part of one transaction:
Given the guarantees provided by transactions, we should expect that our application does not schedule conflicting meetings. To test this hypothesis, we can set up a test that makes concurrent and identical queries on the database. To implement this test, I’ve used the worker threads module, to create n number of worker threads that make the same query on the database. See the test here: https://github.com/oitee/meetings/blob/33aab6b/test/concurrent_requests.js
When I ran this test for the first time, it passed. But when I ran the same test sequentially for ten times (using for i in `seq 1 10`; do npm test; done), it failed twice out of the ten times. For the next ten tests, it failed three times out of ten
When a test fails, the same meeting (as shown in the above schema) with the same time-slot (between 2021-12-15 05:30:00+00 and 2021-12-15 07:30:00+00) gets inserted multiple times. So, why did my tests fail some of the times? What happened to the ACID properties of transactions?
Different Levels of Isolation
To better understand why my tests were failing sporadically, it is important to understand that databases enforce different degrees of isolation among concurrent transactions. Note that, ‘isolation’ was described above as a guarantee that the database system will execute concurrent transactions in such a manner that it will appear as if they were executed serially, i.e., one after the other. In fact, serial isolation (i.e., converting multiple concurrent transactions into a set of sequential transactions) is rarely used in practice. This is because serializable isolation has substantial performance costs which can slow down the response time of a database system. For this reason, most databases provide weaker levels of isolation:
“Even on a single-node database, the penalties associated with providing serializability can be severe, including decreased concurrency, reduced performance, and the possibility of deadlock. Accordingly, since the earliest database systems such as System R in 1976, databases have provided a range of user configurable “weak isolation” properties. These properties do not guarantee serializability but offer benefits such as increased concurrency and ease of implementation.” [1].
When two or more concurrent transactions try to write on the same object of a database or one transaction reads an object that is being concurrently modified by another concurrent transaction, we can have concurrency issues, which are also called ‘race conditions’. Each level of isolation provides guarantees against some or all race conditions.
Read Committed
When a change made by a transaction has been committed, that change becomes permanent on the database and the transaction loses its right to ‘undo’ that change. However, uncommitted changes are always revocable. Working with uncommitted changes should ideally be avoided. Reading some other transaction’s uncommitted changes is called ‘dirty reads’. Writing on another transaction’s uncommitted changes is called ‘dirty writes’
‘Read committed’—the first (or weakest) level of isolation—provides guarantees against dirty reads and dirty writes. This is the default isolation level in PostgreSQL.
Dirty writes are prevented by locking relevant rows where writes take place. When a transaction needs to write a specific row, the database will lock that row. Only once the transaction is completed (aborted or committed) will this lock be opened. At a time, only one transaction can lock a row. So if there is a second transaction that needs to write on the same row, it needs to wait for the first transaction to be completed.
As for prevention of dirty reads, when a write lock is applied on a row, the database maintains two values for that row: the original value and the uncommitted value. Thus, read-only transactions can access the original value till the time the write lock is lifted.
Read committed will not be adequate for our use-case, as our application does not rely on dirty reads or writes.
Snapshot Isolation and Repeatable Read
In a snapshot isolation, each transaction works on a consistent snapshot of the database, i.e., the database as it stood at the beginning of the transaction.
Snapshot isolation is implemented by a technique called multi-version concurrency control (MVCC). The core principle of snapshot isolation is that for each transaction, they will read a consistent snapshot of the database, as it stood, when that transaction began. As a corollary, if the database progressed further (i.e., some uncommitted changes were committed during the course of a transaction), the transaction will not see those future changes. This ensures that writes do not block reads, and reads do not block writes. This can be especially useful for taking backups of a large database: the transaction making a copy of the database at a particular point in time (a read-only transaction) will not be impeded by other transactions that are writing on some of the rows of that database. However, note that snapshot isolation implements write locks as well, i.e., when a transaction is writing on a row, no other transaction can write it.
When we implement MVCC, the database may potentially need to maintain several versions of the database, each representing the ‘snapshot’ of the database when each ongoing transaction was initiated.
Snapshot isolation level is typically referred to as ‘repeatable read’ in SQL. However, these two terms are not exactly identical.
“…it defines repeatable read, which looks superficially similar to snapshot isolation. PostgreSQL and MySQL call their snapshot isolation level repeatable read because it meets the requirements of the standard, and so they can claim standards compliance.
Unfortunately, the SQL standard’s definition of isolation levels is flawed—it is ambiguous, imprecise, and not as implementation-independent as a standard should be. Even though several databases implement repeatable read, there are big differences in the guarantees they actually provide, despite being ostensibly standardized. There has been a formal definition of repeatable read in the research literature, but most implementations don’t satisfy that formal definition. And to top it off, IBM DB2 uses “repeatable read” to refer to serializability. As a result, nobody really knows what repeatable read means”[1]
In addition to preventing dirty reads and writes, snapshot isolation also prevents non-repeatable reads: re-reading the same set of rows will not yield a different result.
Also, PostgreSQL’s implementation of repeatable read automatically detects lost update. A lost update happens when two concurrent transactions read the same row(s), modify the data and write that modified data on that row(s) (read-modify-write cycle). When two such transactions are executed concurrently, one of the writes will be lost. Take the example of a database that maintains counters. Each transaction is required to read the data of the counter, and increment it by one and write the new data onto the database. Now if two transactions are fired at the same time, they will both read the same value, and they will both update the counter by one. So, while we made two queries for incrementing the value of the same counter, the value actually got increased once. (The other increment was ‘lost’). (I had encountered this particular problem while generating counters for my URL shortening application. Read here).
Source: Designing Data Intensive Applications [1]
There are two explicit ways to prevent lost updates. First, we can use atomic operations, i.e., we read, modify and write the data in one single query. Second, we use explicit locking. When we use explicit locking, we tell the database to prevent any other transaction from reading or writing on the rows on which our transaction is working on, till the present transaction is completed. This can be done by using FOR UPDATE at the end of the SELECT query.
Other than these explicit ways, PostgreSQL also automatically detects if there is a lost update during a repeatable read transaction.
Snapshot isolation will not be adequate for our use-case, as working with consistent snapshots cannot prevent parallel (and concurrent) transactions from making the same writes. Lost update seems close enough to our use case. However, as our writing operation is akin to creating a new row (as opposed to updating an existing row), Postgres cannot automatically prevent concurrent insertions of the same meeting.
Serializable Isolation
Serializable isolation offers the highest degree of protection.
“Serializable isolation is usually regarded as the strongest isolation level. It guarantees that even though transactions may execute in parallel, the end result is the same as if they had executed one at a time, serially, without any concurrency. Thus, the database guarantees that if the transactions behave correctly when run individually, they con‐ tinue to be correct when run concurrently—in other words, the database prevents all possible race conditions.” [1]
There are three alternative implementations of serializability:
Serial Execution: Literally executing transactions serially, by using a single thread always.
Two phase locking: Concurrent reads are permitted on rows where no write operation is underway. If any transaction is making a write operation, it will lock the relevant row and every other transaction that wants to either read or write on that row will need to wait till the first transaction is completed. This approach is also called a pessimistic concurrency control mechanism, because the database system assumes the worst, i.e., every concurrent write operation on a transaction will fail, and guards against that eventuality. (Of course, this is more optimistic than single-threaded serial execution)
Serializable Snapshot Isolation: Unlike two-phase locking, in this approach, the database system allows for writes and reads to take place concurrently. At the time of committing a transaction, the database checks if there is any violation of isolation properties, in which case it will abort that transaction. This approach is referred to as the ‘optimistic concurrency’ approach.
In the case of lost updates (involving a read-modify-write cycle), an easy fix is to use explicit locks on the relevant rows. However, what happens when the first read operation looks for the absence of rows (meeting a certain criteria)? If this criterion is met, i.e., if no rows exist meeting a certain condition, we write the database by inserting a new row. In this case, there is no row we can explicitly lock to prevent a write skew or a lost update. These kinds of concurrency issues are called ‘phantom reads’ and this is exactly the reason why our tests often fail. Because of its nature and the guarantees it provides, serializable isolation can consistently prevent phantom reads.
Using Serializable Isolation to Prevent Time Collisions
Let’s summarize the different levels of isolation and the respective guarantees they provide:
Isolation Level
Dirty Reads and Writes
Non-Repeatable Reads
Lost Updates
Phantom Reads
Read Committed
Not possible
Possible
Possible
Possible
Snapshot isolation / Repeatable Read
Not possible
Not possible
Not Possible (in Postgres)
Possible
Serializable Isolation
Not possible
Not possible
Not possible
Not Possible
We need to use serializable isolation for our meeting scheduling application, as we know that phantom reads are possible when concurrent transactions try to schedule meetings on the database. So, here’s the modified SQL query for inserting a new meeting on the database:
BEGINTRANSACTIONISOLATIONLEVELSERIALIZABLE;SELECTentityFROMbookingsWHEREentityIN(entity1,entity2...entityN)AND((from_ts<=to_timestamp(start_point)ANDto_ts>=to_timestamp(start_point))OR(from_ts>=to_timestamp(start_point)ANDfrom_ts<=to_timestamp(end_point)));-- if rows.length > 0 ROLLBACK;-- elseINSERTINTObookings(meeting_id,entity,from_ts,to_ts,created_at,updated_at)VALUES(...);COMMIT;
When we use the above query, our tests pass, consistently.
References
[1] Martin Kleppmann, Designing Data-Intensive Applications (2017), Ch. 7.
]]>Kicking Around Packets: Understanding How the Internet Works2022-02-02T00:00:00+00:002022-02-02T00:00:00+00:00https://otee.dev/2022/02/02/understanding-the-internetIn this post, I try to answer the question “What is the internet and how does it really work?”
Key terms
Before going into further details, here is a useful list of some of the key terms used in this article.
End-system or host: This is a device (e.g., a computer) that actually wants to use the internet. It is the ultimate consumer of the communication services offered by the internet. It typically executes a network application on behalf of the user and uses the internet to that end. This article uses the terms ‘host’ and ‘end-system’ interchangeably.
Packet switches or routers: Different networks are interconnected to each other through a series of packet-switches. They are mainly responsible for forwarding packets of information from one node to another. Packet switches are of two kinds: link-layer switches and routers (discussed below). For the most part, this article uses the term ‘routers’ to refer to packet-switches in general.
Communication links: These are the physical media that connect routers to one another. Packets travel from one router to the next via a communication link.
Understanding the Internet
The internet is a network of networks.
Each end-system accesses the internet through an Internet Service Provider (ISP). An ISP is typically a telecom company providing internet service to end-users.
The starting point of the internet is an access ISP. The access ISP is the network which connects a host with the rest of the internet.
Each ISP has its own network. The overarching goal of the internet is to enable any two end-systems to communicate with each other, irrespective of where they are situated. Thus, it is not practically feasible for one ISP to provide a direct line of communication between every pair of end-points. Instead, networks have inter-connections with each other. In other words, each network interacts with one or more networks to deliver a message. Here’s how Wikipedia defines ‘inter-networking’
“Internetworking is the practice of interconnecting multiple computer networks, such that any pair of hosts in the connected networks can exchange messages irrespective of their hardware-level networking technology. The resulting system of interconnected networks are called an internetwork, or simply an internet.”
These interconnections can be direct, i.e., between two networks directly. This kind of interconnection is called peering, where two ISPs, at the same level of hierarchy, choose to mutually allow each other access to their respective networks. Most often, though, interconnections take place indirectly, i.e., to reach your destination end-system’s access ISP, you will have to go via one or more intermediary networks (which are typically larger networks with larger coverage). These kinds of indirect interconnections are called transit. Thus, typically, to pass on a message from one end-user to another the “the traffic will often be transferred through several indirect interconnections to reach the end-user” (source)
Largely, this is how the internet works: when a piece of information needs to be sent from one end-system to another, the information is divided into several segments, called packets. These packets are then sent to the destination end-system through the internet, which comprises several communication links and packet switches.
Communication links are the physical media through which packets are transmitted. Examples include optical fibers and twisted copper wires.
Each packet switch has multiple communication links attached to it. Packet switches take each packet it receives on any one of its attached communication links and forwards them to another one of its attached communication links. There are two main types of packet switches: routers and link-layer switches. While both forward packets towards the destination host, link-layer switches are used for connecting devices within a network. They are mostly used at the access network (for connecting a device to an access ISP). On the other hand, routers are used in the network core, i.e., for sending packets across networks. As both link-layers and routers do the fundamental task of forwarding packets towards the destination end-point, the rest of this article will refer to packet-switches as routers.
How does a router know which communication link to forward a given packet to? Every message transmitted by a source host will contain the IP address of the destination host. An IP address is hierarchical in nature and by looking at a specific part of the packet’s IP address and consulting its own forwarding table (which maps IP addresses to outbound communication links), it determines the correct communication link to forward the packet to.
On the destination end-system, the original message is re-composed by re-assembling all the packets. This is similar to how cargo gets delivered over roads. First, you divide the delivery goods into small packets and load them onto trucks. Then each of these trucks travel on high-ways (communication links) and at each intersection (packet switches) they are redirected to the next highway, which will eventually take them to their destination.
To ensure consistency, each actor involved in the exchange of information over the internet, are governed by a set of protocols. Two of the most fundamental protocols are the Internet Protocol (IP) and Transfer Control Protocol (TCP).
Here’s a brief description of how a bit gets “kicked around” between two end-systems:
“Consider a bit traveling from one end system, through a series of links and routers, to another end system. This poor bit gets kicked around and transmitted many, many times! The source end system first transmits the bit, and shortly thereafter the first router in the series receives the bit; the first router then transmits the bit, and shortly thereafter the second router receives the bit; and so on. Thus our bit, when traveling from source to destination, passes through a series of transmitter-receiver pairs” (Source)
A layered approach to Understanding the Internet
Given the complexity and the numerous moving parts involved in the working of the internet, it is more elegant to organise the many happenings between two hosts, as a series of top-down layers of protocols.
“To provide structure to the design of network protocols, network designers organize protocols—and the network hardware and software that implement the protocols— in layers.”(Source)
To understand this layered architecture, it is important to note that at each layer, the protocol provides some service to the layer above, by performing certain actions and by utilising the services provided by the layer immediately below it
A layered architecture provides the advantage of simplifying the architecture of the internet by breaking it down into specific, well-defined parts, each of which performs a specific task by depending on the service provided by the layer beneath it. More importantly, it adds modularity to the entire system, thereby allowing one to modify the implementation of the service provided by a specific layer, without affecting the rest of the system (so long as the modified implementation provides the same service and depends on the same layer to provide that service).
A protocol layer can be composed solely of software (application and transport layer), hardware (physical layer and data link layer) or a mixture of both (network link)
When taken together, the protocols of the various layers are called the protocol stack. According to the the TCP/IP architecture, the Internet protocol stack consists of five layers: the application layer, the transport layer, the network layer, the link layer and the physical layer.
Here’s a brief summary of what each of the protocol layers do:
Application layer: This layer provides services to the end-user, by executing an application that uses the internet. It does this by sending and receiving a series of messages from and to another host(s). It hides the complexities of how the network is laid out.
Transport layer: This layer provides services to the application layer, by ensuring that the application layer messages are transported to and from the correct destination. It hides how the underlying network is managed (e.g., how packets of information is actually delivered to the correct destination).
Network layer: This layer provides services to the transport layer by finding a “best effort” path via multiple routers, to deliver packets of information from the source host to the destination host. It hides how each router forwards packets to the successive router.
Link layer: It provides services to the network layer by actually determining how to deliver packets of information between two given routers.
Physical layer: It provides the service of physically transmitting “bits” of information to the next router, in a way that the receiver can understand and reconstruct the same.
Here’s an Analogy involving a King and a Queen
Before moving forward and delving into the details of each protocol layer, it would be convenient to take an analogy to better understand the services provided by each layer. Let’s go back to a time before the advent of modern communication technologies: a world where one relied on postal services to send messages over a large distance. Now let’s imagine the King of a distant kingdom—presently at the battlefield—wishes to send a letter to his beloved Queen—presently residing at the Royal Palace. Let’s follow how the letter gets delivered to the Queen, from the prism of the five network protocol layers.
Application Layer: The King at the battlefield dictates the letter to his personal assistant, who writes its out and sends it to the Royal Secretary to deliver the letter he has written.
Here, the King represents the user, the battlefield is the end-system, the King’s personal assistant represents the application layer and the Royal Secretary represents the transport layer.
Transport Layer: The Royal Secretary puts the Royal Seal, adds the address of the Royal Palace and hands it over to the local post office.
Here, the local post office represents the first router in the network layer and the Royal Palace is the destination end-system.
Network Layer: The postal officer realizes that the destination is far-off and that there is no direct route to the Royal Palace. But, looking at the address, the postal officer understands that the Royal Palace is due East. So, they deliver the letter to the next postal caravan moving East.
Link Layer: The caravan carries a load of postal traffic from this postal office to the next postal office in the East. The caravan leader draws out the route to the next postal office in the eastward direction.
Here, the caravan leader is the link layer, connecting the first postal office (a router) to the next postal office (another router).
Physical Layer: The caravan leader employs their trusted nephew to carry the King’s letter. The nephew, along with their camel-driven carriage, carries the King’s letter, following the route drawn by the caravan leader.
The nephew and their carriage, is the physical layer, physically carrying the postal traffic from one postal office (router) to the next.
Physical Layer: Once the carriage reaches the next postal office, the nephew off-loads its load from their carriage.
Link Layer: The guard at the next postal office unpacks the postal load and hands over all its contents to the postal officer of that postal office.
The guard is the link layer, passing on the load received from the carriage driven by the nephew (physical layer). The postal officer is the second router.
Network Layer: The postal officer at the next postal office, receives the postal load delivered by the caravan leader and examines it. They find the King’s letter, and realize that the Royal Post Office is in the next town (it is only natural that the Royal Family has a post office of their own!). However, they have to find a way to physically transport the letter to the Royal Post Office. Luckily, they have a local pigeon sender who regularly delivers mail locally. The postal officer delivers the King’s letter to the local pigeon sender.
Here, the local pigeon sender is the link layer and the pigeon is the physical layer.
Link Layer: The pigeon sender knows the way to the Royal Post Office. They instruct the pigeon to fly to the Royal Post Office.
Physical Layer: The pigeon flies away towards the Royal Post Office.
Physical Layer: The pigeon successfully lands at the entrance of the Royal Pigeon Receiver at the Royal Post Office.
Link Layer: The Royal Pigeon Receiver at the Royal Post Office receives the pigeon’s parcel, cleans it up and hands it over to the postal officer at the Royal Post Office.
Here, the Royal Guard represents the link layer, which hands over the information received from the pigeon (physical layer) to the postal office (network layer).
Network Layer: The postal officer at the Royal Post Office accepts the mail from the Royal Pigeon Receiver. They pass on the King’s letter to the Queen’s Secretary.
The postal officer represents the network layer, while the Queen’s Secretary is the transport layer.
Transport Layer: The Queen’s Secretary receives the King’s letter from the Royal Post Office, verifies the Royal Seal, ensures that it is complete and hands it over to the Queen’s personal assistant.
Here, the Queen’s personal assistant represents the application layer, accepting the message delivered by the Queen’s Secretary (transport layer)
Application Layer: The Queen’s personal assistant knock’s on the Queen’s room and reads out loud what the King had to say. The Queen is delighted to hear from the King!
Now, let’s try to understand how each of these layers actually work.
Application layer
First, it is important to understand what a ‘network application’ is. It is a program that runs on one host and communicates with another host, using the internet. In fact, programs running on different hosts, communicating over the internet are called processes:
“In the jargon of operating systems, it is not actually programs but processes that communicate. A process can be thought of as a program that is running within an end system” (Source)
Processes on different hosts communicate with each other via messages. Typically, a network application consists of two different processes running on two different hosts, that send messages to each other (eg., server and client process on the web).
Now each process sends and receives messages over the internet by using a programming interface called the socket. Sockets act like doors- when the network application needs to send a message across the internet, it simply delivers it across the door, with the expectation that the rest will be taken care of by the network. Sockets in an end-system are identified by port numbers
In fact, sockets are the interface between the application layer and the transport layer.
Now, coming back to the application layer itself, it is defined as the “communications protocols and interface methods used in process-to-process communications across an Internet Protocol (IP) computer network”(source). Mainly, the application layer protocol standardized the manner in which data should be exchanged between two hosts. It does not deal with how this data will, in fact, be transferred.
Examples of application layer protocol include: HTTP(Hyper Text Transfer Protocol), FTP (File Transfer Protocol), SMTP (Simple Mail Transfer Protocol).
Transport layer
The transport layer transports the application-layer messages (received over the socket) between two application end-points (or hosts). The main objective of this layer is to provide logical communication between two application processes. This means that this layer will abstract out the details of the internet before the application layer, giving the impression to the application that it is directly connected to the other end-system’s application.
It is important to note that the transport layer protocols are implemented in the end-systems (or hosts). They are not implemented by intermediate routers or communication links.
In the internet, there are two transport layer protocols both of which can transport application layer messages:
Transmission Control Protocol (TCP): TCP provides reliable data transfer. It guarantees delivery of messages and manages flow control. The latter ensures that messages are delivered in the correct order. It also provides congestion control, by preventing over-congestion of traffic on routers and links between two hosts. This is done by dividing messages into smaller segments, and reducing the transmission rate when the network is congested.
User Datagram Protocol (UDP): UDP provides a connectionless service. It provides no guarantees towards reliability, flow control, or congestion control.
The transport layer protocols convert each application layer message into several transport layer packets called segments each of which is then sent across the network layer, via the routers and links.
Network layer
The network layer receives a transport layer segment and a destination address from the transport layer protocol of the source host. The network layer provides the service of delivering this segment to the correct destination host’s transport layer.
The Internet Protocol (IP) is the only network layer protocol of the internet. The IP implements a best effort service, meaning that while it makes a “best effort” to deliver segments between hosts, it does not make any guarantees regarding their delivery, i.e., the IP cannot guarantee the eventual delivery of a packet. Neither can it assure that the packets will be delivered in an orderly fashion nor that there will not be any tampering of the packets in the process of transmission.
The network layer protocol converts the transport layer segments into network layer segments called “datagrams”. Each datagram contains the address of the destination host. Once this is done, the source host pops each datagram into the network, i.e., it sends the datagram to its nearby router. For this datagram to reach the ultimate destination host, it needs to traverse through several routers. Each router forwards a datagram that it receives in any of its input links to the next appropriate router. The determination of which router to forward the datagram to, is done by looking at the destination address and its own forwarding table. Thus, to send one packet from one host to another, each router has to make a local decision that is compatible with its next router. In this way, we get a global convergence on packet delivery using a series of local decisions at the router-level.
Note that routing, i.e., the route or path that a packet will traverse within the network layer, is implemented by each router and is not determined by the end-systems. This is very convenient, as it insulates the host application from the changes in the internet’s routing architecture. (RFC1122) This is a direct consequence of having a modular protocol stack.
Also, as another corollary of the modular architecture of the protocol stack, routers remain stateless. They do not maintain any state of the end-to-end flow of information. Each router inspects the IP address of the destination of a packet, and determines the next router to forward it to. This enables the effective utilization of redundant paths: two packets headed for the same destination, may take two different routes.(RFC1122)
The network layer provides the service of transporting packets between two end-systems, via a series of routers. The link layer protocols provide the service of actually transporting packets from one node (a host, or a router) to another node (a router or a host). This is done by establishing connectivity with the next router and breaking up network layer datagrams into “frames” to transmit them to the next router. In other words, link layer protocols provide the service of converting datagrams into frames and transporting them across each individual link connecting two specific nodes. At each node, the network layer (after determining the next forwarding router/host) passes the Network Layer datagram to the respective link layer protocol, which sends it across to the next node. At the next node, the link layer protocol hands over the packet to the network layer protocol of that node.
Examples of link layer protocols include WiFi, Ethernet and Point-to-Point protocol.
Physical Layer
The physical layer provides the service of transporting each individual bit of a given frame from one node to the next. The protocol of this layer will depend on the actual link connecting the two nodes. In other words, physical layer protocols determine how to encode digital information and transmit them across a communication link.
End-to-End Argument
As shown above, the network layer does not make any guarantees regarding delivery of messages. It is the job of the end-systems (transfer layer and application layer) to ensure integrity, encryption and orderly delivery of information. Thus, as lower layers of the protocol stack do not implement complex functionalities, the internet is designed as a “dumb” network that relies on “smart” end-systems to carry out complex tasks. To put it in another way, the lower level actors (routers) perform simple tasks efficiently, while higher level actors are expected to use the services of the lower level actors to perform complex tasks. By pushing the complexity to the edges, the internet can be scaled to be compatible with any kind of network or device.
]]>Recurse Center: Week Three2022-01-26T00:00:00+00:002022-01-26T00:00:00+00:00https://otee.dev/2022/01/26/recurse-center-week-threeThis post is about my third week at the Recurse Center.
During the previous week, while reading Brave Clojure’s 4th and 5th chapters, I was briefly introduced to some of the key tenets of functional programming. I found these concepts very fascinating. This week, I spent a considerable amount of time reading about them. While I would like to write a separate post on my findings, here are some of the points I have taken on my notebook:
Functional programming is characterized by its use of pure functions.
A pure function is one that has no side effects.
The values evaluated by a pure function are solely dependent on its arguments. The return value for a given set of arguments, will always be the same, irrespective of the number of times the function is called or when it is being invoked. This makes it easier to do parallel programming, testing, and using higher-order functions
Because of their deterministic nature, a pure function can be replaced with its return value, in an expression (referential transparency)
The use of impure functions is not prohibited. But as a functional programmer, it is important to be wary of using side effects.
As changing the value of a data-structure can cause side effects, functional programming emphasizes the use of immutable data-structures.
Now that I was getting slightly familiar with the world of Clojure, I thought it would be the perfect time to embark on a journey to build my very first app using Clojure. I decided to build an online version of flash cards, called RemindMe.
Given that it would be my first project in Clojure, I thought it would be useful to first build the app using JavaScript, the language I am most familiar with.
So, I spent a day writing the server-side logic of RemindMe, using Node.js. I then proceeded to build the front-end part of the project. To implement the front-end part, I used the Bootstrap framework. This Bootstrap tutorial was very helpful, to get started. (Side note: Long after finishing the front-end part, I realized that the hamburger menu on the navigation bar is not working on a mobile display. I’ll need to fix this; would love to pair with anyone who is good with Bootstrap/CSS!).
After completing the app in Node.js, I proceeded to re-implement the backend of RemindMe, using Clojure. I used the Ring framework and it’s jetty-adaptor to build my web service. I chose Compojure as a routing library, for better handling of requests.
]]>My First Clojure Backend Using Ring, Jetty and Compojure2022-01-25T00:00:00+00:002022-01-25T00:00:00+00:00https://otee.dev/2022/01/25/clojure-backend-using-ring-jetty-compojureIn this post I discuss how I built my first web-app, RemindMe, using Clojure! The app is deployed here: https://remind.otee.dev
Project Scope
The RemindMe app aims to replicate how flash cards work in the real world. Quoting from Wikipedia:
A flashcard or flash card (also known as an index card) is a card bearing information on both sides, which is intended to be used as an aid in memorization. Each flashcard bears a question on one side and an answer on the other
Similar to real world flash cards, RemindMe displays a question, which is selected randomly from a set of questions. The user has the option to see a hint or go to the next question (which will again be randomly selected from the question pool). If the user chooses to the see the hint, the hint to that question will be displayed and the user will again have the choice to either proceed to the next question or see the solution. Lastly, if the user chooses to see the solution, the relevant solution is displayed along with the option to proceed to the next question.
I wanted to build RemindMe, as a tool to recall my solutions to LeetCode problems. For this reason, the app describes each question as a ‘problem’. Also, while the text of each problem is lifted from LeetCode, the hints and solutions are mine. (Always happy to update my solutions with more optimal ones. Pull requests are welcome!)
The app reads a JSON file (data.txt) to access the set of problems, solutions and hints. When we want to add a new problem, we will need to update the JSON file.
Here are the routes supported by this app:
/problem/:id: This route loads the problem corresponding to the id path parameter.
/next: This route redirects to a randomly selected /problem/:id route
/: Same as /next, it redirects to a new problem route
/hint/:id: Generates a JSON payload containing the hint for the respective problem (to be used by client-side JavaScript).
:/solution/:id: Generates a JSON payload containing the hint and solution for the respective problem (to be used by client-side JavaScript).
Here’s a demo of the RemindMe
Goal
The primary goal is to learn how to build and deploy a project using Clojure. But before embarking on this journey, I first built an identical app using Node.js and the Express framework. This was helpful, as I did not have to spend much time focusing on the core business logic of the app and its front-end, while working with Clojure and the Ring framework. The Node.js version of the application is hosted on this GitHub Repository: https://github.com/oitee/remind-me
Thus, the goal of this project is to re-implement the backend of RemindMe, using Clojure, and the Ring framework. This project is hosted on a separate repository, called aspire.
Step 0: Adding Dependencies to a Leiningen Project
In a Leiningen project, dependencies and versions are added to the project.clj file (similar to the package.JSON file in Node projects). Each time we add or remove any dependency, we should run lein deps to install/remove the project’s dependencies.
For this project, we will need three dependencies:
Ring: Ring is a web-framework which is analogous to Express in Node.js. It is used for easier management of HTTP requests, through routes, handlers and middlewares. To quote from its own documentation,
By abstracting the details of HTTP into a simple, unified API, Ring allows web applications to be constructed of modular components that can be shared among a variety of applications, web servers, and web frameworks.
Ring-Jetty: Since Clojure does not come with a built-in HTTP server, unlike Node.js, we need to implement a HTTP server. Ring comes with a default support for Jetty, a Java web-server.
Ring-Jetty is the web server that comes with Ring. It is a Clojure wrapper around Jetty. It is perfectly fine and acceptable and has the easiest setup in my opinion. If you are going to use Ring, it’s the best option. In addition, it is well-maintained and has a lot of users. (Eric Normand)
We will use Ring-Jetty in this project. We can do this by including ring-jetty-adapter in our dependencies.
Compojure: This is a routing library for Ring that allows for easy handling of routes.
Once the above three dependencies are added, the project.clj will look like this:
(defprojectaspire"0.1.0-SNAPSHOT":description"FIXME: write description":url"http://example.com/FIXME":license{:name"EPL-2.0 OR GPL-2.0-or-later WITH Classpath-exception-2.0":url"https://www.eclipse.org/legal/epl-2.0/"}:dependencies[[org.clojure/clojure"1.10.0"][ring/ring-core"1.9.5"][ring/ring-jetty-adapter"1.9.5"][compojure"1.6.2"]]:main^:skip-aotaspire.core:target-path"target/%s":profiles{:uberjar{:aot:all}})
Now, we need to run lein deps on Linux command line:
Beware: it may download the entire internet, when we first run lein deps. 😛
Step 1: Starting a ‘hello world’ server
To start a simple “hello world” server, we need to first write a handler function that will respond to every request.(documentation)
We then pass this handler to the run-jetty function to respond to requests. The run-jetty function starts an HTTP server that listens on a port and when a request is received on this port, calls the handler function. Later on, when we write more complicated code, the handler function will decide (based on request parameters, routes, methods etc) on how to respond to a certain request. In short, jetty is the HTTP server, run-jetty converts Clojure functions to work well with the Java jetty library.
Note that we always respond with hello world, and therefore requests to any path will receive the same response. If we print the request object, we can see all the necessary information that will be required for routing requests (eg. URI).
In the above snippet, we used one function to respond to all requests. However, this would be hard while managing requests with different HTTP methods and/or paths.
As our project supports multiple routes, we will use routes provided by compojure to determine how a request on a certain route should be responded to (similar to an Express.js app).
So, the previous handler function should be replaced with app:
(nsaspire.core(:gen-class)(:require[ring.adapter.jetty:asjetty][clojure.pprint][compojure.core:ascompojure][compojure.route:ascompojure-route]))(compojure/defroutesapp(compojure/GET"/"[]"Hello World")(compojure-route/not-found"Page not found"))(defn-main[&args](jetty/run-jettyapp{:port3000:join?true}))
Note that:
defroutes is a macro that returns a Ring handler function. It allows us to define and combine multiple routes under one umbrella (rather, handler).
Individual compojure routes are macros as well (and they can be used on a stand-alone basis, i.e., without defroutes)
compojure route macros are based on HTTP methods, i.e., GETPUT etc.
The path of a request is matched with the first argument of a route macro (notice the "/" after compojure/GET)
The last argument forms the response for that specific route. Once the HTTP method and path match, this argument will form the response. Instead of being a data value, it can be a function as well, which will have access to the incoming request to form the response.
The second argument in the GET macro is used for parameters: form and query parameters (we will not be using it for this project)
There is a way to have a fall-back route, i.e., not-found, to send a response when none of the routes match (here, we are using the Page not found response, when nothing matches).
We can re-write these routes using compojure and provide specific route handler functions for each route:
(compojure/defroutesapp(compojure/GET"/"paramshome)(compojure/GET"/problem/:id"paramsproblem-by-id)(compojure/GET"/next"paramsnext-problem)(compojure/GET"/hint/:id"paramshint)(compojure/GET"/solution/:id"paramssolution)(compojure-route/not-found"Page not found"))
There are two things to note here:
If the URL has a parameter in the path, we can use : to refer to that parameter in the route. This kind of route definition is similar to that of Express.js.
The second argument to GET is not a vector any more; we are just giving it a name (params) although we may not use it.
Now, let’s write the route handlers (i.e., the third argument to each compojure route). Let’s start with problem-by-id. Note that we need to access the parameter in the URL path. This can be accessed from the request map. We can print the request map to see the relevant params key.
We can access the params key to see the value of our path parameter and write our handlers accordingly:
(defnhome[request]"Home Page")(defnproblem-by-id[request](let[id(:id(:paramsrequest))](str"Problem Page for "id)))(defnnext-problem[request]"Next Problem Page")(defnhint[request](let[id(:id(:paramsrequest))](str"Hint for "id)))(defnsolution[request](let[id(:id(:paramsrequest))](str"Solution for "id)))
Each of the above functions are mentioned in our compojure routes, such that when the request path and method match, the relevant function will be invoked.
Step 4: Code Reorganisation
Currently, all the handlers and routes are in the same namespace. We can split them into three namespaces: one for starting the server, one for defining the routes, and one for the route handlers:
aspire.core: This will start the server
aspire.routes: This will define the routes
aspire.handlers: This will have the route handler functions
At this point, the project structure looks like this:
aspire.handlers contains the route handler functions:
(nsaspire.handlers)(defnhome[request]"Home Page")(defnproblem-by-id[request](let[id(:id(:paramsrequest))](str"Problem Page for "id)))(defnnext-problem[request]"Next Problem Page")(defnhint[request](let[id(:id(:paramsrequest))](str"Hint for "id)))(defnsolution[request](let[id(:id(:paramsrequest))](str"Solution for "id)))(defnnot-found[request]"404: Page not Found")
Step 5: Implementing the Business Logic
Now that we have up the web server, defined our routes and route-handlers, we can build the actual product features, by appropriately defining the route handler functions. To do this, we need a templating engine and a way to read data from a JSON file.
Templating Engine: The Node.js version of the app used Mustache templating engine. So, we need to support Mustache in Clojure as well. This will ensure that the front-end remains un-changed. For this, we can use de.ubercode.clostache/clostache as a dependency (documentation)
Data reading: The actual contents of the project (such as descriptions and solutions of problems etc) are stored in a JSON file which we need to read for serving the request. For this, we can use org.clojure/data.json as a dependency (documentation)
Model Component
The JSON file containing the problem sets looks like this:
[{"id":"find-peak-element","problemTitle":"Find Peak Element","problemDescription":"A peak element is an element that is strictly greater than its neighbors. Given an integer array nums, find a peak element, and return its index. If the array contains multiple peaks, return the index to any of the peaks. You may imagine that nums[-1] = nums[n] = -∞. You must write an algorithm that runs in O(log n) time.","hint":"Start with the middle element","solution":"Start with mid element.\n If this is a peak, then return it.\n If this element is less than the next element, it means this element is part of an asceding slope. So, make lo = mid + 1,\n If this element is less than the earlier element, move to the earlier sub-array, ie, hi = mid - 1\n At the end, if lo === hi, lo is the peak element. Because it would mean we have reached the end of the array. And edges are peaks, if their adjacent element are smaller than them."},{"id":"boats-to-save-people","problemTitle":"Boats to Save People","problemDescription":"You are given an array people where people[i] is the weight of the ith person, and an infinite number of boats where each boat can carry a maximum weight of limit. Each boat carries at most two people at the same time, provided the sum of the weight of those people is at most limit.\n\n Return the minimum number of boats to carry every given person.\n ","hint":"Start with sorting the array","solution":"Sort the array.\n For each people[hi] + people[lo] > limit, hi-- and boats++.\n For others, hi-- lo++ boats++\n At the end, if hi == lo (indicating that there was an odd number of elements), boats++"},...]
We need to parse this JSON file by using clojure.data.json
The read-str function (from the namespace clojure.data.json), takes a JSON string and converts it into a valid Clojure data-structure. Owing to the nature of the data contained in the JSON file, data will be a vector of hash-maps.
Note that we pass an additional argument to read-str, called :key-fn keyword. This ensures that the keys of the hash-maps generated by read-str are keywords instead of strings. Here’s how the hash-map looks like:
[{:id"find-peak-element",:problemTitle"Find Peak Element",:problemDescription"A peak element is an element that is strictly greater than its neighbors. Given an integer array nums, find a peak element, and return its index. If the array contains multiple peaks, return the index to any of the peaks. You may imagine that nums[-1] = nums[n] = -∞. You must write an algorithm that runs in O(log n) time.",:hint"Start with the middle element",:solution"Start with mid element.\n If this is a peak, then return it.\n If this element is less than the next element, it means this element is part of an asceding slope. So, make lo = mid + 1,\n If this element is less than the earlier element, move to the earlier sub-array, ie, hi = mid - 1\n At the end, if lo === hi, lo is the peak element. Because it would mean we have reached the end of the array. And edges are peaks, if their adjacent element are smaller than them."}{:id"boats-to-save-people",:problemTitle"Boats to Save People",:problemDescription"You are given an array people where people[i] is the weight of the ith person, and an infinite number of boats where each boat can carry a maximum weight of limit. Each boat carries at most two people at the same time, provided the sum of the weight of those people is at most limit.\n\n Return the minimum number of boats to carry every given person.\n ",:hint"Start with sorting the array",:solution"Sort the array.\n For each people[hi] + people[lo] > limit, hi-- and boats++.\n For others, hi-- lo++ boats++\n At the end, if hi == lo (indicating that there was an odd number of elements), boats++"}...]
Now, we need to convert this vector into a hash-map, where each id will map to the respective hash-map (representing a problem set)
We need to write the route handlers for each route. Let’s start with the route handler for /problem/:id
If the id parameter is correct, this route handler should generate the home page, with the necessary details of the respective problem set.
To achieve this, we need to render a template file (home.mustache), using clostache
(nsaspire.handlers(:require[clostache.parser:asmustache][aspire.db:asdb]))(defnproblem-by-id[request](let[id(:id(:paramsrequest))data(db/getid)](if-notdata(str"Problem cannot be loaded, as ID is not valid "id)(mustache/render-resource"templates/home.mustache"{:title(:problemTitledata):description(:problemDescriptiondata)}))))
The render-resource function of clostache.parser is a templating engine. The first argument should be the location of the template file. As per its documentation, render-resource can “render a resource from the classpath”.
A classpath is a “a sequence of paths that Clojure (or Java) checks when looking for a Clojure source file”. In a Leiningen project, the following directories are included in the classpath by default: the src, test, classes, test-resources, and resources directories(source). This means that our template file should be placed in any of these directories, to allow render-resource to access it. Accordingly, the template file (home.mustache) is placed in a sub-directory (templates) inside the resources directory of the project.
To write the route-handler for /next, we need to know how to send a redirection response. For this, we need to require ring.util.response name-space and use the redirect function therein:
Install an NGINX config file and set up a DNS entry
Compiling the Code to JAR
Because Clojure is hosted on the Java Virtual Machine, Clojure applications are run the same way as Java applications are run.
How does Java source code get complied?
First, the Java Compiler converts the source code to Java Byte Code
Once a program is converted into Java Byte Code, it can be executed by the Java Virtual Machine, which is the runtime environment for Java (Source)
The Java Byte Code gets stored in class files; a Java ARchive file, also called JAR, can store a collection of class files (source)
Clojure source code gets converted to a specific JAR which can be executed by the JVM.
As our project is built using Leiningen, we can use lein jar to create the JAR file of our project. This file will be stored in the target directory of our project. Instead of simply using lein jar, we can use lein uberjar, which will create a JAR file containing the source code of our project, along with all its dependencies. A uberjar is a “a single standalone executable jar file”, which makes it easier to deploy. Once a uberjar is prepared, we can run it by simply using java -jar command. Optionally, we can use lein clean to clean our target directory. To run multiple commands successively, we can use lein do. So, here is how we first clean our target directory and then compile our project into a single JAR file:
$ lein do clean, uberjar
Java HotSpot(TM) 64-Bit Server VM warning: Options -Xverify:none and -noverify were deprecated in JDK 13 and will likely be removed in a future release.
Compiling aspire.core
2022-01-24 18:38:32.790:INFO::main: Logging initialized @1495ms to org.eclipse.jetty.util.log.StdErrLog
WARNING: seqable? already refers to: #'clojure.core/seqable? in namespace: clojure.core.incubator, being replaced by: #'clojure.core.incubator/seqable?
WARNING: seqable? already refers to: #'clojure.core/seqable? in namespace: clostache.parser, being replaced by: #'clojure.core.incubator/seqable?
WARNING: get already refers to: #'clojure.core/get in namespace: aspire.db, being replaced by: #'aspire.db/get
Compiling aspire.db
WARNING: get already refers to: #'clojure.core/get in namespace: aspire.db, being replaced by: #'aspire.db/get
Compiling aspire.handlers
Compiling aspire.routes
Created /home/otee/projects/aspire/target/uberjar/aspire-0.1.0-SNAPSHOT.jar
Created /home/otee/projects/aspire/target/uberjar/aspire-0.1.0-SNAPSHOT-standalone.jar
In order to run the project from a JAR, we need to ensure that we are not reading any files from the local file system. In the present project, all the non-Clojure files are read from the classpath. In the case of data.txt, which hosts the JSON data-set, we cannot directly use the file-path while slurping it. Instead, we have to use the resource method to read file from the classpath instead:
(slurp (clojure.java.io/resource "data.txt"))
Deploying the uberjar to GCP VM
Now that we have our JAR file, we need to send it across to our VM on GCP (alias calculus). We can do this by using the rsync command, which enables the transfer of files over SSH. Interestingly, it synchronizes the data being transferred between the different machines: ensuring that only those files are transferred which are new or updated.
The -Xmx flag is used to specify the maximum heap memory allocation for running a Java program. When we use -Xmx32m, we restrict the memory allocation to 32 MB.
Registering with systemd
We need to use systemd, to ensure that our application runs consistently. To set up systemd for our application, we need to write a new configuration file (remind.service) in /lib/systemd/system.
Note that we need to specify the value of the internal port(PORT=4003) where our server will be listening under the Environment entry. Also, under the ExecStart entry, we cannot use java; instead we have to mention the location of the executable Java file, i.e., /usr/bin/java. (/usr/bin is the “primary directory of executable commands on the system”).
Now, we need to run the following commands, to have systemd run our application (for more on this, read this previous post):
This should run our application. To see the status of the status of our application, we can use the following command:
sudo systemctl status remind.service
● remind.service - remind
Loaded: loaded (/lib/systemd/system/remind.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2022-01-24 09:25:30 UTC; 18s ago
Docs: https://github.com/oitee/aspire#readme
Main PID: 178875 (java)
Tasks: 22 (limit: 1159)
Memory: 119.4M
CGroup: /system.slice/remind.service
└─178875 /usr/bin/java -Xmx32m-jar /home/oitee.codes/projects/aspire-0.1.0-SNAPSHOT-standalone.jar
Jan 24 09:25:30 calculus systemd[1]: Started remind.
Jan 24 09:25:33 calculus java[178875]: 2022-01-24 09:25:33.385:INFO::main: Logging initialized @2707ms to org.eclipse.jetty.util.log.StdErrLog
Jan 24 09:25:35 calculus java[178875]: WARNING: seqable? already refers to: #'clojure.core/seqable? in namespace: clojure.core.incubator, being replaced by: #'clojure.core.incubator/seqable?
Jan 24 09:25:35 calculus java[178875]: WARNING: seqable? already refers to: #'clojure.core/seqable? in namespace: clostache.parser, being replaced by: #'clojure.core.incubator/seqable?
Jan 24 09:25:35 calculus java[178875]: WARNING: get already refers to: #'clojure.core/get in namespace: aspire.db, being replaced by: #'aspire.db/get
Jan 24 09:25:35 calculus java[178875]: 2022-01-24 09:25:35.264:INFO:oejs.Server:main: jetty-9.4.44.v20210927; built: 2021-09-27T23:02:44.612Z; git: 8da83308eeca865e495e53ef315a249d63ba9332; jvm 17.0.1+12-Ubuntu-120.04
Jan 24 09:25:35 calculus java[178875]: 2022-01-24 09:25:35.454:INFO:oejs.AbstractConnector:main: Started ServerConnector@4052913c{HTTP/1.1, (http/1.1)}{0.0.0.0:4003}
Jan 24 09:25:35 calculus java[178875]: 2022-01-24 09:25:35.456:INFO:oejs.Server:main: Started @4815ms
Using NGINX to redirect traffic from Port 80
To redirect requests to port 80 of our VM to the specific internal port our server will be listening to (4003) , we need to write a configuration file for NGINX. First, we should add a configuration file remind.otee.dev to the /etc/nginx/sites-available.
sudo systemctl status nginx
sudo systemctl restart nginx
Now that NGINX has been configured to redirect requests to remind.otee.dev to the internal port 4003, we need to set up the custom domain remind.otee.dev and then enforce HTTPS, by using the freely available Certbot tool provided by Lets Encrypt.
Here are some of the improvements that can be added in future:
Move the data-set to SQLite. This will allow us to write data on the file. For example, to track analytics of our app: the number of views, requests for hints and solutions
Enable adding of new problem sets from the UI:
This can be done by creating a ‘users’ table, with admin usernames and passwords (using good password storage principles as we did with Twirl)
Maintaining sessions on addition routes (Middleware for cookie parsing)
Source: Designing Data Intensive Applications [1]
