ByteSize: Performant Storage with Flask + React

Combining storage types for maximum performance

Photo by Cobro on Unsplash

In the last tutorial we started putting together a rudimentary link preview component. Starting with the Microlink library, we were able to allow our users to enter a URL address and see an automatically generated OpenGraph preview of the link.

It’s rough though. If we are outside of the caching windows, Microlink will rescrape the entire site. While this sounds wonderful from a “freshness” point of view, it can be death for our site, with loading times approaching ~20 seconds for uncached results.

Brutal.

We have to find a way to bring down that latency to a much more acceptable range.

Sketching it out, we have a few options:

1. Cache the response ourselves for a longer period of time.
2. Store the data persistently in a database

Caching ourselves could help quite a bit, as our latency would drop to ~5–10ms, but depending on the amount of users and the length of time we store it for the costs could quickly add up.

A database would be more economical for long term storage, but that brings our latency back up into the ~50–100ms range, which when you factor in component rendering that puts us back into an area we’d rather not be.

What we could do is combine the two, using the database to store all the requests, and a smaller caching window to ensure frequently accessed routes are accessed much faster.

If you just want to get the code, you can find the repo here

Photo by benjamin lehman on Unsplash

Starting up the Database

We’ll start our task with the database. We will need to import two libraries to communicate with our db and manage upgrades: Flask-SQLAlchemy and Flask-Migrate.

pip install flask-sqlalchemy flask-migrate

Next up we have to import and initialize them for our app. We’ll create an extensions file to hold create our db object (in app/extensions.py).

and after that we’ll initialize the db in the create_app function (in app/__init__.py)

We can use SQLite as a lightweight database, but because of SQLAlchemy you can use any db you feel comfortable with. (in config.py)

As a final setup step we’ll initialize flask-migrate to help us navigate db migrations and upgrades (in application.py)

Setting the Table

With our database is set up, we will start to create the table to store our data. We can look at the Microlink API response to figure out what data we’ll need for our response.

It seems like there some extra fields that are returned for video, audio and iFrame if they are a part of the response, so I ran a YouTube link through a request to make sure that I got the requisite data. Once we have all of the fields we need, we can create our table (in ‘app/models.py’)

Now we have to create a class method to convert an table entry to JSON format to be returned to the user (in app/models.py)

We’ll need to accommodate some conditional fields for iFrames, audio and video, as well as trying to navigate using an array field in a database that doesn’t use arrays (the iFrame scripts field). I plan on converting the iFrame scripts array to a string when I insert it into the database, and using ast.literal_eval I can “rehydrate” the type, so to speak. I figure the input would be sanitized coming from Microlink, but in production I would most likely do some kind of internal validation to ensure that I wasn’t accepting any dangerous code, or at the very least limiting the scope of damage it could do.

With that finished, we also need a way to convert a JSON response into a db entry (in app/models.py)

Now we initialize our database and its first migration in the CLI

set FLASK_APP=application.py
flask db init
flask db migrate
flask db upgrade

Photo by Paul Gilmore on Unsplash

Creating the View

With the database set up and ready, we’ll get our endpoint together.

There’s a couple things to keep in mind here. Since we can have multiple scenarios where a URL is entered with or without a “www” subdomain, I created a conditional to check if it’s present, and appends it in the case that it’s not to normalize the input. We’ve also made it as a backup that if for whatever reason the URL isn’t found in the database even though a derivative of it might be, we just perform a Microlink query instead. This makes sure that no matter what our front end component will get a response, rather than leaving it hanging. We also check to make sure that a specific request was successful before adding it to the database.

Cache Money

Trying out our endpoint, we can see that we’ve got the latency down around where we predicted, somewhere around ~50–80ms for the requests that are stored, with another 100ms or so to render the component. Since we’re trying to get this number down as much as possible, caching is the next weapon in our arsenal. We’ll start off with importing Flask-Caching:

pip install flask-caching

Much like Flask-SQLAlchemy, we’ll have to initialize the package and perform some initial configuration (in app/extensions.py)

and initialize the package (in app/__init__.py)

Finally, we configure our cache type (in config.py)

We ran through some of the finer points of function cache construction in an earlier tutorial (check it out here), so we’ll adapt the principles here. We’ll start off with creating our custom cache key (in app/utils.py)

and use it for our view (in app/main/views.py)

We now have caching enabled on our endpoint for 60 seconds. Testing it out, our endpoint is now down to a lean 5ms response time! Even with component rendering, it’s at just over 150ms to full render.

Photo by Anna Kolosyuk on Unsplash

Finished

With that we’ve completed our component!

As always thanks for reading, and be sure to follow and check back for more tutorials on creating functional full stack components with Flask and React.