Debugging deployment

SoftDev students are hard at work on their final projects. By now, they all have fairly complex code bases. This limits how much I can help them with debugging.

There are some problems, though, that they have to contend with that even with experience, are hard to spot. Notably because the very tools you use to debug these errors are part of the problem.

Last week, this happened twice. Both cases were brought to me by really strong students which just goes to underscore how insidious these problems can be.

Here's a fake code snippet of a Flask application that illustrates both problems.

 1: from flask import Flask, render_template
 2: 
 3: app = Flask(__name__)
 4: 
 5: @app.route("/")
 6: def index():
 7:     return "hello"
 8: 
 9: @app.route('/test/<some_data>')
10: def test():
11:     picture_url = build_url(some_data)
12:     result = api_test(picture_url)
13:     do_something(result)
14: 
15: if __name__ == "__main__":
16:    app.debug = True
17:    app.secret_key = "some secret key"
18:    app.run(host="0.0.0.0", port=8000)

First, the "easy" one. The student was trying to deploy the application. We use Green Unicorn to deploy our applications, ultimately on Digital Ocean servers in the cloud.

The student was using the correct command:

gunicorn -W 4 -b 0.0.0.0:8000 app:app

but it wasn't working. It ran, but whenever he went to the site, it came back with an error.

The problem?

He had to change:

if __name__ == "__main__":
   app.debug = True
   app.secret_key = "some secret key"
   app.run(host="0.0.0.0", port=8000)

to

app.secret_key = "some secret key"
if __name__ == "__main__":
    app.debug = True
    app.run(host="0.0.0.0", port=8000)

Normally, when developing and testing our applications, we use the test server that's bundled with Flask. The line that reads "app.run…" takes care of this.

When running the application as a "main program" - "python app.py" the if statement is true and it runs the indented lines, setting the secret key which is required for session management.

When running under gunicorn, the gunicorn server loads the application as a module and then runs it. In this case name isn't main so it never sets secret key and so we have a problem.

Pretty subtle and even though we did cover this in class, it comes up pretty rarely so it's not an easy catch.

Then there was this problem.

The setup for this one's a little more complicated. The group was using a facial recognition api. You provide the API with the url to an image, it fetches it and does recognition.

It's also important to note that when Flask is running, it will serve files from a static directory, so, if I'm running my flask server on myhost, port 800 and you stored an image named picture.jpg in the static directory, going to:

would get that image.

The group did things right. They ran the Flask test server to serve the static files and then wrote a small python program to test the api:

picture_url = build_url(some_data)
result = api_test(picture_url)
do_something(result)

Everything worked fine.

But, when they put this code in as a route in their web app (as in the top code fragment), it froze.

They couldn't figure it out.

The code worked as a "stand alone" but not in the web app.

The problem?

Once again, the built in Flask development server.

The development server runs in a single thread / process. This means it can only do one thing at a time. When they ran their test as a separate program, the api they used made a request to their app to serve up the static picture file and it worked.

When they ran from the Flask application itself, their app made a call to the web api (line 12) and then blocked while waiting for the response. The web api tried to request the image from the Flask app but it was blocked – deadlock.

Again, the solution was to run the web app using a server that could handle multiple requests - gunicorn.

Once again, that solved the problem.

Both of these problems were fairly subtle and very hard to catch - even with experience. I remember the hours I lost when I was learning this stuff.

Some times kids get caught up in algorithms or poor code design but sometimes, it's just the tools.