Playing with W3C Annotation AKA Web Annotation

Today I was playing around with W3C Annotation AKA Web Annotation. I needed to convert the JSON-LD examples to RDF XML and test XPath expressions. I justed a couple of online tools. Just some quick notes about these tools.

  • Convert JSON-LD to RDF XML: Select JSON-LD as input format and Pretty RDF/XML as output format.
  • Test XPath expressions: Copy and past the XML into the input field. Type the XPath into the XPath input field. Hit Test XPath button and you should see the result.
  • XML Formatter: A XML formatter that lets you specify the number of leading spaces.

You can find the latest information about Web Annotation over here and example code on Github.

Adding a ssh public to Gitlab

I needed to add my ssh public key to Gitlab. Pretty straight forward generating it using PuttyGen and saving the public and private keys. When I pasted they public key into the dialog in Gitlab and hit “Add key” I got the following error.

The form contains the following errors:

  • Key is invalid
  • Key should be a single line
  • Fingerprint has already been taken
  • Fingerprint cannot be generated

I checked the public key and saw that it was not a single line. To correct this I did the following.

  • Deleted the “—- BEGIN SSH2 PUBLIC KEY —-” line
  • Deleted the “—- END SSH2 PUBLIC KEY —-” line
  • Deleted the “Comment: …” line
  • Removed all line ends from the generated key
  • Added “ssh-rsa ” before the key
  • Make sure there is an empty line after the key

That’s it.

Flask application factory pattern and testing

This is a project I setup to show the use the application factory pattern in Flask. I set it up because I was stuck while refactoring one of my projects to use the factory pattern. The code would run normally, but my test cases where failing with the following two errors:

RuntimeError: application not registered on db instance and no application bound to current context
RuntimeError: working outside of application context

Most Flask examples don’t use the factory pattern so I spent a lot of time searching around to solve the problem. So I thought I would work it out and share it. Hopefully it saves someone else time.

The problem

Once your project starts to grow, code organization is everything. Flask provides a number of mechanisms for code organization. One of these mechanism’s is blueprints. Combined with the factory pattern provides a nice way to structure and organise code.

Another problem that the factory pattern helps solve is circular dependencies.

Getting the factory pattern to work isn’t hard. Getting it to work correctly, it turned out, was a little harder. The problem I had was caused in the testing code. In the following section I will briefly explain how to setup and use the factory pattern correctly.

Lessons learned

I have added more code than strictly necessary to show the concept of the factory pattern working as a realistic example.

The structure and contents of this example project is:

src
│ .gitignore
│ readme.md
│ manage.py
│ requirements.txt
├── instance
│   sensitive.cfg
├── test_app_factory
│   │ __init__.py
│   │   appliction.py
│   │   config.py
│   │   extensions.py
│   │   models.py
│   ├── helpers
│   │   __init__.py
│   │   misc.py
│   ├── module
│   │   __init__.py
│   │   viws.py
│   ├── static
│   │   favicon-16x16.png
│   │   favicon-32x32.png
│   └── templates
│   index.html
└── tests
    test_basics.py

Okay, what’s important to point out here?

The core of the factory pattern is setup in application.py and extensions.py. All extensions are initialized in extensions.py. If you add additional extensions make sure to add them to the import statement in test_app_factory/__init__.py. This is a convent way to shorten import statements.

The actual heavy lifting is done in application.py. Each part of the application initialization is a separate function, which are called by the main function app_factory. This function takes a string, which specifies the environment the configuration should be loaded for. The configuration is defined in config.py.

The factory pattern in application.py looks like this:

def app_factory(config, name):
app = Flask(...)

...

return app

The function calls a number of functions that load the configuration settings, extensions, blueprints, etc.

Using the factory is really easy, just use the following call:

app = app_factory('TST')

To access the app object in modules after the application has been initialized is done using the proxy provided by Flask:

from flask import current_app as app

Now for the part that was driving met crazy, the testing code. I still do not understand fully why it is the only place in my code that was causing a problem, probably has to do with the way unittest works. Anyway, to get the factory pattern to work you need to add app_context to specific statements. Here is an example.

class TestCase(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        cls.app = app_factory('TST')

    def setUp(self):
        with self.app.app_context():
            self.client = app.test_client()
            db.create_all()

    def tearDown(self):
        with self.app.app_context():
            db.session.remove()
            db.drop_all()

    def test_add_user(self):
        with self.app.app_context():
            db.session.add(User(name='test user', email='test@mail.com'))
            db.session.commit()

Conclusion

Finding good examples isn’t always easy. The factory pattern can really help to organize the code and make it more readable and maintainable.

Any suggestions how I can further improve the code? I would love to hear from you!

Inspecting and replaying requests with ngrok

A very cool feature of ngrok is the web interface. With this interface you can inspect requests and replay them. Very handy for debugging purposes.

The web interface can be reached on http://localhost:4040.

In the web interface you can view both the request received and response sent. The default view of the request and response is a summary of the post data, but it can be viewed in raw format or binary format. This is really useful when debugging webhooks.

Another great feature of this web interface is the replay functionality. Instead of having to run through the code again to get to the problem, you can just hit the resend button and the request will be resent.

Running a specific test in nose2

Today I needed to run a specific nose2 test case to fix an error. This can be achieved the following ways.

My test cases are located in project/tests.

To run all the test cases in a test.

nose2 project.tests.test_file

To run a specific test case in a test.

nose2 project.tests.test_file.TestCase.test_method

That’s it. Happy coding.

Flask error loading config file in instance directory

I have been dealing with a really frustrating problem today. Yesterday my code was working and today it was failed without any code changes.

This is the piece of code causing the problem.

app = Flask(__name__, instance_relative_config=True)
app.config.from_object('config')
app.config.from_pyfile('config.cfg')

And this is the error.

Traceback (most recent call last):
  File "run.py", line 1, in 
    from project import app
  File "C:\Python27\env\vod_test\src\project\__init__.py", line 37, in 
    load_config()
  File "C:\Python27\env\vod_test\src\project\__init__.py", line 33, in load_config
    app.config.from_pyfile('config.cfg')
  File "C:\Python27\env\vod_test\lib\site-packages\flask\config.py", line 129, in from_pyfile
    with open(filename) as config_file:
IOError: [Errno 2] Unable to load configuration file (No such file or directory): 'C:\\Python27\\env\\vod_test\\var\\project-instance\\config.cfg'

Flask is trying to load the config file from C:\\Python27\\env\\vod_test\\var\\project-instance\\config.cfg instead of from C:\\Python27\\env\\vod_test\\project\\instance\\config.cfg. Where is the var coming from? And why is there a hyphen between project and instance?

The strange thing was that the problem only occurred when I was running my test cases from the command line. In PyCharm the code was working. How frustrating. It turned out that PyCharm was using a different virtualenv than the command line. After comparing the output of ‘pip freeze’ I discovered that some libraries had different versions.

To solve the problem I added the following code.

import os
from flask import Flask


def load_instance_config_fix(filename):
    config = {}
    with app.open_instance_resource(filename) as f:
        for line in f:
            if line[0] == "#":
                continue
            name, value = line.partition("=")[::2]
            config[name.strip()] = value.strip()
    return config


def handle_incorrect_instance_path(f):
    def handle_load_instance_config():
        try:
            f()
        except IOError:
            app.instance_path = os.path.join(os.path.abspath(os.curdir), 'instance')
            config = load_instance_config_fix('config.cfg')
            app.config.update(config)

    return handle_load_instance_config


@handle_incorrect_instance_path
def load_config():
    app.config.from_object('config')
    app.config.from_pyfile('config.cfg')


app = Flask(__name__, instance_relative_config=True)

load_config()


@app.route('/')
def hello_world():
    return 'Hello, World!'

Just some nice syntactic sugar with a decorator to keep the code nice and concise.

Another way to solve the problem is by setting the instance_path when initializing Flask. What fun would that be 🙂

import os
from flask import Flask


app = Flask(__name__, instance_path=os.path.join(os.path.abspath(os.curdir), 'instance'), instance_relative_config=True)
app.config.from_object('config')
app.config.from_pyfile('config.cfg')


@app.route('/')
def hello_world():
    return 'Hello, World!'

Back to real coding 🙂

Test API webhooks from localhost

To test the Mollie payment API on my localhost I needed a way to receive the webhook callback on my localhost. To be able to do this I needed a service to tunnel a public URL to localhost (my local machine, as opposed to a publicly accessible URL). Here are few examples of services that provide this service:

ngrok (free plan)

localtunnel (free)

pagekite

forwardhq

These types of tools give you a publicly accessible URL that forwards the request to the localhost port, which can be used for the API callback URL while testing.

Create a new project in PyCharm based on an existing virtualenv

Today I cloned a project from Github and needed to setup a project in PyCharm for it. These are some quick notes how to setup a new project in PyCharm based on an existing virutalenv.

  1. Clone the project with: git clone <URI>.
  2. Open PyCharm.
  3. Create a new project: File -> New project…
  4. Select project type: python
  5. Click create.
  6. Select the root directory of the virtualenv.
  7. Select local type by clicking the button behind the source field.
  8. From the Scripts directory of the virtualenv select python.exe.
  9. Clicke create.
  10. PyCharm will display a message that the directory is not empty (“The directory <…> is not empty. Would you like to create a project from the existing source instead?”).
  11. Click Yes.
  12. Select File -> Settings…
  13. Select Project […] -> Project Structure.
  14. Delete current source directory.
  15. Add the source directory in the virtualenv.
  16. Click OK.

You’re ready to rock. Have fun!

 

Quick test Flask, Flask-Restplus and Flask-Marshmallow

I needed to quickly test a few things today using Flask, Flask-Restplus and Flask-Marshmallow. I created the following boilerplate code that can easily be modified for further quick tests.

from flask import Flask, Blueprint, url_for, jsonify
from flask_restplus import Api, Resource
from flask_marshmallow import Marshmallow, base_fields
from marshmallow import post_dump

# Setup Flask app

url_prefix = '/api/v4'
doc = '/apidoc/'
app = Flask(__name__)
app.config['JSON_SORT_KEYS'] = False
ma = Marshmallow()
blueprint = Blueprint('api', __name__, url_prefix=url_prefix)
api = Api(blueprint, doc=doc, version='3.0')

# Models

class Bunch(dict):
    def __init__(self, *args, **kwargs):
        super(Bunch, self).__init__(*args, **kwargs)
        self.__dict__ = self

class Author(Bunch):
    pass

def mock_author():
    author = Author(id=123, name='Fred Douglass')
    return author

def mock_author_list():
    a1 = Author(id=1, name="Alice")
    a2 = Author(id=2, name="Bob")
    a3 = Author(id=3, name="Carol")
    return [a1, a2, a3]

# Schemas

class AuthorSchema(ma.Schema):
    id = base_fields.Int(dump_only=True)

    absolute_url = ma.AbsoluteURLFor('api.author', id='<id>')

    links = ma.Hyperlinks({
        'self': ma.URLFor('api.author', id='<id>'),
        'collection': ma.URLFor('api.authors')
    })

    @post_dump(pass_many=True)
    def wrap(self, data, many):
        key = 'authors' if many else 'author'
        return {
            key: data
        }

    class Meta:
        fields = ('id', 'name', 'links', 'absolute_url')
        ordered = True

# Setup operations

ns = api.namespace('authors', description='desc')
ep_1 = 'authors'
ep_2 = 'author'

@ns.route('/', endpoint=ep_1)
class AuthorsCollection(Resource):
    def get(self):
        print " ---- GET - result=" + url_for('api.' + ep_1)
        s = AuthorSchema(many=True)
        result = s.dump(mock_author_list())
        return jsonify(result.data)

    def post(self):
        print " ---- POST - result=" + url_for('api.' + ep_1)
        return None, 201

@ns.route('/<int:id>', endpoint=ep_2)
class AuthorsDetail(Resource):
    def get(self, id):
        print " ---- GET - result=" + url_for('api.' + ep_2, id=id)
        print " ---- GET - result=" + url_for('api.' + ep_1)
        s = AuthorSchema()
        result = s.dump(mock_author())
        return jsonify(result.data)

    def put(self, id):
        print " ---- PUT - result=" + url_for('api.' + ep_2, id=id)
        print " ---- PUT - result=" + url_for('api.' + ep_1)
        return None, 204

    def delete(self, id):
        print " ---- DELETE - result=" + url_for('api.' + ep_2, id=id)
        print " ---- DELETE - result=" + url_for('api.' + ep_1)
        return None, 204

app.register_blueprint(blueprint)

if __name__ == '__main__':
    print '>>>>> Starting server at http://localhost:8080{url_prefix}{doc}'.\
        format(url_prefix=url_prefix, doc=doc)
    app.run(port=8080, debug=True)