Articles with the tag python

Use more values

Posted on 2021-11-12

While working with Django ORM very often .only() is used to speed up DB query. Unfortunately is also opens code to some nasty db queries related bugs. Let's for example take simple model:

1
2
3
4
5

class Foo(models.Model):

    field_a = models.CharField(max_length=16)
    field_b = models.CharField(max_length=50)
    field_c = models.BooleanField()

Everything looks good here. Now we have a developer that should implement listing of all the names of model Foo. As they are thinking about performance they use "only" to limit size of the query (Django ORM will query only for the mentioned field).

1
2
3
4
5
6

def some_view():
    ret = []
    foos = Foo.object.all().only("field_a")
    for foo in foos:
        ret.append({"field_name": foo.field_a})
    return ret

And every body is happy. This view is making one query to db that returns only the required data.

But after few weeks new feature is added that need also field_b value. This task is being taken by some other developer. So they jump into code and modify the view to look like this:

1
2
3
4
5
6

def some_view():
    ret = []
    foos = Foo.object.all().only("field_a")
    for foo in foos:
        ret.append({"field_name": foo.field_a, "field_bool": foo.field_b})
    return ret

Code is working. Great work. But because new dev didn't notice only("field_a") or was not experienced enough to know what it does. But the code is working. Test were adjusted to get new return dicts. Things are well. Or are they? Because that simple oversight view that was generating one query to db is generating now n queries, where n is number of Foo instances. And that can get big pretty fast.

"But that should be caught by code review" you can say. Yes. You would be right, but unfortunately we are people. We make mistakes. It's very easy to miss that especially if the queryset is build way earlier in the code.

I think it's better to use values_list or values in such situations. Not only they are faster and take less memory (as instances of Foo model doesn't have to be created), but they are also protect you in a way from making such mistake.

For example this code:

1
2
3
4
5
6

def some_view():
    ret = []
    foos = Foo.object.all().values_list("field_a", "field_b", named=True)
    for foo in foos:
        ret.append({"field_name": foo.field_a, "field_bool": foo.field_b})
    return ret

will be faster and will also cause exception to be thrown in case somebody tries to access .field_c on foo.

PySeaweed update

Posted on 2016-06-12

I've updated my package PySeaweed (and changed it name from PyWeed).

Changes:

• Overall code cleanup
• Added option yo use reqests.Session() for better performance while dealing with huge ammount of files
• Fixed some tests
• Merge in one fork

PySeaweed on PyPi and Sources

Getting package version from code without importing

Posted on 2015-04-07

I use this code for getting version info from package code without importing it, as it has to be installed to be imported and it can't be installed without version number. Classic 'Chicken or the egg' dilema.

I'm using ast module from Python stdlib to parse file from package code. In example I'm using __init__.py but one can use any other file.

It's works with module level variables (only string or number) and it's alternative to using eval() or exec().

setup.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

# coding=utf-8
import os
import ast
from setuptools import setup, find_packages


def get_info(filename):
    info = {}
    with open(filename) as _file:
        data = ast.parse(_file.read())
        for node in data.body:
            if type(node) != ast.Assign:
                continue
            if type(node.value) not in [ast.Str, ast.Num]:
                continue
            name = None
            for target in node.targets:
                name = target.id
            if type(node.value) == ast.Str:
                info[name] = node.value.s
            elif type(node.value) == ast.Num:
                info[name] = node.value.n
    return info

file_with_packageinfo = "packagename/__init__.py"
info = get_info(file_with_packageinfo)

here = os.path.abspath(os.path.dirname(__file__))
# README = open(os.path.join(here, 'README.txt')).read()
# CHANGES = open(os.path.join(here, 'CHANGES.txt')).read()


requires = [
]

setup(name='packagename',
      version=info.get('__version__', '0.0.0'),
      description='Package description',
      # long_description=README + '\n\n' + CHANGES,
      classifiers=[
          "Programming Language :: Python ",
      ],
      author=info.get('__author__', 'Łukasz Bołdys'),
      author_email='[email protected]',
      url='http://dev.utek.pl',
      keywords='',
      packages=find_packages(),
      include_package_data=True,
      zip_safe=False,
      test_suite='packagename',
      install_requires=requires,
      entry_points="""\
      [console_scripts]
          ddd = packagename.scripts.analyze:main
      """,
      )

packagename/__init__.py

1
2
3
4

# coding=utf-8

__version__ = "0.5.1"
__author__ = "Łukasz Bołdys"

Extending setup.py install commands

Posted on 2015-03-11

Lately one of my associates asked if there is a way to use different install_requires lists in setup.py depending on way of installing.

If you are doing python setup.py install you would get one set of requires and doing python setup.pl develop would give you another set of requires (either extended or totally different).

Here's my solution:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

#!/usr/bin/env python
# coding=utf-8

from setuptools import setup, find_packages
from setuptools.command.develop import develop

requires = [
    'fabric',
    'pelican'
]


class ExtendedDevel(develop):
    ''' Adding ipython to requirements '''
    def run(self):
        requires.append('ipython')
        develop.run(self)


setup(name='lalala',
      version='0.0.1',
      description='That\'s stupid',
      long_description='DESCRIPTION',
      classifiers=[
      ],
      author='Łukasz Bołdys',
      author_email='[email protected]',
      url='',
      keywords='',
      license='LICENSE',
      packages=find_packages(),
      include_package_data=True,
      zip_safe=False,
      test_suite='test',
      install_requires=requires,
      cmdclass={'develop': ExtendedDevel}
      )

Using this code in setup.py will install fabric, pelican and ipython when doing python setup.py develop but will only install fabric and pelican if installing via python setup.py install.

If you don't want to change default behavior of develop you can add new command with changing cmdclass={'develop': ExtendedDevel} to cmdclass={'newcommand': ExtendedDevel}.

Ignoring tables in Alembic

Posted on 2013-08-12

While working on spatial enabled application I came up to a problem with spatial table in my postgres database (spatial_ref_sys). Alembic insisted on deleting this table as it wasn't declared in my models.py.

I didn't wanted to define it just to keep Alembic from removing it so I've added following changes to .ini and env.py files:

1
2

[alembic:exclude]
tables = spatial_ref_sys

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

def exclude_tables_from_config(config_):
    tables_ = config_.get("tables", None)
    if tables_ is not None:
        tables = tables_.split(",")
    return tables

exclude_tables = exclude_tables_from_config(config.get_section('alembic:exclude'))

def include_object(object, name, type_, reflected, compare_to):
    if type_ == "table" and name in exclude_tables:
        return False
    else:
        return True

def run_migrations_online():
    if isinstance(engine, Engine):
        connection = engine.connect()
    else:
        raise Exception('Expected engine instance got %s instead' % type(engine))

    context.configure(
        connection=connection,
        target_metadata=target_metadata,
        include_object=include_object
    )

    try:
        with context.begin_transaction():
            context.run_migrations()
    finally:
        connection.close()

WebPy middleware authorization

Posted on 2012-11-12

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

class AuthApp(SimpleHTTPRequestHandler):
    def __init__(self, environ, start_response):
        self.headers = []
        self.environ = environ
        self.start_response = start_response

    def send_response(self, status, msg=""):
        self.status = str(status) + " " + msg

    def send_header(self, name, value):
        self.headers.append((name, value))

    def end_headers(self):
        pass

    def log_message(*a): pass

    def __iter__(self):
        environ = self.environ
        self.send_header('WWW-Authenticate','Basic realm="App authorization"')
        self.send_response(401, "Unauthorized")
        self.start_response(self.status, self.headers)
        yield "Unauthorized"

    AUTH = settings.AUTH

    class AuthMiddleware:
        """WSGI Middleware for authentication."""
        def __init__(self, app):
            self.app = app
        def __call__(self, environ, start_response):
            #allowed = [("test", "test")]
            allowed = AUTH
            auth = environ.get('HTTP_AUTHORIZATION')
            authreq = False
            if auth is None:
                authreq = True
            else:
                auth = re.sub('^Basic ','',auth)
                username,password = base64.decodestring(auth).split(':')
                if (username,password) in allowed:
                    return self.app(environ, start_response)
                else:
                    authreq = True
            if authreq:
                return AuthApp(environ, start_response)

1
2
3
4
5
6
7
8

app = web.application(urls, locals())
app.add_processor(load_sqla)

application = app.wsgifunc(AuthMiddleware) # for WSGI

if __name__ == "__main__":
    pass
    app.run(AuthMiddleware) # for direct execution