aaronoellis.com

Loading CSV Files in Django with psycopg 2 and 3

contact@aaronoellis.com (Aaron O. Ellis) — Sat, 13 Apr 2024 00:00:00 +0000

Recently, I wanted to make use of the Postgres COPY methods within Django. Most people recommended django-postgres-copy, however, as of April 2024, the package only supports psycopg2 and Django 4.2. It also makes use of a temporary table, whereas I felt comfortable enough with copying the values directly to their destination table.

Psycopg2

If you’re using the older version of the psycopg Postgres database adapter, psycopg2, then Django’s underlying database connection cursor will have copy_from, copy_to, and copy_expert methods available. You can read their full API documentation here.

I recommend accessing the cursor using a with statement:

from django.db import connection

with connection.cursor() as cursor:
    print(hasattr(cursor, "copy_from"))

These methods expect files, or a file-like object with read() and readline() methods. For instance, you can upload a file uploaded to Django’s request.FILES.

f = self.request.FILES["file"]
with connection.cursor() as cursor:
    cursor.copy_from(f, "db_table_name")

By default, the method expects the CSV to be tab-separated and have all columns present, including the primary key that is assigned by default to Django models. For instance, the following model class:

class Item(models.Model):
    name = models.CharField(max_length=128)
    amount = models.FloatField(null=True)
    modified = models.DateTimeField(null=True)

Would require a CSV with four columns by default: id, name, amount, modified.

Generally, we want to leave the primary key assignment to the underlying database. Thankfully, we can use the model’s meta attributes to find or dynamically assign these parameters:

columns = [f.name for f in Item._meta.get_fields() if not f.primary_key]
with connection.cursor() as cursor:
    cursor.copy_from(f, Item._meta.db_table, sep=",", columns=columns)

We can also specify null values for the various database columns that support them by using the special \N string. This special string value can also be changed via the null keyword argument.

Accepting user uploads directly to the database is risky behavior. We may want to sanitize or alter the uploaded file in some way. We may even have some other sort of non-file data that we want to copy. To do so, we can dynamically create a file-like csv object:

import csv
from io import StringIO

f = StringIO()
writer = csv.writer(f)
writer.writerow(["A", 1.001, timezone.now()])

f.seek(0)  # Make sure the SQL cursor reads the CSV stream from its start
columns = [f.name for f in Item._meta.get_fields() if not f.primary_key]
with connection.cursor() as cursor:
    cursor.copy_from(f, Item._meta.db_table, sep=",", columns=columns)

Note that the defaults of the Python standard library csv package are different from the copy_from method. The csv package will emit comma separated values by default and will transform None values to empty strings. Sadly, I was unable to find an easy way to change the csv writer to write None as \N.

If you need to support null values with csv, the easiest ways are to either manually write \N values or to change the copy_from method to use empty strings as null:

f = StringIO()
writer = csv.writer(f, dialect=csv.excel_tab)
writer.writerow(["B", 1, None])
writer.writerow(["C", None, timezone.now()])

f.seek(0)  # Make sure the SQL cursor reads the CSV stream from its start
columns = [f.name for f in Item._meta.get_fields() if not f.primary_key]
with connection.cursor() as cursor:
    cursor.copy_from(f, Item._meta.db_table, null="", columns=columns)

But watch out, doing so will prevent you from loading any values that are actually empty strings!

If you can install the pandas package, you can use the above approach to copy pandas.DataFrame objects directly to the database, including the expected null format:

from io import StringIO
import pandas as pd

columns = [f.name for f in Item._meta.get_fields() if not f.primary_key]
df = pd.DataFrame(columns=columns)
df.loc[0] = ["D", 42.0, None]
df.loc[1] = ["E", -0.1, None]

f = StringIO()
df.to_csv(f, sep="\t", na_rep=r"\N", index=False, header=False)

f.seek(0)  # Make sure the SQL cursor reads the CSV stream from its start
with connection.cursor() as cursor:
    cursor.copy_from(f, Item._meta.db_table, columns=columns)

Psycopg Version 3

The latest version of psycopg (which has the package name psycopg, but is technically version 3), moves away from the above file-based methods and instead introduces a single copy method.

f = self.request.FILES["file"]
with connection.cursor() as cursor:
    with cursor.copy(f"COPY table_name (name, amount) FROM STDIN DELIMITER ','") as copy:
        copy.write(f.read())

This new approach allows rows to be written iteratively:

records = [
    ["D", 42.0, None],
    ["E", -0.1, None]
]
with connection.cursor() as cursor:
    with cursor.copy("COPY table_name (name, amount, modified) FROM STDIN") as copy:
        for record in records:
            copy.write_row(record)

I have mixed feelings on having to construct the COPY statement yourself: although highly flexible, it does require some extra coding in the simpliest cases compared to the psycopg2 methods like copy_from.

A few things to be mindful of regardless of your psycopg version:

If you are using additional databases or advanced routing, make sure you have the correct connection for your Django model.
You can specify auto-increment values in your uploads (such as id in the example Item above), but know that the underlying sequence in Postgres will not be updated. You may need to manually update the sequence before resuming normal inserts on your table.
Similarly, identical or repeat uploads with manually specified auto-increment primary keys will raise UniqueViolation exceptions and cause the copy to be canceled.
Also regarding auto-increment values: errors during copy will result in the operation being canceled, but any auto-increment values created before the error will be discarded.
Using copy methods will not trigger Model signals, such as pre_save and post_save.

Using copy has a significant speed improvement over Django’s bulk_create, which itself is already an improvement over saving individual models. Here are the results of some rudimentary tests on my M1 MacBook Pro (total elapsed time in seconds):

rows	save()	save() in transaction	bulk_create()	copy_from()
1,000	0.69	0.30	0.10	0.08
10,000	2.26	1.25	0.36	0.10
100,000	22.35	9.41	2.81	0.39

Happy hacking!

Postgres Full Text Search on Django JSON Fields with KT Expressions

contact@aaronoellis.com (Aaron O. Ellis) — Sun, 09 Apr 2023 00:00:00 +0000

The Django developers have worked hard to integrate Postgres full text search into the web framework’s ORM. Unfortunately, there are few quirks with the system, especially when working with JSON model fields. In this post, we’ll discuss those issues and how to work around them.

Search Lookup Versus SearchVector

If you have a simple Document model class, such as:

class Document(models.Model):
    title = models.CharField(default="")
    body = models.CharField(default="")

You can perform a basic full text search of body with:

Document.objects.filter(body__search="Example")

Which produces the SQL:

SELECT "document"."id", "document"."title", "document"."body"
FROM "document"
WHERE to_tsvector(COALESCE("document"."body", '')) @@ (plainto_tsquery('Example'));

The above will match any document containing the case insensitive word stem exampl. This includes words such as examples and EXAMPLING.

To perform the above query, you’ll need to be using Postgres as your database. You also need to add django.contrib.postgres to your INSTALLED_APPS or you will receive a django.core.exceptions.FieldError: Unsupported lookup 'search' error message when performing the above query.

To improve search results, you can implement ranking, weighting, and different languages using SearchVector:

title_vector = SearchVector("title", config="French", weight="A")
body_vector = SearchVector("body", config="French", weight="B")
query = SearchQuery("Exemple", config="French")
results = (
    Document.objects.annotate(
        rank=SearchRank(title_vector + body_vector, query)
    )
    .filter(rank__gte=0.1)
    .order_by("-rank")
)

If you’re changing the language, make sure to set the config for both SearchVector and SearchQuery. More features and examples are shown in the Django documentation for full text search.

JSON Fields

The previous examples used a model with CharField content. We could also put all those fields into a JSONfield, allowing us to extend the Document without additional migrations:

class Document(models.Model):
    content = models.JSONField(default=dict)

For instance, by adding an author field:

Document.objects.create(
    content={
        "title": "Example Title",
        "author": "Myself",
        "body": "The body of text that we would want to search.",
    }
)

These fields can be filtered by using a double underscore __ path lookup such as:

Document.objects.filter(content__author="Myself")

The ORM translates this to a query using the -> operator, which extracts the jsonb object at the given key, and by casting the filter term to jsonb before testing for a match:

SELECT "document"."id", "document"."content"
FROM "document"
WHERE ("document"."content" -> 'author') = '"Myself"'::JSONB;

Unfortunately, this lookup uses the same double underscore syntax as the first search lookup we used before. Attempting a basic full text search with search results in a nested JSON lookup using the #> operator instead of the full text lookup we expected:

Document.objects.filter(content__body__search="Example")

SELECT "document"."id", "document"."content"
FROM "document"
WHERE ("document"."content" #> '{body,search}') = '"Example"'::JSONB;

Thankfully, the above can be fixed with Django 4.2’s newly introduced KT expression.

Document.objects.annotate(body=KT("content__body")).filter(body__search="Example")

A bit unwieldy, but it generates the SQL we expect:

SELECT "document"."id", "document"."content",
       ("document"."content" ->> 'body') AS "body"
FROM "document"
WHERE to_tsvector(coalesce(("document"."content" ->> 'body'), '')) @@ (plainto_tsquery('Example'));

Bad Tokenization

When using SearchVector, it may appear that the KT expression isn’t needed:

title_vector = SearchVector("content__title", weight="A")
body_vector = SearchVector("content__body", weight="B")
query = SearchQuery("Example")
results = (
    Document.objects.annotate(
        rank=SearchRank(title_vector + body_vector, query)
    )
    .filter(rank__gte=0.1)
    .order_by("-rank")
)

The above may even return the results you expect, but there is a subtle bug hiding in its SQL:

SELECT "document"."id", "document"."content",
ts_rank((setweight(
        to_tsvector(coalesce((("document"."content" -> 'title'))::text, '')), 'A') || setweight(
        to_tsvector(coalesce((("document"."content" -> 'body'))::text, '')), 'B')), plainto_tsquery('Example')) AS "rank"
FROM "document"
WHERE ts_rank((setweight(
    to_tsvector(coalesce((("document"."content" -> 'title'))::text, '')), 'A') || setweight(
    to_tsvector(coalesce((("document"."content" -> 'body'))::text, '')), 'B')), plainto_tsquery('Example')) >= 0.1
ORDER BY 3 DESC;

The SearchVector is generating its expression using the -> operator, as opposed to the KT expression’s ->>. The -> operator will return jsonb, which then needs to be cast to text in order to be used in to_tsvector, while ->> directly returns a text type.

These are not equivalent operations, as you can see from the following example:

SELECT to_tsvector(E'A string of \ntext with a newline');
          to_tsvector
--------------------------------
 'newlin':7 'string':2 'text':4

SELECT to_tsvector('"A string of \ntext with a newline"'::jsonb::text);
           to_tsvector
---------------------------------
 'newlin':7 'ntext':4 'string':2

If you’re using SearchVector with a JSONField, it’s possible that you’ve been generating the wrong stem words. Thankfully, we can also correct this with KT expressions:

vector = SearchVector("title", weight="A") + SearchVector("body", weight="B")
query = SearchQuery("Example")

results = (
    Document.objects.annotate(
        title=KT("content__title"),
        body=KT("content__body"),
        rank=SearchRank(vector, query),
    )
    .filter(rank__gte=0.1)
    .order_by("-rank")
)

Which produces SQL that uses the ->> operator:

SELECT "document"."id", "document"."content",
("document"."content" ->> 'title') AS "title",
("document"."content" ->> 'body') AS "body",
ts_rank((setweight(
    to_tsvector(COALESCE(("document"."content" ->> 'title'), '')), 'A') || setweight(
    to_tsvector(COALESCE(("document"."content" ->> 'body'), '')), 'B')),
plainto_tsquery('Example')) AS "rank"
FROM "document"
WHERE ts_rank((setweight(
    to_tsvector(COALESCE(("document"."content" ->> 'title'), '')), 'A') || setweight(
    to_tsvector(COALESCE(("document"."content" ->> 'body'), '')), 'B')), plainto_tsquery('Example')) >= 0.1
ORDER BY 5 DESC;

Other Common Mistakes

Even with the correct syntax, you may be disappointed with your results for certain search terms. In addition to creating stem words, the default english dictionary will drop commonly used words, also known as stop words. For instance:

SELECT to_tsvector('english', 'A string with some of the example stop words');
               to_tsvector
-----------------------------------------
 'exampl':7 'stop':8 'string':2 'word':9

For more info on stem and stop words, use ts_debug:

SELECT * FROM ts_debug('english', 'A string with some of the example stop words');

   alias   |   description   |  token  |  dictionaries  |  dictionary  | lexemes
-----------+-----------------+---------+----------------+--------------+----------
 asciiword | Word, all ASCII | A       | {english_stem} | english_stem | {}
 blank     | Space symbols   |         | {}             |              |
 asciiword | Word, all ASCII | string  | {english_stem} | english_stem | {string}
 blank     | Space symbols   |         | {}             |              |
 asciiword | Word, all ASCII | with    | {english_stem} | english_stem | {}
 blank     | Space symbols   |         | {}             |              |
 asciiword | Word, all ASCII | some    | {english_stem} | english_stem | {}
 blank     | Space symbols   |         | {}             |              |
 asciiword | Word, all ASCII | of      | {english_stem} | english_stem | {}
 blank     | Space symbols   |         | {}             |              |
 asciiword | Word, all ASCII | the     | {english_stem} | english_stem | {}
 blank     | Space symbols   |         | {}             |              |
 asciiword | Word, all ASCII | example | {english_stem} | english_stem | {exampl}
 blank     | Space symbols   |         | {}             |              |
 asciiword | Word, all ASCII | stop    | {english_stem} | english_stem | {stop}
 blank     | Space symbols   |         | {}             |              |
 asciiword | Word, all ASCII | words   | {english_stem} | english_stem | {word}

If you need to search strings with relevant stop words, you can also use the built-in simple dictionary:

SELECT to_tsvector('simple', 'A string with some of the example stop words');
                                   to_tsvector
----------------------------------------------------------------------------------
 'a':1 'example':7 'of':5 'some':4 'stop':8 'string':2 'the':6 'with':3 'words':9

Note that this also disables stemming, so stem variants, such as plurals and verbs, will no longer match. For more information on dictionaries, including how to create your own, see the Postgres documentation.

You can see a full list of available dictionaries on your Postgres installation with \dFd and the parameter you’d need to pass to Django’s config option with SELECT cfgname FROM pg_ts_config;

Happy hacking!

Fixing the Postgres Query Planner with enable_seqscan = off

contact@aaronoellis.com (Aaron O. Ellis) — Sat, 25 Feb 2023 00:00:00 +0000

If you’re querying a table with a small number of rows in Postgres, your query will most likely use a “sequential scan”, which scans the entirety of the table to find your result. This is true even when tables have an index for the field or fields you’re querying.

Take for instance a small table with datetime and float columns, such as:

CREATE TABLE "items" (
    "timestamp" TIMESTAMP WITH TIME ZONE,
    "value" FLOAT
);
CREATE INDEX ON "items" ("timestamp");

You can see the query plan chosen by Postgres with EXPLAIN ANALYZE. For instance, with the query:

EXPLAIN ANALYZE SELECT * FROM "items" WHERE "timestamp" = '2023-02-24 12:44:27';

When not using the index:

                                           QUERY PLAN
-------------------------------------------------------------------------------------------------
 Seq Scan on items  (cost=0.00..18.50 rows=1 width=16) (actual time=0.277..0.279 rows=0 loops=1)
   Filter: ("timestamp" = '2023-02-24 12:44:27-07'::timestamp with time zone)
   Rows Removed by Filter: 1000
 Planning Time: 0.839 ms
 Execution Time: 0.325 ms

With the index:

                                                         QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------
 Index Scan using items_timestamp_idx on items  (cost=0.28..8.29 rows=1 width=16) (actual time=0.010..0.011 rows=0 loops=1)
   Index Cond: ("timestamp" = '2023-02-24 12:44:27-07'::timestamp with time zone)
 Planning Time: 0.277 ms
 Execution Time: 0.036 ms

On my current system, the threshold for using the index is around 800 rows. Afterwards, the Postgres query planner will choose the index rather than performing a sequential scan of the table. Even with only a few thousand rows, the performance gain from the index is significant.

A common mistake when using indexes is to write a query that applies a function to a column, when only the column itself is indexed. The above examples are querying a full timestamp, but what if we wanted all results for the day (UTC of course)? Naively, a programmer might write the following query:

SELECT * FROM "items" WHERE DATE("timestamp" AT TIME ZONE 'utc') = '2023-02-24';

But even with the index and 10,000 rows in the table, the query planner still performs a sequential scan:

                                              QUERY PLAN
------------------------------------------------------------------------------------------------------
 Seq Scan on items  (cost=0.00..235.00 rows=50 width=16) (actual time=0.328..4.883 rows=1609 loops=1)
   Filter: (date(timezone('utc'::text, "timestamp")) = '2023-02-24'::date)
   Rows Removed by Filter: 8391
 Planning Time: 0.377 ms
 Execution Time: 5.273 ms

There are several ways to fix the above problem, including querying with a range of timestamps that will use the existing index, but you can also create a new index for the function’s result:

CREATE INDEX ON "items" (DATE("timestamp" AT TIME ZONE 'utc'));

Which will be used by our previous query (and significantly improve performance):

                                                        QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on items  (cost=4.67..65.89 rows=50 width=16) (actual time=0.111..0.541 rows=1609 loops=1)
   Recheck Cond: (date(timezone('utc'::text, "timestamp")) = '2023-02-24'::date)
   Heap Blocks: exact=55
   ->  Bitmap Index Scan on items_date_idx  (cost=0.00..4.66 rows=50 width=0) (actual time=0.082..0.083 rows=1609 loops=1)
         Index Cond: (date(timezone('utc'::text, "timestamp")) = '2023-02-24'::date)
 Planning Time: 0.120 ms
 Execution Time: 0.722 ms

Recently, I came across a query that had made the above mistake. The query was taking over 52 seconds to scan over all 3.2 million rows in the table. However, after adding the correct index, the query was still performing a sequential scan.

It’s in these moments that a programmer begins to doubt their sanity, and question everything they’ve ever learned. But in this case, I hadn’t made a mistake, the Postgres query planner was wrong. The un-indexed query had been used for so long that the internal statistics indicated that a sequential scan should be used even with the new index present.

Thankfully, Postgres provides ways to manually alter the query planner. The documentation is clear that these should be temporary measures. In this case, since I knew the index should be more performant than a sequential scan, I ran SET enable_seqscan = off; for my current session and re-ran the query, which immediately made use of the new index. The new execution time? 2.1 milliseconds.

The Postgres documentation actively discourages use of these parameters, calling them “crude”. The documentation suggests running ANALYZE manually with an increased default_statistics_target value or by increasing the amount of collected statistics.

But at least this time, I was right.

Using Postgres Schemas

contact@aaronoellis.com (Aaron O. Ellis) — Wed, 08 Jun 2022 00:00:00 +0000

Schemas are a shockingly underappreciated feature of relational databases. With minimal overhead, they enable segmentation and isolation of tables within a database.

If you are interested in using schemas with Postgres, I recommend starting with the official documentation’s fantastic introduction. Although I will reiterate some of the introduction below, I will mainly focus on the exceptional behavior and issues that arise when using schemas.

For starters, if you are a Postgres user, then you are already using a schema! By default, Postgres places tables into the public schema. The hierarchy is simple: databases can have multiple schemas, and schemas can have multiple tables:

Or working backwards: a table belongs to one and only one schema and a schema belongs to one and only one database. Different schemas, however, can have tables with the same name:

CREATE SCHEMA "private";
CREATE TABLE "private"."example" (id SERIAL PRIMARY KEY);
CREATE TABLE "public"."example" (id SERIAL PRIMARY KEY);

When querying with the unqualified table name example, precedence will be determined by the user’s search_path, which you can view with SHOW search_path;

       search_path
--------------------------
 "$user", public, private

You can alter the search_path for the current user’s session with:

SET search_path TO "$user", "private", "public";

Note that this is only for the current session. Starting a new session, for instance by re-connecting to the database, will reset the search_path to the user’s default. To permanently set the default search_path, you’ll need to alter the user’s role:

ALTER ROLE postgres SET search_path = "$user", "private", "public";

But the above will only take affect when a new session is started! To set the search_path on the current session and all future sessions you’ll have to run both:

SET search_path TO "$user", "private", "public";
ALTER ROLE postgres SET search_path = "$user", "private", "public";

Additionally, any changes made to search_path in a transaction will be ignored if the transaction is rolled back.

Fully Qualified Names

You can avoid issues with search_path by using a table’s fully qualified name with the schema and tables names separated by a .:

INSERT INTO private.example DEFAULT VALUES;

When quoting names, the schema and table should be quoted separately:

INSERT INTO "private"."example" DEFAULT VALUES;

Quoting the entire qualified name in our current example will error:

INSERT INTO "private.example" DEFAULT VALUES;

ERROR:  relation "private.example" does not exist

Note that . is a perfectly valid character for names as long as it is quoted, but also an endless source of madness:

CREATE TABLE "private"."private.example" (id SERIAL PRIMARY KEY);

You can set a user’s search_path to an empty string to force queries to use these fully qualified names. Note that this includes the \d command and its ilk: you’ll need to either use a wildcard or specify a schema to return results, such as \d public.*.

If you don’t know a table’s schema, a variety of commands will display it, including \dt in the left-most column:

 Schema  | Name    | Type  |  Owner
---------+---------+-------+----------
 private | example | table | postgres
 public  | example | table | postgres

You can also list all schemas in a database with \dn:

  Name   |  Owner
---------+----------
 private | postgres
 public  | postgres

Or query every table in a schema with:

SELECT table_name FROM information_schema.tables WHERE table_schema = 'private';

Privileges

Like all objects in Postgres, schemas will be owned by the role that created them unless otherwise specified. The following will create a schema with a different owner:

CREATE USER website WITH PASSWORD 'secret';
CREATE SCHEMA orm AUTHORIZATION website;

The above will make the website user the owner of the orm schema, allowing it to create objects in that schema. Remember to update the search_path for each user, otherwise commands with unqualified names may return unexpected results.

But just because website is the owner of the orm schema, that doesn’t make it the owner of every object in the schema! When connected as the postgres superuser (or any user other than website that has privileges to create in the orm schema) we can create a table:

CREATE TABLE orm.privileged (id SERIAL PRIMARY KEY);

And the website user will be unable to read from it:

SELECT * FROM orm.privileged;

ERROR:  permission denied for table privileged

Running \dt reveals the issue:

 Schema |     Name      | Type  |  Owner
--------+---------------+-------+----------
 orm    | example       | table | website
 orm    | privileged    | table | postgres

In order to use this new table, the postgres user must grant access to the website user:

GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA orm TO website;

The above grants all privileges. For specific privileges, please refer to the full list in the Postgres documentation.

We can confirm that these new privileges have been set with the \z command:

 Schema |         Name         |   Type   |     Access privileges     
--------+----------------------+----------+---------------------------
 orm    | privileged           | table    | postgres=arwdDxt/postgres+
        |                      |          | website=arwdDxt/postgres
 orm    | privileged_id_seq    | sequence |

There is still an issue with the website user’s privileges, however, which becomes apparent when we attempt an insert:

INSERT INTO orm.privileged DEFAULT VALUES;

ERROR:  permission denied for sequence privileged_id_seq

As we can see from the output of the \z command above, the website user doesn’t have access to the sequence that generates the primary key! The postgres user also needs to grant access to it with:

GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA orm TO website;

And even though we’ve granted all privileges to all tables and sequences currently in the orm schema, any subsequent objects created by the postgres user will need the above commands repeated before they can be accessed by the website user.

Thankfully, Postgres has the following commands that will grant all privileges to the website user for all future tables and sequences created by the postgres user in the orm schema:

ALTER DEFAULT PRIVILEGES FOR USER postgres IN SCHEMA orm GRANT ALL ON TABLES TO website;
ALTER DEFAULT PRIVILEGES FOR USER postgres IN SCHEMA orm GRANT ALL ON SEQUENCES TO website;

Debugging

Schemas are a powerful feature that can provide segmentation and isolation to a database with minimal overhead. Their quirks, however, can cause a variety of frustrating errors, such as users being unable to modify, access, or even know that their tables exist. If you’re starting with schemas, remember:

Update your search_path - order matters!
Know the owner of the schema and its objects: use the \dn+ and \dt commands
Check access privileges for objects with the \z command and update as needed

Happy hacking!

Aggregation with django-filter via Proxy Models

contact@aaronoellis.com (Aaron O. Ellis) — Thu, 26 May 2022 00:00:00 +0000

The django-filter package adds a quick way to add URL parameter driven filtering to existing Django database models. For example, take the following model:

from django.db import models

class Purchase(models.Model):
    item = models.TextField()
    amount = models.FloatField()
    at = models.DateTimeField()

    class Meta:
        db_table = "purchases"

We can then use django-filter to quickly create a FilterSet class that wraps the above database model:

import django_filters
from .models import Purchase

class PurchaseFilter(django_filters.FilterSet):
    item = django_filters.CharFilter(lookup_expr="iexact")
    sort = django_filters.OrderingFilter(
        fields=(
            ("item", "item"),
            ("amount", "amount"),
            ("at", "at"),
        ),
    )

    class Meta:
        model = Purchase
        fields = ("amount", "at")

The above can then be integrated into views, such as ListView:

from django.views.generic.list import ListView
from .filters import PurchaseFilter

class Purchases(ListView):
    def get_queryset(self, **kwargs):
        return PurchaseFilter(self.request.GET).qs

This configuration allows GET parameters to control the filtering and ordering of database queries in views with a minimal amount of glue code. For instance, if the above Purchases view is at the /purchases URL, requesting /purchases?item=lunch&sort=amount will generate the following SQL:

SELECT "purchases"."id", "purchases"."item", "purchases"."amount", "purchases"."at"
FROM "purchases"
WHERE "purchases"."item" LIKE 'lunch'
ORDER BY "purchases"."amount";

The FilterSet can be extended by overriding its qs property. For instance, to aggregate the results by item and sum their amounts:

class AggregatedPurchaseFilter(django_filters.FilterSet):
    item = django_filters.CharFilter(lookup_expr="iexact")
    sort = django_filters.OrderingFilter(
        fields=(
            ("item", "item"),
            ("total", "total"),
        ),
    )

    @property
    def qs(self):
        return super().qs.values("item").annotate(total=models.Sum("amount"))

    class Meta:
        model = Purchase
        fields = ("item",)

The above produces a familiar GROUP BY statement in SQL:

SELECT "purchases"."item", SUM("purchases"."amount") AS "total"
FROM "purchases"
GROUP BY "purchases"."item";

However, when attempting to sort the filter by total using the URL sort parameter, the following error is returned:

FieldError
Cannot resolve keyword 'total' into field. Choices are: amount, at, id, item

Order Matters

This error originates from the Django ORM query planner when the parent FilterSet class attempts to add ordering to the queryset. In brief, the query planner is being told to order by a field that has not yet been created! We can reproduce the same error without django-filter using the following query:

Purchase.objects.order_by("total").values("item").annotate(total=models.Sum("amount"))

Developers familiar with SQL might consider this an obvious mistake. Of course order matters! SQL is fairly strict about its syntax and placing an ORDER BY clause before GROUP BY would result in a syntax error. The Django ORM, however, abstracts query construction, and in our examples above, that construction is spread across several classes and their sub-classes.

To fix our FieldError issue, we need to aggregate the queryset before it is passed to the FilterSet. Thankfully, the FilterSet includes an optional queryset parameter that does exactly this:

queryset = Purchase.objects.values("item").annotate(total=models.Sum("amount"))
PurchaseFilter(request.GET, queryset=queryset)

Which produces the expected SQL without issue:

SELECT "purchases"."item", SUM("purchases"."amount") AS "total"
FROM "purchases"
GROUP BY "purchases"."item"
ORDER BY "total";

Proxy Models

Unfortunately, having to filter the queryset before initializing the FilterSet adds a considerable amount of glue code to our application and prevents us from writing views with modular filters that can be specified by class.

Thankfully, we can also aggregate our queryset before the FilterSet via proxy models and custom managers. We can add the following alongside our Purchase class:

class AggregatedPurchaseManager(models.Manager):
    def get_queryset(self):
        return super().get_queryset().values("item").annotate(total=models.Sum("amount"))


class AggregatedPurchase(Purchase):
    objects = AggregatedPurchaseManager()

    class Meta:
        proxy = True

The above proxy model allows us to change the default queryset without additional database tables. We also no longer need to extend the FilterSet queryset or pass the filtered queryset to it during initialization. We only need to change its model property to our proxy:

class AggregatedPurchaseFilter(django_filters.FilterSet):
    item = django_filters.CharFilter(lookup_expr="iexact")
    sort = django_filters.OrderingFilter(
        fields=(
            ("item", "item"),
            ("total", "total"),
        ),
    )

    class Meta:
        model = AggregatedPurchase
        fields = ("item",)

By using the above FilterSet in our view, we can now order by the aggregated field via URL parameters such as /purchases?sort=total.

Python datetime.utcnow() Considered Harmful

contact@aaronoellis.com (Aaron O. Ellis) — Mon, 14 Jun 2021 00:00:00 +0000

- An update on November 20, 2023

Success! The datetime.utcnow function is deprecated in Python 3.12 along with datetime.utcfromtimestamp. It may not be soon, but one day, these functions will be gone from maintained codebases.

The Python datetime.utcnow() classmethod is harmful and should be replaced with:

from datetime import datetime, timezone
datetime.now(timezone.utc)

Naive Time Traveling

What makes utcnow() so dangerous? Although the function gives the correct time in the UTC timezone, the returned datetime does not include a timezone:

from datetime import datetime
now = datetime.utcnow()
now.tzinfo is None # True

These timezone-less datetime instances are referred to as naive, and a naive datetime can have unexpected behavior. For instance, the timestamp() method of a datetime instance will assume that any naive datetime is local time! Although you might not notice this difference on a server set to UTC time, any timestamps will be off by your local time’s UTC offset. Here’s the offset in the US/Mountain timezone:

from datetime import datetime
datetime.now(timezone.utc).timestamp() - datetime.utcnow().timestamp() # -21600

That puts the utcnow timestamp 6 hours into the future! If you’re interacting with a service that expects a UTC-based unix timestamp, you’re likely to encounter undefined behavior. Since the naive UTC datetime is assuming that it’s local, timezones with a negative offset will generate timestamps in the future and timezones with positive offsets will be in the past.

The lack of timezone also makes comparing now() and utcnow() unpredictable. In the US/Mountain timezone, a datetime generated after another can report that it was generated before!

from datetime import datetime
in_the_past = datetime.utcnow()
datetime.now() < in_the_past # True

Although the Python documentation is clear that this function should be avoided, it has yet to be marked as deprecated, even in the upcoming 3.10 release. So double check your code!

By the way, if you want a unix timestamp, I recommend the succinct time.time():

import time
time.time() # 1623721105.8261912

New York Times Misleads on COVID-19 Cases in Small U.S. Counties

contact@aaronoellis.com (Aaron O. Ellis) — Mon, 26 Oct 2020 00:00:00 +0000

Last week, the New York Times published an excellent series of visualizations tracking the COVID-19 outbreak in rural areas across the United States. At the very end of the article, however, they added a misleading table of the counties with the highest average daily case rates. The intent was to show that the most recent cases are hitting rural areas harder than urban and suburban areas of the country, and they did so by highlighting the counties with less than 10,000 people.

In reaction, Professor Scott E. Page of the University of Michigan tweeted:

Duh? Of course the counties with the highest per capita outbreaks have small populations. So do the counties with the lowest per capita rates. This is basic statistics! ⁦@nytimes⁩ pic.twitter.com/QJIlyZJ5ay
— Scott Page (@Scott_E_Page) October 25, 2020

Professor Page is exactly correct. Unfortunately, while his ire is appropriate when directed at the New York Times - they should know better - a more detailed explanation is warranted for the underlying statistics.

But first, a disclaimer: I am merely demonstrating a property of statistics using COVID-19 data as a background. The spread of COVID-19 is far from random, and cases should never be treated as a simplistic probability distribution.

A Simulation

The U.S. Census Bureau maintains an estimate of county populations, most recently for 2019. The 2020 data will no longer be an estimate, as the census was taken this year, but the data is still being tabulated.

Let’s take a look at this dataset using pandas:

import pandas
df = pandas.read_csv('co-est2019-alldata.csv', encoding='ISO-8859-1')
df.shape
# (3193, 164)

counties = df[df['SUMLEV'] == 50]
counties = counties.filter(['CTYNAME', 'STNAME', 'POPESTIMATE2019'])
counties.rename(columns={"CTYNAME": "County", "STNAME": "State", "POPESTIMATE2019": "Population"}, inplace=True)
counties.set_index(['County', 'State'], inplace=True)
counties.shape
# (3142, 1)

The dataset contains a lot of information, including population estimates for prior years and state-level aggregations. Narrowing the dataset down, we can determine some basic statistics:

3,142 total counties
Total population of 328,239,523
The least populated is Kalawao County, Hawaii, with 86 people
The most populated is Los Angeles County, California, with 10,039,107 people
The counties have a mean population of 104,468 and median of 25,726 indicating a heavy right-hand skew
There are 718 counties - about 23% - with less than 10,000 people

Shown visually in a histogram with a bar for each 10,000 people:

When this was written, there had been 468,793 new COVID-19 cases in the United States in the past week, which is about 0.1428% of the total population. That is a daily average of 66,970 - or 2.04 daily cases per 10,000 people.

For our simulation, we’ll distribute these cases randomly throughout the United States. Once again: this is not how COVID-19 spreads; this is to simply illustrate a property of statistics.

import random
sample = random.sample(range(328239523), 468793)

We can then create bins that represent each county:

by_population = counties.sort_values(by=['Population'])
last = 0
intervals = []
for value in by_population['Population'].values:
    intervals.append((last, last+value))
    last += value
bins = pandas.IntervalIndex.from_tuples(intervals)

And place the sample set into those bins, determining an average daily case rate per 10,000 people:

out = pandas.cut(sample, bins, right=False)
by_population['Weekly Cases'] = out.value_counts().values
by_population['Avg Daily/10k'] = by_population['Weekly Cases'] / by_population['Population'] * 10000 / 7.0
by_population.sort_values(by=['Avg Daily/10k'], ascending=False, inplace=True)

Here’s the 20 counties with the highest case rates from our simulation - with full names redacted so no one confuses this for real data. Notice anything interesting?

County	State	Population	Weekly Cases	Avg Daily/10k
L* County	Nebraska	664	4	8.61
L* County	Texas	169	1	8.45
S* Municipality	Alaska	1,183	6	7.24
H* County	New Mexico	625	3	6.86
H* County	Nebraska	682	3	6.28
G* County	Texas	1,409	6	6.08
K* County	Texas	762	3	5.62
P* County	Montana	1,077	4	5.31
G* County	Washington	2,225	8	5.14
A* County	California	1,129	4	5.06
Q* County	Georgia	2,299	8	4.97
L* County	Texas	3,233	11	4.86
C* County	Montana	1,252	4	4.56
P* County	West Virginia	8,247	26	4.50
S* County	South Dakota	6,376	20	4.48
W* County	North Dakota	3,834	12	4.47
F* County	Nebraska	2,979	9	4.32
P* County	West Virginia	6,969	20	4.10
C* County	Oklahoma	2,137	6	4.01
F* County	North Dakota	3,210	9	4.01

Shown visually, with counties that have less than 10,000 people in orange:

Note that the striations are caused by case counts being discrete and not continuous - you can’t have half a case!

So, our simulation shows that smaller population counties have both the highest and lowest case rates, proving Professor Page’s point.

But Why?

Let’s look at the county that had the highest case rate in our simulation. With a population of just 664, in order to have greater than the national average of of 2.04 daily cases per 10,000 people, the county would only need to have a single case the entire week! We can determine the chance of that occurring in our simulation using a binomial distribution given that there was a 0.1428% of any resident having a case in the preceding week:

from scipy.stats import binom

p = binom(664, 0.001428)
binom(664, 0.001428).pmf(1)
# 0.3676444783149298

So there was about a 36.8% percent chance of the county having one new case during the week. Once again, this is possible because our cases are randomly determined - in reality, cases are far from random.

But that’s the probability of exactly one case. We can find the probability of one or more cases with:

1 - binom(664, 0.001428).pmf(0)
# 0.6128215783303926

Which increases the chance that this county would have higher than the national average of cases to 61.3%.

And the chance of 4 or more cases?

rv = binom(664, 0.001428)
1 - (rv.pmf(0) + rv.pmf(1) + rv.pmf(2) + rv.pmf(3))
# 0.01589406080799649

So there was a 1.6% chance of this county having more than 4 times the national average!

To represent all these likelihoods visually:

To have that same rate, the largest county in America, Los Angeles, would have to have 60,476 cases or more in a week. And what is the probability of that occurring if the cases are random? Once again we can use the binomial distribution:

rv = binom(10039107, 0.001428)
1 - sum([rv.pmf(x) for x in range(0, 60477)])
# 1.2370320878751784e-08

Or about 0.000001237%. In actuality, Los Angeles had 10,801 cases over the previous week, which was below the 14,335 we’d expect if the county had the national case rate.

The more sample points we have, the easier it is to determine if a conclusion is significant. And that’s what the New York Times forgot: if you’re going to compare small populations to large populations, then naturally your small populations will be more varied to the extremes - both high and low!

Outbreak Likelihoods

What if we can use these likelihoods to re-examine the original question: are recent COVID-19 cases hitting rural areas harder? Thankfully, the New York Times does an excellent job in regards to data provenance, and uploaded the case counts used in the original article to a GitHub repository.

Let’s create a table with the counties that have the least likely case rates above the national average:

County	State	Weekly Cases	Population	Likelihood	Avg Daily/10k
Norton	Kansas	279	5,361	~0%	74.35
Grand Forks	North Dakota	699	69,451	~0%	14.38
Burleigh	North Dakota	925	95,626	~0%	13.82
Dodge	Wisconsin	836	87,839	~0%	13.60
Sheboygan	Wisconsin	1,027	115,340	~0%	12.72
Marathon	Wisconsin	1,125	135,692	~0%	11.84
Winnebago	Wisconsin	1,339	171,907	~0%	11.13
El Paso	Texas	6,494	839,238	~0%	11.05
Outagamie	Wisconsin	1,451	187,885	~0%	11.03
Fond du Lac	Wisconsin	783	103,403	~0%	10.82
Minnehaha	South Dakota	1,312	193,134	~0%	9.70
Mobile	Alabama	2,632	413,210	~0%	9.10
Brown	Wisconsin	1,626	264,542	~0%	8.78
Cass	North Dakota	1,075	181,923	~0%	8.44
Lubbock	Texas	1,510	310,569	~0%	6.95
Waukesha	Wisconsin	1,789	404,198	~0%	6.32
Milwaukee	Wisconsin	4,145	945,726	~0%	6.26
Utah	Utah	2,394	636,235	~0%	5.38
Salt Lake	Utah	4,078	1,160,437	~0%	5.02
Cook	Illinois	11,597	5,150,233	~0%	3.22

This table is not a ranking by likelihood, since all these counties have likelihoods approaching zero! These zero values are a strong indication that the random distribution we’ve chosen for our model is completely inappropriate. We’ve also run into an unexpected limit: the probabilities are so small that they’ve reached the minimum supported float value of our computer.

We can visualize these likelihoods for all counties, with those with less than 10,000 people once again in orange:

In the above visualization, the counties with case rates near the national average will be in the center of the y-axis. Counties with rates higher than expected will be towards the top of the y-axis, and counties with lower rates towards the bottom.

We could also re-examine what it means for a county to be rural. Instead of using a population limit, we can use the definition of the Federal Office of Rural Health Policy: a rural county is any county that is not part of a metropolitan area.

We can create a new visualization, with our newly defined rural counties still in orange:

It’s difficult to draw any conclusion from this visualization. While there appears to be more rural counties with likelihoods above the national average and urban counties below it, a more rigorous numerical analysis is needed. If anything, we’ve just introduced new biases by using a naive model.

Perhaps with a more appropriate probability distribution these results could have more meaning, but our analysis will always suffer from the original mistake committed by the New York Times: in order to make definitive conclusions about the spread of COVID-19 we need more sample points, and in the case of a county-by-county analysis the smaller population counties will always have less. Forgetting this underlying property of statistics led them to publish a misleading table of data.

Allow Whitespace to be a Valid CharField Value in Django Admin

contact@aaronoellis.com (Aaron O. Ellis) — Mon, 12 Oct 2020 00:00:00 +0000

Let’s attempt to create a basic Django model that allows whitespace as a valid value. Here’s an example models.py:

class Example(models.Model):
    title = models.CharField(max_length=100)

    def __str__(self):
        return self.title

Using the Django shell, everything works as expected:

>>> from example.models import Example
>>> created = Example.objects.create(title=' ')
>>> created.title == ' '
True

But if we add the Example model to admin.py:

from django.contrib import admin
from .models import Example

@admin.register(Example)
class ExampleAdmin(admin.ModelAdmin):
    ordering = ('title',)

And then try to create the same model via the Django admin, using an empty space for the title, we receive an error:

Adding blank=True to the title property makes the result even more confusing:

class Example(models.Model):
    title = models.CharField(max_length=100, blank=True)

The model will be created, but the title property is empty!

The Django admin is doing something hidden and non-obvious when creating the example model’s admin form: it’s enabling whitespace stripping by default! This stripping process removes our whitespace character and - in the case of blank=False (the default for fields) - fails validation!

This default can be seen in the keyword arguments of the form CharField that the Django admin uses to represent the title model field.

Even worse, it’s not clear how to change this behavior. Adding a strip=False keyword to the model CharField results in an unexpected keyword argument error. We could build a custom form for the admin, but we’d be re-creating an entire ModelForm to change a single keyword of one property. The ModelAdmin has a property called formfield_overrides, but since it operates via classes, we’d be altering the behavior of all CharField inputs - and still need to create a custom field input class!

The easiest solution is to override the behavior of the ModelAdmin method formfield_for_dbfield in admin.py, which determines which form class should be used to represent each model property. We just need to add strip=False to the field keywords when it encounters our title property:

@admin.register(Example)
class ExampleAdmin(admin.ModelAdmin):
    ordering = ('title',)

    def formfield_for_dbfield(self, db_field, request, **kwargs):
        if db_field.name == 'title':
            kwargs['strip'] = False
        return super().formfield_for_dbfield(db_field, request, **kwargs)

Success! The admin has an undesirable behavior, and that’s where our override lives - no changes to models.py needed!

Fixing SimpleLazyObject Errors in Django 3.1

contact@aaronoellis.com (Aaron O. Ellis) — Fri, 21 Aug 2020 00:00:00 +0000

The Quick Fix

int() argument must be a string, a bytes-like object or a number, not 'SimpleLazyObject'.

If you’re seeing the above TypeError in Django 3.1, you are likely doing something like this in a View class with context data:

class ExampleView(TemplateView):
    def get_context_data(self, **kwargs):
        get_object_or_404(MyObject, pk=kwargs.get('pk'))

This error occurs even if the pk parameter is converted to an int in the path, e.g. path('<int:pk>', ....

To fix, use self.kwargs instead:

class ExampleView(TemplateView):
    def get_context_data(self, **kwargs):
        get_object_or_404(MyObject, pk=self.kwargs.get('pk'))

You’ll see the same TypeError when just calling int() directly on the kwargs parameter, such as int(kwargs.get('pk')).

You may also see django.db.utils.ProgrammingError if you’re passing any of the keyword arguments to a database model. There is an ongoing ticket for this issue.

Using self.kwargs (or view.kwargs in templates) is the change you’d have to make by Django 4.0 anyway, since URL parameter keyword arguments will no longer be passed directly to the context. The underlying issue is caused by a mishandled deprecation warning, which unfortunately has no easy fix.

A More Detailed Explanation

Class-based views were introduced in Django 1.3 and included the highly useful TemplateView. Originally, the TemplateView passed any matched URL parameters to its template context data under the key params, allowing the user to access these parameters in the templates without any additional coding.

With the introduction of the ContextMixin in Django 1.5, the URL parameters were added directly to the template context. For example, the named URL parameter item could be accessed in templates with {{ item }}. This was done by having the TemplateView pass kwargs directly to the inherited get_context_data method:

def get(self, request, *args, **kwargs):
    context = self.get_context_data(**kwargs)

But the introduction of ContextMixin created an issue that persists today: the View instance wants to add itself to the context, but will be prevented from doing so if there is a URL parameter named view.

def get_context_data(self, **kwargs):
    if 'view' not in kwargs:
        kwargs['view'] = self

URL parameters have also been accessible in templates under the view’s kwargs property (if the view was correctly added to the context). For example with the URL parameter item: {{ view.kwargs.item }}.

Not until Django 3.1 was a change made to solve this issue. The direct passing of URL kwargs to the context is now deprecated and will be removed in Django 4.0 (approximately 16 months from now). Users should just use the kwargs attached to the view instance.

To warn users of this deprecated feature in Django 3.1, the passed kwargs are wrapped in a closure and transformed into a SimpleLazyObject (code edited for brevity):

context_kwargs = {}
for key, value in url_kwargs.items():
    @SimpleLazyObject
    def access_value(key=key, value=value):
        warnings.warn('<warning message')
        return value
    context_kwargs[key] = access_value

SimpleLazyObject is a creation of the Django project and lives in its functional utilities. When a variable matching the passed URL parameter is called in a template, the SimpleLazyObject behaves as intended, rendering the parameter and issuing a deprecation warning (which you can enable by running the process with python -Wa):

RemovedInDjango40Warning: TemplateView passing URL kwargs to the context is deprecated.
Reference item in your template through view.kwargs instead.

The SimpleLazyObject works in Django templates because the default behavior of template rendering is to call str() on any non-str variable types (plus some escaping), and SimpleLazyObject proxies this function to the inner value of its wrapped function. But SimpleLazyObject doesn’t know what to do with the built-in function int(), since this function is not proxied. The lack of this proxy method also explains the failure of get_object_or_404() when querying with an integer primary key, since int() is used during prep value.

And while adding __int__ = new_method_proxy(int) to SimpleLazyObject would solve some of our issues, model managers still wouldn’t know what to do with these objects, causing errors such as: Error binding parameter - probably unsupported type or django.db.utils.ProgrammingError: can't adapt type '__proxy__'.

Wrapping URL parameter values was likely a bad idea, even if the goal was to indicate an upcoming deprecation. URL parameters are mostly built-in types, and converting them to lazy-evaluated objects is asking for trouble. It makes more sense to wrap accessor methods on the context dictionary, but the structure of Django’s ContextMixin makes it difficult if not impossible to add a deprecation to the parameter kwargs.

A bug like this gets through rigorous testing and QA because it is a perfect case of use-case blindness: the Django developers intended the passed URL parameters to be rendered in templates, but the code to enable that feature also allowed the parameters to be used anywhere in the context body. Any testing of the recent changes would pass, because only the intended use was tested, instead of all possible uses.

If testing all possible use cases sounds impossible, you’re right. Even a test suite with 100% coverage could miss this bug, since template rendering would successfully execute all involved lines of code.

If you’re building a framework for other users, expect them to invent new use cases for your code - cases you may even consider misuse. And if issues arise during this misuse, know that it was your code that enabled it. After all, there are really only two types of frameworks in the world: those that users will abuse, and those with no users.

Replacing Ansible's AWS ec2.py Script with the aws_ec2 Plugin

contact@aaronoellis.com (Aaron O. Ellis) — Thu, 23 May 2019 00:00:00 +0000

For the past few years, I’ve been using Ansible to create reproducible builds on my AWS EC2 infrastructure.

To keep my deployments flexible, I’ve been making use of Ansible’s dynamic inventory. For AWS EC2, this takes the form of an external python script and ini file that can be called via the --inventory (or --i) flag:

ansible -i ec2.py -m ping

By default, the ec2.py script will create a large number of groups that can be used directly by Ansible’s hosts field, including groupings by region, AMI, and tags.

The returned hosts can also be filtered with a variety of options. In the example lines below, I configured the ec2.ini file to filter by the us-west-2 region and a specific tag:

regions = us-west-2
instance_filters = tag:Name=redacted

I can then use the dynamically created tag group name to target specific hosts:

ansible tag_Name_redacted -i ec2.py -m ping

The dynamic inventory script has worked well for years, but when upgrading to Ansible 2.8, I noticed the following deprecation warning:

[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad
characters in group names by default, this will change, but still be user configurable on
deprecation. This feature will be removed in version 2.10. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.

While attempting to fix this deprecation, I discovered that the Ansible developers have been hard at work on an official inventory plugin ecosystem, including an EC2 inventory source named aws_ec2.

This plugin replaces the ec2.py and ec2.ini files with a YAML file, such as:

plugin: aws_ec2
regions:
  - us-west-2
filters:
  tag:Name: redacted

It can be called in a manner similar to the external python script:

ansible -i aws_ec2.yml -m ping

There are, however, a few changes in defaults between the external script and the new plugin. These can be easily seen with the ansible-inventory command:

ansible-inventory -i aws_ec2.yml --list

For instance, using the external script, the above filters will create the following host group (among many others), while also returning the public IP of the hosts if available:

"tag_Name_redacted": {
    "hosts": [
        "###.###.###.###"
    ]
}

But the new plugin with the above filters will only return:

"aws_ec2": {
    "hosts": [
        "ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal"
    ]
}

While it might be possible to adapt your deployment script to these changes, the aws_ec2 plugin is highly configurable, and with a few options we can make a true drop-in replacement for the external script:

plugin: aws_ec2
regions:
  - us-west-2
filters:
  tag:Name: redacted
keyed_groups:
  - key: tags.Name
    prefix: tag_Name
hostnames:
  - ip-address

The only issue I had during this process was determining the correct hostnames value to use. By default the aws_ec2 plugin will return dns-name, falling back to private-dns-name. These defaults are not listed on the plugin’s documentation page, and instead I had to find them in the plugin code. Since all my hosts have public IP addresses, I wanted to reproduce the behavior of the external script, and thankfully ip-address is one of many options available for hostnames.

I have already replaced the external script with the new plugin throughout my deployment infrastructure. With a small YML file, I was able to remove several hundred lines of external scripts and configuration that I had to maintain myself.

Subclassing datetime.date in Python 3

contact@aaronoellis.com (Aaron O. Ellis) — Sun, 23 Sep 2018 00:00:00 +0000

I’ve been building an application that uses Python’s datetime.date class to represent financial quarters. The system hardcodes the month and day of each quarter, so the first quarter, for example, is equal to March 31st, and does not vary year to year.

Since these month and day combinations are already hardcoded, I decided to subclass datetime.date to better handle quarters throughout the system. The process was more complicated than I expected.

The subclass started simply:

import datetime

class Quarter(datetime.date):
    pass

Since it inherited the functionality of date, this class was ready to use:

q1 = Quarter(2018, 3, 31)

print(type(q1)) # <class '__main__.Quarter'>
print(repr(q1)) # Quarter(2018, 3, 31)
print(q1) # 2018-03-31

But I had yet to improve upon date. To make this class useful for my application, I wanted a different constructor. Since each quarter has a corresponding month and day, I just wanted to pass a year and quarter, such as Quarter(2018, 1).

Knowing the basics of Python classes, I started by overriding the initialization function:

_q1 = (3, 31)
_q2 = (6, 30)
_q3 = (9, 30)
_q4 = (12, 31)

_qs = [_q1, _q2, _q3, _q4]

class Quarter(datetime.date):
    def __init__(self, year, q):
        if q < 1 or q > 4:
            raise ValueError("Quarter must be 1, 2, 3 or 4")
        month, day = _qs[q-1]
        super().__init__(year, month, day)

But when attempting to call the constructor Quarter(2018, 1), I received the following error:

TypeError: Required argument 'day' (pos 3) not found

What’s going on here? I thought I had removed the day argument? Did I screw up the initialization of the super type?

Stepping through the debugger, I discovered that the __init__ function is never called, and the above error originates from a function called __new__. This function is present on immutable types, which includes all types in the datetime package.

The __new__ function is not mentioned anywhere in the Python tutorial on classes. The best description I could find is in the data model reference. In brief, __new__ is called to create a new instance of a class, which will then be initialized.

The function __new__ is a static method, though a special case since it does not need to be declared as one. It takes cls as its first argument instead of self and returns an instance of our desired class. Just as I did with my first attempt at initialization, I wanted to use super() and trust the behavior of date.__new__:

class Quarter(datetime.date):
    def __new__(cls, year, q):
        if q < 1 or q > 4:
            raise ValueError("Quarter must be 1, 2, 3 or 4")
        month, day = _qs[q-1]
        return super().__new__(cls, year, month, day)

It worked! And the type() and repr() outputs were the same as before!

But wait, the repr() output was now a problem, since the constructor changed. Calling eval(repr(Quarter(2018, 1))) with the above code throws:

TypeError: __new__() takes 3 positional arguments but 4 were given

I needed to override the inherited __repr__ magic method and return a value appropriate for the new constructor (if you’re unfamiliar with __repr__ you can read more here). Since doing so required the instance’s q value, which I suspected would be used frequently, I added it as a computed property:

class Quarter(datetime.date):
    def __new__(cls, year, q):
        if q < 1 or q > 4:
            raise ValueError("Quarter must be 1, 2, 3 or 4")
        month, day = _qs[q-1]
        return super().__new__(cls, year, month, day)

    @property
    def q(self):
        return self.month // 3 # We want an int type

    def __repr__(self):
        return '{0}({1}, {2})'.format(self.__class__.__name__, self.year, self.q)

Wth the eval issue fixed, I started using the new subclass throughout the system. But there was still a significant problem with the above code, and I only discovered it when pickling:

q1 = Quarter(2018, 1)
pickle.loads(pickle.dumps(q1))

Which throws:

TypeError: new() missing 1 required positional argument: &#39;q&#39;

Another constructor issue!

Finding and fixing issues such as the above would be easier with the datetime package source code, but since the major implementation of Python is cpython, you’ll need to know some C to read its source.

There is also a Python implementation in pure Python called pypy, but we cannot guarantee that its implementation is equivalent to cpython.

Returning to the issue with pickle, I attempted to find the cause of the TypeError. The pickle documentation lists six magic methods that can all alter the behavior of pickle; some take precedence over others, and most have a variety of return types. There are also five different pickle protocols in use as of Python 3.4, which affect the usage of these methods. Additionally, four of these methods aren’t even unique to pickle, and are instead part of the copy protocol.

So which methods did I need to implement to fix the pickle (and copy) behavior the subclass? Since the pickle documentation explicitly warned against writing your own __reduce__, calling it “error prone”, I started with __getnewargs__, since it controls arguments passed to __new__:

class Quarter(datetime.date):
    # Other methods omitted...
    def __getnewargs__(self):
        return (self.year, self.q)

But when re-trying pickle, I received the same TypeError as before. And when stepping through the code, I discovered that __getnewargs__ is never called!

I forgot that I was writing a subclass! Looking through the cpython source, I found that date has a __reduce__ method the uses a private getstate method (which is different from the __getstate__ magic method). According to the pickle documentation, the __reduce__ method takes precedence over __getnewargs__, so it is never called!

Thankfully, I had found the solution to the error - re-implement __reduce__ despite the documentation’s warning:

class Quarter(datetime.date):
    # Other methods omitted...
    def __reduce__(self):
        return (self.__class__, (self.year, self.q))

I could now safely copy and pickle the subclass:

q1 = Quarter(2018, 1)
copy.copy(q1)
copy.deepcopy(q1)
pickle.loads(pickle.dumps(q1))

I also added other magic methods to customize the subclass. I only needed a few, since the subclass inherits existing features of date, including comparisons and hashing.

Here’s a gist of the complete Quarter class, which shows off some of this inherited behavior.

Overall, I was disappointed by the complexity of subclassing a built-in, immutable type.

Preventing Max Connection Errors in Go

contact@aaronoellis.com (Aaron O. Ellis) — Sat, 23 Apr 2016 00:00:00 +0000

Earlier this week, I had to debug a production Go system that was crashing with the friendly PostGres error pq: sorry, too many clients already. The culprit was a function that left database transactions open under certain conditions. Although the solution was easy (close your transactions!), the problem was damaging enough that I wanted to find an automated way to guarantee every transaction was being closed throughout the application.

The error I experienced can be replicated by opening more transactions than the database server allows (a common max_connections setting for PostGres is 100):

package main

import "database/sql"
import "log"
import _ "github.com/lib/pq"

func main() {
    conn, err := sql.Open("postgres", "<credentials>")
    if err != nil {
        log.Fatal(err)
    }

    for i := 0; i < 101; i += 1 {
        if _, err := conn.Begin(); err != nil {
            log.Fatal(err)
        }
    }
}

Recent Go versions have introduced more ways to control the database connection pool, such as SetMaxOpenConns, which sets a hard limit on the number of active connections. If set, and when the limit is reached, any requests for new connections will block until another connection is closed. It will not, however, clean up long-running connections (such as unclosed transactions). Therefore, instead of a returning a direct error message when the limit has been reached, the service will timeout - arguably a worse error state.

For PostGres, there are a number of standalone programs (such as PgBouncer) that will implement connection pooling that includes long-running connection clean-up. But not only is this another layer of complexity for the service, its use could mask significant bugs in an application.

The good news is, thanks to Go 1.4’s introduction of TestMain, we can create a test harness for our application that guarantees all connections have been closed upon completion of our tests.

Our first challenge is that any number of testing functions may need to access the database connection pool. Having each call sql.Open, however, will create a new pool per function and prevent us from intelligently managing the active connections. By using sync.Once, we can guarantee that only one connection pool will be created per test run:

var conn *sql.DB // Set package-wide, but not exported
var once sync.Once

func GetConnection() *sql.DB {
    once.Do(func() {
        var err error
        if conn, err = sql.Open("postgres", "<credentials>"); err != nil {
            log.Panic(err)
        }
        conn.SetMaxOpenConns(20) // Sane default
        conn.SetMaxIdleConns(0)
        conn.SetConnMaxLifetime(time.Nanosecond)
    })
    return conn
}

We can then call GetConnection in any test that requires database access:

func TestSomething(t *testing.T) {
    conn := GetConnection()
    for i := 0; i < 10; i++ {
        if err := Something(conn); err != nil {
            t.Error(err)
        }
    }
}

GetConnection also sets some sane defaults around the connection pool that will be useful later.

Next, we need to create a TestMain function. Since our various testing functions will create one and only one connection pool, we only need a setup if there is additional work to be done, such as starting a container image or loading initial SQL fixtures into the database.

func TestMain(m *testing.M) {
    setup()
    retCode := m.Run()
    teardown() // Can't use defer because os.Exit :(
    os.Exit(retCode)
}

The magic of our harness is done in teardown, which is run after all tests have completed. Through a call to Stats we can access - after a brief pause - the current number of open connections. Even one is an error.

func teardown() {
    time.Sleep(time.Millisecond)
    if conn == nil {
        // No tests used the database!
        return
    }
    open := conn.Stats().OpenConnections
    if open > 0 {
        // This could also modify the return code...
        log.Panicf("failed to close %d connections", open)
    }
}

Tests that close their connections will pass as usual:

go test -run=TestGood
PASS
ok      github.com/aodin/go-sql-test-harness    0.071s

But buggy ones will panic during teardown (even if their actual tests pass):

go test -run=TestBad
PASS
2016/05/23 02:04:58 failed to close 10 connections
panic: failed to close 10 connections

Unfortunately, the testing harness cannot tell us in which function(s) the lingering open connections were created. With judicious use of Go test’s -run flag, however, we can quickly locate the buggy function - even if the harness is at best a smoke test.

You can see a basic implementation of this harness on GitHub. Happy hacking!

URL Parsing

contact@aaronoellis.com (Aaron O. Ellis) — Sun, 17 May 2015 00:00:00 +0000

I have a URL parsing problem.

How do we get a computer to recognize a URL in a block of text? Having a method to do this allows links such as google.com to be made clickable: google.com.

This automatic link generation has become such a feature of mainstream social media sites that its absence is notably jarring to the user. But despite this being a critical UX component, it often goes wrong. For instance, on my Android text messenger:

So how can we parse URLs? And how does it go wrong?

One way to perform this parsing is through a regular expression, which can match any given sequence of characters. Unfortunately, we can’t guarantee that the most distinguishing component of a URL, the scheme (e.g. http:// or https://), will be present. And since we can’t expect subdomains to be included as well, at minimum our desired regular expression is no more than a few whitelisted characters joined by a period.

Honoring the first rule of development - has someone else written this code - we found a Stack Overflow response with a pretty good expression if you don’t require IP addresses or internationalization:

[-a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,6}\b([-a-zA-Z0-9@:%_\+.~#?&//=]*)

Even with those caveats, this expression fails on some important URLs. It places multiple restrictions on the size of the domain name (e.g. twitter) and the top level domain, a.k.a. TLD (e.g. .com). It would miss t.co and entangled.ventures, along with any other single character domain name or TLD over six characters.

Instead of imposing character limits, other attempts at parsing URLs have just whitelisted domains, such as Gruber’s:

(?i)\b((?:https?:(?:/{1,3}|[a-z0-9%])|[a-z0-9.\-]+[.](?:com|net|org|edu|gov|mil|aero|asia|biz|cat|coop|info|int|jobs|mobi|museum|name|post|pro|tel|travel|xxx|ac|ad|ae|af|ag|ai|al|am|an|ao|aq|ar|as|at|au|aw|ax|az|ba|bb|bd|be|bf|bg|bh|bi|bj|bm|bn|bo|br|bs|bt|bv|bw|by|bz|ca|cc|cd|cf|cg|ch|ci|ck|cl|cm|cn|co|cr|cs|cu|cv|cx|cy|cz|dd|de|dj|dk|dm|do|dz|ec|ee|eg|eh|er|es|et|eu|fi|fj|fk|fm|fo|fr|ga|gb|gd|ge|gf|gg|gh|gi|gl|gm|gn|gp|gq|gr|gs|gt|gu|gw|gy|hk|hm|hn|hr|ht|hu|id|ie|il|im|in|io|iq|ir|is|it|je|jm|jo|jp|ke|kg|kh|ki|km|kn|kp|kr|kw|ky|kz|la|lb|lc|li|lk|lr|ls|lt|lu|lv|ly|ma|mc|md|me|mg|mh|mk|ml|mm|mn|mo|mp|mq|mr|ms|mt|mu|mv|mw|mx|my|mz|na|nc|ne|nf|ng|ni|nl|no|np|nr|nu|nz|om|pa|pe|pf|pg|ph|pk|pl|pm|pn|pr|ps|pt|pw|py|qa|re|ro|rs|ru|rw|sa|sb|sc|sd|se|sg|sh|si|sj|Ja|sk|sl|sm|sn|so|sr|ss|st|su|sv|sx|sy|sz|tc|td|tf|tg|th|tj|tk|tl|tm|tn|to|tp|tr|tt|tv|tw|tz|ua|ug|uk|us|uy|uz|va|vc|ve|vg|vi|vn|vu|wf|ws|ye|yt|yu|za|zm|zw)/)(?:[^\s()<>{}\[\]]+|\([^\s()]*?\([^\s()]+\)[^\s()]*?\)|\([^\s]+?\))+(?:\([^\s()]*?\([^\s()]+\)[^\s()]*?\)|\([^\s]+?\)|[^\s`!()\[\]{};:'".,<>?«»“”‘’])|(?:(?<!@)[a-z0-9]+(?:[.\-][a-z0-9]+)*[.](?:com|net|org|edu|gov|mil|aero|asia|biz|cat|coop|info|int|jobs|mobi|museum|name|post|pro|tel|travel|xxx|ac|ad|ae|af|ag|ai|al|am|an|ao|aq|ar|as|at|au|aw|ax|az|ba|bb|bd|be|bf|bg|bh|bi|bj|bm|bn|bo|br|bs|bt|bv|bw|by|bz|ca|cc|cd|cf|cg|ch|ci|ck|cl|cm|cn|co|cr|cs|cu|cv|cx|cy|cz|dd|de|dj|dk|dm|do|dz|ec|ee|eg|eh|er|es|et|eu|fi|fj|fk|fm|fo|fr|ga|gb|gd|ge|gf|gg|gh|gi|gl|gm|gn|gp|gq|gr|gs|gt|gu|gw|gy|hk|hm|hn|hr|ht|hu|id|ie|il|im|in|io|iq|ir|is|it|je|jm|jo|jp|ke|kg|kh|ki|km|kn|kp|kr|kw|ky|kz|la|lb|lc|li|lk|lr|ls|lt|lu|lv|ly|ma|mc|md|me|mg|mh|mk|ml|mm|mn|mo|mp|mq|mr|ms|mt|mu|mv|mw|mx|my|mz|na|nc|ne|nf|ng|ni|nl|no|np|nr|nu|nz|om|pa|pe|pf|pg|ph|pk|pl|pm|pn|pr|ps|pt|pw|py|qa|re|ro|rs|ru|rw|sa|sb|sc|sd|se|sg|sh|si|sj|Ja|sk|sl|sm|sn|so|sr|ss|st|su|sv|sx|sy|sz|tc|td|tf|tg|th|tj|tk|tl|tm|tn|to|tp|tr|tt|tv|tw|tz|ua|ug|uk|us|uy|uz|va|vc|ve|vg|vi|vn|vu|wf|ws|ye|yt|yu|za|zm|zw)\b/?(?!@)))

But this expression would miss any link with an emerging TLD, such as one of Google’s 2015 April Fools’ pranks: com.google.

A since the list of new TLDs continues to grow - there are currently over 900 - any whitelist would quickly grow stale or unwieldy. And returning to our non-whitelisted regular expression, we’d even have to watch for new TLDs that push the character length limitations, such as .international and .cancerresearch.

And if we resign ourselves to create a link for any words joined with a ., we’re going to generate a number of false positives for files (e.g. bower.json or report.xlsx).

So normally this is where the developer gives up, tells the designer to create a new <input> field and calls it an early day.

Or, you could keep digging that hole, and build infrastructure that would perform the ultimate test of whether a loosely matched sequence of characters is a URL or not: send an http request at it.

But that’s another story…

Arrays in Go

contact@aaronoellis.com (Aaron O. Ellis) — Sun, 27 Jul 2014 00:00:00 +0000

If you’re looking to learn more about arrays in Go, the best place to start is Go Slices: usage and internals on the Go blog. The following merely expands on the article’s points and demonstrates a few gotchas.

For instance, there are multiple ways to initialize both arrays and slices, and it’s not always obvious which one will be created:

Initialization	Type
`[3]int{1, 2, 3}`	array
`[...]int{1, 2, 3}`	array
`var a [3]int`	array
`a[:]`	slice
`var s []int`	slice
`[]int{1, 2, 3}`	slice
`make([]int, 3)`	slice

The zero value of a slice may be nil, but don’t expect it to have a length:

log.Println(len(nil))

./nil.go:6: use of untyped nil

Unless you specify a type:

var slice []int = nil
log.Println(len(slice))
// 0

You can use nil as a valid parameter or field value:

type Package struct {
    Name         string
    Dependencies []string
}

func main() {
    unsafe := Package{Name: "unsafe", Dependencies: nil}
    log.Println(unsafe, len(unsafe.Dependencies))
    // {unsafe []} 0 
}

Despite a newly created slice being nil, it is able to undergo append operations immediately. No call to make is necessary:

var Primes []int
Primes = append(Primes, 2, 3, 5, 7)

This also works if the slice is the field of a struct:

type Node struct {
    Name     string
    Children []Node
}

func Nodes() {
    A := Node{Name: "A"}
    B := Node{Name: "B"}
    A.Children = append(A.Children, B)
}

Or a named return value:

func CreateFibonacci() (Fibonacci []int) {
    Fibonacci = append(Fibonacci, 1, 1, 2, 3, 5, 8)
    return
}

Or my favorite, the value of a map:

regions := make(map[string][]string)
regions["Europe"] = append(regions["Europe"], "France", "Germany")
regions["Africa"] = append(regions["Africa"], "Djibouti", "Egypt")

But watch out, the JSON output of a slice will vary depending on the initialization method:

var a []int
jsonA, _ := json.Marshal(a)

b := make([]int, 0)
jsonB, _ := json.Marshal(b)

log.Printf("jsonA: %s\n", jsonA)
log.Printf("jsonB: %s\n", jsonB)
// jsonA: null
// jsonB: []

Golint will suggest replacing the latter syntax despite the difference in output:

slices.go:12:2: can probably use "var b []int" instead

Slices are unhashable and cannot be used as keys:

xy := make(map[[]int]string)

./xy.go:6: invalid map key type []int

But arrays are hashable:

xy := make(map[[2]int]string)
xy[[2]int{3, 4}] = "Battleship"
log.Println(xy)
// map[[3 4]:Battleship]

They are not, however, valid keys in JSON objects:

output, err := json.Marshal(xy)
if err != nil {
    log.Fatal(err)
}

json: unsupported type: map[[2]int]string

Happy Hacking!

Opinions on Go

contact@aaronoellis.com (Aaron O. Ellis) — Thu, 29 May 2014 00:00:00 +0000

Last week I delivered a talk on teaching Go at the Denver Gophers meetup. The talk included a number of what I called “controversial opinions” on Go and I have foolishly decided to make them a matter of public record.

Go is not a first language

The Go language is missing many features - objects, exceptions, and generics to name a few - but I’m not missing them. These omissions were done intentionally and were meant to turn several mainstream programming paradigms on their heads.

But it’s impossible to teach a paradigm in a language where it doesn’t exist. If I could guarantee that a new programmer would only ever write Go, I would gladly teach them its opinionated ways. That belief, however, is folly, and programmers should be exposed as soon as possible to more extensive languages, such as Python and Java.

Go is ugly

The tooling built into Go, especially go fmt and integrated testing and benchmarking, ended numerous debates before they even began. By embracing strict semantics, Go also shrugged off superfluous semicolons and parentheses that have haunted many C-esque languages.

Despite these advances, I still believe Go to be ugly. It is mostly the verbosity and duplication of error checking, even though I agree with the purpose of errors. An over abundance of left-sided whitespace and repeated variables can activate the Spidey sense of programmer brains. Aesthetic arguments, however, are subjective and formed by experience. And if callbacks can be beautiful in Node.js, it is only a matter of time…

Go is young

The first stable release of Go was in March 2012. And despite its rapid adoption, it is still a language that was conceived after the first release of Python 3.

With this in mind, it seems unnecessary to state that the ecosystem is young and incomplete. But this problem is exacerbated by Go’s dependency management, which pulls from the master branch of remote repositories without an option to specify an alternative. Combined with project maintainers that commit to master (myself included), the only solution is forking or third-party tools.

Its youth also means a lack of best practice. Even though I would consider Go a small language, numerous questions still abound: Should the reflect package be avoided? When is it acceptable to use panic? Type-casts? Embedding? What shouldn’t be exported from a library? Can I refer to a package as a framework without being yelled at on the go-nuts mailing list?

Go loves purpose

During the talk, I stated “Go loves requirements” and that it’s a perfect language for building version two of a product. While I still believe this, discussion with the team at SupportLocal convinced me to make a slightly more inclusive point.

Go is being used to replace legacy systems at Google, SendGrid, Cloudflare, and others. But Go is also being used to write software that solves long-standing problems, such as Heka, Docker, and Doozer. By providing a new set of paradigms and primitives, Go has brought a number of problems from the “we’ll solve it later” to the “let’s solve it now” category.

And while Go is perfectly suited for this type of purpose driven development, I also believe the inverse: there is a major penalty to be paid when writing open-ended Go. A lack of generics means code duplication. Usage of the reflect package removes type safety at compilation. The standard library is comprehensive, but often goes too far with unexported identifiers.

Go is here to stay

I started programming Go over two and a half years ago. I was looking for a language that was built for the web and could help me build concurrent and distributed systems. I had almost settled on using Erlang (and its one-based indexing) before I discovered Go. I immediately feel in love with channels and first-class interfaces.

Go did not get everything right and it did little to advance the state of programming languages. But Go has refined the practice of programming more than any other modern language. It is small, readable, and includes a brilliant standard library. It is also here to stay.

How to Complicate Open Data

contact@aaronoellis.com (Aaron O. Ellis) — Tue, 01 Apr 2014 00:00:00 +0000

Cities are increasingly driven by data. When that data is released to the public the city is rightfully treating information as infrastructure. But as with all infrastructure, it must be constructed wisely and maintained properly.

Here are a few ways open data can go wrong.

Call a Field “UniqueId” and Make It Not Unique

Liquor licenses in the city of Denver are available for download in a variety of formats and are updated daily.

The schema for this data includes the field UNIQUE_ID, which surprisingly or not surprisingly depending on your level of cynicism, is not actually unique.

Unique identifiers, when actually unique, provide a way to quickly index and reference objects. Duplication makes this process impossible. Anyone who shares a first name with a friend knows this dilemma well.

Luckily in the case of the licenses, the duplicate ids do not differ and can be removed. But this is an unnecessary additional step in taking the data from a raw to usable state.

Don’t Tell Anyone Your Coordinate System

The spherical nature of the Earth complicates geographical calculations. The solution used in many geographic information systems is to transform spherical coordinates onto a flat plane. Within a large area, even one as large as a third of Colorado, errors due to this transformation will be less than 1:10,000.

The liquor license schema also includes the fields X_COORD and Y_COORD, but does not name its coordinate system or zone in the file or metadata. They happen to belong to the Colorado Central Grid, just one of the zones in the State Plane Coordinate System.

Transforming the given x and y values to more recognizable coordinates, such as latitude and longitude, is a quick calculation, but only if you know what coordinate system you’re using.

Randomly Sort the File

Checking if two files differ should not be a difficult operation. On Unix-based systems there are a number of tools installed by default to compare files. Continuing with the city of Denver liquor licenses, we can see the byte count of the downloaded files over successive days with ls -l:

392152 Mar 28 03:00 licenses_2014-03-28.csv
393545 Mar 29 03:00 licenses_2014-03-29.csv
393545 Mar 30 03:00 licenses_2014-03-30.csv

The files on the 29^th and 30^th have the same number of bytes, but are the actual bytes the same? We can quickly calculate a checksum of these files with sha1sum:

08260d863858ceacecdeda3f63727ba954e63cb7  licenses_2014-03-28.csv
bde511e788140672682ad985c2710022772f5034  licenses_2014-03-29.csv
db235e5ce4c4236417f265bd85645257d123d6b2  licenses_2014-03-30.csv

Despite the last two files having the same number of bytes, they actually differ. Unfortunately, a SHA1 checksum cannot tell us the magnitude of these differences, but we can use diff to perform a line by line comparison. For instance, if the name on a license changed, we’d see the following:

< 331902_LI32,1063820,331902,WHOLE FOODS MARKET,7400 E HAMPDEN AVE # D-1,LI32,1,LIQUOR-3.2 % BEER,3.2 % Beer - City,2014-03-06 00:00:00,2015-02-19 00:00:00,C,0,180030.0,3,4,06809,V,3168440.0,1663019.0
---
> 331902_LI32,1063820,331902,HOLE FOODIE MARKET,7400 E HAMPDEN AVE # D-1,LI32,1,LIQUOR-3.2 % BEER,3.2 % Beer - City,2014-03-06 00:00:00,2015-02-19 00:00:00,C,0,180030.0,3,4,06809,V,3168440.0,1663019.0

But running diff on the files from the 29^th and 30^th tells us that 1,525 lines have changed, nearly three quarters of their total 2,097 lines. What dramatic changes could have occurred in one day? Did a dutiful government worker spend his Saturday rewriting liquor licenses?

Spoiler alert: no. Nothing changed.

On a hunch, I sorted these files. Unix provides a tool called sort, however, its usage is made slightly more complicated by the need to skip the header in the first row:

(head -n 1 licenses_2014-03-30.csv && tail -n +2 licenses_2014-03-30.csv | sort) > sorted_2014-03-30.csv

After sorting both the files, we can rerun the checksum:

d978ff44979f0fa7c3d33b7f4db7df909856f6f8  sorted_2014-03-29.csv
d978ff44979f0fa7c3d33b7f4db7df909856f6f8  sorted_2014-03-30.csv

These files contain exactly the same information. They were randomly sorted before publishing.

Don’t Escape Values

The comma-separated values file format has been used for decades and is supported by nearly every spreadsheet and database. Other characters can be used to separate values, such as tabs, but no matter the implementation there must be a way to include the character used to separate within actual values. A widely implemented solution for this dilemma is to use double quotes to wrap a value that itself contains separating characters. For instance, instead of the ambiguous:

Eats, Shoots & Leaves,Truss, Lynne

The values would be wrapped in double quotes to show that this row only contains a book title and the author by last name:

"Eats, Shoots & Leaves","Truss, Lynne"

I’ve seen naive implementations of CSV writers forget to escape values in this manner, corrupting their output as a result. But there is another way this process can go wrong. Since the double quote now has significance, any double quotes in the value must be escaped. For instance:

"\"You miss 100% of the shots you don't take.\" -Wayne Gretzky." -Michael Scott

The city of Denver releases many files in CSV format, including their checkbook and assessor sales. But both these files contain escaping errors, such as the following:

"'Maintenance" RFID/Tech (temp)"
"GAMBILL,SHANONNON O" QUINN &"

A simple rule: if your file cannot be parsed by a CSV reader, it shouldn’t end in .csv.

Only Provide a PDF

Street closure data for the city of Denver is updated every two weeks. It is released in Portable Document Format, which until 2008 was a proprietary format controlled by Adobe Systems. The city recommends the following method for finding a closure important to you:

If looking for a particular street, hit Ctrl + F to find it within the document.

Releasing data as a PDF complicates parsing and introduces unnecessary roadblocks for developers looking to build something better than Ctrl + F.

Benford's Law and the Denver City Checkbook

contact@aaronoellis.com (Aaron O. Ellis) — Thu, 30 Jan 2014 00:00:00 +0000

The City and County of Denver Checkbook is open to the public, minus some confidential transactions and payment information.

The version I download on January 28th, 2014 contained 941,686 transactions between the dates of January 5th, 2009 and January 23rd, 2014. They totaled over $6.5 billion.

Transactions can also be summed by the day of the week that they occurred:

Day	Total
Monday	$849,048,448
Tuesday	$1,125,252,608
Wednesday	$1,445,859,328
Thursday	$1,705,470,848
Friday	$1,412,129,280
Saturday	$1,011,808
Sunday	$0

While the majority of the transactions are under $1,000, the checkbook also includes payments for general operations and contractual services. Some of the largest include:

Payee	Department	Date	Amount
Denver Health Medical Center	Environmental Health	2011-02-11	$15,505,120
Denver Health Medical Center	Environmental Health	2009-02-19	$13,304,044
United Airlines	DIA	2009-12-18	$12,500,000
Metro Wastewater Reclamation	DIA	2012-12-18	$12,399,824
Manager of Finance	Dept of Finance	2011-06-29	$11,436,182
Manager of Finance	Wastewater Manage	2011-03-14	$11,252,650

Since the transaction amounts vary across several orders of magnitude, this data should abide to Benford’s law. If we take the first digit of each amount, we should find that the digit 1 occurs much more frequently than 9. Performing this count on the checkbook:

First Digit	Count
1	286,341
2	168,059
3	106,017
4	85,579
5	70,787
6	63,240
7	61,767
8	46,758
9	37,612

Expressed as a percentage of the total, along with the expected percentages:

First Digit	Expected	Observed
1	30.1%	30.9%
2	17.6%	18.2%
3	12.5%	11.5%
4	9.7%	9.2%
5	7.9%	7.6%
6	6.7%	6.8%
7	5.8%	6.7%
8	5.1%	5.1%
9	4.6%	4.1%

These observed counts are completely within the range of the expected percentages and the checkbook is a great real life example of Benford’s law.

Queries

With a copy of the checkbook.csv you can produce these results yourself. The following schema was used to load the checkbook into a PostGres database:

CREATE TABLE "checkbook" (
    "id" varchar PRIMARY KEY,
    "payee" varchar,
    "city" varchar,
    "state" varchar,
    "program_area" varchar,
    "department" varchar,
    "payment_id" varchar,
    "payment_date" timestamp,
    "year" int,
    "amount" real,
    "purchase_order" varchar,
    "funding_source" varchar,
    "project" varchar,
    "project_description" varchar,
    "expense_category" varchar,
    "expense_subcategory" varchar
);

The only change made to the data was the removal of a double quote in the Project Description "'Maintenance" RFID/Tech (temp)" that exists on multiple lines.

The PostGres COPY command was then used to quickly load the data:

COPY "checkbook" FROM 'checkbook.csv' WITH CSV HEADER;

Since the checkbook includes negative numbers, the sum was calculated using an absolute value:

SELECT to_char(SUM(@"amount"),'9,999,999,999D99') FROM "checkbook";

The largest transactions were found with:

SELECT
    "payee",
    "department",
    to_char("payment_date", 'Mon DD, YYYY') AS "date",
    to_char("amount", '999,999,999')
FROM "checkbook"
ORDER BY "amount" DESC
LIMIT 6

The transactions by quarter:

SELECT
    "year",
    date_part('quarter', "payment_date") AS "quarter",
    to_char(SUM(@"amount"),'9,999,999,999D99')
FROM "checkbook"
GROUP BY 1, 2
ORDER BY 1, 2

And by day of the week:

SELECT
    date_part('isodow', "payment_date") AS "day",
    to_char(SUM(@"amount"),'9,999,999,999D99')
FROM "checkbook"
GROUP BY 1
ORDER BY 1

To count the leading digits:

SELECT
    left("amount"::text, 1) AS "digit",
    count("id")
FROM "checkbook"
WHERE "amount" >= 1.0
GROUP BY 1

And the percentages:

SELECT
    left("amount"::text, 1) AS "digit",
    to_char(count("id")::real / (SELECT count("id")FROM "checkbook" WHERE "amount" > 1.0) * 100, '99D99') AS "%"
FROM "checkbook"
WHERE "amount" >= 1.0
GROUP BY 1

Collisions in Django Sessions

contact@aaronoellis.com (Aaron O. Ellis) — Wed, 22 Jan 2014 00:00:00 +0000

I often program in the Django web framework. Recently, I became interested in how it tracks the information of users that have already logged in.

By default, Django uses a cookie named sessionid and contains a 32 character string built from lowercase letters and digits, or 36 possible characters. Therefore, there are a total of 36³² or 63,340,286,662,973,277,706,162,286,946,811,886,609,896,461,828,096 possibilities for your session key. This number is between 2¹⁶⁵ and 2¹⁶⁶ and is slightly larger than the possibilities produced by a good cryptographic hash.

But what are the chances that two randomly generated session keys in a large collection of session keys will be equal? The solution to this problem is actually counter-intuitive and is summarized in a similar problem:

The Birthday Problem is to find the probability that, in a group of N people, there is at least one pair of people who have the same birthday. Birthday Problem

While some people estimate this number to be over 100,000, it is much less than that, since to determine a “collision” of birthdays we have to compare not just your birthday to everyone, but everybody’s birthday to each other.

Turns out, there only needs to be 23 people before the chance of two of them sharing a birthday is greater than 50%.

So how many sessions have to exist before a key collision becomes likely? We can estimate this value using the equation:

Q(H) = √(π / 2 * H)

Where H is the number of possibilities. In our case, H is 36³², the number of unique sessions, giving a result of 9.97 * 10²⁴ or about 2⁸³.

That’s still 10 septillion sessions before our first collision is likely.

No worries!

Visualizing City Density

contact@aaronoellis.com (Aaron O. Ellis) — Wed, 08 Jan 2014 00:00:00 +0000

Using a list of Denver County addresses with latitude and longitude, I was able to generate a density map of the downtown area:

The low density area to the left of center includes Confluence Park, which is adjacent to the high density residential district of LoDo.

At a higher resolution, many of the high density blocks disappear as addresses are condensed in large loft and condo units.

I also had a list of Washoe County addresses with latitude and longitude:

This map is approximately centered on the I-80 and I-580 junction. The large empty section just to the southeast of the junction is the airport.

At higher resolution, the varying density of suburbs becomes apparent.

You can follow further development of this project on its GitHub page.

Superlative Buildings in Washoe

contact@aaronoellis.com (Aaron O. Ellis) — Mon, 18 Nov 2013 00:00:00 +0000

Here are some superlatives that I’ve found in the data from the Washoe County Assessor’s office.

All buildings constructed before 1900 with the oldest first:

Address	Name	Year	Purpose
7250 FRANKTOWN RD		1860	Single Family Residence
12945 S VIRGINIA ST		1860	Single Family Residence
1020 S US 395 N CC HWY	WINTERS RANCH	1862	Single Family Residence
4005 S US 395 N CC HWY	BOWERS MANSION	1864	Single Family Residence
1495 S MARSH AVE		1869	Single Family Residence
753 MILL ST	TAYLOR REALTY	1874	Single Family Residence
100 S US 395 N CC HWY		1875	Office Building
333 RALSTON ST		1875	Conversion
1150 W PECKHAM LN		1875	Single Family Residence
250 COURT ST	LAKE MANSION	1877	Conversion
9825 S VIRGINIA ST		1881	Multiple Residence (Low Rise)
320 WONDER ST		1884	Single Family Residence
195 N VIRGINIA ST		1885	Retail Store
98 W COMMERCIAL ROW		1888	Retail Store
1140 W PECKHAM LN		1888	Single Family Residence
557 WASHINGTON ST		1890	Office Building
18 STEWART ST		1890	Conversion
7 ELM CT		1890	Single Family Residence
75 COURT ST		1890	Governmental Building
840 WASHINGTON ST		1890	Single Family Residence
650 N CENTER ST		1890	Apartment Residential
118 CALIENTE ST		1890	Single Family Residence
14355 PYRAMID WAY		1890	Single Family Residence
5565 LEON DR		1895	Single Family Residence
829 N CENTER ST		1895	Conversion
532 VALLEY RD		1896	Single Family Residence
542 PLUMAS ST		1896	Conversion
3636 MAYBERRY DR		1896	Single Family Residence
310 W SUNSET BLVD		1898	Single Family Residence
310 SPOKANE ST	SPICE HOUSE	1898	Cocktail Lounge
428 HILL ST		1898	Conversion
375 RALSTON ST		1898	Conversion

All buildings above 12 floors with the tallest first:

Address	Name	Year	Floors	Purpose
411 N SIERRA ST	SILVER LEGACY	1995	39	Hotel, Full Service
10455 DOUBLE R BLVD		2004	30	Dental Office/Clinic
1100 NUGGET AVE	J A NUGGET	1996	28	Hotel, Full Service
2500 E 2ND ST	GRAND SIERRA RESORT	1978	27	Hotel, Full Service
130 W 6TH ST	CIRCUS CIRCUS	1985	27	Hotel, Full Service
516 WEST ST	CIRCUS CIRCUS	1985	27	Hotel, Full Service
200 N CENTER ST	HAMPTON INN	1995	26	Hotel, Full Service
345 N SIERRA ST	ELDORADO	1989	24	Casino
219 N CENTER ST	HARRAHS	1969	24	Hotel, Full Service
100 N ARLINGTON AVE	ARLINGTON TOWERS	1966	23	Apartment (High Rise)
255 N SIERRA ST		1978	23	Luxury High Rise Apartment
500 N SIERRA ST		1981	22	Hotel, Full Service
3800 S VIRGINIA ST	ATLANTIS RESORT & CASINO	1970	20	Hotel, Full Service
450 N ARLINGTON AVE		1976	19	Hotel, Full Service
280 ISLAND AVE		1961	18	Apartment (High Rise)
201 N CENTER ST	HARRAHS	1969	17	Hotel, Full Service
2707 S VIRGINIA ST		1996	17	Hotel, Full Service
209 N CENTER ST	HARRAHS	1969	17	Hotel, Full Service
121 WEST ST		1967	16	Parking Levels
255 N VIRGINIA ST	COMMROW	1976	16	Hotel, Full Service
133 N VIRGINIA ST	CAL NEVA-NV TOWER	1977	16	Hotel, Limited Service
345 N ARLINGTON AVE		1979	16	Hotel, Full Service
5190 SPECTRUM BLVD		2001	15	Equipment (Shop) Building
300 E 2ND ST		1981	15	Parking Structure
5195 SPECTRUM BLVD		2001	15	Equipment (Shop) Building
1 E 1ST ST		1963	15	Office Building
130 N VIRGINIA ST	VIRGINIAN	1987	15	Hotel, Full Service
140 N VIRGINIA ST	VIRGINIAN	1987	15	Hotel, Full Service
200 W 2ND ST	RIVERWALK CONDOMINIUMS	1978	14	Luxury High Rise Apartment
60 W 1ST ST		2006	13	Apartment (High Rise)
80 W 1ST ST		2006	13	Luxury High Rise Apartment
1000 E 6TH ST	RAMADA INN	1973	13	Hotel, Full Service
50 W 1ST ST	PALLADIO	2006	13	Luxury High Rise Apartment
50 N SIERRA ST		2006	13	Luxury High Rise Apartment

A 30 floor dental office? Looks like there might be some errors in the assessor data.

And the largest by square footage:

Address	Name	Year	Floors	Ft²	Purpose
2500 E 2ND ST		1978	27	2,592,414	Hotel, Full Service
1155 MILL ST	RENOWN	2005	10	2,020,510	Hospital
411 N SIERRA ST	SILVER LEGACY	1995	39	1,865,345	Hotel, Full Service
77 RENOWN WAY	WASHOE MEDICAL CENTER INC	2005	10	1,838,189	Hospital
2707 S VIRGINIA ST	PEPPERMILL	1973	4	1,633,019	Casino
11111 STEAD BLVD		1979	2	1,629,177	Office Building
1402 S MC CARRAN BLVD		1970	1	1,550,700	Mega Warehouse
9295 PROTOTYPE DR		2004	2	1,051,043	Industrial Engineering Building
516 WEST ST	CIRCUS CIRCUS	1985	27	1,019,138	Hotel, Full Service
14551 INDUSTRY CIR		1992	1	806,920	Mega Warehouse
3800 S VIRGINIA ST	ATLANTIS RESORT & CASINO	2008	3	799,762	Casino
4590 S VIRGINIA ST	CONVENTION CEN	1900	2	762,377	Single Family Residence
130 W 6TH ST	CIRCUS CIRCUS	1985	27	732,418	Hotel, Full Service
0 LONGLEY LN		2000	1	718,791	Mega Warehouse
0 S ROCK BLVD	AIRPORT	1997	2	706,541	Parking Structure
12090 SAGE POINT CT	SHERWIN WILLIA	1997	1	676,431	Mega Warehouse
0 ROCK BLVD		2007	1	663,112	Fire Station (With Staff)
255 N SIERRA ST		1978	22	644,943	Luxury High Rise Apartment
1095 ROCK BLVD		2005	1	637,868	Car Wash
0 N SIERRA ST		1978	23	624061	Luxury High Rise Apartment
1100 NUGGET AVE	SPARKS NUGGET	1978	3	622,628	Casino
555 VISTA BLVD		1980	1	622,550	Mega Warehouse
12660 OLD VIRGINIA RD		2000	1	601,250	Mega Warehouse
2985 S VIRGINIA ST	PEPPERMILL	2006	5	591,000	Parking Structure
500 N SIERRA ST	CIRCUS CIRCUS	1957	5	587,380	Hotel, Full Service
650 N SIERRA ST	CIRCUS CIRCUS	1996	5	569,285	Parking Structure
1500 TERMINAL WAY		1997	2	550,368	Parking Structure
12035 MOYA BLVD		2008	1	545,550	Mega Warehouse
6450 LONGLEY LN		2000	1	545,232	Mega Warehouse
300 E 2ND ST		1981	15	519,840	Parking Structure
450 LILLARD DR	OWEN DISTRIBUT	1988	1	513,152	Mega Warehouse
200 N CENTER ST	HARRAHS	1975	2	504,466	Casino
1001 E 9TH ST	WASHOE COUNTY	1977	2	497,713	Governmental Building
5200 SUMMIT RIDGE DR	RECREATION BLDG	1997	1	477,926	Health Club

Not just a WAREHOUSE, it’s a MEGA WAREHOUSE!

A sub-query was used to remove duplicate parcels on the same address. To get the oldest buildings:

SELECT
    concat_ws(' ', p.number, p.direction, p.street) AS address,
    p.name,
    p.original_year AS year,
    p.type AS purpose
FROM (
    SELECT DISTINCT ON (number, direction, street) * FROM properties
        JOIN buildings ON buildings.parcel = properties.parcel
        ORDER BY number, direction, street, original_year
    ) p
WHERE
    p.original_year <> 0
    AND p.original_year < 1900
ORDER BY year NULLS LAST

To get the tallest:

SELECT
    concat_ws(' ', p.number, p.direction, p.street) AS address,
    p.name,
    p.original_year AS year,
    p.stories AS floors,
    p.type AS purpose
FROM (
    SELECT DISTINCT ON (number, direction, street) * FROM properties
    JOIN buildings ON buildings.parcel = properties.parcel
    WHERE number <> '0'
    ORDER BY
        number,
        direction,
        street,
        stories DESC NULLS LAST,
        name DESC NULLS LAST
    ) p
ORDER BY stories DESC NULLS LAST;

And the largest by square footage:

SELECT
    concat_ws(' ', p.number, p.direction, p.street) AS address,
    p.name, p.original_year AS year,
    p.stories AS floors,
    p.building_sqft AS sqft,
    p.type AS purpose
FROM (
    SELECT DISTINCT ON (number, direction, street) * FROM properties
    JOIN buildings ON buildings.parcel = properties.parcel
    ORDER BY
        number,
        direction,
        street,
        building_sqft DESC,
        name DESC NULLS LAST,
        original_year DESC NULLS LAST
    ) p
ORDER BY building_sqft DESC NULLS LAST

Get Going with Web Servers in Go

contact@aaronoellis.com (Aaron O. Ellis) — Wed, 13 Nov 2013 00:00:00 +0000

Go, initially developed at Google, recently celebrated its fourth anniversary as an open source project. I have found Go to be unrivaled when building web services and see a bright future for it as micro service architecture gains wider appeal.

I wanted to share a few code snippets that I have found indispensable. I also have a boilerplate Go web server that shows them in action. Even more examples can be found on my GitHub project.

Hopefully this will help to get your first web server in Go going.

Execute a Template while Avoiding Globals

Are your files being polluted with globals? Attach variables and handlers to a struct:

type Website struct {
    attrs     map[string]interface{}
    homepage  *template.Template
}

func (web *Website) HomepageHandler(w http.ResponseWriter, r *http.Request) {
    web.homepage.Execute(w, web.attrs)
}

Execute Nested Templates

Getting nested templates to work in Go can be a frustrating experience. Instead of generating an error, many mistakes in templates will result in blank output. Here’s the boilerplate you’ll need to get started:

type Website struct {
    attrs     map[string]interface{}
    homepage  *template.Template
}

func (web *Website) HomepageHandler(w http.ResponseWriter, r *http.Request) {
    web.homepage.ExecuteTemplate(w, "base", web.attrs)
}

func Create(attrs map[string]interface{}) *Website {
    h := template.Must(template.New("home").ParseFiles("home.html", "base.html"))
    return &Website{attrs: attrs, homepage: h}
}

The file base.html contains:

{{ define "base" }}<!doctype html>
<html>
  <head>
    <meta charset="utf-8">
    <title>Nested Templates</title>
    {{ template "header" . }}
  </head>
  <body>
    {{ template "content" . }}
    <p>I am a paragraph from base.html</p>
    {{ template "script" . }}
  </body>
</html>{{ end }}

And the file home.html:

{{ define "header" }}{{ end }}

{{ define "content" }}
<p>I am a paragraph from home.html</p>
{{ end }}

{{ define "script" }}{{ end }}

Nested template files must be parsed together in the same template.Template and their define template actions (what’s between the double curly braces) must be coordinated. And don’t forget to call the top-level template (in this example it’s base) during execution.

Write a Status Code

Status codes must be written before any content. Once content is written, http.StatusOK will be set automatically by the ResponseWriter. To specify your own status code (these will do the same thing):

w.WriteHeader(http.StatusCreated)
w.WriteHeader(201)

There is also a built-in 404 response:

http.NotFound(w, r)

Write a Content-Type Header

This order of operations is also true for the content type of the response. Set it before any content is written or the type will be determined by the function DetectContentType:

w.Header().Set("Content-Type", "application/json")

Serve a Static Directory

To serve an entire directory (and its sub-directories) as static files:

handler := http.StripPrefix("/static/", http.FileServer(http.Dir("./static")))
http.Handle("/static/", handler)

Serve a Single File

If you need to serve a static file, but don’t need a full directory:

func IconHandler(w http.ResponseWriter, r *http.Request) {
    http.ServeFile(w, r, "favicon.ico")
}

Command Line Options

Instead of hard-coding your server port, use Go’s flag package to pass it on the command line:

port := flag.Int("port", 8000, "Server Port")
flag.Parse()
err := http.ListenAndServe(fmt.Sprintf(":%d", *port), nil)

The port can then be specified with:

./website --port=80

Log to a File

By default, Go’s log package will write to stdout. The following will append the log output to site.log, creating the file if necessary:

f, err := os.OpenFile("site.log", os.O_APPEND | os.O_WRONLY | os.O_CREATE, 0644)
if err != nil {
    panic(err)
}
defer f.Close()
log.SetOutput(f)

Deployment by Forking

Go’s net/http package contains a robust, production-ready server. While it might be a good idea to place the binary behind a more familiar server like nginx, or at least daemonize the server process, you can deploy using:

./website &

When the Mortgage Crisis Comes to Town

contact@aaronoellis.com (Aaron O. Ellis) — Thu, 07 Nov 2013 00:00:00 +0000

For Hack4Reno 2013 my team worked with data from the Washoe County Assessor’s office. The entire raw dataset can be downloaded here. You can also follow my team’s work on GitHub.

During the hackathon, we performed some queries on the aggregate data that would be nearly impossible with the existing API. Here are a few that focus on buildings classified as a Single Family Residence. As of November 7, 2013, there were 109,093 of these buildings listed in Washoe county.

New home construction peaked in 2005 and dropped significantly until 2010. Despite the slight increase in recent years, construction remains at a level not seen since the early 1980s.

Year	Homes
2000	2,868
2001	3,884
2002	4,305
2003	3,590
2004	4,101
2005	4,803
2006	3,732
2007	2,021
2008	1,071
2009	533
2010	385
2011	496
2012	593
2013	672*

Likewise, sales of single family homes peaked in 2005 and are currently at about 60% of that high.

Year	Sales
2000	18,221
2001	25,376
2002	29,618
2003	33,690
2004	32,798
2005	34,207
2006	25,170
2007	18,935
2008	16,586
2009	18,176
2010	17,621
2011	18,365
2012	19,973
2013	16,353*

The sale price per square foot of homes underwent a similar decrease. Because of extreme outliers in this query, results were limited to sales less than $1.5 million and homes less than 10,000 square feet. A large number of zero dollar sales were also removed - nearly 50% of the data set.

Year	Sales	Avg. Ft²	$ / Ft²
2000	9,482	2,086.2	114.88
2001	11,470	2,040.1	125.13
2002	12,509	2,042.7	144.73
2003	13,521	2,013.0	150.01
2004	14,514	2,015.0	173.45
2005	13,181	2,035.6	211.91
2006	9,536	2,099.9	206.74
2007	7,361	2,054.9	187.37
2008	7,763	1,987.9	161.71
2009	9,216	1,919.0	128.02
2010	9,327	1,929.1	122.30
2011	10,030	1,924.9	103.78
2012	8,997	1,989.2	106.62
2013	7,232*	1,986.4*	127.71*

* All 2013 data is year to date.

The queries were performed using the following SQL statements. To find the number of single family homes:

SELECT
    COUNT(buildings.parcel)
FROM buildings
WHERE
    type = 'Single Family Residence'

To produce the table and graph of new home construction:

SELECT
    original_year AS year,
    COUNT(buildings.parcel)
FROM buildings
WHERE
    type = 'Single Family Residence'
GROUP BY 1
ORDER BY 1

For home sales:

SELECT
    EXTRACT(year from sales.date) as year,
    COUNT(sales.id)
FROM sales
    JOIN buildings ON buildings.parcel = sales.parcel
WHERE
    type = 'Single Family Residence'
GROUP BY 1
ORDER BY 1

And the price per square foot:

SELECT
    EXTRACT(year from sales.date) as year,
    COUNT(sales.id),
    AVG(properties.building_sqft),
    AVG(sales.amount / properties.building_sqft)
FROM sales
    JOIN buildings ON buildings.parcel = sales.parcel
    JOIN properties ON properties.parcel = sales.parcel
WHERE
    type = 'Single Family Residence'
    AND sales.amount <> 0 AND sales.amount < 1500000
    AND properties.building_sqft <> 0
    AND properties.building_sqft < 10000
GROUP BY 1
ORDER BY 1

My Blog

contact@aaronoellis.com (Aaron O. Ellis) — Fri, 25 Oct 2013 00:00:00 +0000

In the beginning the Universe was created.
This has made a lot of people very angry and been widely regarded as a bad move. Douglas Adams, “The Restaurant at the End of the Universe”

Hello.

This is my blog.

I hope its creation will not be widely regarded as a bad move.