index.xml

<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>awm&#39;s blog</title>
    <link>https://ders.github.io/index.xml</link>
    <description>Recent content on awm&#39;s blog</description>
    <generator>Hugo -- gohugo.io</generator>
    <language>en-us</language>
    <lastBuildDate>Tue, 28 Feb 2017 15:51:00 +0900</lastBuildDate>
    <atom:link href="https://ders.github.io/index.xml" rel="self" type="application/rss+xml" />
    
    <item>
      <title>BigQuery</title>
      <link>https://ders.github.io/post/2017-02-28-big-query/</link>
      <pubDate>Tue, 28 Feb 2017 15:51:00 +0900</pubDate>
      
      <guid>https://ders.github.io/post/2017-02-28-big-query/</guid>
      <description>

&lt;p&gt;A recent project involves a script to do a regular data slurp, process it, and write the results to Google BigQuery.  The script runs once an hour via cron.&lt;/p&gt;

&lt;p&gt;The data slurp is such that my script always requests a specific range of data.  Thus if processing fails at any point, I can easily re-slurp the same data and process it again.&lt;/p&gt;

&lt;p&gt;On the BigQuery side, however, I need to ensure data integrity.  I need to ensure that no data are lost, nor are any data inserted twice.  This must be accomplished without ever querying the BigQuery tables, as queries are expensive.&lt;/p&gt;

&lt;p&gt;The strategy then is to fail hard during the data slurp and processing phases, so that if something goes wrong, nothing goes into BigQuery, and we try again in an hour.  This works well for recovering from the occasional communication errors encountered during the data slurp.&lt;/p&gt;

&lt;p&gt;On the other hand, an error during the BigQuery insert phase must not fail hard, as that would leave us in an indeterminate state of having some of our data written.  Instead, BigQuery inserts that fail should be retried and retried again until they succeed.  (Of course I need to make sure that the failures we&amp;rsquo;re retrying are transient, but that&amp;rsquo;s a separate topic.)&lt;/p&gt;

&lt;h2 id=&#34;the-incident&#34;&gt;The Incident&lt;/h2&gt;

&lt;p&gt;Today in the log I found an &amp;ldquo;unknown error&amp;rdquo; entry, which means that something raised an exception in an unexpected place.&lt;/p&gt;

&lt;p&gt;Inspecting the log file, I saw that one of the BigQuery insert calls had encountered a 500 (service temporarily unavailable) response.  This was supposed to trigger an automatic retry, but the retry failed on account of one line of errant logging code.  The script failed hard and marked the job as not done, even though several thousand rows had already made it into BigQuery.&lt;/p&gt;

&lt;p&gt;On the next run an hour later, the script dutifully played catch-up, reprocessing the data that had gone astray and inserting it, this time successfully, into BigQuery.&lt;/p&gt;

&lt;p&gt;So no data have been lost, but I&amp;rsquo;ve failed at preventing duplication.&lt;/p&gt;

&lt;p&gt;Fortunately, we have have a timestamp on every insert, so it should be a relatively simple matter to manually delete everything that was inserted at that particular hour.&lt;/p&gt;

&lt;p&gt;So imagine my surprise and confusion when I discovered that there were exactly zero records timstamped in that range.  The logger clearly showed several batches of 500 inserts successfully completed before the crash; where had all the records gone?&lt;/p&gt;

&lt;p&gt;As it turns out, it&amp;rsquo;s the &lt;a href=&#34;https://cloud.google.com/bigquery/streaming-data-into-bigquery#dataconsistency&#34;&gt;insert ID&lt;/a&gt; that saved us.  Each data point is sent with a unique insert ID which is generated as a function of the data itself.  When BigQuery received insert IDs that it had seen before, it silently deduped the data for us.&lt;/p&gt;

&lt;p&gt;Two observations to note:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The documentation states that BigQuery will remember the insert IDs for &amp;ldquo;at least one minute.&amp;rdquo;  In our case, the duplicate data showed up an hour later and was still detected.&lt;/li&gt;
&lt;li&gt;The deduping resulted in the earlier inserts being discarded and the later inserts being kept.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I&amp;rsquo;ve fixed the errant logging code, by the way.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>A lean, clean Golang machine</title>
      <link>https://ders.github.io/post/2016-12-23-lean-clean-golang-machine/</link>
      <pubDate>Fri, 23 Dec 2016 14:49:00 +0900</pubDate>
      
      <guid>https://ders.github.io/post/2016-12-23-lean-clean-golang-machine/</guid>
      <description>

&lt;p&gt;Writing a &lt;a href=&#34;https://golang.org/&#34;&gt;Go&lt;/a&gt; package that interacts with a relational
data store such as Postgres is full of messiness.&lt;/p&gt;

&lt;p&gt;Those of us who appreciate the strong-typedness of Go probably also appreciate
the strong-typedness of SQL, and vice versa.  Unfortunately, communication
between Go and SQL is less than ideal. This is due partly to the mostly
free-form text format of data exchange (queries) and partly to some subtle
differences in data types.&lt;/p&gt;

&lt;p&gt;Database nulls are a particular headache, leading to the contortions of defining
types such &lt;a href=&#34;https://golang.org/pkg/database/sql/#NullString&#34;&gt;NullString&lt;/a&gt;,
&lt;a href=&#34;https://golang.org/pkg/database/sql/#NullInt64&#34;&gt;NullInt64&lt;/a&gt;, and
&lt;a href=&#34;https://golang.org/pkg/database/sql/#NullBool&#34;&gt;NullBool&lt;/a&gt;, and an extra check
is required every time you want distinguish a null from a zero value.&lt;/p&gt;

&lt;p&gt;Why not use an ORM? There has been
&lt;a href=&#34;http://www.hydrogen18.com/blog/golang-orms-and-why-im-still-not-using-one.html&#34;&gt;a lot written&lt;/a&gt;
&lt;a href=&#34;https://blog.codinghorror.com/object-relational-mapping-is-the-vietnam-of-computer-science/&#34;&gt;on this already&lt;/a&gt;,
but in a nutshell, the level of generality required means that
&lt;a href=&#34;https://godoc.org/github.com/jinzhu/gorm&#34;&gt;pretty much everything is an interface{}&lt;/a&gt;
with runtime checks to cast stuff into the types you need, and at this point
we&amp;rsquo;ve lost the benefits of Go&amp;rsquo;s strong typing and may as well write our whole
application in Ruby.&lt;/p&gt;

&lt;p&gt;I&amp;rsquo;ve found that programmers who appreciate the power and control that comes from
writing in a low-level compiled language such as Go also appreciate the power
can control that comes from writing queries yourself in SQL.&lt;/p&gt;

&lt;h2 id=&#34;so-what-s-the-problem-really&#34;&gt;So what&amp;rsquo;s the problem, really?&lt;/h2&gt;

&lt;p&gt;The real headache of &lt;a href=&#34;https://golang.org/pkg/database/sql/&#34;&gt;Go + SQL&lt;/a&gt; is the
volume of boilerplate code that goes with even relatively simple operations.&lt;/p&gt;

&lt;p&gt;(1) Run a query that doesn&amp;rsquo;t return any results.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;_, err := db.Exec(query, ...args)
if err != nil {
	return err
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;(1a) Run a query that doesn&amp;rsquo;t return any results, but we want to know how many
rows were changed.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;res, err := db.Exec(query, ...args)
if err != nil {
	return err
}
count, err := res.RowsAffected()
if err != nil {
	return err
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;(1b) Run a query that doesn&amp;rsquo;t return any results, and we&amp;rsquo;d like to catch and
process integrity violations (e.g. duplicate entry on a unique field). This one
requires some database-specific code; the example here is for Postgres.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;_, err := db.Exec(query, ...args)
duplicate := false
if err != nil {
	if pgerr, ok := err.(*pq.Error); ok {
		duplicate = pgerr.Code.Class().Name() == &amp;quot;integrity_constraint_violation&amp;quot;
	}
	if !duplicate {
		return err
	}
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;(1c) Run a query that doesn&amp;rsquo;t return any results, and we&amp;rsquo;d like to catch and
process data exceptions (e.g. number out of range). This uses the same strategy as 1b and can be combined with it.&lt;/p&gt;

&lt;p&gt;(2) Run a query that returns one row.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;err := db.QueryRow(query, ...args).Scan(&amp;amp;arg1, &amp;amp;arg2, ... )
if err != nil {
	return err
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;(2a) Run a query that returns one row, and we&amp;rsquo;d like to catch and process the
case where no rows are returned.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;err := db.QueryRow(query, ...args).Scan(&amp;amp;arg1, &amp;amp;arg2, ... )
noRows := err == ErrNoRows
if err != nil &amp;amp;&amp;amp; !noRows {
	return err
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;(3) Run a query that returns multiple rows.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;rows, err := db.Query(query, ...args)
if err != nil {
	return err
}
defer rows.Close()
for rows.Next() {
	err := rows.Scan(&amp;amp;arg1, &amp;amp;arg2, ... )
	if err != nil {
		return err
	}
}
err = rows.Err()
if err != nil {
	return err
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;None of these is particularly bad as far as boilerplate goes, but unless we&amp;rsquo;re
writing an ORM (and we&amp;rsquo;ve already decided we&amp;rsquo;re not), we&amp;rsquo;re going to have tens,
perhaps hundreds of these scattered throughout our application.  Add to that an
other &lt;code&gt;if err != nil&lt;/code&gt; every time we start a transaction, and I&amp;rsquo;m thinking
there&amp;rsquo;s got to be a better way.&lt;/p&gt;

&lt;h2 id=&#34;organizing-database-access-around-high-level-functionality&#34;&gt;Organizing database access around high-level functionality&lt;/h2&gt;

&lt;p&gt;We would like to follow the
&lt;a href=&#34;http://martinfowler.com/eaaCatalog/unitOfWork.html&#34;&gt;unit of work&lt;/a&gt;
pattern and create something akin to the
&lt;a href=&#34;http://docs.sqlalchemy.org/en/latest/orm/session_basics.html&#34;&gt;session model&lt;/a&gt;
of SQLAlchemy.&lt;/p&gt;

&lt;p&gt;A simple example of a unit of work is a password reset, which checks for an
email match, and then generates, saves, and returns a reset code. This will
involve a minimum of two queries, which need to be in the same transaction.
(Much more complicated units of work are possible, of course, both read-only
and read-write.)&lt;/p&gt;

&lt;p&gt;Our goal then is to find a way to have just one copy of all the boilerplate above and be able to substitute queries and argument lists as needed.&lt;/p&gt;

&lt;p&gt;I&amp;rsquo;m going to propose that it&amp;rsquo;s straightforward to implement such a thing Go
by defining a custom transaction handler which extends
&lt;a href=&#34;https://golang.org/pkg/database/sql/#Tx&#34;&gt;the one in database/sql&lt;/a&gt;.
This is done within the package that uses it.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;type Tx struct {
	sql.Tx
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;We extend &lt;code&gt;sql.Tx&lt;/code&gt; with methods to (a) convert all database errors to panics so
that we can catch and process them all in one place, and (b) easily iterate over
result sets.&lt;/p&gt;

&lt;p&gt;To accomplish (a), we add the methods &lt;code&gt;MustExec&lt;/code&gt;, &lt;code&gt;MustQuery&lt;/code&gt;, and &lt;code&gt;MustQueryRow&lt;/code&gt;.
These are identical to &lt;code&gt;Exec&lt;/code&gt;, &lt;code&gt;Query&lt;/code&gt;, and &lt;code&gt;QueryRow&lt;/code&gt; except that they panic
instead of returning an error code. Also, in the case of &lt;code&gt;MustQuery&lt;/code&gt; and &lt;code&gt;MustQueryRow&lt;/code&gt;,
they return custom &lt;code&gt;Rows&lt;/code&gt; and &lt;code&gt;Row&lt;/code&gt; objects that have similar extensions.&lt;/p&gt;

&lt;p&gt;To accomplish (b), we add the method &lt;code&gt;Each&lt;/code&gt; to the custom &lt;code&gt;Rows&lt;/code&gt; object returned
by &lt;code&gt;MustQuery&lt;/code&gt;.  Method &lt;code&gt;Each&lt;/code&gt; iterates over the result set and calls a
callback function for each row.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;ourError&lt;/code&gt; type is used to wrap errors that we want to convert back to error
codes. It distinguishes them from other kinds of panics (e.g. out of memory).&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;type ourError struct {
	err error
}

func (tx Tx) MustExec(query string, args ...interface{}) sql.Result {
	res, err := tx.Exec(query, args...)
	if err != nil {
		panic(ourError{err})
	}
	return res
}

func (tx Tx) MustQuery(query string, args ...interface{}) *Rows {
	rows, err := tx.Query(query, args...)
	if err != nil {
		panic(ourError{err})
	}
	return &amp;amp;Rows{*rows}
}

func (tx Tx) MustQueryRow(query string, args ...interface{}) *Row {
	row := tx.QueryRow(query, args...)
	return &amp;amp;Row{*row}
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;The custom &lt;code&gt;Row&lt;/code&gt; and &lt;code&gt;Rows&lt;/code&gt; types are defined analogously.
&lt;code&gt;Row&lt;/code&gt; is extended with a &lt;code&gt;MustScan&lt;/code&gt; method:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;type Row struct {
	sql.Row
}

func (row Row) MustScan(args ...interface{}) {
	err := row.Scan(args...)
	if err != nil {
		panic(ourError{err})
	}
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;&lt;code&gt;Rows&lt;/code&gt; is extended with a &lt;code&gt;MustScan&lt;/code&gt; method and also with the &lt;code&gt;Each&lt;/code&gt; iterator
described above.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;type Rows struct {
	sql.Rows
}

func (rows Rows) MustScan(args ...interface{}) {
	err := rows.Scan(args...)
	if err != nil {
		panic(ourError{err})
	}
}

func (rows *Rows) Each(f func(*Rows)) {
	defer rows.Close()
	for rows.Next() {
		f(rows)
	}
	err := rows.Err()
	if err != nil {
		panic(ourError{err})
	}
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Now to make it all work, we define a custom transaction function.  It
sets up the transaction, provides the custom transaction handler to our
callback, and then catches the panics.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;func Xaction(db *sql.DB, f func(*Tx)) (err error) {

	var tx *sql.Tx
	tx, err = db.Begin()
	if err != nil {
		return
	}

	defer func() {
		if r := recover(); r != nil {
			if ourerr, ok := r.(ourError); ok {
				// This panic of from tx.Fail() or the equivalent.  Unwrap it,
				// process it, and return it as an error code.
				tx.Rollback()
				err = ourerr.err
				if err == sql.ErrNoRows {
					err = ErrDoesNotExist
				} else if pgerr, ok := err.(*pq.Error); ok {
					switch pgerr.Code.Class().Name() {
					case &amp;quot;data_exception&amp;quot;:
						err = ErrInvalidValue
					case &amp;quot;integrity_constraint_violation&amp;quot;:
						// This could be lots of things: foreign key violation,
						// non-null constraint violation, etc., but we&#39;re generally
						// checking those in advance. As long as our code is in
						// order, unique constraints will be the only things we&#39;re
						// actually relying on the database to check for us.
						err = ErrDuplicate
					}
				}
			} else {
				// not our panic, so propagate it
				panic(r)
			}
		}
	}()

	f(&amp;amp;Tx{*tx}) // this runs the queries

	tx.Commit()
	return
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;This covers all of our boilerplate needs except for (1a) above.
To accommodate (1a), we could extend &lt;code&gt;sql.Result&lt;/code&gt; the same way we extended
the others, but I haven&amp;rsquo;t really needed it yet, so I&amp;rsquo;ll leave it as an
exercise for the reader.&lt;/p&gt;

&lt;p&gt;One final method that&amp;rsquo;s there just to make everything neat and tidy is a &lt;code&gt;Fail&lt;/code&gt;
method on the transaction which can be used to return an arbitrary error.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;func (tx Tx) Fail(err error) {
	panic(ourError{err})
}
&lt;/code&gt;&lt;/pre&gt;

&lt;h2 id=&#34;the-result&#34;&gt;The result&lt;/h2&gt;

&lt;p&gt;Our application code is now a lot neater.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;err := Xaction(func(tx *Tx) {

	// Run a query that doesn&#39;t return any results.
	tx.MustExec(query1, ...args)

	// Run a query that returns one row.
	tx.MustQueryRow(query2, ...args).MustScan(&amp;amp;arg1, &amp;amp;arg2, ... )

	// Run a query that returns multiple rows.
	tx.MustQuery(query3, ...args).Each(func(r *Rows) {
		r.MustScan(&amp;amp;arg1, &amp;amp;arg2, ... )
	})
})

if err != nil {
	switch err {

	case ErrDoesNotExist:
		// query2 returned no rows

	case ErrInvalidValue:
		// data exception

	case ErrDuplicate:
		// integrity violation

	default:
		return err
	}
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;And since this is an extension to the stock transaction handler rather than
a replacement for it, we can still use the original non-must methods for
any edge case that might require a different kind of error handling.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>CRUD APIs are crud</title>
      <link>https://ders.github.io/post/2016-06-21-crud-is-crud/</link>
      <pubDate>Tue, 21 Jun 2016 17:20:00 +0900</pubDate>
      
      <guid>https://ders.github.io/post/2016-06-21-crud-is-crud/</guid>
      <description>

&lt;h2 id=&#34;crud-apis-are-crud&#34;&gt;CRUD APIs are crud&lt;/h2&gt;

&lt;p&gt;I&amp;rsquo;m making the case specifically about &lt;a href=&#34;http://web.archive.org/web/20130116005443/http://tomayko.com/writings/rest-to-my-wife&#34;&gt;REST&lt;/a&gt; APIs, but in fact everything here applies to any API, REST or not.&lt;/p&gt;

&lt;p&gt;It&amp;rsquo;s a common paradigm to create a data model as a collection of tables in a relational database and then access the data from some client app (mobile or web).  &lt;a href=&#34;https://en.wikipedia.org/wiki/Create,_read,_update_and_delete&#34;&gt;CRUD&lt;/a&gt; has become a popular way to access the data, perhaps because it&amp;rsquo;s easy to make and easy to explain.&lt;/p&gt;

&lt;p&gt;In CRUD, we&amp;rsquo;re essentially giving the caller direct access to INSERT, SELECT, UPDATE and DELETE commands on our SQL database.  Or something analogous if you&amp;rsquo;re into NoSQL.  It comes with some permissions checking, of course, but as far as the capabilities of the API, that&amp;rsquo;s pretty much it.&lt;/p&gt;

&lt;p&gt;The worst thing this does is expose the schema to the client, making it difficult to change the internal structure later on.  Want to fix how tags are stored?  Too bad, you&amp;rsquo;re going to break the API.&lt;/p&gt;

&lt;p&gt;Besides that, there&amp;rsquo;s a lot of &lt;a href=&#34;http://www.agiledata.org/essays/relationalDatabases.html#AdvancedFeatures&#34;&gt;database capability&lt;/a&gt; that&amp;rsquo;s missing.&lt;/p&gt;

&lt;p&gt;What happens when we have some business logic, e.g. in a stored procedure?  We&amp;rsquo;ll have to create a separate endpoint for that.&lt;/p&gt;

&lt;p&gt;What happens when we have some limited resource that we need to allocate on a first-come, first-served basis, e.g. room reservations.  Again, we need some special processing to ensure that only one of two simultaneous requests succeed.&lt;/p&gt;

&lt;p&gt;What happens when we need some concept of transactions, that when a series of operations can&amp;rsquo;t be completed we revert back to the original state?  Once again, we need to handle this separately.&lt;/p&gt;

&lt;p&gt;What happens when we need to enforce some consistency between tables?  In the case of foreign key constraints, it&amp;rsquo;s usually enough just to do the updates in the proper order, but other more complicated constraints will either need their own separate endpoints or will need to be momentarily violated.  And being violated is never acceptable, even for just a moment.&lt;/p&gt;

&lt;p&gt;The biggest problem with a CRUD API is that it&amp;rsquo;s &lt;a href=&#34;https://lostechies.com/chrispatterson/2014/01/03/crud-is-not-a-service/&#34;&gt;shifting all the business logic to the caller&lt;/a&gt;, whereas it should instead be invisible to the caller.  Even Microsoft &lt;a href=&#34;https://msdn.microsoft.com/en-us/library/ms954638.aspx#soade_topic3&#34;&gt;recognized CRUD as an anti-pattern&lt;/a&gt;, and that was way back in 2005.  Even when we&amp;rsquo;re only doing read and display, it&amp;rsquo;s often necessary to make several API calls to produce one document, unnecessarily slowing down load times.&lt;/p&gt;

&lt;p&gt;The second-biggest problem with a CRUD API is specific to the update operation.  Update does not represent any realistic use case.  When do you ever want to rewrite an entire database row?  We carry this mistake all the way to the UI, where we press &lt;code&gt;edit&lt;/code&gt; on our profile, get back all of our data in input fields, change one field, and then write everything back.&lt;/p&gt;

&lt;h2 id=&#34;apis-that-work&#34;&gt;APIs that work&lt;/h2&gt;

&lt;p&gt;I&amp;rsquo;m proposing a way to approach APIs, a way that avoids the pitfalls of CRUD.  If you&amp;rsquo;re practicing &lt;a href=&#34;http://dddcommunity.org/learning-ddd/what_is_ddd/&#34;&gt;domain-driven design (DDD)&lt;/a&gt;, this will happen naturally.  (Side note: at our company, we&amp;rsquo;ve been using DDD since day one, but no one here knew there was a buzzword for it.)  None of what I&amp;rsquo;m proposing is new or groundbreaking; it&amp;rsquo;s just the way we should be doing things.&lt;/p&gt;

&lt;p&gt;For read operations, there is one API call per display operation.  Everything needed to render the requested view comes back as one bundle.  Dynamic web content that&amp;rsquo;s generated server-side is done this way, and the API can too.  As a bonus, we can use the same API as internal for server-generated pages and as external for client-generated views.&lt;/p&gt;

&lt;p&gt;For write operations, there is a one-to-one correspondence between a user action and an API call.  On the backend, one API call is one transaction, and if any part fails, then the whole thing fails.  (Side note: one should never, ever build a system where it&amp;rsquo;s possible for only part of a user action to succeed.  Usability nightmare.)&lt;/p&gt;

&lt;p&gt;If we absolutely need some CRUD-style functionality (e.g. updating one&amp;rsquo;s profile), we should make our updates one field at a time.  Not only does this match more closely what the average user will be doing, but it gives us an easy way to manage concurrency: simply require an update call to specify both the old and new value.  If the old value doesn&amp;rsquo;t match, it&amp;rsquo;s an error.&lt;/p&gt;

&lt;h2 id=&#34;tracking-changes-and-archiving&#34;&gt;Tracking changes and archiving&lt;/h2&gt;

&lt;p&gt;Tracking changes and archiving are two capabilities that are often added to a data store as an afterthought.  I&amp;rsquo;d like to be proactive and incorporate them into the data design from the beginning.&lt;/p&gt;

&lt;p&gt;The simplest way to track changes is with created-at and updated-at fields on every db model, and most database engines have neat ways to auto-update these fields.  This level of tracking is of limited use, however, as we don&amp;rsquo;t know what changed or who changed it.&lt;/p&gt;

&lt;p&gt;There are plenty of add-ons to do detailed revision tracking (&lt;a href=&#34;https://django-reversion.readthedocs.io/en/stable/&#34;&gt;django-reversion&lt;/a&gt; is one I like), but I&amp;rsquo;m a little bit concerned about the performance hit.  Also, such add-ons make the created-at and updated-at fields redundant.  That&amp;rsquo;s probably a good thing.&lt;/p&gt;

&lt;p&gt;As for archiving, a common technique is to add a boolean field called &lt;code&gt;archived&lt;/code&gt; to every model you want to be able to archive.  On this plus side, it&amp;rsquo;s easy not to break references when you have non-archived data that refers to archived data, but &lt;a href=&#34;https://en.wikipedia.org/wiki/Design_smell&#34;&gt;we really shouldn&amp;rsquo;t have that happening&lt;/a&gt;.  On the minus side, we end up adding &lt;code&gt;and not archived&lt;/code&gt; to nearly every query.&lt;/p&gt;

&lt;p&gt;We also might want to be able to permanently delete some archived material after a certain expiration time.  We&amp;rsquo;d then need an &lt;code&gt;archived_at&lt;/code&gt; field as well.&lt;/p&gt;

&lt;p&gt;Here&amp;rsquo;s where CRUD fails again:  Archive a record by setting &lt;code&gt;archived&lt;/code&gt; to true and write it back.  Unarchive it similarly.  Determine the age of data by reading the created-at and updated-at fields on the model.&lt;/p&gt;

&lt;p&gt;I propose that archiving and revision tracking can be implemented together in a way that&amp;rsquo;s clean and transparent to the client.&lt;/p&gt;

&lt;p&gt;Instead of adding extra fields to the models, all the archive and tracking information goes into a read/append-only journal, which may or may not be implemented as a database table.&lt;/p&gt;

&lt;p&gt;The journal contains one entry for each user action (see above). If there are system actions (e.g. daily aggregations) that get written to the database, those get included as well.  Each entry contains a before-and-after detail of all changes.  Since this before-and-after detail will only ever be accessed as a whole, it&amp;rsquo;s reasonable to make it one json bundle in a text field.&lt;/p&gt;

&lt;p&gt;Archiving simply becomes a delete operation, as all the details are archived in the history.  This means, of course, that related data needs to be archived together, which is a good thing.  Furthermore, it&amp;rsquo;s trivial to put a time limit on data retention; simply delete old journal entries.&lt;/p&gt;

&lt;p&gt;My next API is going to rock.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>The Django REST framework</title>
      <link>https://ders.github.io/post/2016-04-14-django-rest-framework/</link>
      <pubDate>Thu, 14 Apr 2016 10:22:00 +0900</pubDate>
      
      <guid>https://ders.github.io/post/2016-04-14-django-rest-framework/</guid>
      <description>&lt;p&gt;I may have to reconsider choosing Go for some server applications.&lt;/p&gt;

&lt;p&gt;There&amp;rsquo;s a bit of a learning curve, but
version 3 of the &lt;a href=&#34;http://www.django-rest-framework.org/&#34;&gt;Django REST framework&lt;/a&gt;
packs a lot of nice features.
The web browsable API is the one that won me over.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Why I code in Go for server applications</title>
      <link>https://ders.github.io/post/2016-03-16-why-i-code-in-go/</link>
      <pubDate>Wed, 16 Mar 2016 16:47:00 +0900</pubDate>
      
      <guid>https://ders.github.io/post/2016-03-16-why-i-code-in-go/</guid>
      <description>&lt;p&gt;I&amp;rsquo;ve written server applications in Ruby, Python, and Go.  With Ruby I&amp;rsquo;ve tried out both Sinatra and Rails; with Python I&amp;rsquo;ve used Flask and Django; with Go I&amp;rsquo;ve used the net/http package.&lt;/p&gt;

&lt;p&gt;There are endless arguments for and against using this framework or that language, and there are many valid reasons to like or dislike a set of tools.  I personally like Django a lot.  But Go has two features that beat the competition when it comes to writing web services: static typing and explicit error handling.&lt;/p&gt;

&lt;p&gt;In Ruby, we often find ourselves having to check if a value is nil before processing it.  Anything can be nil, and unexpected inputs often create nil values where we least expect.  If we forget to check just one place in the code, sooner or later it shows up as a 500 error and our service is &lt;a href=&#34;https://www.youtube.com/watch?v=nZiDS-4Xd2k&#34;&gt;broken&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Test suites should cover this, but it&amp;rsquo;s just as easy to miss one edge case in a test suite as it is to miss one in the main code.&lt;/p&gt;

&lt;p&gt;In Go, nothing can be nil (unless it&amp;rsquo;s a pointer, but it&amp;rsquo;s easy to know when a pointer might not have been initialized).  In the case of unexpected input, a variable is set to its zero value (e.g. 0, &amp;#39;&amp;#39;, {}), and the fact that there was unexpected input &lt;a href=&#34;http://dave.cheney.net/2015/01/26/errors-and-exceptions-redux&#34;&gt;is conveyed separately&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;In Python, we often find ourselves having to convert types, especially in the case of numerical inputs into string variables.  Using a string where an int is required will raise a TypeError exception, and casting a non-numeric string to int will raise a ValueError exception.  Here too, it&amp;rsquo;s all too easy to miss one try-except block and get a 500 error.&lt;/p&gt;

&lt;p&gt;Again, test suites should cover this, but that means a test for every possible branch in the code.  Again, it&amp;rsquo;s just as easy to miss one edge case in a test suite as it is to miss one in the main code.&lt;/p&gt;

&lt;p&gt;In Go, compatible types are checked at compile time, thereby eliminating this source of errors.&lt;/p&gt;

&lt;p&gt;I choose Go for the simple reason that most 500-inducing code bugs can be either caught at compile time or avoided entirely.  The result is faster and more stable deployments than the alternatives.&lt;/p&gt;

&lt;p&gt;OK, I lied.  I choose Go because I like it.  But this is a great way to justify my choice.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>A simple sentiment analysis of two US presidential candidates</title>
      <link>https://ders.github.io/post/2016-02-18-a-simple-sentiment-analysis/</link>
      <pubDate>Wed, 03 Feb 2016 17:51:00 +0900</pubDate>
      
      <guid>https://ders.github.io/post/2016-02-18-a-simple-sentiment-analysis/</guid>
      <description>

&lt;p&gt;&lt;strong&gt;Goal:&lt;/strong&gt;  To do some basic &lt;a href=&#34;https://en.wikipedia.org/wiki/Sentiment_analysis&#34;&gt;sentiment analysis&lt;/a&gt; on video content.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Test cases:&lt;/strong&gt;  Two 5-minute clips of US presidential candidate speeches.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Strategy:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Extract 5 minutes of audio from the beginning of each video.&lt;/li&gt;
&lt;li&gt;Generate a transcript using a speech-to-text program.&lt;/li&gt;
&lt;li&gt;Feed the transcript into a sentiment analyzer.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&#34;the-original-content&#34;&gt;The original content&lt;/h2&gt;

&lt;iframe width=&#34;560&#34; height=&#34;315&#34; src=&#34;https://www.youtube.com/embed/qOQCw7Hcwic&#34; frameborder=&#34;0&#34; allowfullscreen&gt;&lt;/iframe&gt;

&lt;iframe width=&#34;560&#34; height=&#34;315&#34; src=&#34;https://www.youtube.com/embed/p5ZB8Lg1tcA&#34; frameborder=&#34;0&#34; allowfullscreen&gt;&lt;/iframe&gt;

&lt;h2 id=&#34;extracting-a-5-minute-audio-clip&#34;&gt;Extracting a 5-minute audio clip&lt;/h2&gt;

&lt;p&gt;There are many ways to do this.  One way is to download the video using a browser add-on.  Browser add-ons are easy to find but are also fickle, as they make it easy to download material in violation of copyright.  And if you&amp;rsquo;re downloading from YouTube, you&amp;rsquo;re violating their terms of service, even if you&amp;rsquo;re not infringing on copyright.  (We maintain that this exercise falls under &lt;a href=&#34;https://en.wikipedia.org/wiki/Fair_use&#34;&gt;fair use&lt;/a&gt;.)&lt;/p&gt;

&lt;p&gt;Another way is to turn on audio capture while playing the video.&lt;/p&gt;

&lt;p&gt;After the capture is complete, we&amp;rsquo;ll want to convert to FLAC if we&amp;rsquo;re not there already.  We&amp;rsquo;ll use &lt;a href=&#34;https://ffmpeg.org/&#34;&gt;ffmpeg&lt;/a&gt; for this, e.g.:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;ffmpeg -i captured-content.mp4 captured-content.flac
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;And then to extract the first 5 minutes:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;flac --until=5:00 captured-content.flac -o five-minute-clip.flac
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Both &lt;code&gt;ffmpeg&lt;/code&gt; and &lt;code&gt;flac&lt;/code&gt; are available via homebrew.&lt;/p&gt;

&lt;h2 id=&#34;generating-a-transcript&#34;&gt;Generating a transcript&lt;/h2&gt;

&lt;p&gt;The IBM Watson Developer Cloud has a &lt;a href=&#34;http://www.ibm.com/smarterplanet/us/en/ibmwatson/developercloud/speech-to-text.html&#34;&gt;speech-to-text&lt;/a&gt; service which is available through an API and also has a &lt;a href=&#34;https://speech-to-text-demo.mybluemix.net/&#34;&gt;demo page&lt;/a&gt;.  In theory, one can get limited free access to the API after going through a mildly annoying sign-up process, but in practice I was unable to convince the API to accept the credentials I&amp;rsquo;d obtained.&lt;/p&gt;

&lt;p&gt;Fortunately, the demo page allows file uploads and produced the following transcripts from the content above:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Trump&lt;/strong&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;This is no way I&amp;rsquo;m leaving South Carolina. And I was gonna leave for tonight come back as it upsets up saying you have a five days we got a win on Saturday we&amp;rsquo;re going to win. Make America great again we&amp;rsquo;re gonna make America. We&amp;rsquo;re going to win. You know. It&amp;rsquo;s been an amazing friend Ruben all over the state today I love you too girly looking out dnmt love you I love you all. I love you. So many things are happening for a country. And it&amp;rsquo;s this is a movement time magazine last week at the most beautiful story cover. And they talk about its improvement they&amp;rsquo;ve never seen they say there&amp;rsquo;s not been anything like this I don&amp;rsquo;t know ever but they actually say ever. We went to Tampa the other day Tampa Florida would like two days notice fifteen thousand people that are turned away five thousand and by the way for all of the people in this room I can&amp;rsquo;t believe it this is a huge room but downstairs to filling up another one and there sadly sending people away we don&amp;rsquo;t like that right. No okay why don&amp;rsquo;t we all get up go let&amp;rsquo;s have that now. Now we have one of the great veterans outside I&amp;rsquo;ll stand up while one of the great great you are great. Love this guy. He loves the veterans and I love the veterans are we going to take care of our veterans I&amp;rsquo;ll tell you that we&amp;rsquo;re going to take it did not. They are not properly taken care of so we&amp;rsquo;re going to take a right we have sent a look at this. I knew you guys would say that I can spot a veteran a long ways off. But we are we going to take a break here we&amp;rsquo;re gonna take you have a military we&amp;rsquo;re going to take you have a military because our military is being whittled away whittled away we&amp;rsquo;re going to make our military so big so strong so powerful nobody&amp;rsquo;s going to mess with us anymore nobody nobody. Nobody. So Nikki Haley a very nice woman she better speech the other day you saw that and she was talking about anger and she said there&amp;rsquo;s a lot of anger and I guess she was applying all of us you know really referring to us. And by the end of the day she was actually saying that Donald Trump is a friend of mine he&amp;rsquo;s been a supporter of mine everything else you know the tone at the beginning was designed by the time that she was just barraged with people she said I think we better change our path here. And by the end of the day and it was fun it was great but she said you know there is anger but I said there is a group and I was asked during not this debate but the previous debate I was asked. I by the way did you love this last debate dnmt. Listen to like. They came at me from every angle possible. Don&amp;rsquo;t know they came out before every angle you know sort of interesting. They were hitting me with things like and such until as you know I never realized I&amp;rsquo;ve always don&amp;rsquo;t politicians it is honest but I&amp;rsquo;ve never known the level of dishonesty. And I deal in industries a lot of different but mostly real estate and like in Manhattan at different places but I&amp;rsquo;ve never seen people as dishonest as politicians. They will say anything. Like okay so a lot of you people understand that you you get when you&amp;rsquo;ve seen the speeches that you see in a lot of it and you know that I protect the second amendment more than anybody by far dnmt more than. And this guy Ted Cruz gets upset Donald Trump does not respect a second amendment and the more that anybody I&amp;rsquo;m with the second amendment. I saw no no it&amp;rsquo;s lies. And then they do commercials and you know he did it to Ben Carson and him in particular in all fairness. Jeff is represents but these are minor misrepresentations and he&amp;rsquo;s not going anywhere anyway so what would how casual. Not as. Well Jeb was talking about eminent domain Donald Trump used eminent domain privately then I see there&amp;rsquo;s a big story I had to bring this out luck. Proof Jeb bush under eminent domain took a disabled veterans property. Something about me. No state. Honestly these guys are these guys are the worst. Eminent domain without eminent domain by the way you don&amp;rsquo;t up highways roads airports hospitals you know not bridges you drive anything so. They say Donald Trump does look like Eminem and I don&amp;rsquo;t even tell me but you need to road you need a highway you need you know it&amp;rsquo;s funny they all want they all want the keystone pipeline right but without eminent domain without think of it without eminent domain you can&amp;rsquo;t have the keystone pipeline and we&amp;rsquo;re going to get the keystone pipeline approved but but fluids jumps. It&amp;rsquo;s jobs but remember this when it gets approved a politicians go to baby approve it.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;strong&gt;Sanders&lt;/strong&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;President Falwell and. David. Ok thank you very much for inviting my wife Jane and. Ought to be with you this morning we appreciate the invitation. Very much. And let me start off by acknowledging what I think. All of you already know. And that is the views. That many here at liberty university have. And all I. On a number of important issues. A very very different. I believe in women&amp;rsquo;s rights dnmt. In the light of the woman to control her own body dnmt. I believe in gay rights. And now. Those of my views. And it is no secret. But I came here today. Because I believe from the bottom of my heart. That it is vitally important for those of us. Who hold different views. To be able to engage in any civil discourse. Who often in our country and I think both sides. Bear responsibility for us. There is too much shouting at each other. There is too much making fun of each other. Now in my view then are you can say this is somebody who whose voice is hoarse because I have given dozens of speeches. And the last few months it is easy. To go out and talk to people who agree with you are missing Greensboro North Carolina just last night. Alright. We are nine thousand people out. Mostly they agreed with me tonight. We&amp;rsquo;re going to be a Manassas and thousands out they agree with me. It&amp;rsquo;s not a whole lot to do. That&amp;rsquo;s what politicians by and large do we go out and we talk to people who agree with us. But it is harder. But not less important. For us to try and communicate with those who do not agree with us on every issue. After. And it is important to see where if possible and I do believe it&amp;rsquo;s possible we can find common grounds. No liberty university. Is a religious school obviously. Pn. All of you are proud of the. You already school. Which as all of us in our own way. Tries to understand the meaning of morality. What does it mean. To live a moral life. And you try to understand in this very complicated modern world that we live in. What the words of the Bible me in today&amp;rsquo;s society. You are in school which tries to teach its students. How to behave with decency and with honesty and how you can best relates. To your fellow human beings and I applaud. You for trying to achieve those goals. Let me. Take a moment. Or a few moments. To tell you what motivates me. And the work that I do. As a public servant as a Sentinel. From the state of Vermont. And let me tell you that it goes without saying I am flaws foh throw me being a perfect human being. But all I am motivated by a vision.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2 id=&#34;sentiment-analysis&#34;&gt;Sentiment Analysis&lt;/h2&gt;

&lt;p&gt;Again, there are a variety of tools to do this, including the &lt;a href=&#34;http://www.nltk.org/&#34;&gt;Natural Language Toolkit Project&lt;/a&gt;, a free python library. Taking advantage of a &lt;a href=&#34;http://text-processing.com/demo/sentiment/&#34;&gt;simple demo site&lt;/a&gt; which uses the NLTK, we can see that both Sanders and Trump are polar, but Sanders is more positive.  Who would&amp;rsquo;ve known?&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Trump&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Overall: negative&lt;/li&gt;
&lt;li&gt;Subjectivity

&lt;ul&gt;
&lt;li&gt;neutral: 0.2&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;polar: 0.8&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;li&gt;Polarity

&lt;ul&gt;
&lt;li&gt;pos: 0.4&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;neg: 0.6&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Sanders&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Overall: positive&lt;/li&gt;
&lt;li&gt;Subjectivity

&lt;ul&gt;
&lt;li&gt;neutral: 0.2&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;polar: 0.8&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;li&gt;Polarity

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;pos: 0.8&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;neg: 0.2&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For the adventuresome, &lt;a href=&#34;http://www.nltk.org/howto/sentiment.html&#34;&gt;here are more detailed instructions&lt;/a&gt; on using the NLTK for sentiment analysis.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Using Grunt to Manage Static Assets</title>
      <link>https://ders.github.io/post/2016-02-03-static-assets-with-grunt/</link>
      <pubDate>Wed, 03 Feb 2016 17:51:00 +0900</pubDate>
      
      <guid>https://ders.github.io/post/2016-02-03-static-assets-with-grunt/</guid>
      <description>

&lt;p&gt;I &lt;a href=&#34;https://ders.github.io/post/2016-01-14-static-assets-for-websites/&#34;&gt;previously posted&lt;/a&gt; about using GNU Make to manage front-end assets for a website.  A colleague suggested that I should check out &lt;a href=&#34;http://gruntjs.com/getting-started&#34;&gt;Grunt&lt;/a&gt; as it does everything I need to do and more.  So here it is.&lt;/p&gt;

&lt;p&gt;I have the same goals as I did last week:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;concatenate an arbitrary combination of js files, minifying them in the process&lt;/li&gt;
&lt;li&gt;preprocess css with sass&lt;/li&gt;
&lt;li&gt;copy directories i and lib untouched&lt;/li&gt;
&lt;li&gt;run a watch process to update files as they&amp;rsquo;re changed&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&#34;installing-grunt&#34;&gt;Installing grunt&lt;/h2&gt;

&lt;p&gt;Grunt is part of the &lt;a href=&#34;https://nodejs.org/&#34;&gt;node.js&lt;/a&gt; ecosystem, and as such is available via &lt;a href=&#34;https://www.npmjs.com/&#34;&gt;the node package manager (npm)&lt;/a&gt;.  Npm is available on OS X via Homebrew.&lt;/p&gt;

&lt;h3 id=&#34;basic-npm-concepts&#34;&gt;Basic npm concepts&lt;/h3&gt;

&lt;p&gt;There are a few things that we need to understand about npm.  The biggest headache was recognizing the difference between local and global installs and knowing when to use which.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Npm installs packages into a project (unless the &lt;code&gt;-g&lt;/code&gt; global option is specified, more on that later) and needs to be run in project root.  Packages then go into a subdirectory called &lt;code&gt;node_packages&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;If you&amp;rsquo;re in some other directory when running npm, the packages will go into a &lt;code&gt;node_packages&lt;/code&gt; subdirectory there and confuse you.&lt;/li&gt;
&lt;li&gt;Npm expects to see a file called &lt;code&gt;package.json&lt;/code&gt; in the project root directory and complains if it&amp;rsquo;s not there.
&lt;code&gt;package.json&lt;/code&gt; includes a list of packages that the project depends on, and the default &lt;code&gt;npm install&lt;/code&gt; without any parameters installs those packages.&lt;/li&gt;
&lt;li&gt;When installing a package explicitly, there is in an option to add an entry to &lt;code&gt;package.json&lt;/code&gt; so that someone else will be able to use &lt;code&gt;npm install&lt;/code&gt; and get everything.  Note that this is an option and not the default behavior.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&#34;creating-the-package-json-file&#34;&gt;Creating the package.json file&lt;/h3&gt;

&lt;p&gt;According to &lt;a href=&#34;http://gruntjs.com/getting-started#package.json&#34;&gt;the documentation&lt;/a&gt;,
the command to use is &lt;code&gt;npm init&lt;/code&gt;, and it must be run in project root.  Running it starts a dialog on the terminal, asking some mostly irrelevant questions: name (defaults to the name of the project directory), version (defaults to 1.0.0), description, entry point (defaults to index.js), test command, git repository, keywords, author, and license (defaults to ISC).  These questions can be suppressed by using &lt;code&gt;npm init --yes&lt;/code&gt;, which defaults everything.&lt;/p&gt;

&lt;p&gt;Unfortunately, npm will complain if it doesn&amp;rsquo;t see a description, a repository field and a license field.  The defaults only cover the license field, leaving the description blank and the repository field missing altogether.&lt;/p&gt;

&lt;p&gt;The minimum &lt;code&gt;package.json&lt;/code&gt; has &lt;a href=&#34;https://docs.npmjs.com/getting-started/using-a-package.json#requirements&#34;&gt;just a name and a version&lt;/a&gt;.
But since I&amp;rsquo;m &lt;a href=&#34;https://www.bignerdranch.com/blog/a-bit-on-warnings/&#34;&gt;a stickler for getting rid of warnings&lt;/a&gt;, I&amp;rsquo;m going to have to create my own &lt;code&gt;package.json&lt;/code&gt; that includes name, version, description, repository and license.  None of this information is relevant; its only purpose is to make the warnings go away.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;{
  &amp;quot;name&amp;quot;: &amp;quot;taco&amp;quot;,
  &amp;quot;version&amp;quot;: &amp;quot;1.0.0&amp;quot;,
  &amp;quot;description&amp;quot;: &amp;quot;xyz&amp;quot;,
  &amp;quot;repository&amp;quot;: {
    &amp;quot;type&amp;quot;: &amp;quot;git&amp;quot;,
    &amp;quot;url&amp;quot;: &amp;quot;xyz&amp;quot;
  },
  &amp;quot;license&amp;quot;: &amp;quot;ISC&amp;quot;
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Unfortunately there&amp;rsquo;s one warning I can&amp;rsquo;t get rid of.  At the time of this writing, &lt;code&gt;npm install grunt&lt;/code&gt; produces this:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;npm WARN deprecated lodash@0.9.2: lodash@&amp;lt;2.0.0 is no longer maintained. Upgrade to lodash@^3.0.0
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;According to &lt;a href=&#34;https://github.com/lodash/lodash/wiki/Changelog#v092&#34;&gt;the changelog for lodash&lt;/a&gt;,
version 0.9.2 was released in 2012, and the current version is 4.0.0.  Even the &amp;ldquo;upgrade to&amp;rdquo; version of 3.0.0 is a year old already.  This is a red flag; how and why are these dependencies not getting maintained?  That said, it appears that &lt;a href=&#34;https://github.com/gruntjs/grunt/issues/1419&#34;&gt;an update is on the way&lt;/a&gt;.  Will have to ignore this warning for now.&lt;/p&gt;

&lt;h3 id=&#34;grunt-plugins&#34;&gt;Grunt plugins&lt;/h3&gt;

&lt;p&gt;Grunt itself is just the overlord; to do any real work we&amp;rsquo;re going to need some plugins.  After a lot of googling, I&amp;rsquo;ve come up with this list:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;To minify and combine javascript files, we can use &lt;code&gt;grunt-contrib-uglify&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;To compile scss into css, we can use &lt;code&gt;grunt-contrib-sass&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;To copy directories, we can use &lt;code&gt;grunt-contrib-copy&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;To delete old files, we can use &lt;code&gt;grunt-contrib-clean&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;To watch for changes and recompile, we can use &lt;code&gt;grunt-contrib-watch&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;All of these are &lt;a href=&#34;http://gruntjs.com/plugins&#34;&gt;marked as officially maintained&lt;/a&gt;, giving us the warm, fuzzy feeling that everything is going to work.&lt;/p&gt;

&lt;p&gt;We can now install grunt and the plugins.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;npm install grunt grunt-contrib-uglify grunt-contrib-sass grunt-contrib-copy grunt-contrib-clean grunt-contrib-watch --save-dev
&lt;/code&gt;&lt;/pre&gt;

&lt;h3 id=&#34;grunt-command-line&#34;&gt;Grunt command line&lt;/h3&gt;

&lt;p&gt;There is one more install required if we are to be able to run grunt from the command line.  The package is &lt;code&gt;grunt-cli&lt;/code&gt;, and needs to be installed globally so that the grunt executable goes into /usr/local/bin and is available in the system path.&lt;/p&gt;

&lt;p&gt;npm install grunt-cli -g&lt;/p&gt;

&lt;p&gt;It&amp;rsquo;s possible to install &lt;code&gt;grunt-cli&lt;/code&gt; in the project directory, but then the executable will be in node_modules/.bin instead of /usr/local/bin, and that makes more headaches for us&lt;/p&gt;

&lt;p&gt;One gotcha is that the global grunt-cli requires a local grunt or it will fail.  Grunt-cli is a wrapper to find the locally installed grunt to whatever project you&amp;rsquo;re in.  The global grunt-cli will not find a global grunt.&lt;/p&gt;

&lt;h3 id=&#34;summary-of-grunt-installation&#34;&gt;Summary of grunt installation&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Install npm (e.g. &lt;code&gt;brew install npm&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;Create the package.json file shown above.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;npm install grunt grunt-contrib-uglify grunt-contrib-sass grunt-contrib-copy grunt-contrib-clean grunt-contrib-* watch --save-dev&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;npm install grunt-cli -g&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;code&gt;package.json&lt;/code&gt; should go into source control, and &lt;code&gt;node_modules&lt;/code&gt; should be excluded from source control with the appropriate entry in &lt;code&gt;.gitignore&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Once we have &lt;code&gt;package.json&lt;/code&gt; as updated by the npm install &amp;ndash;save-dev command, steps 2 and 3 can be replaced by a simple &lt;code&gt;npm install&lt;/code&gt;.  We still need to keep step 4; global packages can&amp;rsquo;t go into &lt;code&gt;package.json&lt;/code&gt; (npm will ignore &lt;code&gt;--save-dev&lt;/code&gt; when &lt;code&gt;-g&lt;/code&gt; is specified).&lt;/p&gt;

&lt;h3 id=&#34;optionally-installing-grunt-cli-locally&#34;&gt;Optionally installing grunt-cli locally&lt;/h3&gt;

&lt;p&gt;Installing &lt;code&gt;grunt-cli&lt;/code&gt; locally instead of globally will allow it to be included in &lt;code&gt;package.json&lt;/code&gt;, but it has the side effect of not having the grunt executable in the path.  A possible workaround to this side effect is to add a script section to &lt;code&gt;package.json&lt;/code&gt; with all the grunts you want to do.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;&amp;quot;scripts&amp;quot;: { &amp;quot;watch&amp;quot;: &amp;quot;grunt watch&amp;quot; }
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Then you can type &lt;code&gt;npm run watch&lt;/code&gt; instead of &lt;code&gt;grunt watch&lt;/code&gt;.  This may or may not be worth the trouble.&lt;/p&gt;

&lt;h2 id=&#34;writing-a-gruntfile&#34;&gt;Writing a gruntfile&lt;/h2&gt;

&lt;h3 id=&#34;basic-gruntfile-concept&#34;&gt;Basic gruntfile concept&lt;/h3&gt;

&lt;p&gt;The gruntfile is a bit of javascript initialization that gets run whenever grunt is invoked.  The gruntfile needs to define an initialization function and assign that to the global &lt;code&gt;module.exports&lt;/code&gt;.  Within the initialization function, we&amp;rsquo;ll need to list the modules we need (grunt-contrib-uglify, etc.), specify some configuration for each module, define the default task, and optionally define additional tasks.&lt;/p&gt;

&lt;p&gt;Each plugin defines a task of the same name as the plugin (e.g. grunt-contrib-uglify defines an &amp;ldquo;uglify&amp;rdquo; task, under which any number of subtasks may be defined).&lt;/p&gt;

&lt;p&gt;The gruntfile is named &lt;code&gt;Gruntfile.js&lt;/code&gt; and resides in project root.  The basic gruntfile structure is:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;module.exports = function(grunt) {
  grunt.initConfig({
    pluginname: { ... }  // one of these for each plugin
  };
  grunt.loadNpmTasks( ... );  // one of these for each plugin
  grunt.registerTask(&#39;default&#39;, ... );  // define the default behavior of `grunt` with no parameters
  grunt.registerTask( ... );  // optional additional tasks
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Each plugin defines a task of the same name as the plugin (e.g. grunt-contrib-uglify defines an &amp;ldquo;uglify&amp;rdquo; task, under which any number of subtasks may be defined).  Defining additional tasks is useful for combining tasks into a single command.&lt;/p&gt;

&lt;p&gt;A thorough read of &lt;a href=&#34;http://gruntjs.com/getting-started&#34;&gt;the docs&lt;/a&gt; along with &lt;a href=&#34;https://www.google.co.kr/search?q=gruntfile+examples&#34;&gt;some examples&lt;/a&gt; gives us enough information to build a single gruntfile, giving us the following commands:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;grunt&lt;/code&gt; does a clean build, deleting &lt;code&gt;pub&lt;/code&gt; if it exists and building everything from &lt;code&gt;src&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;grunt build&lt;/code&gt; does an incremental build of js and css files, updating only those files whose source has changed.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;grunt copy&lt;/code&gt; syncs the directories &lt;code&gt;i&lt;/code&gt; and &lt;code&gt;lib&lt;/code&gt; from &lt;code&gt;src&lt;/code&gt; to &lt;code&gt;pub&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;grunt watch&lt;/code&gt; runs until you kill it, watching for changes in &lt;code&gt;src&lt;/code&gt; and updating &lt;code&gt;pub&lt;/code&gt; as necessary.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Note that &lt;code&gt;grunt&lt;/code&gt; is short for &lt;code&gt;grunt all&lt;/code&gt;, which does &lt;code&gt;grunt clean&lt;/code&gt; + &lt;code&gt;grunt copy&lt;/code&gt; + &lt;code&gt;grunt build&lt;/code&gt;.&lt;/p&gt;

&lt;script src=&#34;https://gist.github.com/ders/3ca946b14641e5efe783.js&#34;&gt;&lt;/script&gt;

&lt;h3 id=&#34;observations&#34;&gt;Observations&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Overall, the quality of documentation is poor.  I had to resort to copying examples and then modifying them by trial and error until I got the results I wanted.  There are many alternate syntaxes, causing further confusion.&lt;/li&gt;
&lt;li&gt;Could not find a way to do incremental updates with uglify.  The entire js collection is rebuilt whenever any js source file changes.&lt;/li&gt;
&lt;li&gt;The sass plugin depends on having command-line sass installed as a ruby gem, a dependency that I grudgingly accepted when writing the previous makefile and was hoping to avoid.&lt;/li&gt;
&lt;li&gt;Dependencies from &lt;code&gt;@import&lt;/code&gt; statements in scss source files are handled nicely; the dependencies are honored when doing an incremental build and don&amp;rsquo;t need to be included in the gruntfile.  This is nice.&lt;/li&gt;
&lt;li&gt;The &lt;code&gt;grunt-contrib-copy&lt;/code&gt; plugin doesn&amp;rsquo;t know how to sync. The &lt;code&gt;i&lt;/code&gt; and &lt;code&gt;lib&lt;/code&gt; directories are copied in their entireties every time there&amp;rsquo;s a change.  There is &lt;a href=&#34;https://github.com/tomusdrw/grunt-sync&#34;&gt;another plugin&lt;/a&gt; which claims to know how to sync, but I haven&amp;rsquo;t tested it.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&#34;conclusion&#34;&gt;Conclusion&lt;/h3&gt;

&lt;p&gt;This was a whole lot of trouble to set up a relatively simple build system. Grunt is a powerful tool, and I can see the value of using it when you&amp;rsquo;re already in a node-based project, but it to use it as an isolated build tool is not worth the effort.&lt;/p&gt;

&lt;p&gt;The only thing we gained with Grunt is the ability to auto-detect imports in .scss files and do incremental updates accordingly.  At the same time we lost the ability to incremental updates of the Javascript files, at least with the standard plugin.&lt;/p&gt;

&lt;p&gt;I was also hoping to avoid the ruby sass dependency by using the plugin, but no luck there since the plugin is just a wrapper for the command line sass.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Static assets for websites</title>
      <link>https://ders.github.io/post/2016-01-14-static-assets-for-websites/</link>
      <pubDate>Thu, 14 Jan 2016 14:22:00 +0900</pubDate>
      
      <guid>https://ders.github.io/post/2016-01-14-static-assets-for-websites/</guid>
      <description>

&lt;p&gt;Count me in on the developers who believe that &lt;a href=&#34;http://www.gnu.org/software/make/manual/make.html&#34;&gt;GNU make&lt;/a&gt; is the best tool for assembling static assets.&lt;/p&gt;

&lt;h3 id=&#34;the-general-problem&#34;&gt;The general problem&lt;/h3&gt;

&lt;p&gt;We need to maintain a set of files B that is derived from another set of files A through some known (and possibly complicated) transformation.  We edit the files in set A but not in set B.  We would like a simple way to (1) create B from A, and (2) update B when A changes, only recreating the parts that are necessary.&lt;/p&gt;

&lt;h3 id=&#34;the-more-specific-problem&#34;&gt;The more specific problem&lt;/h3&gt;

&lt;p&gt;B is the set of static assets for a web service, and A is the set of source files used to make them.  Only A will be checked into source control, and only B will be uploaded to the web server.&lt;/p&gt;

&lt;p&gt;There are different kinds of assets in A that need to be treated differently.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Javascript&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;My Javascript source files are formatted nicely and full of meaningful, well-thought-out comments.  I would like the js files sent with the web pages to be devoid of comments and mashed together so as to be almost unreadable.  This can be accomplished by piping the files through &lt;a href=&#34;http://www.crockford.com/javascript/jsmin.html&#34;&gt;JSMin&lt;/a&gt; on the way from A to B.&lt;/p&gt;&lt;/li&gt;

&lt;li&gt;&lt;p&gt;My Javascript source files are modular, and one page may need several files.  These are best combined into one file for faster loading.  Also, any source file could be included in several combination files.  I would like the ability to have each js file in B created from an arbitrary combination of source files from A.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;CSS&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;All my css is written as scss and needs to be processed with an scss compiler such as  &lt;a href=&#34;http://sass-lang.com/&#34;&gt;Sass&lt;/a&gt;.  Scss files may import other sccs files, a fact we need to be aware of when detecting changes.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Other assets such as images and precompiled libraries can be copied from A to B without modification.&lt;/p&gt;

&lt;h3 id=&#34;what-to-do&#34;&gt;What to do&lt;/h3&gt;

&lt;p&gt;The first thing is to define a directory structure.&lt;/p&gt;

&lt;p&gt;For set A we&amp;rsquo;ll make a subdirectory &lt;code&gt;src&lt;/code&gt; in project root with four subdirectories: &lt;code&gt;js&lt;/code&gt; for Javascript sources, &lt;code&gt;css&lt;/code&gt; for scss sources, &lt;code&gt;i&lt;/code&gt; for image files, and &lt;code&gt;lib&lt;/code&gt; for precompiled libraries.&lt;/p&gt;

&lt;p&gt;For set B we&amp;rsquo;ll make a subdirectory &lt;code&gt;pub&lt;/code&gt; in project root.  Compiled js and css files will go directly in &lt;code&gt;pub&lt;/code&gt;, and the two subdirectories &lt;code&gt;i&lt;/code&gt; and &lt;code&gt;lib&lt;/code&gt; will mirror &lt;code&gt;src/i&lt;/code&gt; and &lt;code&gt;src/lib&lt;/code&gt;.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;.
├── src
│   ├── js
│   ├── css
│   ├── i
│   └── lib
└── pub
    ├── i
    └── lib
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Next we need to make a list of the js and css files we would like generated and placed into &lt;code&gt;pub&lt;/code&gt;.  We&amp;rsquo;ll do that by defining variables &lt;code&gt;JSFILES&lt;/code&gt; and &lt;code&gt;CSSFILES&lt;/code&gt;, e.g.:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;JSFILES := main.js eggs.js pancake.js
CSSFILES := blueberry.css yogurt.css
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;After that, we need to define the dependencies for each of these files, e.g.:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;pub/main.js: src/js/main.js
pub/eggs.js: src/js/eggs.js src/js/milk.js
pub/pancake.js: src/js/milk.js src/js/flour.js src/js/eggs.js

pub/blueberry.css: src/css/blueberry.scss src/css/fruit.scss
pub/yogurt.css: src/css/yogurt.scss
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;To simplify things, we&amp;rsquo;ll define the default dependeny to be one source file of the same name, so we can omit dependency definitions for &lt;code&gt;main.js&lt;/code&gt; and &lt;code&gt;yogurt.css&lt;/code&gt;.  We&amp;rsquo;ll also define &lt;code&gt;JS := src/js&lt;/code&gt;, &lt;code&gt;CSS := src/css&lt;/code&gt; and &lt;code&gt;PUB := pub&lt;/code&gt;.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;$(PUB)/eggs.js: $(JS)/eggs.js $(JS)/milk.js
$(PUB)/pancake.js: $(JS)/milk.js $(JS)/flour.js $(JS)/eggs.js
$(PUB)/blueberry.css: $(CSS)/blueberry.scss $(CSS)/fruit.scss
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Finally, we need to make a list of directories to be copied directly from &lt;code&gt;src&lt;/code&gt; to &lt;code&gt;pub&lt;/code&gt;.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;COPYDIRS := lib i
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;This is now enough information for us to build a simple makefile, giving us (at least) the following commands:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;make&lt;/code&gt; does a clean build, deleting &lt;code&gt;pub&lt;/code&gt; if it exists and building everything from src.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;make build&lt;/code&gt; does an incremental build of js and css files, updating only those files whose source has changed.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;make copy&lt;/code&gt; syncs the directories &lt;code&gt;i&lt;/code&gt; and &lt;code&gt;lib&lt;/code&gt; from &lt;code&gt;src&lt;/code&gt; to &lt;code&gt;pub&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;make watch&lt;/code&gt; runs until you kill it, watching for changes in &lt;code&gt;src&lt;/code&gt; and updating &lt;code&gt;pub&lt;/code&gt; as necessary.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Note that &lt;code&gt;make&lt;/code&gt; is short for &lt;code&gt;make all&lt;/code&gt;, which does &lt;code&gt;make clean&lt;/code&gt; + &lt;code&gt;make copy&lt;/code&gt; + &lt;code&gt;make build&lt;/code&gt;.&lt;/p&gt;

&lt;script src=&#34;https://gist.github.com/ders/627147bf67544c96f8be.js&#34;&gt;&lt;/script&gt;

&lt;h3 id=&#34;how-it-works&#34;&gt;How it works&lt;/h3&gt;

&lt;p&gt;The meat of this makefile is in the pattern rules (lines 43-55).  Quick cheat sheet:  &lt;code&gt;$@&lt;/code&gt; = target, &lt;code&gt;$^&lt;/code&gt; = all dependencies, &lt;code&gt;$&amp;lt;&lt;/code&gt; = the first dependency.  &lt;a href=&#34;http://www.gnu.org/software/make/manual/make.html#Automatic-Variables&#34;&gt;Details are here.&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The first rule takes care of &lt;code&gt;main.js&lt;/code&gt; and &lt;code&gt;eggs.js&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The second rule takes care of &lt;code&gt;pancake.js&lt;/code&gt;.  Note that &lt;code&gt;pancake.js&lt;/code&gt; doesn&amp;rsquo;t match the first rule because there is no source file called pancake.&lt;/p&gt;

&lt;p&gt;The third rule takes care of &lt;code&gt;blueberry.css&lt;/code&gt; and &lt;code&gt;yogurt.css&lt;/code&gt;.  Note that on line 55 &lt;code&gt;fruit.scss&lt;/code&gt; is &lt;strong&gt;not&lt;/strong&gt; supplied as an argument to sass.  It&amp;rsquo;s only listed as a dependency because &lt;code&gt;blueberry.scss&lt;/code&gt; contains an &lt;code&gt;@import &amp;quot;sass&amp;quot;;&lt;/code&gt; directive.&lt;/p&gt;

&lt;p&gt;Finally, lines 32-36 take care of syncing directories &lt;code&gt;i&lt;/code&gt; and &lt;code&gt;lib&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;In the end, our filesystem looks like this:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;.
├── src
│   ├── js
│   │   ├── eggs.js
│   │   ├── flour.js
│   │   ├── main.js
│   │   └── milk.js
│   ├── css
│   │   ├── blueberry.scss
│   │   ├── fruit.scss
│   │   └── yogurt.scss
│   ├── i
│   │   ├── hanjan.jpg
│   │   └── ikant.png
│   └── lib
│       └── MooTools-Core-1.5.2-compressed.js
└── pub
│   ├── i
│   │   ├── hanjan.jpg
│   │   └── ikant.png
│   ├── lib
│   │   └── MooTools-Core-1.5.2-compressed.js
│   ├── blueberry.css
│   ├── eggs.js
│   ├── main.js
│   ├── pancake.js
│   └── yogurt.css
└── Makefile
&lt;/code&gt;&lt;/pre&gt;

&lt;h3 id=&#34;dependencies&#34;&gt;Dependencies&lt;/h3&gt;

&lt;p&gt;This makefile requires &lt;code&gt;jsmin&lt;/code&gt;, &lt;code&gt;sass&lt;/code&gt; and &lt;code&gt;watchman-make&lt;/code&gt; to be available at the command line.&lt;/p&gt;

&lt;p&gt;Jsmin and &lt;a href=&#34;https://facebook.github.io/watchman/docs/install.html&#34;&gt;Watchman&lt;/a&gt; (which includes watchman-make) are available on OS X via Homebrew.  Sass is not (yet), but it can be installed as a system-wide ruby gem.  I&amp;rsquo;m not a fan of requiring rubygems for my decidedly anti-rails build system, but since Sass runs nicely from the command line I&amp;rsquo;ll turn a blind eye for now.&lt;/p&gt;

&lt;p&gt;Jsmin is also &lt;a href=&#34;https://libraries.io/npm/jsmin&#34;&gt;available via npm&lt;/a&gt;.&lt;/p&gt;

&lt;h3 id=&#34;other-features-i-d-like-to-include&#34;&gt;Other features I&amp;rsquo;d like to include&lt;/h3&gt;

&lt;p&gt;Would be nice to automatically detect @import statements in scss source files and generate dependency lists based on that.  I&amp;rsquo;m aware that the Sass package has it&amp;rsquo;s own watcher that handles dependencies, but using that would mean bypassing a significant part of the makefile, thereby making a mess.&lt;/p&gt;

&lt;p&gt;It would be pretty simple to add a &lt;code&gt;make deploy&lt;/code&gt; command to rsync the server.  I&amp;rsquo;ll probably do that later.&lt;/p&gt;

&lt;h3 id=&#34;a-feature-i-excluded-on-purpose&#34;&gt;A feature I excluded on purpose&lt;/h3&gt;

&lt;p&gt;Many web frameworks automatically append timestamps or version numbers to static assets in order to defeat browser caching.  This adds a whole lot of complexity for a pretty minor benefit.  Once a site is in production, I expect updates to be few and far between, and I&amp;rsquo;m happy to manually add a version number to a target filename as necessary.&lt;/p&gt;

&lt;h3 id=&#34;credits&#34;&gt;Credits&lt;/h3&gt;

&lt;p&gt;This Makefile was heavily influenced by and owes thanks to &lt;a href=&#34;http://west.io/post/2015/04/11-frontend-builds-with-makefiles/&#34;&gt;this blog post&lt;/a&gt;.  Thank you!&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Google Sign-in</title>
      <link>https://ders.github.io/post/2016-01-08-google-signin/</link>
      <pubDate>Fri, 08 Jan 2016 15:50:00 +0900</pubDate>
      
      <guid>https://ders.github.io/post/2016-01-08-google-signin/</guid>
      <description>

&lt;h2 id=&#34;using-google-sign-in-on-website-x&#34;&gt;Using Google Sign-in on Website X&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Disclaimer:&lt;/strong&gt;  &lt;a href=&#34;https://developers.google.com/identity/sign-in/web/sign-in&#34;&gt;Read the docs&lt;/a&gt; too.  This post doesn&amp;rsquo;t cover everything.&lt;/p&gt;

&lt;p&gt;A week ago I was completely clueless as to how Google sign-in works.  I set out to write about it and learned a few things.&lt;/p&gt;

&lt;h3 id=&#34;overview&#34;&gt;Overview&lt;/h3&gt;

&lt;p&gt;Using Google sign-in on a website requires first doing the following in the &lt;a href=&#34;https://console.developers.google.com/home/dashboard&#34;&gt;Google developer&amp;rsquo;s console&lt;/a&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;creating a project&lt;/li&gt;
&lt;li&gt;creating a sign-in client ID for that project&lt;/li&gt;
&lt;li&gt;associating the domain(s) of the website with the sign-in client ID&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Sign-in is done using javascript on the web page to talk directly to Google&amp;rsquo;s servers.  The javascript is loaded from Google&amp;rsquo;s servers.  It is not necessary to involve the server for website X at all.&lt;/p&gt;

&lt;p&gt;When Joe the Hacker attempts to sign in to website X, a popup dialog appears.  The contents of the dialog depend on Joe&amp;rsquo;s current signed-in state.&lt;/p&gt;

&lt;p&gt;If Joe is not signed in to Google at all, then a sign-in dialog appears.  If he&amp;rsquo;s signed in to more than one account, then an account chooser dialog appears.  If he&amp;rsquo;s signed into exactly one account, then the sign-in part is skipped.&lt;/p&gt;

&lt;p&gt;If this is the first time he&amp;rsquo;s attempted to sign in to website X, then he&amp;rsquo;ll be asked to give permission for website X to have access to his profile information (name, picture) and email address.&lt;/p&gt;

&lt;p&gt;In the case that Joe needs neither the sign-in dialog nor the permissions dialog (i.e. he&amp;rsquo;s already signed in to exactly one account and is a returning user), then the pop-up closes itself immediately without any user action.&lt;/p&gt;

&lt;p&gt;The browser remembers that Joe is signed in to website X using Google sign-in.  He can sign out of website X and still be signed into Google.  However, if he signs out of Google, then he&amp;rsquo;ll automatically be signed out of website X as well.  He can&amp;rsquo;t be signed in to website X using his Google ID and not also be signed in to Google.&lt;/p&gt;

&lt;p&gt;If the webpage making the sign-in call is served from a domain that has not been registered in the console, then Joe will see a 400 error (redirect_uri_mismatch) and a picture of a broken robot when trying to sign in.  The error page also exposes the email address of the account that the project is made under.&lt;/p&gt;

&lt;h3 id=&#34;javascript-details&#34;&gt;Javascript details&lt;/h3&gt;

&lt;p&gt;The file platform.js provides the global Google API object called &lt;code&gt;gapi&lt;/code&gt; and the auth2 module.  The auth2 module must be explicitly loaded into gapi with the &lt;code&gt;gapi.load&lt;/code&gt; method before it&amp;rsquo;s used.  This method provides an optional callback for when/if the module is loaded successfully.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;gapi.load(&amp;quot;auth2&amp;quot;, callback);
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Once the module is loaded, it must be initialized with the sign-in client ID (see above).  The client ID may either be provided as &lt;a href=&#34;https://developers.google.com/identity/sign-in/web/reference#gapiauth2initwzxhzdk20paramswzxhzdk21&#34;&gt;an option to the init method&lt;/a&gt; or in &lt;a href=&#34;https://developers.google.com/identity/sign-in/web/sign-in#specify_your_apps_client_id&#34;&gt;a meta tag in the document&lt;/a&gt;.  The init function returns a GoogleAuth object.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;gauth = gapi.auth2.init(options);
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;A logical initialization flow would be to have the initialization in the load callback.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;gapi.load(&amp;quot;auth2&amp;quot;, function() { gapi.auth2.init(); });
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;The GoogleAuth object may also be obtained any time after it&amp;rsquo;s initialized using the &lt;a href=&#34;https://developers.google.com/identity/sign-in/web/reference#gapiauth2getauthinstance&#34;&gt;&lt;code&gt;getAuthInstance&lt;/code&gt;&lt;/a&gt; method.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;gauth = gapi.auth2.getAuthInstance();
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;At this point we can find out whether Joe is or isn&amp;rsquo;t signed in to website X.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;if (gauth.isSignedIn.get()) { alert(&amp;quot;User is signed in, and I debug with alerts.&amp;quot;); }
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;If he isn&amp;rsquo;t signed in, we can try to sign him in.  This is done with &lt;code&gt;GoogleAuth.signIn&lt;/code&gt;.  In most cases, we should wait for him to click a button (e.g. &amp;ldquo;Sign In with Google&amp;rdquo;) before doing so.  There are &lt;a href=&#34;https://developers.google.com/identity/sign-in/web/reference#googleauthsigninwzxhzdk72optionswzxhzdk73&#34;&gt;a few options&lt;/a&gt; that we can ignore for now.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;gauth.SignIn(options)
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;This function returns &lt;a href=&#34;https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise&#34;&gt;a Promise&lt;/a&gt;, making it easy to do stuff when it finishes.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;gauth.SignIn().then(function() {
    alert(&amp;quot;We&#39;re in!&amp;quot;);
}, function() {
    alert(&amp;quot;You FAILED!&amp;quot;);
});
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;There is also this thing called a GoogleUser object, which we can get after Joe has signed in.  (We can also get it before he&amp;rsquo;s signed in, but it would be useless.)  The GoogleUser object reveals Joe&amp;rsquo;s name, email address and profile picture if he has one.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;user = gauth.currentUser.get();
profile = user.getBasicProfile();
alert(profile.getName() + &amp;quot; &amp;quot; + profile.getEmail());
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Finally, use can use &lt;code&gt;GoogleAuth.signOut&lt;/code&gt; to sign Joe out of website X.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;gauth.signOut();
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Although this signs him out, it doesn&amp;rsquo;t forget about him.  The GoogleUser object is still available and still has his ID.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;user = gauth.currentUser.get();
alert(user.getId());
alert(user.isSignedIn());    // false
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;And we can sign him in again using the grant method.  This behaves slightly differently from &lt;code&gt;GoogleAuth.signIn&lt;/code&gt; in that it doesn&amp;rsquo;t give Joe a chance to choose a different account.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;user.grant();
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;According to the documentation, &lt;code&gt;GoogleUser.signIn&lt;/code&gt; is equivalent to &lt;code&gt;GoogleUser.grant&lt;/code&gt;, but my Firebug tells me that there&amp;rsquo;s in fact no method called signIn on the GoogleUser object.  Bad documentation.&lt;/p&gt;

&lt;h3 id=&#34;what-usually-happens-in-practice&#34;&gt;What usually happens in practice&lt;/h3&gt;

&lt;p&gt;Google provides &lt;a href=&#34;https://developers.google.com/identity/sign-in/web/sign-in#add_a_google_sign-in_button&#34;&gt;a handy shortcut&lt;/a&gt; so that anyone can add Google sign-in to their page without knowing any javascript.  As soon as platform.js is loaded, it checks for the existence of a &lt;code&gt;div.g-signin2&lt;/code&gt; (&amp;lt;div class=&amp;ldquo;g-signin2&amp;rdquo;&amp;gt;) and springs into action if found.&lt;/p&gt;

&lt;p&gt;It loads the auth2 module and puts a standard sign-in button in the div (clobbering what was there, watch out), and wires up the button so that when you press it, it logs you in.  It can also call a function of your choosing on sign-in, passing in Joe&amp;rsquo;s GoogleUser object.&lt;/p&gt;

&lt;p&gt;The button appears to be rendered with &lt;a href=&#34;https://developers.google.com/identity/sign-in/web/reference#gapisignin2renderwzxhzdk114idwzxhzdk115_wzxhzdk116optionswzxhzdk117&#34;&gt;&lt;code&gt;gapi.signin2.render&lt;/code&gt;&lt;/a&gt;, and specifying options like this works as expected:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;&amp;lt;div class=&amp;quot;g-signin2&amp;quot; data-onsuccess=&amp;quot;onSignIn&amp;quot; data-longtitle=&amp;quot;true&amp;quot; data-width=&amp;quot;200&amp;quot;&amp;gt;&amp;lt;/div&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;One caution here is that data-longtitle is a string value that gets cast to a boolean, and, as such, the string &amp;ldquo;false&amp;rdquo; will get cast to true.  The way to make longtitle false is set data-longtitle=&amp;quot;&amp;quot; (or to omit it entirely, as false is the default).&lt;/p&gt;

&lt;h3 id=&#34;involving-the-server-for-website-x&#34;&gt;Involving the server for website X&lt;/h3&gt;

&lt;p&gt;Chances are website X will want to know about Joe&amp;rsquo;s sign-in on the server side.  This will require some javascript to send a token to the server, which the server can then decode to get Joe&amp;rsquo;s information.&lt;/p&gt;

&lt;p&gt;The token can be obtained from the GoogleUser object.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;token = user.getAuthResponse().id_token
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;The token is signed to prevent spoofing, and it&amp;rsquo;s up to the server code to &lt;a href=&#34;https://developers.google.com/identity/sign-in/web/backend-auth#verify-the-integrity-of-the-id-token&#34;&gt;verify its integrity&lt;/a&gt;.&lt;/p&gt;

&lt;h3 id=&#34;scopes&#34;&gt;Scopes&lt;/h3&gt;

&lt;p&gt;Scopes are the permissions that we&amp;rsquo;re asking Joe to give to website X.  The default for sign-in is scopes profile and email, and we get Joe&amp;rsquo;s name, email, and picture if available.  &lt;em&gt;(Nerdy detail: the name and picture are not part of any scope; see below.)&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;With the default settings, Joe will see the a request for the following permissions the first time he attempts to sign in to website X:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Know who you are on Google&lt;/li&gt;
&lt;li&gt;View your email address&lt;/li&gt;
&lt;li&gt;View your basic profile info&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If other permissions (e.g. calendar) are needed, they can be added with the &lt;code&gt;scope&lt;/code&gt; option in GoogleAuth.init, the &lt;code&gt;scope&lt;/code&gt; option in GoogleAuth.signIn, or the &lt;code&gt;data-scope&lt;/code&gt; tag on div.g-signin2.  Most scopes are expressed as urls, e.g. &amp;quot;https:&amp;#47;&amp;#47;www.googleapis.com/auth/calendar.readonly&amp;rdquo;.&lt;/p&gt;

&lt;p&gt;Here is &lt;a href=&#34;https://developers.google.com/identity/protocols/googlescopes&#34;&gt;a list of all known scopes&lt;/a&gt;.&lt;/p&gt;

&lt;h3 id=&#34;fewer-scopes&#34;&gt;Fewer Scopes&lt;/h3&gt;

&lt;p&gt;The default scopes can be turned off with by sending &lt;code&gt;fetch_basic_profile: false&lt;/code&gt; as one of the options to GoogleAuth.init.  (Note that this precludes using the auto magic button.)  In this case, at least one scope must be explicitly specified with the scope parameter.&lt;/p&gt;

&lt;p&gt;The two scopes that are included by default are &lt;code&gt;profile&lt;/code&gt; and &lt;code&gt;email&lt;/code&gt;.  Adding the &lt;code&gt;profile&lt;/code&gt; scope does nothing.  It only provides Joe&amp;rsquo;s 21-digit ID, which we get anyway.  Adding the &lt;code&gt;email&lt;/code&gt; scope gives us Joe&amp;rsquo;s email address, but the only way to get it is to decode &lt;code&gt;id_token&lt;/code&gt;, as &lt;code&gt;GoogleUser.getBasicProfile&lt;/code&gt; only works when &lt;code&gt;fetch_basic_profile&lt;/code&gt; is true.&lt;/p&gt;

&lt;p&gt;Strangely, requesting only the &lt;code&gt;email&lt;/code&gt; scope causes permission requests #1 and #2 to be displayed, even though we only get the email access.  Requesting only the &lt;code&gt;profile&lt;/code&gt; scope causes only #3 to be displayed.  Requesting both &lt;code&gt;email&lt;/code&gt; and &lt;code&gt;profile&lt;/code&gt; displays #2 and #3 but not #1.  I have yet to understand the logic behind this behavior.&lt;/p&gt;

&lt;p&gt;Also strange is the fact that neither the &lt;code&gt;profile&lt;/code&gt; nor &lt;code&gt;email&lt;/code&gt; scope provides Joe&amp;rsquo;s name and picture.  As far as I can tell, the only way to get his name and picture is to stick to the default &lt;code&gt;fetch_basic_profile: true&lt;/code&gt;.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Happy new year 11111100000</title>
      <link>https://ders.github.io/post/2016-01-05-happy-new-year-11111100000/</link>
      <pubDate>Tue, 05 Jan 2016 11:45:00 +0900</pubDate>
      
      <guid>https://ders.github.io/post/2016-01-05-happy-new-year-11111100000/</guid>
      <description>&lt;p&gt;Or, happy new year (2&lt;sup&gt;6−1&lt;/sup&gt;)(2&lt;sup&gt;6&lt;/sup&gt;−1).&lt;/p&gt;

&lt;p&gt;We just wrapped up a long-running big data project.  While I didn&amp;rsquo;t find the
project itself particularly interesting, I learned quite a bit of interesting
stuff doing it.&lt;/p&gt;

&lt;p&gt;On the skills side, I learned &lt;a href=&#34;https://golang.org&#34;&gt;Go&lt;/a&gt;,
learned how to navigate AWS,
and gained a good understanding of how cluster data stores work.&lt;/p&gt;

&lt;p&gt;But probably the most important takeaway from last year is summarized
neatly in &lt;a href=&#34;http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html&#34;&gt;this blog post&lt;/a&gt; from 2010, recently sent to me by a coworker.
In a nutshell, there are a lot of big data systems that are far
larger and more complicated than they need to be for the data that
they are designed to process.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>New blog engine</title>
      <link>https://ders.github.io/post/2015-07-02-new-blog-engine/</link>
      <pubDate>Thu, 02 Jul 2015 11:14:56 +0900</pubDate>
      
      <guid>https://ders.github.io/post/2015-07-02-new-blog-engine/</guid>
      <description>&lt;p&gt;Trying out &lt;a href=&#34;http://gohugo.io/&#34;&gt;Hugo&lt;/a&gt; as a new blog engine.  If you&amp;rsquo;re reading this, then I&amp;rsquo;ve succeeded.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Why you should always understand what&#39;s going on under the hood</title>
      <link>https://ders.github.io/post/2014-01-27-why-you-should-always-understand-whats-going-on-under-the-hood/</link>
      <pubDate>Mon, 27 Jan 2014 10:07:00 +0900</pubDate>
      
      <guid>https://ders.github.io/post/2014-01-27-why-you-should-always-understand-whats-going-on-under-the-hood/</guid>
      <description>

&lt;p&gt;This is about how I got bitten by &lt;a href=&#34;http://datamapper.org/&#34;&gt;DataMapper&lt;/a&gt;.
No infection and I didn&amp;rsquo;t get rabies, but it still hurt.&lt;/p&gt;

&lt;p&gt;I should mention that overall I think DataMapper is pretty good.
It has most of the core features that an ORM should have without unneeded complexity.
It beats the pants off of Active Somethingorother when it comes to making clean, backend-independent models.&lt;/p&gt;

&lt;p&gt;Unfortunately &lt;em&gt;most of the core features&lt;/em&gt; isn&amp;rsquo;t quite &lt;em&gt;all of the core features&lt;/em&gt;, and when you really need that one missing piece, that&amp;rsquo;s when the trouble starts.&lt;/p&gt;

&lt;p&gt;What bit me was the combination of two unrelated things:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;I needed an atomic update with conditions attached.&lt;/li&gt;
&lt;li&gt;Datamapper has different representations of the Boolean type, even between different SQLs.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&#34;the-missing-feature&#34;&gt;The Missing Feature&lt;/h3&gt;

&lt;p&gt;The glaringly missing feature is the ability to update a value and know
whether or not we actually changed anything.&lt;/p&gt;

&lt;p&gt;In SQL this would look like this:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;UPDATE &amp;quot;quarks&amp;quot; SET &amp;quot;seen&amp;quot; = &#39;t&#39;, &amp;quot;whosaw&amp;quot; = &#39;ders&#39; WHERE &amp;quot;id&amp;quot; = 42 AND &amp;quot;seen&amp;quot; = &#39;f&#39;
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Then I would get the number of affected rows to see if a change was made
(i.e. whether or not the quark had already been seen).&lt;/p&gt;

&lt;p&gt;So let&amp;rsquo;s try it with DataMapper:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;Quark.all(:id =&amp;gt; 42, :seen =&amp;gt; false).update(:seen =&amp;gt; true, :whosaw =&amp;gt; &#39;ders&#39;)
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Two problems here.
The first problem is that the return value is always true, and we don&amp;rsquo;t
get to see how many rows were updated.
This isn&amp;rsquo;t too big a deal;
we can work around it by using &lt;code&gt;Quark.first&lt;/code&gt; instead of &lt;code&gt;Quark.all&lt;/code&gt;,
generating an exception if no records are found.&lt;/p&gt;

&lt;p&gt;The second problem is the dealbreaker.
Datamapper insists on generating two separate queries for the single
update statement:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;SELECT &amp;quot;id&amp;quot;, &amp;quot;size&amp;quot;, &amp;quot;name&amp;quot;, &amp;quot;seen&amp;quot;, &amp;quot;whosaw&amp;quot; FROM &amp;quot;quarks&amp;quot; WHERE (&amp;quot;id&amp;quot; = 42 AND &amp;quot;seen&amp;quot; = &#39;f&#39;) ORDER BY &amp;quot;id&amp;quot;
UPDATE &amp;quot;quarks&amp;quot; SET &amp;quot;seen&amp;quot; = &#39;t&#39;, &amp;quot;whosaw&amp;quot; = &#39;ders&#39; WHERE &amp;quot;id&amp;quot; = 42
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;This obviously won&amp;rsquo;t do, as it&amp;rsquo;s not thread-safe.
Two users running this code at the same time would both believe that they saw the quark first.&lt;/p&gt;

&lt;h3 id=&#34;the-solution&#34;&gt;The Solution&lt;/h3&gt;

&lt;p&gt;The solution was to write the update query in SQL.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;da = DataMapper.repository(:default).adapter
r = da.execute(&amp;quot;UPDATE quarks SET seen=&#39;t&#39;, whosaw=&#39;ders&#39; WHERE id=42 AND seen=&#39;f&#39;&amp;quot;)
if r.affected_rows &amp;gt; 0
   ... # we saw it first
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Kind of defeats the purpose of having an ORM, but it gets the job done.
And as it turns out, I&amp;rsquo;m
&lt;a href=&#34;http://stackoverflow.com/questions/18650932/how-to-add-a-where-clause-in-update-query-in-datamapper&#34;&gt;not the first one&lt;/a&gt;
to run into this issue.&lt;/p&gt;

&lt;h3 id=&#34;how-i-got-bitten&#34;&gt;How I Got Bitten&lt;/h3&gt;

&lt;p&gt;Little did I know that the internal mapping of a Boolean field varies
between SQL implementations.
For Sqlite and Postgres, it&amp;rsquo;s a character field with &lt;code&gt;&#39;t&#39;&lt;/code&gt; and &lt;code&gt;&#39;f&#39;&lt;/code&gt; values, whereas for MySQL it&amp;rsquo;s the integers 1 and 0.&lt;/p&gt;

&lt;p&gt;In my case the unit tests all passed, but the live server (with a MySQL backend)
started returning 500s.&lt;/p&gt;

&lt;p&gt;It&amp;rsquo;s easy enough to change the query to work with MySQL:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;r = da.execute(&amp;quot;UPDATE quarks SET seen=1, whosaw=&#39;ders&#39; WHERE id=42 AND seen=0&amp;quot;)
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;But then the unit tests fail.&lt;/p&gt;

&lt;p&gt;In the end, I wrote this bit of horrible code to keep the tests passing
and the live server happy.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;t, f = if da.options[&#39;scheme&#39;] == &#39;sqlite&#39;
  [&amp;quot;&#39;t&#39;&amp;quot;, &amp;quot;&#39;f&#39;&amp;quot;] # sqlite
else
  [1, 0] # mysql
end

  ...

r = da.execute(&amp;quot;UPDATE quarks SET seen=#{t}, whosaw=&#39;ders&#39; WHERE id=42 AND seen=#{f}&amp;quot;)
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;(There may be a way to extract &lt;code&gt;t&lt;/code&gt; and &lt;code&gt;f&lt;/code&gt; directly from the
DataMapper internals, but I&amp;rsquo;m not that good yet.)&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Python</title>
      <link>https://ders.github.io/post/2014-01-17-try-it/</link>
      <pubDate>Fri, 17 Jan 2014 12:20:00 +0900</pubDate>
      
      <guid>https://ders.github.io/post/2014-01-17-try-it/</guid>
      <description>&lt;pre&gt;&lt;code&gt;$ python
&amp;gt;&amp;gt;&amp;gt; import this
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Just try it.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>AWS Locale</title>
      <link>https://ders.github.io/post/2014-01-10-aws-locale/</link>
      <pubDate>Fri, 10 Jan 2014 14:55:00 +0900</pubDate>
      
      <guid>https://ders.github.io/post/2014-01-10-aws-locale/</guid>
      <description>&lt;p&gt;Every time I start a new EC2 Ubuntu instance, I&amp;rsquo;m confronted with the following
warning when I ssh in:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;_____________________________________________________________________
WARNING! Your environment specifies an invalid locale.
 This can affect your user experience significantly, including the
 ability to manage packages. You may install the locales by running:

   sudo apt-get install language-pack-UTF-8
     or
   sudo locale-gen UTF-8

To see all available language packs, run:
   apt-cache search &amp;quot;^language-pack-[a-z][a-z]$&amp;quot;
To disable this message for all users, run:
   sudo touch /var/lib/cloud/instance/locale-check.skip
_____________________________________________________________________
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Furthermore, a variety of package installations fail with some complaint
related to the locale, the default language, or both.
And for some reason the advice to install relevant language packs
is not helpful.&lt;/p&gt;

&lt;p&gt;It turns out that there are some of environment variables
(&lt;code&gt;LANGUAGE&lt;/code&gt;, &lt;code&gt;LC_CTYPE&lt;/code&gt; and &lt;code&gt;LC_ALL&lt;/code&gt; to be specific) that are not
set properly.&lt;/p&gt;

&lt;p&gt;The advice to install language packs assumes that these environment
variables are set to a language that&amp;rsquo;s not installed.  However, in the
case of a new EC2 instance, these variables are not set at all.&lt;/p&gt;

&lt;p&gt;An easy way to get the warnings to go away is to edit the file
&lt;code&gt;/etc/default/locale&lt;/code&gt; so that these variables always get set. I&amp;rsquo;ve found
that the default installation only sets &lt;code&gt;LANG&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;/etc/default/locale&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;LANG=en_US.UTF-8
LANGUAGE=en_US
LC_CTYPE=en_US.UTF-8
LC_ALL=en_US.UTF-8
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;As always, it&amp;rsquo;s also a good idea to make sure you have the latest and
greatest packages:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;$ sudo apt-get update
$ sudo apt-get upgrade
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;And finally, while we&amp;rsquo;re at it, why not set the timezone?&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;$ sudo dpkg-reconfigure tzdata
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Next time I need to set up a new EC2 instance,
I&amp;rsquo;ll come read my own blog and know exactly what to do.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Woke up, fell out of bed</title>
      <link>https://ders.github.io/post/2013-12-17-woke-up/</link>
      <pubDate>Tue, 17 Dec 2013 11:06:00 +0900</pubDate>
      
      <guid>https://ders.github.io/post/2013-12-17-woke-up/</guid>
      <description>&lt;p&gt;Things I learned this year:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ruby.&lt;/strong&gt; Ruby is an interesting language, but I find that the ability to
open up classes and modify methods on the fly is too easily abused.
Also not a fan of all the alternate method names (e.g. map and collect) &amp;ndash;
would prefer just to choose one and stick with it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Git&lt;/strong&gt; and &lt;strong&gt;Github.&lt;/strong&gt; Made crystal clear with SourceTree.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rails.&lt;/strong&gt; Learned to hate it. Was going to write a blog post on why I
hate rails, but someone
&lt;a href=&#34;http://kakubei.blogspot.kr/2012/05/why-i-hate-rails.html&#34;&gt;beat me to it&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rubygems.&lt;/strong&gt; Learned to hate them too. I&amp;rsquo;ve wasted a stupid amount of
time figuring out that my program is broken because the gems I&amp;rsquo;m using
don&amp;rsquo;t work the way they&amp;rsquo;re supposed to. Better to just write it myself.
(I actually think the gem system is pretty cool, but quality control,
ladies and gentlemen, quality control.)
And if you can do something in two lines of ruby code, you don&amp;rsquo;t need a gem.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Data Mapper.&lt;/strong&gt; I wish this were maintained better. It&amp;rsquo;s a handy tool,
especially when you&amp;rsquo;re dealing with simple, table-friendly data.
It&amp;rsquo;s not so good at complex queries, though, and I think that the lofty goal
of making it compatible with all different kinds of database engines
is hampering its ability to work really well with the most common ones (e.g. SQL).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Python.&lt;/strong&gt; Python rocks. My next job will be in a python shop.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Django.&lt;/strong&gt; Django rocks along with python. In my next life (when I get
really good at python and am independently wealthy with a lot of free time)
I&amp;rsquo;m going to be a regular contributor to the django project.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Unfuddle&lt;/strong&gt; and &lt;strong&gt;Pivotal Tracker.&lt;/strong&gt; Say what? What is there to learn there?
In fact there&amp;rsquo;s a lot to learn if you&amp;rsquo;ve never used an issue tracking system before.
I find the Unfuddle UI to be kind of clunky, especially if you&amp;rsquo;re dealing
with a large number of tickets, but it&amp;rsquo;s much more thorough than Pivotal
Tracker in keeping track of comments, ticket disposition, change history,
and so on.
Maybe some clever person will invent a PT-style front end for Unfuddle.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Heroku.&lt;/strong&gt; Five stars.
The first time I attempted a heroku deployment, it was all black magic and I was lost.
Now it&amp;rsquo;s still black magic, but I&amp;rsquo;ve learned that I can use it.
Deployment is one of those things that gets more and more complicated the more you try to understand it.
Heroku allows you to remain ignorant and just have your program run.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Android.&lt;/strong&gt; On my desktop I have a 2200-page book on how to code for Android.
I&amp;rsquo;m going to tell you I&amp;rsquo;ve read it all. I&amp;rsquo;ll be lying, and you won&amp;rsquo;t believe me.
In reality, I&amp;rsquo;ve learned the basics and can make a simple app.
I&amp;rsquo;m still at the tedious stage where I have to look everything up, and Android programming is already tedious by nature, but I&amp;rsquo;m getting it done.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Amazon AWS.&lt;/strong&gt; This probably has the steepest learning curve of all.
The documentation is complete and thorough and is also written to an
audience of Sysadmin Ph.Ds.
So far I&amp;rsquo;ve learned to use S3 buckets and create EC2 instances.&lt;/p&gt;

&lt;p&gt;Things I want to learn next year:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;advanced Python&lt;/li&gt;
&lt;li&gt;advanced Django&lt;/li&gt;
&lt;li&gt;advanced Git&lt;/li&gt;
&lt;li&gt;Neo4j&lt;/li&gt;
&lt;li&gt;Go&lt;/li&gt;
&lt;li&gt;Sass&lt;/li&gt;
&lt;li&gt;node.js&lt;/li&gt;
&lt;li&gt;backbone.js&lt;/li&gt;
&lt;li&gt;how to deploy stuff (why is this so hard, anyway?)&lt;/li&gt;
&lt;li&gt;Haskell&lt;/li&gt;
&lt;/ul&gt;
</description>
    </item>
    
  </channel>
</rss>