But first, I was trying to squeeze out as much performance as I can from single-threaded code itself. Ruby garbage collector is getting quite bad rap these days, but I didn’t want to go to the dark side right from the beginning and mess with reducing allocations or controlling GC cycles manually. My first targets were SQL queries.

Search filters operate on relatively big dataset, and their results can contain nothing more than just collections of database record ids. There is no need to pull all the stuff or instantiate real models because until all filters are applied, we don’t know whether given record would appear in result. So SQL based filters tend to run rather fast queries that return integer arrays of significant size.

It’s no surprise that ActiveRecord spends fair amount of time on object instantiation. ORM plumbing comes at price. There is well known way to avoid paying that cost by calling connection adapter’s select_values directly. It appears however, that with PostgreSQL we can do better. pg gem has column_values function written in C which returns array of string values for any given column of query result. Here are some benchmarks that compare find with select_values and my column_values wrapper:

  Test sample contains 20000 records
  Benchmarking find
  Time: 34.165s 0.6833s per run
  Benchmarking select_values
  Time: 3.686s 0.0737s per run
  Benchmarking to_int_array
  Time: 2.426s 0.0485s per run

I’ve made an attempt to put some envelope around column_values for accessing it from ActiveRecord in more convenient way. The column_queries gem gives ability to quickly pull id columns and is compatible with scopes and other ActiveRecord native facilities.

Ok, that helped, but still response time wasn’t perfect. No matter how hard I tried to optimize filters that run on app’s own database, there were always 2-3 filters making HTTP calls to other components, and at least one of those going to 3rd party service. So by just summing up time spent on those external calls, it easily approached 800ms, which was still far from the goal. The next logical step was to run filters in parallel, but GIL and poor MRI support for concurrency is a well known story. I was rather skeptical at that point, but still wanted to try.

Though one obvious way to get real concurrency is to switch to JRuby, I sticked to MRI. I may try JRuby on that project some day, however at that moment I was still looking at other options that wouldn’t bring significant infrastructure change to the production environment.

After column_values discovery I was looking into pg gem sources for asynchronous interfaces to be used with threads or fibers. And there were such, but one thing looked even more interesting: async_exec method. It’s interesting because despite its name it is a synchronous call. It blocks and returns only after it gets response from database. However internally, it uses asynchronous libpq interface and uses Ruby rb_thread_select call to release global mutex and wait for data on connection socket. So this method behaves just like regular exec, but releases GIL while blocked on I/O. Moreover, there is already relevant ActiveRecord code in place to work with that, it can be enabled by allow_concurrency: true entry in database.yml:

  development:
    adapter: postgresql
    encoding: unicode
    database: sandbox
    username: postgres
    allow_concurrency: true
    pool: 10

I’ve tested that discovery with this simple test:

  puts "Starting"
 
  threads = []
 
  bm = Benchmark.realtime do
    10.times do
      threads << Thread.new { ActiveRecord::Base.connection.select_values('SELECT pg_sleep(2)') }
    end
 
    threads.each(&:join)
  end
 
  puts "Time: #{"%0.3f" % bm}s"
 
  # with :allow_concurrency => true
  # Starting
  # Time: 2.372s
  #
  # with :allow_concurrency => false
  # Starting
  # Time: 20.337s

That meant I could actually get acceptable level of concurrency in Rails and Ruby 1.9 with pg gem and PostgreSQL database. The only thing left was ActiveResource client or HTTP requests in general.

Can threads with SQL queries run concurrently with threads that make ActiveResource calls? It appears that ActiveResource uses Net::HTTP and native I/O which releases GIL with rb_thread_blocking_region, just like rb_thread_select. So the thing works right out of the box:

  # dummy_server.rb:
  #
  require 'rubygems'
  require 'sinatra'
 
  get '/' do
    sleep 2
    "Hello world!"
  end
 
 
  # test.rb:
  #
  puts "Starting"
 
  threads = []
 
  bm = Benchmark.realtime do
    10.times do
      threads << Thread.new { ActiveRecord::Base.connection.select_values('SELECT pg_sleep(2)') }
    end
 
    10.times do
      threads << Thread.new { Net::HTTP.get('localhost', '/', 4567) }
    end
 
    threads.each(&:join)
  end
 
  puts "Time: #{"%0.3f" % bm}s"
 
  # Starting
  # Time: 2.350s

Note that ActiveRecord connection pool needs to to be tuned carefully: the number of concurrent connections to the database shouldn’t be less than the number of filter threads that run in parallel. Also, each thread should play nicely with pool, so it makes sense to package code in separate Scheduler module:

  module Scheduler
 
    def self.run_group_and_reduce(*workers, &block)
      group = ThreadGroup.new
      workers.each do |w|
        group.add(Thread.new do
          begin
            w.call
          ensure
            ActiveRecord::Base.clear_active_connections!
          end
        end)
      end
      group.list.each(&:join)
      yield
    end
 
  end

The above scheduler method gets an array of callable worker procs and regular do...end block. It first runs all the supplied procs in separate threads, waits for them to complete and then runs its own block. For spawned threads scheduler also takes care of releasing connections to pool. Such semantics of barrier synchronized blocks with dedicated “reduce” step allowed me to experiment with threads, fibers and single-threaded sequential execution. Here’s how scheduler looked like during those experiments:

  class Scheduler
 
    def self.run_group_and_reduce_threaded(*workers, &block)
      group = ThreadGroup.new
      workers.each do |w|
        group.add(Thread.new do
          begin
            w.call
          ensure
            ActiveRecord::Base.clear_active_connections!
          end
        end)
      end
      group.list.each(&:join)
      yield
    end
 
    def self.run_group_and_reduce_sequential(*workers, &block)
      workers.each do |w|
        w.call
      end
      yield
    end
 
    def self.run_group_and_reduce_fibered(*workers, &block)
      counter = 0
      fibers = workers.collect do |w|
        Fiber.new do
          begin
            w.call
            counter -= 1
            EventMachine.stop if counter == 0
          ensure
            ActiveRecord::Base.clear_active_connections!
          end
        end
      end
 
      counter = fibers.size
      EventMachine.run do
        fibers.each(&:resume)
      end
 
      yield
    end
 
    class << self
      # Select desired strategy:
 
      # alias_method :run_group_and_reduce, :run_group_and_reduce_sequential
      # alias_method :run_group_and_reduce, :run_group_and_reduce_fibered
      alias_method :run_group_and_reduce, :run_group_and_reduce_threaded
    end
 
  end

As you may see, I’ve also played with EventMachine and fibers inspired by findings of Ilya Grigorik. That even ended up with a bug report and one line fix to Ruby core. I’ve turned back to threads though, mainly because of the discomfort with the amount of monkey patching in PostgreSQL ActiveRecord connection adapter and ActiveResource. EM Reactor pattern is great at I/O handling but under circumstances that are different from mine. Starting EM event loop and shutting it down just for search phase in serving each request seemed a bit unnatural to me. So by putting all my filters into separate threads I was able to come closer to the desired response time which was about 200ms.

Use of threads in Rails is an intriguing endeavor. Rails is declared thread safe, there is plenty of relevant code that takes care of synchronization, but still the threading practice should be approached with care. While threads aren’t that common in Rails applications, some configuration may be suboptimal for multi-threaded code. Some code can get broken just because regression testing for thread safety is a hard thing to achieve. And some deadlocks and race conditions can only reveal themselves under load or when scheduling becomes more “fair” as MRI progresses. Despite all the dangers, I’ve came across just several minor tweaks and tricks along the way of getting my code into production.

One problem that I’ve stepped into is that BufferedLogger default configuration can result in skipping log messages from the production log. The tricky part is that it is only production mode which is affected:

  # railties/lib/rails/application/bootstrap.rb:
  #
  # Initialize the logger early in the stack in case we need to log some deprecation.
  initializer :initialize_logger do
    Rails.logger ||= config.logger || begin
      path = config.paths.log.to_a.first
      logger = ActiveSupport::BufferedLogger.new(path)
      logger.level = ActiveSupport::BufferedLogger.const_get(config.log_level.to_s.upcase)
      logger.auto_flushing = false if Rails.env.production?
      logger
    rescue StandardError => e
      logger = ActiveSupport::BufferedLogger.new(STDERR)
      logger.level = ActiveSupport::BufferedLogger::WARN
      logger.warn(
        "Rails Error: Unable to access log file. Please ensure that #{path} exists and is chmod 0666. " +
        "The log level has been raised to WARN and the output directed to STDERR until the problem is fixed."
      )
      logger
    end
  end

As you may see, production mode disables buffer auto flushing by default. If threads produce less than MAX_BUFFER_SIZE = 1000 log messages, which is very likely given most logging is off in production, all their legacy remains hidden and that can result in memory leak. The cure is rather simple:

  config.after_initialize { Rails.logger.auto_flushing = true }

Another interesting side note that may be useful is that when you use config.threadsafe! it shuts off ActiveSupport dependency loader. That makes total sense, as it has been test proven that all the magic present in that module can’t cope with multi-threaded environment. So booting app by requiring files in proper order is a good thing and the only option for such applications. The “proper order” part is achieved by ActiveSupport::Dependency still, so Rails does something like this:

  Application.eager_load!
  ActiveSupport::Dependencies.unhook!

That is the reason of problems in rake tasks: with config.threadsafe! you have to manually require all the necessary files in right order. Eager loading doesn’t work there because rails dependency loader is already unhooked. One possible way to boot up app in rake task is this:

  task :some_task => :environment do
    ActiveSupport::Dependencies.hook!
    MyApp::Application.eager_load!
    ActiveSupport::Dependencies.unhook!
 
    # ...
  end

Hope that helps!

There are always interesting things happening in Ruby and Rails ecosystem. Our community embraces change, even when change means redefining conventional wisdom. alias_method_chain has stopped being considered a good practice, and it was only beginning. Today we refer to classics of SOLID design principles, while people continue revisiting patterns and practices that have made Ruby stand out as highly dynamic runtime. Take method_missing as another example: it can be harmful just like any other powerful technique. What’s next? Dynamic dependency loading? (It’s evil indeed)

Our understanding of MVC is changing as well. Frontends aren’t what they were before. Driven by desire to match platform native user experience, people find ways to do crazy and beautiful things.

I can’t help myself but rant play with some fundamentals too, so I’d like to focus on OO design and SOLID theme in particular. I don’t think that all five principles are equally applicable to Ruby because of dynamic typing. SRP (Single Responsibility Principle) may be the only one that can be taken as it is. OCP (Open Closed Principle) makes sense as well, but given Ruby open classes, needs an updated statement (i.e. what is meant by modification and extension).

LSP (Liskov Substitution Principle) is much more curious to think of in Ruby. Why would anyone want to consider such boring thing as this:

Let q(x) be a property provable about objects x of type T. Then q(y) should be true for objects y of type S where S is a subtype of T.

when Ruby offers such wonderful and fun version as Duck Typing:

When I see a bird that walks like a duck and swims like a duck and quacks like a duck, I call that bird a duck.

I’m being sarcastic here, of course I don’t think that Duck Typing can substitute LSP. Have a look at behavioral conditions section in the wikipedia article for reasons. The fact that LSP is attracting attention right now can be pretty telling. I’m excited to witness it. Community is finally seeking for some grownup approaches, great! However, the question is: doesn’t that get us closer to some limitations of our beloved highly dynamic language?

Rails code is rapidly evolving, the same is true for the community and ideas behind it. The fact that such static typing rituals as DIP (Dependency Inversion Principle) and ISP (Interface Segregation Principle) start being considered may mean that within the current pace of growth, it becomes more and more of a challenge to manage complexity. It is well known that in order to successfully “Duck Type” one’s way through code, he or she should often mind more details.

Rituals are what we hate in Java and C++, it’s what gets in our way when we want to take the shortest path from an idea to it’s expression. But as codebase becomes bigger and more complex, rituals become inevitable. I think everyone who has got past “15 minutes blog app” state and the euphoria of “I’ll put together a dozen of gems and my CMS is done, than add some more and here is my CRM” knows that in order to build (and scale) apps beyond the framework, some set of carefully picked rituals is very important. Often the very same set that was in the way at the beginning. An extreme example of building beyond framework is Twitter and their reasons behind switching to Scala.

The fact that you can call any private method you like, or access any instance variable, or reopen class and do whatever you want is fun. But doesn’t that lead to less time spent on thinking about fundamentals like encapsulation? How often do we have thoughts like “should this method that I write be public, private or protected” ?

Ruby is famous for its well thought OO paradigm. It has been influenced by Smalltalk design, which seems to mean a lot in the world of OO. The reality however is that de-facto standard attr_accessor doesn’t contribute at all to good design. And even if we put some opinionated decision and use attr_reader instead, doesn’t that still feel like we’re pretending encapsulation? For example if we expose an array trough attr_reader :x, we can’t do direct assignment, but what about x elements? Being able to ensure immutability on language level is good for VM performance optimizations and very important for various concurrency patterns. There are plenty of opportunities to break encapsulation. Of course being able to do something doesn’t necessarily mean having to do that, which brings us to certain aspects of code discipline. But, isn’t it all about rituals again? Implicit rituals unfortunately.

I heard a phrase at Railsconf 2011, that we end up writing more tests because of interface specs (Lint), and it’s ok to do so. It may be true in that particular case, but generally, is it okay to write more code especially when it does the work that interpreter or runtime can do better?

I’ve found this beautiful article on the topic. It has nice explanation of different ways to access current file location in TextMate. I’ve chosen the last option for myself and created a custom bundle, but did a small change in script however. Having file path relative to the file’s project directory is more convenient for me, I don’t want to distract people with details of my home folder organization. Here’s my version of the script:

 
FILEPATH=${TM_FILEPATH/#$TM_PROJECT_DIRECTORY\//}
 
echo -n "$FILEPATH" | pbcopy && echo "Path copied to clipboard: $FILEPATH"

You can create it yourself using Ciarán’s guide and this code snippet, or just grab it from github: http://github.com/bgipsy/files-tmbundle

Code constants that belong to database

Consider this example:

 
SPONSORED_PARTNER_IDS = [
    1234, 5678, 9101 #, ... etc
  ]

Here, prod DB was taken in order to look up the ids, and than those were hard coded into the code constant. It’s wonderful to observe how this seemingly harmless approach expands with time. Sometimes magic numbers don’t become less magic even after being put into well named constants.

Database tables that belong to code

Here’s the inverse pattern. Have you ever seen anything like this:

 
 SELECT * FROM content_statuses;
 
 id |   name    
----+-----------
  1 | Draft
  2 | Published
  3 | Hidden
(3 rows)

Here we have a content status dictionary in the database. But for what reason? For consistency enforced by foreign keys maybe, but I’m not sure it worth the burden of maintaining the table. The problem from design perspective here is more important however. What makes all those statuses different is code itself. If you do an INSERT and add another status, say “rejected”, it won’t change anything until you drop in some code changes that do all the associated special treatment to your content. This approach often breaks Common Closure Principle of component design.

Another way code leaks into database is “table algorithms”. Have you ever seen a table that contains pieces of SQL/PHP/Perl/Ruby code that are used by agile and generic algorithm? Often it comes as anticipation for future changes. The Holy Grail of code reuse probably is to make an algorithm that is so generic that it can handle any business request without change. All you have to do is update some data in the database. I’d rather call it code that leaked into configuration. Sometimes it transforms into a belief that marketing/content/admin folks will be able to make changes without even touching developers team. But more often than not, it ends up with everyone being too afraid to touch any of such code, and only one person being able to make “configuration changes” – the author of the code. With new requirements often come additional generalization and complexity. Same effort put into well thought modular, object oriented design could yield in good code that is complaint with Open Closed Principle (OCP) and located where it belongs – in files that are under control of VCS. It would still require some updates with new marketing requests, but those would land in code, and not DB migrations.

 
class CreateUsers < ActiveRecord::Migration
  def self.up
    create_table :users do |t|
      t.string :name, :null => false
      t.string :email, :null => false
      t.string :hashed_password, :null => false
      t.string :salt, :null => false
      t.timestamps
    end
 
    add_index :users, :email, :unique => true
 
    User.create!(:name => 'Default User',
      :email => 'user@example.com', :password => 'demo',
      :password_confirmation => 'demo')
  end
 
  def self.down
    drop_table :users
  end
end

Does it look good? If the answer is yes, then the rest of this small post worth reading.

Once upon a time

Let’s pay closer attention to the line where we’re creating default user record. The fact that it works and does its job well doesn’t mean that we’re safe. And that is due to the fact that we’re referencing a piece of application code from the database migration script. Migration files and application code have different life cycles. While application code evolves, migration code, once committed to the master branch (i.e. shared with the rest of the team/world), should stay the same forever (well, we’re doing grown up development, right?). We need to be extra careful when referencing any application code from migrations. For example, months after that migration was successfully committed and pushed, User model changes:

 
# new migration gets added...
 
class AddUsersBirthday < ActiveRecord::Migration
  def self.up
    add_column, :users, :birthday, :date, :null => false
  end
 
  def self.down
    remove_column, :users, :birthday
  end
end
 
# and User model gets a new validation:
 
class User < ActiveRecord::Base
 
  # ...
 
  validates_presence_of :birthday
 
  # ...
 
end

Now, if we try to recreate database from scratch, old migration which creates users table will not work for obvious reasons. Better yet, things could get even more interesting as User could have been renamed to Admin at this point.

Special cases

Sometimes using models for data migration is tempting. SQL often gets not as pretty as we would like to; seldom we just can’t do data conversions in SQL at all (consider hashing operations on passwords as an example). But in all such cases validations, relationships and algorithms that we may want to utilize in migrations can easily change as application develops.

One obvious solution for this problem is not to use models in migrations at all. Just DDL, plain SQL updates with execute is not such a bad idea actually. Another way is to declare migration local models like this:

 
class CreateUsers < ActiveRecord::Migration
 
  class User < ActiveRecord::Base
    include Behaviors::HashedPassword
  end
 
  def self.up
    User.reset_column_information
    create_table :users do |t|
      # ...
    end
 
    User.create!(:name => 'Default User',
      :email => 'user@example.com', :password => 'demo')
  end
 
  def self.down
    drop_table :users
  end
end

And if data conversion requires some piece of model implementation, it’s better to copy that code into migration local model class. In such cases it’s not a violation of DRY. It’s rather a code revision freeze, as migration should always run the same way.

Just don’t do it

If all migration data manipulation gravitates around bootstrapping your database with some starter content (like the above example with creating a user instance), then it may better fit into a dedicated rake task. Migrations should deal with schema changes, not data population. And often having to deal with static table content (which is often used as options for various selects, status id/name dictionary for example) can be a sign of more general design problem of putting code constants into db.

I’d like to share a trick that served me well in those triggers and I think it was not the last time. Every once in a while we need to create a row in a table if it isn’t there yet or update it otherwise. It can be implemented as easy as it sounds, with solid feeling of accomplishment, until one day you face process races.

Let’s suppose that we have two processes running the same code. Then the following sequence is possible. You probably already know that if you came here by googling, looking for a solution. But anyway:

• process A does SELECT and finds out that row does not exist
• process B does SELECT – still nothing there
• B inserts a row
• A tries to insert the same row
• BOOM! depending on whether you have unique constraints or not, process A gets an exception or database gets an extra row

Both outcomes, extra row and exception, may not be the worst things that can happen to some “blog” application, but fixing such bugs is good for self esteem. So let’s see what can be done here.

Race conditions are fixed with proper synchronization, and proper synchronization is based on locks. Postgres has nice implementation of shared and exclusive locks. We need an exclusive lock here which is gained by performing SELECT * FROM table WHERE conditions FOR UPDATE. After such query, transaction owns the lock up to it’s end. There is no other pretty way to release locks.

We have locks implementation at hand, but, in above scenario, we may not have a row to lock yet. Okay, let’s make one. Hello, Giant Mutex:

 
-- somewhere in schema initialization / migration
 
CREATE TABLE locks(
  name VARCHAR PRIMARY KEY
);
 
INSERT INTO locks VALUES('MATVIEW_INSERT');
 
-- and then somewhere in materialized view update trigger ...
 
SELECT 1 FROM locks WHERE name='MATVIEW_INSERT' FOR UPDATE;
-- triggers are run inside transactions, so everything below is run under global exclusive lock
 
SELECT INTO id_ id FROM my_matview WHERE ....
IF NOT FOUND THEN
  INSERT INTO my_matview VALUES(...)
  RETURN;
END IF;
UPDATE my_matview SET ... WHERE id = id_;

Well, that’s nice. But we killed concurrency quite a bit. And that’s not nice because rows actually do exist most of the time. There is a pattern which is called “double checked locking” and often used in singleton resources initialization. An adaptation for our example would look like this:

 
SELECT INTO id_ id FROM my_matview WHERE ....
IF NOT FOUND THEN
  SELECT 1 FROM locks WHERE name='MATVIEW_INSERT' FOR UPDATE;
  SELECT INTO id_ id FROM my_matview WHERE ....
  IF NOT FOUND THEN
    INSERT INTO my_matview VALUES(...)
    RETURN;
  END IF;
END IF;
UPDATE my_matview SET ... WHERE id = id_;

Note how we obtain lock only when first “easy” attempt to find row fails. Then we go “serious”, and when we do, we need to perform SELECT check once again. That’s because, like at the very beginning, several threads can get down to serious business at the same time.

The next things to worry about are deadlocks. But that probably deserves a separate post.

Monkey Patching

One possible way to add logic to library classes is to reopen class and add new methods directly, i.e. monkey-patch a class:

 
class UserTracking::SiteVisitor
  def initialize_with_fancy_links(request)
    initialize_without_fancy_links(request)
    @fancy_links = api_get_fancy_links(self.api_id)
  end
 
  alias_method_chain :initialize, :fancy_links
 
  def featured_links_with_fancy_links
    @fancy_links + featured_links_without_fancy_links
  end
 
  alias_method_chain :featured_links, :fancy_links
end

This approach has some limitations. First, the default dependency loader doesn’t really know how to handle classes that are defined in several files. I had to add some explicit require statements in order to get properly loaded models. Second, I had to use alias_method_chain to add logic to methods defined in framework parts of classes. That sometimes produced rather ugly effects. Look how it affected constructor for example.

Injecting a Module

As a slightly modified version of previous approach, I was considering a style taken from active_support’s core extensions:

 
module SiteVisitorExtension
  def self.included(base)
    base.alias_method_chain :initialize, :fancy_links
    base.alias_method_chain :featured_links, :fancy_links
    base.send(:attr_accessor, :fancy_links)
    base.extend(ClassMethods)
  end
 
  def initialize_with_fancy_links(request)
    initialize_without_fancy_links(request)
    @fancy_links = api_get_fancy_links(self.api_id)
  end
 
  def featured_links_with_fancy_links
    @fancy_links + featured_links_without_fancy_links
  end
 
  module ClassMethods
    def find_in_cache(cookie_key)
      # ...
    end
  end
end
 
UserTracking::SiteVisitor.send(:include, SiteVisitorExtension)

It looks just a bit more clever, but adds extra syntax overhead for defining class methods or calling macro methods. Such things all go into def self.included and become messy with time.

So What’s The Problem?

But it wasn’t extra lines of code that kept me looking for other options. I didn’t like the fact that the inherent difference of one particular front end application, the very incentive to create one, was implemented by some kind of monkey patch. I think that described techniques are very good in their own domain. When, for example, one wants to add some formatting to log messages or stick caching into objects that should be unaware of the fact, these methods do their trick. But implementing parts of business logic by alias_method_chains is something I’d really like to avoid.

Subclassing

We don’t create application models by reopening ActiveRecord::Base, we subclass it. All the above examples get simplified dramatically if we use boring inheritance instead of fancy tricks with patching classes. Also, standard dependency loader works perfectly with such classes because they’re just plain regular.

 
class MySiteVisitor < UserTracking::SiteVisitor
  attr_accessor :fancy_links
 
  def initialize(request)
    super
    @fancy_links = api_get_fancy_links(self.api_id)
  end
 
  def featured_links
    @fancy_links + super
  end
 
  def self.find_in_cache(cookie_key)
    # ...
  end
end

But can I easily use inheritance if framework creates instances of models inside itself? How can it know about my new shiny subclasses? People were facing such problems for quite a long time now. And for statically typed languages the answer always was often found near generative patterns or Factories. I used to think that in ruby I will never need a factory. Now I’m going to use this pattern and see how it works out.

Factory Module

Here’s a slightly adapted Factory pattern. I’m trying to be aware that I’m still using a highly dynamic language, and inventory of clever tricks is always at hand. But at the same time I don’t want to place any magic inside model classes – neither framework ancestors, nor app descendants. I mean that I don’t probably want to see some class BaseClass to magically appear as DescendantClass. Tricks like that can be fun but I’d like to see an explicit notion of whether framework code refers to some fixed class or is willing to instantiate it with the help of a factory.

A possible use case example may look like this:

framework code

 
# in user_tracking_plugin/app/models/user_tracking/site_visitor.rb
class UserTracking::SiteVisitor
  def self.api_find_or_create_by_request(http_request)
    api_response = ...
    new(api_response)
  end
 
  def initialize(api_response)
    # ...
  end
 
  def featured_links
    # ...
  end
end
 
# in user_tracking_plugin/app/models/user_tracking/factory.rb
module UserTracking
  create_factory
end
 
# in user_tracking_plugin/app/constrollers/user_tracking/controller_base.rb
class UserTracking::ControllerBase
  before_filter :track_visitor
 
  protected
 
  def track_visitor
    @visitor = UserTracking::Factory::SiteVisitor.api_find_or_create_by_request(request)
  end
end

application code

 
# in config/environment.rb
config.after_initialize do
  UserTracking::Factory.add_mapping :site_visitor, :fancy_visitor
end
 
# in app/models/fancy_visitor.rb
class FancyVisitor < UserTracking::SiteVisitor
  def initialize(request)
    super
    @fancy_links = another_api_get_very_special_links(self.api_id)
  end
 
  def featured_links
    @fancy_links + super # I love super
  end
end

Another option for application code would be to place factory mappings setup in separate initializer instead of config.after_initialize block.

One important thing is to be aware how the factory module gets loaded. Standard class reloading mechanism that us used in development can drop all factory mappings. To avoid that rb file which creates factory module must be loaded explicitly (or just be placed in initializer) or reside under one of the load_once_paths (that is true for plugins by default).

Install

Implementation can be found on github. It is packaged as a separate plugin, so you can play with it by installing:

./script/plugin install git://github.com/bgipsy/factory_module.git

Additional details can be found by looking at spec/lib/proto_factory_spec.rb under the plugin tree.

What walks like duck and quacks like a duck is a duck. That what duck typing is about. It makes development easier. We don’t need interfaces and abstract classes. Many object oriented patterns look much simpler because of such dynamic typing. They come more naturally in ruby code and often we don’t even notice them until we see explicit names like Visitor, Decorator, etc.

After I’ve got used with such a powerful tool, it was too easy to step into dangerous territory without even noticing. Sometimes things do walk and quack like ducks all the time… well, almost all the time. And sometimes in a pack of ducks there are some species that do quack (and that makes you feel safe), but that doesn’t make them ducks.

When all ducks are quacking well

Making real world examples is hard, and the following one is somewhat synthetic. But anyway. Let’s suppose that we have a system with Project model and our system should generate reports by aggregating project instances. We have different types of parametrized reports, and these types share some common features. Let’s see how it fits in code.

 
class Project < ActiveRecord::Base
 
  has_and_belongs_to_many :categories
 
  has_many :items
 
end
 
 
class ReportFilter < ActiveRecord::Base
 
  def filter_project_ids(project_ids)
    # implemented in descendants
  end
 
end
 
class DueDateFilter < ReportFilter
 
  def filter_project_ids(project_ids)
    irrelevant_projects = Project.find(project_ids,
      :conditions => ['due_date > ?', self.cut_off_date])
 
    project_ids - irrelevant_projects.collect(&:id)
  end
 
end
 
class ComplexFilter < ReportFilter
 
  def filter_project_ids(project_ids)
    connection.select_values <<-SQL
      SELECT p.id FROM projects p
        JOIN project_estimates e ON (e.project_id = p.id)
        JOIN metrics m ON (p.id = m.project_id)
        WHERE m.coverage_level < average_coverage_in_group(p.group_id)
          AND e.author_id IN (SELECT id FROM team_members WHERE statisfaction_level > #{desired_level})
        AND p.id IN (#{project_ids.join ','})
    SQL
  end
 
end
 
class CategoryReport < ActiveRecord::Base
 
  belongs_to :category
 
  has_many :report_filters, :order => 'complexity_level'
 
  def projects
    project_ids = category.project_ids
 
    report_items.each do |i|
      project_ids = i.filter_project_ids(project_ids)
    end
 
    Project.find(project_ids)
  end
 
end

Note how CategoryReport generates it’s project set. It iterates through it’s project filters to trim category project list down to some relevant report specific subset. All the report filters implement common interface. Each filter accepts project_ids array and reduces it according to its parameters and semantics. Some filters are fairly easy to calculate, other can be computationally complex. We run simple filters first to reduce set and then refer to complex ones.

Most of our filter implementations should look like DueDateFilter. But we live in real world, and several filters look like ComplexFilter. Don’t look deep into SQL query in it, it was added for dramatic effect only. My point is that due to some (often premature) optimization, complex filters can implement some real black magic under the hood.

And here comes the time for a closer look

Let’s suppose that we always run ComplexFilter as the last filter in our chain. But one day a very good reason for changing that order arises. We place DueDateFilter after ComplexFilter. System works, all the filter rspecs pass, and rspec for Report passes as well. It checks Report behavior with great success by mocking everything except Report (a model’s rspec should be focused on the model it’s testing, right?)

Some time passes and we discover a great surprise. It appears that reports have been working incorrectly for some time now. Some filters that come after ComplexFilter don’t filter projects any more. No exceptions are raised. They’ve just silently became transparent.

Please note how DueDateFilter operates on array of ids that are Fixnums. Then note how ComplexFilter returns an array of strings. They’re still ids, but that doesn’t help when it comes to…

 
project_ids - irrelevant_projects.collect(&:id)

… in other filters.

And a few reflections

A type of bug that is easy to fix, easy to debug, but hard to predict. Who would notice such a thing on code review when all these filters did work previously? And another big question: what can we do to make such bugs harder to sneak into our code?

Well, I don’t know. There’s probably no silver bullet for it. But there’s something we can do however. It’s not new, in fact it’s plain old OOD.

Arrays of fixnums (and arrays of strings) are not domain models for our system. Yes, we don’t want to instantiate full projects for the sake of performance, but passing integers around instead of project models is not the only option that we have.

 
class ProjectSet
 
  def initialize(ids)
    @project_ids = []
    add(ids)
  end
 
  def add(ids)
    @project_ids |= ids.collect(&:to_i)
    self
  end
 
  def exclude(ids)
    @project_ids - ids.collect(&:to_i)
    self
  end
 
  def project_ids
    @project_ids.dup
  end
 
  def projects
    Project.find(@project_ids)
  end
 
  def to_sql_list
    return 'NULL' if @project_ids.empty?
    @project_ids.join ','
  end
 
end

Looks like we’ve avoided the temptation to implement domain specific operations through overriding operators (especially arithmetic). Cool books say that’s not something that grownups do very often. So our methods are named add and exclude. Now we can change our report code to pass an instance of ProjectSet and have a bit more control of operations that are essential to our problem domain.

ProjectSet adds the following benefits:

• we can encapsulate type checks and explicit conversions in it, and keep filter code clean of such details
• we can raise an exception if supplied ids are of incorrect type and be informed about problems earlier by QA (500-page is easier to notice than tricky changes in reports behavior)
• we can add other specific code that otherwise would be scattered across different filters (note how to_sql_list handles empty lists)
• we can open paths to other caching/performance optimizations that can be hidden under the hood of ProjectSet

Not all .rb files that live under app/models directory have to be AR models. Many applications can benefit from having additional models. And those can be no less domain specific than conventional ActiveRecord::Base descendants.

Here’s a rails plugin to add foreign keys for associations. It doesn’t have any dependencies. And it’s code is simple so it can be read in one take. These are the two things in plugins world that I appreciate the most.

In migrations, when a table is created, it recognizes columns which are foreign keys by _id suffix. Then it adds foreign key declarations. Referred table name is guessed from column’s name according to rails conventions. This can be disabled by adding :references => false option.

If a column should refer to a table which name can’t be inferred that way, you can explicitly specify it by using a symbol or a string.

create_table :comments do |t|
  t.text :body
  t.integer :commented_post_id, :references => :posts
end

will produce

CREATE TABLE comments(
  id SERIAL PRIMARY KEY,
  body TEXT,
  commented_post_id INTEGER REFERENCES posts
);

And with strings you can do all the things that you shouldn’t want to do:

create_table :comments do |t|
  t.text :body
  t.integer :commented_post_id, :references => 'whatever(strange_pk) ON DELETE RESTRICT'
end

will become

CREATE TABLE comments(
  id SERIAL PRIMARY KEY,
  body TEXT,
  commented_post_id INTEGER REFERENCES whatever(strange_pk) ON DELETE RESTRICT
);

That can be handy when dealing with nasty legacy schema.

And you can add foreign key constraints for already created tables:

add_foreign_key_constraint(:posts, :comment_id, :comments, :p_id)

ALTER TABLE posts ADD FOREIGN KEY (comment_id) REFERENCES comments(p_id);

You get the idea. For more examples please see spec/migration_spec.rb here

So the next thing you may want after you’ve packed your database with foreign key constraints is to get rid of them. In your tests. Rails disables referential integrity checks before loading fixtures and enables them after that. So missing references in fuxtures are ok. And if you need to disable foreign key constraints for certain table you can always make a call.

remove_all_foreign_key_constraints('comments')

Another option is to automatically drop all constraints for tables that are referenced by test suite fixtures statement. That is what another branch with_fixture_patch of the plugin contains. I’m still considering if it can be useful…

To install run

./script/plugin install git://github.com/bgipsy/fk_constraints.git

from your app root directory.