Sub directories for STI models in Rails

I recently started using Single Table Interitance in my Rails application and immediately wanted to group all the child models in a sub directory inside the models directory.

All over the web, the advice was just create a directory, stick your models in there and then add on of the following into environment.rb:


config.load_paths += Dir["#{RAILS_ROOT}/app/models/**/**]
# or
config.load_paths += Dir["#{RAILS_ROOT}/app/models/mysubdir"]


Well, this doesn’t seem to work, as of Rails 2.3.2 anyway (no idea about earlier versions).

It seems that Rails scans the models directory, and when it finds subdirectories in it, it loads the classes into a namespace that is the subdirectory name. If you have a subdirectory called Actions, your classes will end up loaded as Actions::SubClass, which is not want I wanted.

To get this working, I had to put the extra load path I wanted at the front of the load path array (at least before it loads the models directory):


config.load_paths.unshift "#{RAILS_ROOT}/app/models/**/**"
# or
config.load_paths.unshift "#{RAILS_ROOT}/app/models/mysubdir"


I am not really happy with this approach, but I have already wasted hours trying to get it working at all, so the hack will have to do for now!

Code Comments are not for tracking changes

Recently, I have been reviewing and correcting a lot of broken code, and in doing so, I have concluded that the number of developers who know how to properly use a bug tracker and version control is not nearly high enough (disclaimer – this is in a big faceless corporation).

Commenting Anti-Pattern

So what leads me to this bold conclusion? Have you ever seen code that looks like this:

begin
  -- start change for #3245
  v_myvar varchar2(100)
  -- end change for #3245

  for r in (select c1,
            /* start change for #9870 */
                        c2,
                        c3
            /* end change for #9870 */
              from foo_table
              where c1 in (1, 4, 6)
           -- commented out for #9870
           -- and c2  = 'val1'
           -- start change for #5678
               and c3 = 'val2'
           -- end change for #5678
             ) loop
    null;
  end loop;
end;

Ignore the fact that this code does nothing useful, and is PLSQL – its just there to illustrate a point.

The amount of code I have seen recently that has its changes tracked through inline comments that attempt to encapsulate every single line changed to fix a bug or add a feature is unbelievable.

Some may ask, what is the big deal? Well firstly, when I see code like this, it says to me, the developer who did this, doesn’t have any idea about version control – they are commenting every change to attempt to provide traceability and rollback ability to their change. Often, when I see this anti-pattern, I subsequently find the code hasn’t seen the inside of Subversion or anything like it.

Show me the code

If I ever dare to ask to see the changes that were made to fix a bug in code like this, I get handed a source code file and told to grep through it for the bug number – what scares me, is how do I know he remember to comment every single change? For a big change, it takes a lot of patience to do something like that! And then the Delivery Manager (DM) arrives at the developers desk and the conversation goes like this:

DM: Remember those 10 feature requests we agreed to deliver tomorrow, well how are they going?

Dev: No problems, we have them all finished and tested, ready to roll!

DM: Well, we have a problem – turns out the front end developers have not been able to complete feature #4567 and that impacts us – we need to deliver the other 9 changes, but not that one.

Dev: Huum, thats going to be a problem, #4567 was a big change – I don’t think we can get it out of there without a lot of effort!

DM: Don’t you have all this tracked in your version control system … back in my day we committed individual changes into SCCS and things like this were never a problem!

Dev: Gulp … guess its going to be a long night for me then sir …

Finally, don’t even get me started on how much harder the code is to read with all these annoying ’starting/ending’ change comments.

A simple recipe

If you come across this rant, and have found yourself doing this, then stop a moment to think about how to use version control to track these changes for you – thats what is was designed for!

Here is a tip – when you have to fix a bug or add a feature to existing code:

1. Checkout your code
2. Make changes to whatever files are necessary and forget about pointless start/end change comments.
3. Get your test suite to pass your bug/feature (you have a test suite, right?)
4. Check in you code when its all done, quoting the bug number in the commit comment

Simple, clean code changes, traceable in a much more robust way and you can safely roll back the changes to any bug at any time.

The biggest mistake I see developers making with version control, aside from not using it at all, is to fix about 10 bugs, and then do a commit – please don’t do this, as it removes the ability to examime the changes made to fix a single bug, not to mention removing the changes if the need arises – which happens more than you may think, as our poor Dev found out above!

sqlite3-ruby-1.2.2 is broken on OS X

So I just spent ages trying to figure out why the latest version of sqlite3-ruby wouldn’t run on OS X. Turns out there is a bug …

If you are getting an error that looks like:

dyld: NSLinkModule() error
    dyld: Symbol not found: _RSTRING_PTR
      Referenced from: /usr/local/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.2.2/lib/sqlite3_api.bundle
      Expected in: flat namespace

    Trace/BPT trap

Its best to revert back to 1.2.1 which seems to work fine. This behavior occurs on Tiger and Ruby 1.8.4.

Rails style meta programming in Ruby – part 2

This is part two of this blog post – if you landed on this page, then please checkout part 1 before reading this part.

Last time, we learned that all methods are executed by sending a message to self, where self is set to the appropriate object automatically. We also learned that class definitions are code which is executed at run time, and that a class is nothing but an object of type Class.

What is a metaclass?

Here is the next important point – Classes are a place to store instance methods, objects are a place to store instance variables. An object cannot store a method, only a class can.

But what about class methods? We know that a class is just an object, and when you say something like

class Foo
  def self.hello
    puts "hello"
  end
end

Foo.hello

You are invoking the hello method on the instance of the object stored in Foo, which happens to be an object of type Class (ie your class definition). So where are these class methods stored?

When you invoke a method on an object, Ruby looks at the object’s class pointer, which contains a reference to the ‘class object’ that defines the class. The class object contains the list of methods defined by the class definition, and a pointer to its super class, which contains another list of methods and a pointer to its super class and so on. Ruby searches along this chain until it finds the method it requires, or calls method_missing if it cannot.

To be consistent, a ‘class object’ should behave in the same way. It has a class pointer that points to a class. Ruby magically creates this virtual class when you define a class method, and uses it to store all class methods for the class. This magically created virtual class is what is known as the classes metaclass.

Note that metaclasses follow the same inheritance chain as classes, inheriting from metaclasses up the chain, which is why this works:

class Foo
  def self.hello
    puts "parent says hello"
  end
end

class Bar < Foo
end

Bar.hello

Classes can have instance variables

So if a class is just an object, and an object is a place to store instance variables, then can a class have instance variables too? Yes it can, but think carefully about how to access them.

Normally, we create a class, say Foo, and in that class defintion use attr_reader to provide methods to access the instance variables. So where do we put our attr_reader defintion to allow access to a class instance variable? We have to put it in the objects class defintion, which in this case is the classes metaclass:

class Foo
  @class_instance_var = '1234'

  class<<self
    attr_reader :class_instance_var
 end

  def hello
    puts "hello"
  end
end

puts Foo.class_instance_var

To complete the picture, we just need to know what the following syntax means:

class<<self
 ...
end

'class<<obj' allows us to extend an object by "building a new class just for object obj". In the context above, it basically changes self to be the metaclass instead of the defining class.

Another way of thinking of it is simply that 'class<<obj' gives you access to obj's metaclass.

When we are in the metaclass, we can define the attr_readers required to read the class instance variables.

Finally some Rails magic

Now we know every we need to know make this work:

class User < ActiveRecord::Base
  set_table_name :silly_named_users_table
end

We know that when 'set_table_name' is execute, it is executed against self, which happens to be the object defining the User class, so its just like saying:

class User < ActiveRecord::Base
  User.set_table_name :silly_name_users_table
end

This means that the method 'set_table_name' must be in User's metaclass, or in the inheritance chain for it, which includes ActiveRecord::Base. We know its not in User's metaclass, as we didn't put it there, so it must be in ActiveRecord:

module ActiveRecord
  class Base

    def self.set_table_name (table)
      puts "setting the table name to #{table}"
    end
  end
end

class User < ActiveRecord::Base
  set_table_name :silly_named_users_table
end

In the Users class, we want to store this alternative table name - it needs to be stored in a class instance variable in the User class, which is simple (I will also add an attr_reader method to the meta class so we can test the code does the correct thing):

module ActiveRecord
  class Base

    def self.set_table_name (table)
      puts "setting the table name to in AR #{table}"
      @tablename = table
    end
  end
end

class User < ActiveRecord::Base
  set_table_name :silly_named_users_table

  class<<self
     attr_reader :tablename
  end
end

puts User.tablename

The above code does as expected - why?

When set_table_name is encountered in the User class definition, self is the class object stored in the constant User. It cannot find the set_table_name method method in User's metaclass, so it looks in the parent class, and finds it, but self is not changed to the parent class, its still the class object stored in the constant User that is how inheritance works.

When we set the instance variable in the parent method, self is still User so we are setting an instance variable in the User class object. Anytime we call a method against an instance of a User object, the method can access that tablename variable and use it to generate the correct SQL. How to do that is left as an exercise for the reader

Rails style meta programming in Ruby – part 1

For a long time now, I have pondered and wondered about just how Rails does what it does – surely such wizardy is well outside my basic Ruby skills – well not any more thanks to this presentation by Dave Thomas.

He explains all the concepts you need to know, so that you too can do things in your own code like:

Class Users < ActiveRecord::Base
  set_table_name :unconventional_name
  has_many :friends
end

Dave, I am sure, explains it much better than me, but the video is an hour long, and I had to watch it twice, so here, hopefully is the 10 (or 20) minute summary.

Everything is a method

First things first – every method you call in Ruby is run by sending a message to an object – even at the top level when you wouldn’t have though so:

puts "hello, world!"

In the above code, puts is a method with is executed by sending a message to an object. Infact, no matter which method you call on any object in your code, it is always executed by sending a message to the same object, ’self’.

puts self.to_s
puts self.class

prints

main
Object

So, at the top level, all messages are sent to an object contained in the variable self that is called ‘main’ and is of class Object.

When you type something like

  a = Array.new
  puts a.class

It prints ‘Array’, as the variable ‘a’ contains an instance of an Array object. The trick to remember here is that the ‘a.’ says, set the value of self to the object a, and then execute the method on self. This means that everytime you execute a method, it is always executed as a message passed to ’self’, which will have been set to the correct object.

That is why this works:

class Foo
  def hello
    there          # a method call with no receiver?
    puts " there"
  end
  def there
    puts "hello"
  end
end

b = Foo.new
b.hello

When you are inside an instance method, executing another method without a class or object prefix invokes the method on the object self. Ruby arranges so that the value of self inside an instance method is the correct thing – ie set to the instance of the object the original method was invoked on, in this case the instance of Foo stored in ‘b’.

The following code is equivalent to the above, but self is explicity specified.

class Foo
  def hello
    self.there          # a method call with a receiver?
    puts " there"
  end
  def there
    puts "hello"
  end
end

b = Foo.new
b.hello

I know this seems trivial, but its very important, so I will say it again – a method is always executed by sending a message to self – self is always changing so it contains different objects however.

Class definitions are executable code

Big deal – what has this got to do with this spiffy ‘has_many’ Rails style directives. Well, the next thing you need to learn, is that class defintions are executable code. Remember that any method that is executed, must be executed on self …

class Foo
  puts "hello"

  def bar
  end
end

Classes are just objects

Runnning the above code prints “hello” – we know its doing this because class definitions are executable code, but also that puts is invoked against self, so what is self?

class Foo
  puts self
  puts self.class

  def bar
  end
end

Which outputs

Foo
Class

So inside a class definition, the current object is set to an object with the same name as your class, that is of type Class. This leads to the second thing you need to remember – a class is nothing but an object – yes its a little special, but its an object none the less.

Ruby is even sneakier – when you define a class ‘Foo’, Ruby creates a constant called Foo, and places an object of type Class into it which contains the definition of your klass. Don’t believe me? Run this code -

class Foo
end

puts Foo
puts Foo.class

Which produces

Foo
Class

ie, the contents of a constant, ‘Foo’, (because in Ruby any variable starting with an uppercase letter is a constant) is an instance of Class, called ‘Foo’.

Phew – thats enough for now – next time, I will introduce metaclasses and the other bits and pieces that allow these Rails like ‘has_many’ directives to actually work …

Four things hotel 1.0 and web app 1.0 have in common

For the past week I have been staying in a very nice hotel, which I just learned opened only a month ago. Today it dawned on me, that opening a new hotel is in many ways a lot like getting version 1.0 of a web application launched.

Hotel 1.0 does not need all the features

Both a hotel and a software product require a large amount of investment to get to the launch date. When developing version 1.0, there is a real temptation to keep adding features that you think are essential, continuously delaying the launch. Guess what – its perfectly fine to get the place opened with only the indoor pool done while the outdoor one is not finished, or when the first restaurant is ready but the second one is not. I can imagine the hotel owner stressing out about all the things he thinks are essential, but to be honest, the guests are not going to care so long as the rough edges are kept out of sight and what is there does its job well.

Just like with software, getting the doors open at the earliest opportunity allows feedback to be gathered from guests, and more importantly start at least a small amount of money coming in!

Hotel 1.0 does not need every to be feature perfect

OK, the rooms need to be perfect as does the food and restaurant – that is what the guests came for (to sleep and probably to eat) – in other words the core features need to be perfect.

However it doesn’t matter that much if the heating doesn’t work in the indoor pool yet, so long as its still usable. Same goes for software – the user experience of the main part of the application had better be spot on, but I can live with some discomfort in the data exporter or preferences pane … for now!

Hotel 1.0 can really impress the early guests

When the hotel opens, it will probably not be booked out on week 1, 2 or even 3. This is a special chance to give exceptional service to the first guests. If the hotel treats the guests really well, responds to any problems quickly and explains the reasons behind the unfinished pool and restaurant they will probably leave happy and evangelise the place to friends and colleagues.

The same goes for new software products. The first customers are the most important – so treat them like kings and they will tell others.

Hotel 1.0 will have many of the best rooms empty

Related to point 3 – when the hotel first opens, many of the more expensive rooms will remain empty. Why not upgrade some guests into these rooms? In doing this, the guests will probably be less annoyed by the unfinished features and they will have a positive story to tell colleagues and friends.

In new web application the same thing goes – bandwidth and servers are not going to be taxed for the first while (unless you hit the Facebook jackpot) so why not give early users the top package for the first few months in the hope they there when the trial period ends?

Honesty is the best Policy

One final important thing – don’t advertise the hotel as having a heated indoor and outdoor pool if only the indoor pool works, and it’s cold – guests will just feel insulted when they find out and that will be what sticks in their mind. Honest usually pays dividends if everything else is good.

If, for any reason you happen to be travelling to Hyderabad on business, version 1.0 of the Ella Compass Suites hotel is highly recommended!

Setting up a Sun Enterprise 250 (E250)

Recently I needed to install Solaris 10 on an old creaky Sun Enterprise 250 (E250) and 450 (E450) box. This quick how-to documents what I did to get the machine up and running, and answers a few of the questions it took me a while to figure out along the way! Most of it is pretty applicable to a modern Sun too.

Getting to ok

The first thing you have to do is get to the OK prompt. On most boxes I have used in the past, getting to ok is easy, just power on the machine and it stops at the ok prompt, waiting for someone to type boot, however this machine was different – it just booted straight away, without an ok in sight!

To get the ok prompt, wait for the system to initialise the memory and devices and either press STOP-A (if you are using a VVT220, it will probably have a STOP key) or ctrl-break repeatedly.

Now that you have got the OK prompt, you probably want to stop the machine auto-booting until you have it working the way you want it. To stop auto-booting, enter the following commands at the ok prompt:

ok setenv auto-boot-on-error? false
ok setenv auto-boot? false

Now power off the machine and let it restart (apparently you need to do this, as STOP-a or ctrl-break stops the machine initialising some things, so its best to restart to be sure you don’t hit problems later).

Installing Solaris 10

Installing Solaris 10 is pretty simple at this point – pop the DVD in the drive and enter:

ok boot cdrom

The installer will kick off, asking you a bunch of questions along the way – make sure and have your hostname, domain, network and DNS settings to hand.

Before the OS actually installs, you need to layout your disk. How you set this up is totally up to you – I usually create a parition for each of /, /usr, /export/home, /opt, /tmp and /var/crash on the primary disk. Sizes will very much depend on your setup and requirements.

Getting Rid of the Desktop and Login Server

By default, Solaris 10 attempts to run the desktop environment on boot. If, like me, you want to use this machine as a server and have no need for a desktop, then save yourself some RAM and disable it:

# svcadm disable svc:/application/graphical-login/cde-login:default
# /usr/dt/bin/dtconfig -d

(Thanks to this site for that tip!)

ZFS Rocks

I knew in the back of my mind that Solaris 10 had a fancy new filesystem called ZFS, but I didn’t know what that meant until I came across this article. If you have a sever with more than one disk, or you are not sure what sizes each of your mountpoints should be, or you need striping, mirroring or RAID, then go read up on ZFS. It just might make disk management as pleasurable as writing Ruby!

ActiveRecord, Oracle and stored procedures

After much painful searching for a built in way to get ActiveRecord to execute a stored procedure on Oracle complete with in and out bind variables I have concluded that its just not possible.

The Oracle connection adapter uses OCI8 under the covers to connect to Oracle and many of its methods are exposed to you. To execute a stored procedure with OCI 8, one method is like this:

require 'oci8'
conn = OCI8.new(user, password, db.world)
res = conn.exec('begin my_proc(:in,  ut); end;', 'invalue', 'outvalue')

That’s pretty simple, I guess. To do such magical things with ActiveRecord, a method ‘execute’ is exposed, this looks like this:

def execute(sql, name = nil) #:nodoc:
  log(sql, name) { @connection.exec sql }
end

The method spec for OCI8#exec is however

def exec(sql, *args, &blk)
  ...
end

So ActiveRecord gives us a wrapper to exec, but sadly removes the essential bind variables.

In my application I added the following code:

class ActiveRecord::ConnectionAdapters::OracleAdapter
  def _exec(stm, *args)
    @connection.exec(stm, *args)
  end

  def _cursor(stm)
    @connection.parse(stm)
  end
end

Which lets me execute Oracle stored procedures using my ActiveRecord connection at last!

My application is not a Rails application as such, more an application that uses ActiveRecord, so I put the above code in my boot script after including ActiveRecord – not sure exactly where you would put it in a Rails app.

If there is a better built in way to do this, please comment and let me know as I have not been able to find it!

UPDATE – it seems there are a few other struggling to get this stuff to work – I have added a chunk of code below that makes this stuff work:

require "rubygems"
require_gem "activerecord"

ActiveRecord::Base.establish_connection(:adapter => "oracle",
                                        :database => "avdev7",
                                        :username => "sodonnel2",
                                        :password => "sodonnel2")

class ActiveRecord::ConnectionAdapters::OracleAdapter
  def _exec(stm, *args)
    @connection.exec(stm, *args)
  end

  def _cursor(stm)
    @connection.parse(stm)
  end
end

# create or replace procedure testproc(inxml in varchar2, outxml out varchar2)
# is
# begin
#     outxml := inxml;
# end;
# /

class PLSQL < ActiveRecord::Base
  attr_accessor  utxml
  def initialize
    execute
  end

  def execute
    cursor = connection._cursor("BEGIN testproc(:inxml,  utxml);  end;");
    xmltext = 'some text to print'
    cursor.bind_param("inxml", xmltext)
    cursor.bind_param("outxml", nil, String, 1000)
    cursor.exec
    @outxml = cursor['outxml']
    cursor.close
  end
end

obj = PLSQL.new
puts "The returned value is : #{obj.outxml}"

Of course it’s the Database, I don’t need a Profiler

We all know that when any database backed application starts to behave slowly, its always the databases fault, right? After all, if the application code hasn’t change so what else could it be?

If you have spent much time around Oracle, you are sure to have seen complex queries when Oracle decides to changes its query plan for “no good reason” and application performance changes drastically. Perhaps this is what leads to all this finger pointing at the database, mostly by people who don’t understand how it works.

Last week I was contacted to investigate a performance problem in an application that had everyone baffled. The system works off 15 – 20 queues of requests, and a batch job is responsible for processing each queue, so there are up to 20 batch jobs running at once. Normal throughput for a batch job is several requests per second, but this had dropped to several a minute and things were backing up.

Lost Time

The DBAs had provided me with a level 1 trace file captured during this slowdown that looked very normal – no bad queries in sight. Only strange thing was a lot of walk clock time in the trace compared to CPU seconds (5 or 6 times more elapsed time). My initial reaction was certain there was a locking issue somewhere in the database, so I asked the DBAs to create a level 8 trace file that would surely should show me where the all this “lost time” had gone.

No such luck, Oracle was not waiting on anything, so where had all this time gone?

Perhaps the database server was under heavy load you say? Not so, this was a 24 core server 50% idle through the test, humm.

Luckily, I had read Optimising Oracle Performance and actually understand much of the gibberish inside an SQL Trace file. Still mystified where all the time had gone, I fired up Emacs and used Ruby to tell me the biggest time between two database calls, which stood at an impressive 30 seconds between a fetch on cursor 18 and exec on 19 – ie the “lost time” was in the application somewhere between those two queries, not the database calls.

In this case its a bit greyer, as the application is written in PLSQL, but the problem turned out to be the OS struggling to open log files in a directory containing over 2 million files!

Power to the Profiler

The morale of this story, apart from the need to housekeep your log files, is that its important to have the ability to profile your application code as well as the database calls.

Instead of writing scripts to parse Oracle trace files, I should have ran the code through DBMS_Profiler which would have solved this mystery in about two minutes and saved much head scratching and other people making ridiculous suggestions to change irrelevant database parameters!

Emacs and Oracle

After learning so much about Emacs to get my Rails setup working the way I wanted it, I though it was about time I figured out how to use SQLPLUS in Emacs too. Turns out it was really easy, as sql-mode is built right in, no .emacs changes required or extra files to download.

To connect to SQLPLUS in an emacs buffer, fire up emacs and type ALT-x sql-oracle. Emacs will prompt you for a username and password and a database to connect to. The database name will need to be in your local tnsnames.ora – in other words, if you cannot sqlplus username/password@database, emacs will not be able to connect either.

You can now enter commands into SQLPLUS just like normal, only inside Emacs. The thing that always frustrated me about SQLPLUS is that there is no command history recall, so I always found myself writing a query in an editor, and copying and pasting into SQLPLUS. Not anymore … thanks to Emacs.

When you enter sql-oracle mode and login, Emacs splits your window in two. You can then edit your SQL in the original window and send the query to SQLPLUS in the other buffer in (at least) one of two ways:

• Send the entire buffer by typing ALT-x sql-send-buffer
• Select a region to send by typing ALT-x sql-send-region, or if you buffer is in sql-mode, type CTRL-C CTRL-r

You can of course bind a key sequence to each of these commands to save on the typing if you use them a lot!

If you have an SQLPLUS buffer open for a while, it could get very large. To truncate it, use the command ALT-x comint-truncate-buffer, or add the following to your .emacs:

(add-hook 'sql-interactive-mode-hook
              (function (lambda ()
                          (setq comint-output-filter-functions 'comint-truncate-buffer
                                comint-buffer-maximum-size 5000
                                comint-scroll-show-maximum-output t
                                comint-input-ring-size 500))))

Which will keep your buffer under 5000 lines.