Daddy Fixes Everything - A father and programmer's skewed view of the world

For Heaven’s Sake, At Least Test the CRUD

April 15, 2015 by Karl · 1 Comment

The very first integrated (feature) specs I setup it is to test the CRUD. If you don’t do anything else with feature specs, please, at least do these tests. You will be amazed at how much of your application is actually exercised and covered just with these simple tests.

Yes, you can and should test much more. And they don’t replace good unit and integrated tests. But you will be surprised how many bugs these basic tests will catch. For example, my nemesis, the dreaded undefined method `blahblahblah' for nil:NilClass in views.

These test will ensure the Adult model will:

Display all views (this alone is huge)
Data can be entered, displayed, updated, and destroyed (CRUD)
Model validations are respected
Potentially test model relations

I am going to show you how to test #index, #show, #edit, #update, #new, #create, and #destroy of a resource, an Adult model. I’ll break it down because the full spec file is long.

Here is the entire spec in it’s original project: adults_spec.rb

Setup

A little pre-knowledge would help; we are using Devise (for authentication), Rspec (for testing framework), Capybara (for integration tests). Note, we are not testing JavaScript in this test, and therefore invoke the RackTest as the Capybara driver for maximum speed.

require 'rails_helper'

include Warden::Test::Helpers

RSpec.describe 'Adults' do

before(:all) { Capybara.default_driver = :rack_test }

before(:all) do

Warden.test_mode!

seed_all_records

end

before(:each) do

Warden.test_reset!

login_as(@user, scope: :user)

visit unit_adults_path(@unit)

end

# tests go hear...

end

Warden.test_mode! gives us some nice Warden helper methods to use during testing. Notably the login_as(user, scope) method. Warden is used under-the-hood by Devise.

seed_all_records is just a method that will create a complete structure of records so that we have something to find and display. For example, it could create a user, company, orders, invoices, inventory, etc.

IMPORTANT: I am loading these records in a before(:all) do block so that they are not recreated for every test. If seed_all_records creates a long and complex amount of records, as this real-life version does, it can dramatically speed up the test.

CAVEAT: If we need to create other new records, we need to manually keep track of them to delete them after use. See the #edit example.

For every test, the first thing we do is view the index, visit unit_adults_path(@unit) . I like to start here and “click” the appropriate links to edit/show/delete tests.

#index

So we take a look at the index page, a table with a list of names (adults) and we check that we see some basic data (links and text).

context 'when viewing a list of unit adults' do

it 'shows the adult name, leadership position, and scouts' do

within 'body .container-fluid table' do

expect(page).to have_link('Karl, Tara')

expect(page).to have_text('Committee Chair')

expect(page).to have_link("Aydan Russel")

end

#show

We view the show page by clicking a link from the index page (Adult name). First, we make sure we are looking at the right show page. Then we click the “Return To List” link and make sure we go back to the index page. Finally, we try the “Edit” link of the Adult name and make sure were are taken to the edit form.

We look at the page.current_path and test it to make sure we are on the right page. This may not work in your application as some paths may contain query values or other errata. But for simple specs with standard Rails routes, it works.

context 'when showing an existing adult' do

before do

click_link 'Karl, Tara'

end

it 'clicking the name of the users goes to the edit view' do

expect(page.current_path).to eq(unit_adult_path(@unit, @user))

expect(page).to have_text('Adult: Tara Karl')

end

it 'clicking Return returns to adult list' do

click_link 'Return To List'

expect(page).to have_text('Adults in Pack 134')

expect(page.current_path).to eq(unit_adults_path(@unit))

end

it 'clicking Edit edits the adult' do

click_link "Edit"

expect(page).to have_text('Edit Adult')

expect(page.current_path).to eq(edit_unit_adult_path(@unit, @user))

end

#edit, #update

Like I noted above, we will need to create an Adult because we are going to change their data and we don’t want to affect the existing records. In an after do block, we will delete this Adult.

We create this new Adult, then we make some changes in the edit form. But don’t submit yet! We’ll do that later.

In the first test, we submit the change, and make sure that any form changes we make are reflected on the Adult’s show page. Success! We can edit and submit Adult records. This tests a significant amount of the Adult model, validations, and potentially more (relations, counters, etc).

Lastly, we test that if the user cancels the edit, they are returned to the Adult show page and none of the entered changes are saved.

context 'editing an adult' do

before do

@user2 = FactoryGirl.create(:adult, first_name: 'Bonnie', last_name: 'Doom')

@user2.units << @unit

visit unit_adult_path(@unit, @user2)

click_link "Edit"

within "form#edit_adult_#{@user2.id}" do

fill_in 'adult_first_name', with: 'Fionula'

fill_in 'adult[unit_positions_attributes][0][additional]', with: 'Jester'

end

after do

@user2.destroy

end

it 'updates fields when changed and returns to the show page' do

click_button 'Update Adult'

expect(page.current_path).to eq(unit_adult_path(@unit, @user2))

expect(page).to have_text('Fionula Doom was successfully updated')

expect(page).to have_text('Fionula Doom')

expect(page).to_not have_text('Bonnie Doom')

expect(page).to have_text('Jester')

end

it 'canceling update does not change data and returns to show page' do

click_link 'Cancel'

expect(page.current_path).to eq(unit_adult_path(@unit, @user2))

expect(page).to have_text('Bonnie Doom')

expect(page).to_not have_text('Fionula Doom')

end

#destroy

“I brought you into this world, I can take you out.”

Create a new Adult, then revisit the index page and click on the new Adult’s destroy link, and make sure they are gone and return to the index page. We also check that the flash notification is displayed.

Note: if you use a confirm: 'Are you sure?' like Rails suggests as a default for destroy links, because we are using RackTest, it will not execute that JavaScript. Thus there is not confirmation dialog to click.

context 'deleting an adult' do

before do

@user2 = FactoryGirl.create(:adult, first_name: 'Bonnie', last_name: 'Doom')

@user2.units << @unit

visit unit_adult_path(@unit, @user2)

end

it 'deletes the user and returns to the adult list' do

click_link 'Destroy Bonnie'

expect(page.current_path).to eq(unit_adults_path(@unit))

expect(page).to have_text('Bonnie Doom, and all associated data, was permanently deleted')

expect(page).to_not have_text('Doom, Bonnie')

end

#new, #update

Saved the longest for last. Also, the most important.

We are going to test two contexts: entering valid data, and entering invalid data.

Enter Valid Data

Simple, we click on the New Adult link, enter a bunch of data into the form, and save it. Then we verify that we are taken to the show page and can see all the data we entered.

This particular form has several select elements that allow the user to select from a list of options, as well as establish relations to other models. This is a good place to test this, but it could also be tested in #edit, #update. We verify that this Adult can have related Scouts, and no related Scouts. This also tests your models, validations, and relations. Good tests to have.

Enter Invalid Data

We want to see that when we enter invalid data, that model validations are respected, and that the record is not saved. When trying to submit invalid data, we return to the new form and check that the proper flash message is displayed.

In this example, I only test one field’s validation, that the Adult’s first_name can’t be blank. This is an excellent place to add more test for more validations.

context 'when adding a new adult' do

before { click_link 'New Adult' }

context 'with valid data' do

before do

within 'form#new_adult' do

fill_in 'adult_first_name', with: 'Rusty'

fill_in 'adult_last_name', with: 'Balls'

end

it 'enter new adult info, submit, goto to adult show and display show adult page' do

expect(page).to have_text('New Adult in Cub Scout Pack 134')

within 'form#new_adult' do

fill_in 'Email', with: '[email protected]'

select 'Webmaster', from: 'adult_unit_positions_attributes_0_leadership'

fill_in 'adult_unit_positions_attributes_0_additional', with: 'Joker'

select '1955', from: 'adult_birth_1i'

select 'February', from: 'adult_birth_2i'

select '14', from: 'adult_birth_3i'

fill_in 'Address1', with: '3660 N Lake Shore Dr'

fill_in 'adult_city', with: 'Chicago'

select 'Illinois', from: 'adult_state'

fill_in 'adult_zip_code', with: '60657'

# this should be tested later with javascript

select 'Russel, Aydan', from: 'adult_scout_ids'

end

click_button 'Create Adult'

rusty_balls = User.find_by_first_name('Rusty')

expect(page.current_path).to eq(unit_adult_path(@unit, rusty_balls))

expect(page).to have_text('Rusty Balls was successfully created')

expect(page).to have_text('Adult: Rusty Balls')

expect(page).to have_text('Cub Scout Pack 134')

expect(page).to have_text('[email protected]')

expect(page).to have_text('Webmaster, Joker')

expect(page).to have_text('3660 N Lake Shore Dr')

expect(page).to have_text('Chicago, IL 60657')

expect(page).to have_link('Aydan Russel')

end

it 'allows the adult to have multiple related scouts' do

within 'form#new_adult' do

select 'Russel, Aydan', from: 'adult_scout_ids'

select 'Jones, Bocephus', from: 'adult_scout_ids'

end

click_button 'Create Adult'

expect(page).to have_text('Rusty Balls was successfully created')

expect(page).to have_link('Aydan Russel')

expect(page).to have_link('Bocephus Jones')

end

it 'allows an adult to have no related scouts' do

click_button 'Create Adult'

expect(page).to have_text('Rusty Balls was successfully created')

expect(page).to_not have_link('Aydan Russel')

expect(page).to_not have_link('Bocephus Jones')

end

context 'with invalid data' do

it 'displays flash messages with errors' do

click_button 'Create Adult'

expect(page).to have_text('First name can\'t be blank')

expect(page).to have_text('Last name can\'t be blank')

end

it 'entered data is persisted' do

within 'form#new_adult' do

fill_in 'adult_last_name', with: 'Eustus'

end

click_button 'Create Adult'

expect(page).to have_text('First name can\'t be blank')

expect(page).to have_field('adult_last_name', with: 'Eustus')

end

context 'clicking cancel' do

it 'returns to adult list' do

click_link 'Cancel'

expect(page.current_path).to eq(unit_adults_path(@unit))

end

Conclusion

Put it all together, and you are testing a lot. I know they seem like simple tests, maybe too simple for some. But I think they are boilerplate for much more.

I said it before, but it bears repeating: setting up these basic CRUD integrated tests will catch a surprising amount of bugs.

Posted in Rails, Rspec. Tagged with capybara, Rails.

Creating Cache Keys for Rails Relations

April 9, 2015 by Karl · Comments Off

For years I have been using the cache_key_for_* method, as seen in Rails Guides and many other places. Here is an example.

def cache_key_for_products(products)

count = products.count

max_updated_at = products.maximum(:updated_at).try(:utc).try(:to_s, :number)

"products/all-#{count}-#{max_updated_at}"

end

It works. But even this venerable snippet couldn’t escape the refactoring razor.

def cache_key_for_products(products)

count_max = [products.size, products.maximum(:updated_at)].map(&:to_i).join('-')

"products/all-#{count_max}"

end

.size vs .count

First change is using products.size instead of products.count . A minor change, but if ActiveRecord has already executed the sql query, using the .size method will prevent it from executing another query, as .count may cause.

Sean Griffin has a quick say in this battle. TL;DR Rails users should always use .size .

DateTime Number

Second, the original used a long chain of trys and then converting the DateTime of updated_at to number format. You might be breaking the law… of Demeter that is. That ‘number’ format looks like this: 20150410030422. It’s just the date and time.

Well, you can accomplish the same thing without chaining so many methods: use .to_i. Read about it at the docs. This one method uses the UTC time and gives you the number of seconds since the Unix Epoch. It looks like this: 1428635331.

Both has the same 1 second resolution, but the refactored version is shorter, which may save memcached memory if you have a lot of keys. I just think it’s cleaner, and avoids all the method chains.

Assembly

Finally, we just need to assemble it all in a string. By putting the size and time value into an array, we can use all those wonderful enumerable methods. You could just as easily use string interpolation and avoid the map and join.

Important Footnote

Rails has been using nanosecond resolution for their cache_key method for a while, assuming your backend supports factional second datetime values. My method only has 1 second resolution. If you require more granularity for your cache keys, by all means, use the original cache_keys_for_* method and change the .to_s(:number) to .to_s(:nsec).

Posted in Rails. Tagged with Rails.

Create an In-memory Temporary ActiveRecord Table for Testing

February 5, 2015 by Karl · 2 Comments

I was reading Greg Niewisiewicz’s blog post on 6 Gems to Make Rails Development Easier and saw the first recommendation was for a gem called temping. Nice. But here is my take on the same thing without adding a gem (sort of) and doing it with in-memory SQLite tables.

I’m using Rspec3, so these example will need modification if you are using MiniTest.

Gemfile

Ok, so you will need to add the sqlite3 gem to your Gemfile, if you don’t already have it in there. Because I am only using SQLite for testing, I put it in the :test group.

group :test do

gem 'sqlite3'

end

Before(:all)

We need to create the table in SQLite via migration, then tell ActiveRecord to switch it’s database connection (temporarily) to use this in-memory db as the test environment database. We also add the ActiveRecord class so that we can, well, CRUD records.

before(:all) do

# don't output all the migration activity

ActiveRecord::Migration.verbose = false

# switch the active database connection to an SQLite, in-memory database

ActiveRecord::Base.establish_connection(:adapter => "sqlite3", :database => ":memory:")

# execute the migration, creating a table (dirty_items) and columns (body, email, name)

ActiveRecord::Schema.define(:version => 1) do

create_table :dirty_items do |t|

t.text :body

t.text :email

t.text :name

end

# this creates the ActiveRecord class we will use, just like any other AR class

class DirtyItem < ActiveRecord::Base

# add anything you need for this class

# this is the concern we are testing!

include AttributeSanitizer

end

After(:all)

We need to switch back to using your “test” database after all the tests are done in this spec. DON’T FORGET THIS STEP! Or all your other tests may fail, and you will be scratching your head why.

after(:all) do

ActiveRecord::Base.establish_connection(ActiveRecord::Base.configurations['test'])

end

Good To Go

So now, you can operate on your temporary, in-memory, ActiveRecord table at lightning speed. And you don’t even need a gem. Just a few lines added to your test.

1	let(:dirty_item) { DirtyItem.new(body: 'humpty', email: 'dumpty', name: 'Ginger Ninja') }

Caveat

Well, there is one downside, by using SQLite for the spec, you may not be using the database you would be running in production. This is true, unless you are using SQLite in production.

But, to be honest, you should try to be as database agnostic as possible. What if you have to switch from MySQL to PostgreSQL in the future for your production backend. Any MySQL database specific language you are using may fail in PostgreSQL.

On the plus side, the in-memory SQLite tables are really, really fast.

Posted in Rails, Rspec.

Why Rails and Their Committers Are Awesome

January 30, 2015 by Karl · 1 Comment

I love keyword arguments when they were introduced in Ruby 2.0, and use them every chance I get. As I was hacking around with ActiveJob today, and thought it would be cool to pass keyword arguments into the ActiveJob#perform_later method.

I thought it should work. It didn’t.

class MyAwesomeJob < ActiveJob::Base

def perform(ginger_ninjas: 1)

attack!(ginger_ninjas)

end

MyAwesomeJob.perform_later(ginger_ninjas: 5)

You get an ArgumentError: wrong number of arguments (1 for 0) error. The reason is the hash is converted to string keys upon serialization.

So I was thrilled when I did a bit of searching to see if anyone else had the same issue. Nope. Looked at the ActiveJob::Arguments module. Passing keyword arguments are not supported. I actually get to report a real feature request to Rails!

So here is my ActiveJob should support passing of keyword arguments to perform method.

This this is where the magic began… within a few minutes Sean Griffin (@sgrif) picked up on the issue report. I went to lunch, planning to fork ActiveJob and start hacking away when I got back. But Sean beat me to it. Sean added the ability to pass keyword arguments to ActiveJob jobs, and even backported it to Rails 4.2.1.

Within an hour, it was all done. Check out the full commit.

Want a reason to love Rails and their committers? This is one, of many.

Thank you Sean.

Posted in Rails, Ruby. Tagged with Rails, Ruby.

Self-joined HABTM with Cache Updates

January 1, 2015 by Karl · Comments Off

You have a self-joined HABTM table. It’s probably called User too. All is well, until you realize that when relationships change, your site does not display those changes. The reason is that the related objects (users) are not being updated when the relation is created/removed or one of the objects are updated/destroyed. And the cache keys that rails generate for the parent object are still referencing old data.

Here is how to fix it.

Self-Joined Users

has_and_belongs_to_many :children, class_name: 'User', join_table: "user_relationships", foreign_key: "adult_id", association_foreign_key: 'child_id'

has_and_belongs_to_many :adults, class_name: 'User', join_table: "user_relationships", foreign_key: "child_id", association_foreign_key: 'adult_id'

Pretty simple really. There is a table in the database called user_relationships with two fields: adult_id , child_id . You can create it with this migration:

class CreateUserRelationships < ActiveRecord::Migration

def change

create_table :user_relationships, id: false do |t|

t.references :adult

t.references :child

end

add_index :user_relationships, [:adult_id, :child_id]

add_index :user_relationships, [:child_id, :adult_id]

end

Now every User can have .adults and .children. And those adults and children can have adults and children. And so on. And so on.

User Views

You are using rails fragment caching aren’t you? If not, stop right now and read up on how to implement caching in your rails app because even if you don’t need it now, you most like will need it later. Your users will love if for it!

Watch more: Greg Pollack has an excellent video on Dalli & Cache Digests. He does a much better job explaining caching, cache keys, and fragment caching that I.

Rails uses the concept of a cache_key for each object. Cache keys are simply the model name, record id, and the datetime of the updated_at attribute for the ActiveRecord object. Check it out:

~ Adult.first.cache_key

Adult Load (1.0ms) SELECT "users".* FROM "users" WHERE "users"."type" IN ('Adult') ORDER BY "users"."id" ASC LIMIT 1

=> "adults/317-20150102035234372794000"

Where cache keys come into play are in views. When a view generates it’s html, the cache method will ask your caching system (most likely memcached) if a key exists. If it does, then it pulls that fragment from the cache without the expense of generating the html. If they key doesn’t exists, then it renders the html and then stores the result in the cache. Here is a very simple example.

= cache @user do

%strong Name:

= @user.name

%strong Children:

= @user.children.map(&:name).join(', ')

When rendered, this will display the user’s name, and the names of all their children.

Changing Relationship Doesn’t Change the Rendered View

All is going well, until you start getting reports of problems. A user deletes one of her children, but when she views her show page (see above view), she still see the deleted child’s name. The reason is that when you make a change to a HABTM relationship, it only removes the relationship in the uses_relationships table. It doesn’t update the updated_at datetime for the parent record. Thus, the cache key for the parent record does not change, and the old (incorrect) information is read from cache.

How To Fix

I wan’t able to find an definitive example of how to fix this completely and correctly for all cases. So here is my implementation.

What you need to do is “touch” all related objects in a relationship anytime that relationship changes. And those changes could be:

Parent (adult) object is updated.
Any children objects are updated.
Relationships are created.
Relationships are deleted.

To fix the first two cases, we just need a couple of callbacks.

after_update :touch_related_users

before_destroy :touch_related_users

def touch_related_users

if self.child?

self.adults.update_all(updated_at: Time.now)

elsif self.adult?

self.children.update_all(updated_at: Time.now)

end

Astute coders will notice that we used update_all instead of touch here. While we could have used touch, there is the possibility that you could create a circular update infinite loop. The touch method will cause all touched objects to also execute any touches that are defined on their relationships.

In our particular case, we are not concerned about touching any additional relationships here because were are only updated the updated_at attribute to trigger a cache invalidation. We aren’t actually changing the related objects data like a name or birthday.

Now, we need to fix the last two cases when relationships are created or deleted.

has_and_belongs_to_many :children, class_name: 'User', join_table: "user_relationships", foreign_key: "adult_id", association_foreign_key: 'child_id', after_add: :touch_updated_at, after_remove: :touch_updated_at

has_and_belongs_to_many :adults, class_name: 'User', join_table: "user_relationships", foreign_key: "child_id", association_foreign_key: 'adult_id', after_add: :touch_updated_at, after_remove: :touch_updated_at

def touch_updated_at(user)

user.update_column(:updated_at, Time.now)

end

Note that we added after_add: :touch_updated_at, after_remove: :touch_updated_at to the HABTM definition. These are callbacks that are triggered upon relationship additions or deletions. They both call the touch_updated_at method that like above, will update the updated_at attribute. But in this case, the HABTM will pass a reference to the modified user object.

Here we use the update_column method because we do not want to trigger any callbacks, validations, or cascading touches.

Fixed!

Now, anytime a user adds a child, deletes a child, or any data on all related children is changed, the updated_at attribute is set to the current datetime. This will change the calculated cache key for the user objects, and voilà, the view will regenerate the html and users will see the modifications.

Posted in Uncategorized.

Rails HABTM with Unique Scope and Select Columns

May 20, 2014 by Karl · 2 Comments

This one left me scratching my head for a while. Not quite sure I would call it a bug, but this one definitely has a GOTCHA.

Let say you have a Users model and Units model. Each can have_and_belong_to_many of each other. But, you want to make sure there are no multiple User->Unit relationships, or Unit->User relationships. So you use the association scope -> { uniq } .

It won’t work. Here is why.

Models

So we have these models:

class Unit < ActiveRecord::Base

has_and_belongs_to_many :users, -> { uniq }

end

class User < ActiveRecord::Base

has_and_belongs_to_many :units, -> { uniq }

end

And the relationship table:

class CreateUnitsUsers < ActiveRecord::Migration

def change

create_table :units_users, id: false do |t|

t.references :unit

t.references :user

end

add_index :units_users, [:unit_id, :user_id]

add_index :units_users, [:user_id, :unit_id]

end

You can read more about ActiveRecord associations and queries at Rails Guides.

Queries

When you query a Unit for Users, you get only DISTINCT users, and vice-versa.

> Unit.find(8).users.count

Unit Load (0.2ms) SELECT "units".* FROM "units" WHERE "units"."id" = ? LIMIT 1 [["id", 8]]

(0.5ms) SELECT DISTINCT COUNT(DISTINCT "users"."id") FROM "users" INNER JOIN "units_users" ON "users"."id" = "units_users"."user_id" WHERE "units_users"."unit_id" = ? [["unit_id", 8]]

=> 130

> User.find_by_first_name_and_last_name('Karl', 'Smith').units.count

User Load (0.4ms) SELECT "users".* FROM "users" WHERE "users"."first_name" = 'Karl' AND "users"."last_name" = 'Smith' LIMIT 1

(0.3ms) SELECT DISTINCT COUNT(DISTINCT "units"."id") FROM "units" INNER JOIN "units_users" ON "units"."id" = "units_users"."unit_id" WHERE "units_users"."user_id" = ? [["user_id", 617]]

=> 1

But All is Not Well

Let’s check out what happens if we use a .select(:first_name) with the Unit.users query.

> Unit.find(8).users.select(:first_name).count

Unit Load (0.1ms) SELECT "units".* FROM "units" WHERE "units"."id" = ? LIMIT 1 [["id", 8]]

(0.4ms) SELECT DISTINCT COUNT(DISTINCT "users"."first_name") FROM "users" INNER JOIN "units_users" ON "users"."id" = "units_users"."user_id" WHERE "units_users"."unit_id" = ? [["unit_id", 8]]

=> 108

Whoa there… our count of users went from 130 (the correct number) to just 108. Why? Take a look at the SQL with the .select(:first_name) query. Specifically the SELECT DISTINCT COUNT(DISTINCT "users"."first_name") . The query that ActiveRecord is generating is doing a DISTINCT (alias for uniq) on the "users"."first_name" . This query will only find users with DISTINCT first names. What if you have 3 people with the first name of “John”. Well, it will only find and return 1 of those John’s.

This is not what I expected. POLA any one?

Documentation

Rails Guides has a good section on Selecting Specific Fields that goes into detail about using .select() with .unique . But it doesn’t mention that using the -> { uniq } scope on an ActiveRecord association will have the same affect. Well, it does.

So, it got me to thinking… could I add -> { distinct(:id) } to the HABTM declaration? Yes you can. But it doesn’t work with .select() . For some reason when you use .select() ActiveRecord will override any column declarations you used with the distinct() .

Proof eh?

class User < ActiveRecord::Base

has_and_belongs_to_many :units, -> { distinct(:id) }

end

class Unit < ActiveRecord::Base

has_and_belongs_to_many :users, -> { distinct(:id) }

end

> Unit.find(8).users.count

Unit Load (0.1ms) SELECT "units".* FROM "units" WHERE "units"."id" = ? LIMIT 1 [["id", 8]]

(0.4ms) SELECT DISTINCT COUNT(DISTINCT "users"."id") FROM "users" INNER JOIN "units_users" ON "users"."id" = "units_users"."user_id" WHERE "units_users"."unit_id" = ? [["unit_id", 8]]

=> 130

> Unit.find(8).users.select(:first_name).count

Unit Load (0.1ms) SELECT "units".* FROM "units" WHERE "units"."id" = ? LIMIT 1 [["id", 8]]

(0.4ms) SELECT DISTINCT COUNT(DISTINCT "users"."first_name") FROM "users" INNER JOIN "units_users" ON "users"."id" = "units_users"."user_id" WHERE "units_users"."unit_id" = ? [["unit_id", 8]]

=> 108

I find it interesting that the first query, without specifying a .select() generates the correct DISTINCT query with the "users"."id" column. But as you can see on the second query, it is overriding the specified scope with the .select() columns.

This is my logic: if I use .select() that doesn’t necessarily mean I want DISTINCT columns, regardless if weather I specify a .distinct() or not. In the above case, .select() always includes the :id column wether or not you are also using .distinct() . So why would the .select() override my .distinct(:id) declaration?

Solution?

First solution is to not use .select() with a query. Duh.

But I really like .select() . It’s a great way to reduce memory size when instantiating hundreds or thousands of User ActiveRecord objects. And in my real-world use case, the User table has a couple of text columns that contain large amounts of html. So I like to use .select() where ever I can.

So I guess there really isn’t a solution that I know of, except not to use .select() with ActiveRecord queries on HABTM tables that use the -> { uniq } scope.

Bummer.

Best Solution!

You knew it was coming, didn’t cha. I wouldn’t leave you hanging.

The best solution is to not use the -> { uniq } scope with HABTM relations, but enforce the uniqueness condition at the database level with unique compound indexes.

In your migration for the relationship table just add unique: true to the index declaration.

class CreateUnitsUsers < ActiveRecord::Migration

def change

create_table :units_users, id: false do |t|

t.references :unit

t.references :user

end

add_index :units_users, [:unit_id, :user_id], unique: true

add_index :units_users, [:user_id, :unit_id], unique: true

end

Posted in Rails.

Model Doesn’t Save After Adding before_save Callback

June 27, 2013 by Karl · Comments Off

File this one under the Duh folder… I added a new before_save callback yesterday, and suddenly nothing in the model would save. Here is the offending code:

before_save :sms_verification

def sms_verification

if sms_number_changed?

self.sms_number_verified = false

end

Spot the problem? I didn’t. At least for a while.

In a callback, if you return false the save is cancelled. I knew that. So, when setting the sms_number_verified attribute to false , it also returns false from the callback, and therefore the model does not save. I did not expect that. Seems evident now that you see it.

Here is the fix:

before_save :sms_verification

def sms_verification

if sms_number_changed?

self.sms_number_verified = false

true

end

Just return true from the callback.

Would have been nice if rails notified me in the logs with something like “save cancelled, before_save callback returned false on line XXX”.

Posted in Rails.

SQLite3::ConstraintException: table.created_at may not be NULL

June 14, 2013 by Karl · Comments Off

So, I’m working along in Rails today, added a new migration, when all of a sudden tests that were passing only a few minutes ago suddenly start failing. The error I in rspec is something along the lines of:

Failure/Error: before { @event.update_attribute(:signup_deadline, 1.day.ago) }

ActiveRecord::StatementInvalid:

SQLite3::ConstraintException: event_signups.created_at may not be NULL: INSERT INTO "event_signups" DEFAULT VALUES

In test.log:

1 2	[1m[35mSQL (0.5ms)[0m INSERT INTO "event_signups" DEFAULT VALUES SQLite3::ConstraintException: event_signups.created_at may not be NULL: INSERT INTO "event_signups" DEFAULT VALUES

WTF? This was working only a few minutes ago.

If you google this, you will read endless others who experience the same error… when using HABTM relations. Well, EventSignup has no HABTM!

This has happened to me a couple of times, and each time it leaves me puzzled for far too many minutes, and there is no real solution.

Okay, well there is only one solution I have found to work: delete the SQLite database file and recreate. Yeah, that sux. Oddly enough, just a quick:

1 2	rm db/test.sqlite3 rake db:test:prepare

And all is good again.

If anyone has an idea why this happens, I would love to hear why.

Posted in Rails. Tagged with Rails, sqlite.

Rails Nested Attributes with Scoped Uniqueness Validation of Association

December 21, 2012 by Karl · Comments Off

This is going to be a long one, so grab your favorite beverage…

Problem

I have a fairly standard belongs_to and has_many model association. And I want to save nested attributes along with the parent when creating a new parent record. But, I don’t want the user to be able to submit duplicate values for the child records.

Sounds easy. That’s what I thought when I started.

My Solution

Models

We have an Organization and a SubUnit models, pretty standard stuff here.

lass SubUnit < ActiveRecord::Base

attr_accessible :name, :organization_id

belongs_to :organization

validates :name,

presence: true,

uniqueness: { scope: :organization_id }

end

class Organization < ActiveRecord::Base

attr_accessible :name, :sub_units_attributes

has_many :sub_units, dependent: :destroy

accepts_nested_attributes_for :sub_units, allow_destroy: true, reject_if: proc { |a| a["name"].blank? }

validates_presence_of :name

end

Controller

The controller is simple (standard actions removed). Keep scrolling down.

# controllers/organizations_controller.rb

class OrganizationsController < ApplicationController

def new

@organization = Organization.new

@organization.sub_units.build

end

def create

@organization = Organization.new(params[:organization])

if @organization.save

redirect_to @organization, notice: 'Organization was successfully created.'

else

render :new

end

View

Now it’s getting interesting.

# organizations/new.html.haml

= form_for @organization, :html => { :class => 'form-horizontal' } do |f|

.row-fluid

.span6

.well

%h2 Organization Info

.control-group

= f.label :name, :class => 'control-label'

.controls

= f.text_field :name, :class => 'text_field input-medium'

.span6

.well.sub-units

%h2 SubUnit Info

= f.fields_for :sub_units do |f_sub|

= render 'sub_unit_fields', f: f_sub

.control-group.add-sub-unit

.controls

= link_to 'add a SubUnit', '#', class: 'btn btn-primary add-sub-unit-button'

#new-sub-unit-fields.hidden

= f.fields_for :sub_units, SubUnit.new, child_index: 'new_sub_unit' do |f_sub|

= render 'sub_unit_fields', f: f_sub

.span12

.form-actions

= link_to 'Create Organization', '#', class: 'btn btn-primary create-organization-button'

= link_to t('.cancel', :default => t("helpers.links.cancel")), organizations_path, :class => 'btn'

# _sub_unit_fields.html.haml

.control-group

= f.label :name, class: 'control-label'

.controls.sub-unit

= f.text_field :name, class: 'input-mini'

= f.hidden_field :_destroy

= link_to "remove", '#', class: 'btn btn-warning btn-mini remove-sub-unit-button'

Your form should look like this.

Polluting Links with Javascript

Like everyone else who has started down the road to nested attribute bliss, you have watched Ryan Bates’ Railscast on Complex Forms. It’s superlative.

But there’s one thing I didn’t like… the heavy handed approach of using link_to_function. Especially when including a fair amount of escaped html into an onclick script. I’m not saying it’s wrong, just that I believe html should stay where it belongs… in the DOM. Heck, they even created a gem nested_form.

I think we can make it simpler, painless, elementary.

That’s where the following little ditty comes in handy. We are creating the exact same field structure that is used for entering a SubUnit, but hiding the html so that we can use JavaScript to copy-and-append to the visible entry area.

#new-sub-unit-fields.hidden

= f.fields_for :sub_units, SubUnit.new, child_index: 'new_sub_unit' do |f_sub|

= render 'sub_unit_fields', f: f_sub

Line #1: hide the div.

Line #2: use fields_for to generate the proper field id and name attributes so that it is submitted as a child object to our Organization. Notice the SubUnit.new? That prevents this block from iterating over all the @organization.sub_units that may be built, either as a new object (controller line #5), or when re-rendered due to a validation error with @organization.

Line #2: don’t forget the child_index: ‘new_sub_unit’ option! Without it, rails will attempt to index the fields id and name attributes with numbers, which in this case would be 0 (zero), and mess up the prior sub_unit fields. IOW, you will end up with two params[:organization][:sub_units_attributes][0] elements.

If you don’t understand all the details, take a few minutes and read the ActiveRecord::NestedAttributes::ClassMethods.

Now for some JS magic:

jQuery ->

$('.well.sub-units').on('click', 'a.remove-sub-unit-button', ->

remove_sub_unit(@)

)

$('.add-sub-unit-button').click ->

add_sub_unit(@)

add_sub_unit = (link) ->

new_sub_unit = $(".hidden#new-sub-unit-fields").html()

new_id = new Date().getTime()

regexp = new RegExp("new_sub_unit", "g")

$(link).closest('.add-sub-unit').before(new_sub_unit.replace(regexp, new_id))

$(link).closest('.add-sub-unit').prev().find('input[type=text]').focus()

remove_sub_unit = (link) ->

$(link).prev("input[type=hidden]").val("1")

$(link).closest(".control-group").hide()

Pretty simple, bind to the click events of the add-sub-unit-button and remove-sub-unit-button to functions.

add_sub_unit function copies the new-sub-unit-fields div, replacing the text of ‘new-sub-unit’ in the id and name attributes of the input/label elements with an integer.

remove_sub_unit function hides the input field group, and set the ‘_destroy’ hidden field to “1” (‘true’ also works) so that it will not be created when the form is submitted.

Inspiration for this JavaScript goes to Ryan Bates.

Status Check

At this point we have a parent model, Organizations, with nested attributes for children SubUnits. When creating a new Organization, you can add and remove SubUnits. The entire relationship is submitted with a single form post.

Eliminate Duplicates in the DOM

Everyone else does it in model validations, manipulation of the params hash, or some other convoluted way. But I decided to eliminate duplicates before the submission of the parent form. Pretty easy, actually.

jQuery ->

$('a.create-organization-button').click ->

# remove duplicate form values, then submit the form

uniqueNames = []

$.each($('.controls.sub-unit input[type=text]'), (i, el) ->

if $.inArray($(@).val(), uniqueNames) == -1

uniqueNames.push($(@).val())

else

$(@).parent().find('input[type=hidden]').val("true")

)

$(@).closest('form').submit()

When the Create Organization button is clicked, the above JavaScript loops through all the SubUnit form field elements, marking duplicate entries hidden ‘_destroy’ field with ‘true’, thus preventing them from being saved. Could you just remove the elements? Sure.

Inspiration for the JavaScript remove duplicates in an array to Roman Bataev in this post.

If you watch your logs, you’ll see that all the SubUnits attributes are submitted, but only those marked with the hidden field ‘_destroy’ == ‘false’ will be created.

Started POST "/organizations" for 127.0.0.1 at 2012-12-21 16:32:39 -0700

Processing by OrganizationsController#create as HTML

Parameters: {"utf8"=>"✓", "authenticity_token"=>"Dkphnf9nkc+5cxqRgo7mkQLcz2nvxHIUxezlOPA85cA=", "organization"=>{"name"=>"My Org", "sub_units_attributes"=>{"0"=>{"name"=>"Unit1", "_destroy"=>"false"}, "1356132751848"=>{"name"=>"Unit1", "_destroy"=>"true"}, "1356132755193"=>{"name"=>"Unit2", "_destroy"=>"false"}, "new_sub_unit"=>{"name"=>"", "_destroy"=>"false"}}}}

(3.5ms) begin transaction

SubUnit Exists (0.4ms) SELECT 1 AS one FROM "sub_units" WHERE ("sub_units"."name" = 'Unit1' AND "sub_units"."organization_id" IS NULL) LIMIT 1

SubUnit Exists (0.1ms) SELECT 1 AS one FROM "sub_units" WHERE ("sub_units"."name" = 'Unit2' AND "sub_units"."organization_id" IS NULL) LIMIT 1

SQL (28.4ms) INSERT INTO "organizations" ("created_at", "name", "updated_at") VALUES (?, ?, ?) [["created_at", Fri, 21 Dec 2012 23:32:39 UTC +00:00], ["name", "My Org"], ["updated_at", Fri, 21 Dec 2012 23:32:39 UTC +00:00]]

SQL (2.1ms) INSERT INTO "sub_units" ("created_at", "name", "organization_id", "updated_at") VALUES (?, ?, ?, ?) [["created_at", Fri, 21 Dec 2012 23:32:39 UTC +00:00], ["name", "Unit1"], ["organization_id", 6], ["updated_at", Fri, 21 Dec 2012 23:32:39 UTC +00:00]]

SQL (0.2ms) INSERT INTO "sub_units" ("created_at", "name", "organization_id", "updated_at") VALUES (?, ?, ?, ?) [["created_at", Fri, 21 Dec 2012 23:32:39 UTC +00:00], ["name", "Unit2"], ["organization_id", 6], ["updated_at", Fri, 21 Dec 2012 23:32:39 UTC +00:00]]

(48.5ms) commit transaction

The astute reader will also notice that the SubUnit model is attempting to validate the uniqueness of the name attribute, scoping it to a non-existant Organization. I admit it would be best to skip this validation, but I will leave that exercise to you, humble reader.

Sample App

What? You wanted a demo app on the GitHubs? You shall be rewarded, post haste.

Synopsis

The Good

Submission of a parent Organization with children SubUnits utilizing nested attributes.
Duplicate SubUnits are removed.
Darn simple, easy to reproduce. Pretty darn unobtrusive, too.
No funky helpers to jack some html into a link element.
Is it better than Ryan Bates’ solution? No. Just simpler.

The Bad

Only works on creating a new Organization. Does not work for editing an Organization with the ability to add additional SubUnits or destroy existing SubUnits. Future post?
My JavaScript is not abstracted to make it easily reusable. Fork it, and submit a patch if you so desire.
The SubUnit validation should be skipped when creating through an Organization.
No fallback if JavaScript is disabled.

Additional Reading

StackOverflow: validates_uniqueness_of in nested model rails, a great discussion of how to handle scoped validation when editing existing child attributes.
Rails issue: validates_uniqueness_of and accepts_nested_attributes_for only validates existing records, macfanatic shows a nice way to handle the same issue.
Rails pull request: Validates associated uniqueness for nested attributes. The paint is still wet on this one. Hopefully is stirs some discussion.
Ryan Bates’ gem: nested_form.

Posted in Rails. Tagged with nested attributes, Rails.

Small Revelation – FactoryGirl, build_stubbed, associations, and let

July 9, 2012 by Karl · 6 Comments

Not really a problem, but something I didn’t really look out for because all my tests were passing… and hey, if tests are passing, I sleep well.

TL;DR Dumped let{}, removed FactoryGirl associations, back to before(:all) and instance variable/FG.build pattern = super speedy tests.

TL;DR.1 And read the docs.

Background

I need to test a fairly complex permission matrix at the model level. To properly test it, I need to build out several complete company structures, with a multitude of users at various authorization levels. And 200+ actual tests.

Being a fan of rspec’s let{}, I decide to use it copiously. 40 lines worth.

In these tests, I don’t really need to have persisted records, so using FactoryGirl.build_stubbed fits the bill here.

let(:company) { FactoryGirl.build_stubbed(:company)}

… an so on, 39 times.

What I Found

I thought one particular spec was running a little slow. So I converted the FactoryGirl.build to the shiny new FactoryGirl.build_stubbed. No difference in time. So I decided to watch the test.log. Well, I quickly saw what was happening… it was persisting thousands of records.

But wait. I was using FactoryGirl.build_stubbed. Where were all these records coming from?

Problem #1

In most of my FactoryGirl factories, I was using ‘association :area‘, ‘association :store‘, and so on. Didn’t think much about these until yesterday.

Turns out that FactoryGirl will build/create and save those associated factories, even if you are using FactoryGirl.build or FactoryGirl.build_stubbed. Learned something new there. Honestly, I didn’t expect this behavior, but I understand why. Shoulda read the docs.

Now, the easy way around this is to pass into a FactoryGirl.create/build/build_stubbed a nil for the association, if it is not needed. Ala:

FactoryGirl.build_stubbed(:store, company: fg_company, area: nil)

Now it won’t build out the associated area. Alas, I had forgotten just one of these nil associations. And at the worst possible relationship level, the bottom. So every time one factory was built, it create the entire supporting structure above. Thus, every hit to an let(:invoice) builds an entire company structure from one single FactoryGirl.build_stubbed(:invoice) call.

But it get’s worse.

Problem #2

I love the let{}. But to be honest, I never read the docs on it. Well, I did read them yesterday. Relavent line being (emphasis added):

The value will be cached across multiple calls in the same example but not across examples.

Uh-oh. Let{} is like a before(:each). Which is what most specs need. But I don’t, not for this spec. I’m never modifying the AR records, just testing some methods within models, which don’t modify AR instances.

Resulting Big Problem

Ok, not really a problem. But certainly very, very inefficient.

By forgetting to nil an association in a FactoryGirld.build_stubbed, and with let{} recreating an entire company structure, to the database, for every 200+ test. Well, you get the picture. It’s slooooow. 22 seconds worth of slow.

Solution

You know I wouldn’t drag you along this far without a solution.

Just remove all the ‘association :model‘ statements from all FactoryGirl definitions. I know they are handy, but I want CONTROL over my factories. And just one small mistake can make a spec run many X-times longer.
Remove the let{} and replace with the good ol’ instance variable/build pattern.
Move all the instance variables into a before(:all).

before(:all) do

@company = FactoryGirl.build_stubbed(:company)

@store = FactoryGirl.build_stubbed(:store, company: company)

… and so on, 38 times.

end

Note for step #1. It caused me to refactor some other specs as well. This turned out to be a good thing, as I was able to speed up several other specs, and add some clarity to those specs that required building out a company structure.

Results

2.5 seconds. Not too shabby.

After the refactoring for all tests, I dropped ~30 more seconds off the entire suite.

Hope this helps someone else out there improve the speed of their specs too.

Posted in Rails, Ruby. Tagged with factory girl, Rails, rspec.