The good, the bad, and the ugly

Created by Ted Winslow / Github: deltreey

Who am I?

I've worked at a lot of cool places

What am I talking about?

Automated Software Testing Less GUI Tests than Acceptance Tests, Less Acceptance tests than Unit Tests

Types of Tests

How will I talk about tests?

  1. All the code
  2. From the tested code all the way down the stack
  3. Some segment of the code in the middle of the stack
  4. Only the method/function under test

Run all the code

  • End to End <-- For clarity
  • Integration
  • Functional
  • Situational
  • Acceptance
[1] Technopedia

Run from the tested code all the way down the stack
and segments of code in the middle of the stack

  • Integration <-- For clarity
  • Functional
  • Unit

[1] Unit, Integration, and Functional Testing

Run only the code under test (one method or function)

  • Unit <-- For clarity
[1] The art of Unit Testing - Definition

The Software Stack

The Software Stack

End to End Tests - The Ugly

There need to be a bunch of these.

  • every possible input to your program at the top of the stack
  • include nulls
  • verify error messages
  • verify results <-- even wierd ones
  • many assertions
And they take FOREVER! [1] Say No to More End-to-End Tests [2] Little know[sic] Details on "End to End Testing"


What's their purpose?

  • Show that your software actually works.
  • Where you find logic errors and high level mistakes in your code.
  • ONLY tests that will actually find bugs in your code base.
  • Don't change often
  • Often are very similar, making them easy to write

How does that look on the stack?

End to End tests start at the front end and go down one path to the bottom of the stack

Integration Tests - The Bad

You build them when stuff doesn't work.

  • As needed when debugging
  • or scaffold a trust stack
  • faster than end to end tests
[1] Integration Testing Tips

The Bad

What's their purpose?

  • Help you write code
  • You can see that new code really works on the stack
  • They test how units work together...
  • You don't have to test to the bottom of the stack!

[1] MSDN [2] Mock Objects - Test Smell

How does that look on the stack?

Integration tests start somewhere in the stack and go down one path to another part of the stack

Unit Tests - The good

  • Super fast!
  • Build a trust stack and rely on one!
  • Super short <-- You still need lots of them
  • Live on Mocks
Trust Stack
[1] Best Practices for writing unit test scripts [2] Characteristics of a Good Unit Test

What's their purpose?

  • Changes to the code base cause them to fail
    • Mutation Testing
  • Any and all possible inputs and outputs are tested
    • Trust Stack
  • You no longer need to look at this code, as long as the tests pass
    • This test and all end to end tests
[1] Writing Great Unit Tests: Best and Worst Practices [2] Unit Test execution speed (how many tests per second?

How does that look on the stack?

Integration tests start somewhere in the stack and go down one path to another part of the stack

Examples Please

End to End

should('Write files to S3 when the user clicks save') do
  # Given: the user has some files to save
  file_path = 'test'
  user_files = [{
    name: file_path,
    data: 'data'
  visit 'pages/s3'
  page.files = user_files

  # When: the user clicks save and we check S3
  page.button({ id: 'save' }).click
  s3 = S3.new
  result = s3.read(file_path)

  # Then: we should see the files are on S3
  assert_false(result == nil)


should('result in the S3 bucket files being read in chunks based ' +
  'on the selected chunk size, ignoring empty files') do
  # Given: a file on S3 with more than 5 bytes of data
  #   and less than 10 bytes, a file on S3 with no data,
  #   and an S3Implementation object with the chunk size set to 5 bytes
  s3 = S3.new
  file_path = 'test'
  s3.write(file_path + "/data", 'some data')
  s3.write(file_path + "/nodata", '')

  s3_implementation = S3Implementation.new(file_path)
  s3_implementation.chunk_size = 5

  # When: we iterate over the data with the s3_implementation object
  count = 0
  s3_implementation.each do |chunk|
    count += 1

  # Then: we should see 2 chunks: 2 for the file with more than 5 bytes,
  #   and none for the empty file
  assert_equal(2, count)


  should('set s3 chunk_size property') do
    # Given: a value to set the chunk_size to and an S3Implementation
    chunk_size = 9000
    s3_implementation = S3Implementation.new('test')

    # When: we set the chunk size to the new value
    s3_implementation.chunk_size = chunk_size

    # We should see that the S3 base object has the new chunk size
    assert_equal chunk_size, s3_implementation.send(:s3).chunk_size

Code Coverage

  • 100% for end to end
  • 100% for unit
  • That's 200%!
  • 0 bug reports and 100% feature requests
  • Not gonna happen overnight

Are you too busy?

Too Busy to Change

[1] http://eedevsln.com/test-goodbadugly/index.html