{"id":1074,"date":"2013-05-11T13:17:25","date_gmt":"2013-05-11T19:17:25","guid":{"rendered":"http:\/\/www.mooreds.com\/wordpress\/?p=1074"},"modified":"2014-09-13T08:37:46","modified_gmt":"2014-09-13T14:37:46","slug":"testing-with-pentaho-pdi-next-steps","status":"publish","type":"post","link":"https:\/\/www.mooreds.com\/wordpress\/archives\/1074","title":{"rendered":"Testing with Pentaho Kettle &#8211; next steps"},"content":{"rendered":"<p>So, to review, we&#8217;ve taken a (very simple) ETL process and written the basic logic, constructed a test case harness around it, built a test suite harness around that test case, and added some logic and a new test case to the suite.\u00a0 In normal development, you&#8217;d continue on, adding more and more test cases and then adding to your core logic to make those test cases pass.<\/p>\n<p>This is the last in a series of blog posts on testing Pentaho Kettle ETL transformations.  Past posts include:<\/p>\n<ul>\n<li>The <a href=\"\/wordpress\/archives\/995\">benefits of automated testing for ETL jobs<\/a><\/li>\n<li><a href=\"\/wordpress\/archives\/1031\">what parts of ETL processes to test<\/a><\/li>\n<li><a href=\"\/wordpress\/archives\/1026\">current options and frameworks for testing Kettle<\/a><\/li>\n<li><a href=\"\/wordpress\/archives\/1010\">writing testable business logic<\/a><\/li>\n<li><a href=\"\/wordpress\/archives\/1033\">running one test using TestCaseRunner<\/a><\/li>\n<li><a href=\"\/wordpress\/archives\/1053\">running multiple tests using TestSuiteRunner<\/a><\/li>\n<li><a href=\"\/wordpress\/archives\/1061\">adding a test for new logic<\/a><\/li>\n<\/ul>\n<p>Here are some other production ready ETL testing framework enhancements.<\/p>\n<ul>\n<li>use database tables instead of text files for your output steps (both regular and golden), if the main process will be writing to a database.<\/li>\n<li>run the tests using kitchen instead of spoon, using ant or whatever build system is best for your operation<\/li>\n<li>integrate with a continuous integration system like hudson or jenkins to be aware when changes break the system<\/li>\n<li>mock up external resources like database tables and web services calls<\/li>\n<\/ul>\n<p>If you are interested in setting up a test of your ETL processes, here are some tips:<\/p>\n<ul>\n<li>use a file based repository, and version your kettle files.\u00a0 Being XML, job and transformation files don&#8217;t handle diffs well, but a file based repository is still far easier to version than in the database.  You may want to try an <a href=\"http:\/\/diffxml.sourceforge.net\/\">XML aware diff tool<\/a> to help with versioning difficultties.<\/li>\n<li>let your testing infrastructure grow with your code&#8211;don&#8217;t try to write your entire harness in a big upfront effort.<\/li>\n<\/ul>\n<p>By the way, testing isn&#8217;t cost free.\u00a0 I went over <a href=\"\/wordpress\/archives\/995\">some of the benefits in this post<\/a>, but it&#8217;s worth examining the costs.\u00a0 They include:<\/p>\n<ul>\n<li>additional time to build the harness<\/li>\n<li>hassle when you add fields to the output, because you have to go back and add them to all the test data as well<\/li>\n<li>additional thought required to decide what to test<\/li>\n<li>running the tests takes time (I have about 35 tests in one of my kettle projects and it can take about 10 minutes to run them all)<\/li>\n<\/ul>\n<p>However, I still think, for any ETL project of decent size (more than one transformation) or that will be around for a while (any time long enough to evolve), an automated testing approach makes sense.\u00a0 <\/p>\n<p>Unless you can guarantee that business requirements won&#8217;t change (and I have news for you, you can&#8217;t!), testing can give you the ability to explore data changes and the confidence to make logic changes.<\/p>\n<p>Happy testing!<\/p>\n<p>Signup for my <a href=\"http:\/\/eepurl.com\/x2Biz\">infrequent emails about pentaho testing<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>So, to review, we&#8217;ve taken a (very simple) ETL process and written the basic logic, constructed a test case harness around it, built a test suite harness around that test [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[63,58],"tags":[],"class_list":["post-1074","post","type-post","status-publish","format-standard","hentry","category-pentaho-data-integration","category-testing"],"_links":{"self":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/posts\/1074","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/comments?post=1074"}],"version-history":[{"count":9,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/posts\/1074\/revisions"}],"predecessor-version":[{"id":1744,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/posts\/1074\/revisions\/1744"}],"wp:attachment":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/media?parent=1074"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/categories?post=1074"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/tags?post=1074"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}