Multi threaded push to google drive with Java Executors

I’ve never really written threaded code in java. Sure, I took a java certification class long ago, and reviewed threading and probably wrote some toy code.

But it had always been one of the areas I was aware of but avoided. And it was easy to avoid–after all, I was often writing for a servlet container, and in that case you are supposed to rely on the platform (Tomcat, etc) to provide multi threading magic–you just have to stay in your servlet box. Other times, I was writing glue code where performance wasn’t an issue.

Recently, I was writing integration code that read from google spreadsheets (using the api) and quickbooks online (using the api) and did some calculations on the gathered data and created PDFs (using jfreechart and jasper reports). These PDFs then get pushed to certain folders on google drive.

I had run through some testing in my development and stage environments, and was ready to do a run in production. I started off the run and, after a few minutes, noticed that performance was not very good. About one report every 80 seconds or so.

Now, I’d run this process before I added google drive integration, and it had completed in a normal amount of time. So I suspected the google drive calls were the issue. When pushing the PDFs to google drive in my dev environment, I’d run into some connectivity issues, so as recommended I’d added a exponential backoff to my requests.

This meant that a lot of clock time was spent just waiting for the google services to recover.

So, an easily parallelizable task that spent a lot of time waiting? Seemed like a great place for multiple threads. I googled multithreading in java and found the Executor framework, which made running such tasks in parallel trivial. It even had futures and everything, and had been a standard part of the library since java 1.5.

I set up a thread pool of 15 threads, spent an hour or so rewriting and testing my file push code to fit the framework, and performance is much better. The push to google drive still takes about 20 seconds per report, but that’s a 4 fold increase in speed.

Next steps–use the spring abstraction layer to make the code more maintainable and clear.


What a pleasurable way to learn a language!

This site was recommended to me, and I have to say, it is a fun way to become more familiar with the syntax of a language. There’s the journey aspect:

things are not what they appear to be: nor are they otherwise
your path thus far [...X______________________________________________] 19/280

and the fact that when you see something you want to investigate further, you just write another unit test:

  def test_slicing_arrays
    array = [:peanut, :butter, :and, :jelly]

    assert_equal [:peanut], array[0,1]
    assert_equal [:peanut,:butter], array[0,2]
    assert_equal [:and,:jelly], array[2,2]
    assert_equal [:and,:jelly], array[2,20]
    assert_equal [], array[4,0]
    assert_equal [], array[3,0] # my addition
    assert_equal [], array[4,100]
    assert_equal nil, array[5,0]
  end

Now, running through these koans certainly isn’t going to make me a Ruby expert, but I will have passing familiarity with the language and be ready to use it on my next small project.

Apparently I’ve been living under a rock, because there appear to be koans projects for quite a few languages: java, haskell, erlang (cue whatsapp reference), and even bash. I was, however, unable to find a koans package for assembler.


What’s wrong with this announcement?

Do you see anything wrong with this announcement, which I received as an email attachment, of what looks to be a very interesting discussion of hunger, food and faith?

faithandfoodsummit flyer

There is no URL.

There is no URL!

Folks, if you are having any kind of gathering or event that you’d like to be shared online, please please provide a URL.

Benefits of a URL:

  • Easy to share
  • Can be updated if details change
  • Lives forever, so can be referenced in the future
  • Can hold vastly more information than a flyer

There are many easy ways to create a URL for your event.

  • Facebook
  • EventBrite
  • Google Sites
  • Weebly
  • wordpress.com
  • Google Forms

And I’m sure I’m missing dozens of other options.

Repeat after me: I am having an event, and my event deserves a URL.


Build your capital

I was working on a post about how important it is to have a side project, but then read this post by patio11: “Don’t End the Week with Nothing”, which could be more accurately titled “Don’t End the Week with Nothing except your Paycheck”. Not that there is anything wrong with just having a paycheck, but Patrick’s point is that when you work on something you own, rather than something you are paid for, you can (in the right circumstances, with hard work and luck) get accumulating returns.

He did such a good job explaining how to move your career forward as a software developer (a superset of the topic I was covering with my “have a side project” post), that I wanted to call your attention to it. The whole article is worth reading, but here’s my favorite part:

Telling people you can do great work is easy: any idiot can do it, and many idiots do. Having people tell people you do great work is an improvement. It suffers because measuring individual productivity on a team effort is famously difficult, and people often have no particular reason to trust the representations of the people doing the endorsements.

This is one of the reasons I blog, it’s why I have spoken at several user’s groups, it is why I wrote a book, and it is why I have a side project.


Accessing more build information from your Cordova CLI hooks

Hooks now give you, as of cordova CLI 3.3.1-0.4.2, much more information about the environment you were executing in.

Before, if I had platform specific code, I needed to check at runtime:

App.config.flurryid = "";
    if (window.device && window.device.platform) {
        if (window.device.platform == "Android") {
            App.config.flurryid = /*REP*/ 'notreallyanandroidflurryid' /*REP*/ ;
        }
        if (window.device.platform == "iOS") {
            App.config.flurryid = /*REP*/ 'notreallyaniosflurryidxxx' /*REP*/ ;
        }
    }

Now, you can inject platform specific code at build time, via hooks. The runtime code is simplified to

App.config.flurryid = /*REP*/ 'myflurryid' /*REP*/;

and you just replace 'myflurryid' with the appropriate value from your configuration file (more on platform specific configuration files).

From the readme in the hooks directory:

All scripts are run from the project’s root directory and have the root directory passes as the first argument. All other options are passed to the script using environment variables:

* CORDOVA_VERSION – The version of the Cordova-CLI.
* CORDOVA_PLATFORMS – Comma separated list of platforms that the command applies to (e.g.: android, ios).
* CORDOVA_PLUGINS – Comma separated list of plugin IDs that the command applies to (e.g.: org.apache.cordova.file, org.apache.cordova.file-transfer)
* CORDOVA_HOOK – Path to the hook that is being executed.
* CORDOVA_CMDLINE – The exact command-line arguments passed to cordova (e.g.: cordova run ios –emulate)

Here’s a sample plugin which you can put in your hooks/before_prepare directory. If you do this and then run cordova -d prepare you can see the values for all the variables.

#!/usr/bin/env node

console.log("process.env.CORDOVA_VERSION: "+process.env.CORDOVA_VERSION);
console.log("process.env.CORDOVA_PLATFORMS: "+process.env.CORDOVA_PLATFORMS);
console.log("process.env.CORDOVA_PLUGINS: "+process.env.CORDOVA_PLUGINS);
console.log("process.env.CORDOVA_HOOK: "+process.env.CORDOVA_HOOK);
console.log("process.env.CORDOVA_CMDLINE: "+process.env.CORDOVA_CMDLINE);

Here’s the output for a test project.

Executing hook 

""/home/mooreds/git/testproject3/hooks/before_prepare/test.js" 

"/home/mooreds/git/testproject3""
process.env.CORDOVA_VERSION: 3.3.1-0.4.2
process.env.CORDOVA_PLATFORMS: android
process.env.CORDOVA_PLUGINS: 
process.env.CORDOVA_HOOK: /home/mooreds/git/testproject3/hooks/before_prepare/test.js
process.env.CORDOVA_CMDLINE: node /usr/local/bin/cordova -d prepare

Note that if you are using this for platform specific code, you will want to run the commands separately: cordova build android and cordova build ios. If you just run cordova build and you have both ios and android platforms installed, the process.env.CORDOVA_PLATFORMS variable will be android,ios, and your code won’t be able to differentiate between them.

If you want to know about more Cordova CLI changes, check out “3 Cordova CLI Changes You Should Know About” which covers version 3.3.1-0.3.1.


How to use platform specific configuration in your Cordova app

This post comes out of a question I answered over at the guest post I did on Devgirl’s blog, ‘Three Hooks Your Cordova Project Needs’.

A commenter asked:

How do you retain the project level settings for cordova Android projects? Platforms folder removes project level setting when you run ‘cordova platform rm android’

I answered over there, but thought I’d expand a bit and write a post here.

When you are doing Cordova development, there are two main tooling paradigms. You can use native tooling (Eclipse, XCode) to manage your source, edit your javascript and CSS, etc–this is called ‘Native Platform Dev’. Or you can use tooling more typically used in web development (a text editor like Sublime or vi) plus Cordova CLI–this is called ‘Web Project Dev’. Here’s a bit more on the names of these paradigms.

In the first case, you are probably not removing the files under platform all that often–you are more likely to work out of that directory. In the second case, everything under platform is derived from your www directory, plus your plugins, so you can remove the platform directory easily.

I can’t really speak to Native Platform Dev, because it isn’t a Cordova workflow I’ve used. My book is entirely about Web Project Dev and how to do it most efficiently. If that is your paradigm, I imagine you won’t have much trouble with platform specific settings, because the native tooling is pretty good about capturing that in version control, so you can rely on it.

If, on the other hand, you are using Web Project Dev, then if you want to modify platform specific settings, you have three options:
You need to either:

  1. Only modify your project in ways that can be expressed in config.xml. Review the config.xml reference and the platform guides to figure out if your needed customization can be captured in this way.
  2. Write an after_platform_add hook which copies your changes over from elsewhere in your source tree (if you have a modified .java file for example).
  3. Write a plugin which modifies an XML files (AndroidManifest.xml) to insert your needed project level config, (an IntentFilter, for example), and add that plugin to your project in an an after_platform_add. Note you can only add XML nodes to config files with plugins, you can’t modify attributes or remove nodes.

Which of these is correct for you depends on exactly which platform specific feature you are trying to modify.


Want your next software purchase to ‘talk’ to your existing systems?

I wrote a post, over at the Geek Estate Blog (one of the best real estate tech blogs, imho) about APIs and why if you are a realtor or a broker, you should look for software with an API

Well, first, let’s talk about what an API is. API stands for “Applications Programming Interface”–not very illuminating. What that means is that the software providing the API has a way to allow other software (applications) to interact (interface) with it, with no human involvement. That is, the two software systems can ‘talk’ to each other.

Full post here.


How to access your QuickBooks Online data via API from the command line

As is so often with this blog, I’m documenting something that took me a long time to figure out.

The problem: the company I work for has data in QuickBooks Online (QBO) that we’d like to distribute elsewhere in our organization.

The answer: QBO has an API. Yahoo! They have several different SDKs (for .NET, Java and PHP) to make access even easier. (The APIs and surrounding docs are called the Intuit Partner Platform, or IPP.)

However…

The issue is that requests to the QBO API must be authenticated with OAuth (I believe it is OAuth 1.0a). And the entire Intuit Partner Platform documentation is focused on needs of developers who are building SaaS applications that consume or augment QBO data and are accessed via the QBO webapp. Which means there’s a heavy focus on web applications and OAuth flows via web applications.

But all I wanted was a command line client that could use the API’s query interface.

I was stymied by OAuth. In particular, I couldn’t find a way to get the accessToken and accessTokenSecret. I tried a number of different tacks (I beat my head against this for the better part of day).

But I just couldn’t find a way to generate the needed tokens, with either the PHP or Java SDK clients (most of the links in this post are for the Java client, because that’s more mature–the PHP client doesn’t support JSON output or have reference documentation).

Desperate, I turned to the IPP forums, which were full of advanced questions. I did stumble on this gem: “Simple way to integrate with our Quickbooks Account”, which took me to the OAuth Playground for IPP Developers. And, voila, if you follow the steps in the playground (I used IE because FireFox failed), you will end up with a valid accessToken and accessTokenSecret.

So, with that sad story told, here’s exactly how you can take access your own QBO data via the command line (I only cover a trial account here, but believe the process is much the same with a paid account):

  1. Start your IE browser (I’m using IE 10 on windows 8)
  2. Go to https://developer.intuit.com/us
  3. Sign up (click the ‘join’ link), click ‘remember me’
  4. Go to your email and find the verify link that was sent to you. Paste it into your IE browser.
  5. Sign in again
  6. Click on ‘My Apps’
  7. Click on ‘Create New App’, then ‘QuickBooks API’
  8. Fill out the name of the app, and the other required items. You can change these all later (I think). I know you can change the URLs later.
  9. Select the level of data access you need. Since this is a test app, you can select ‘All Accounting’
  10. Click ‘Save’
  11. Open up another tab in IE and go to the QuickBooks Online site (We are just adding some dummy data here, so if you have an account, you can skip this.)
  12. Click on ‘Free Trial’
  13. Click on ‘QuickBooks Online Plus’
  14. Click on ‘Already have an Intuit user ID’
  15. Fill out the username and password you used on when you signed up for your developer account.
  16. Ignore the upsell, if any
  17. Click the customers tab
  18. Click on the ‘new customer’ button
  19. Enter a first name and last name then press save
  20. Open a new tab and go to the API Console
  21. Choose the company that you want to access, and note the number next to that name. That is the company ID or the Realm ID.
  22. Open a new tab and go to the OAuth playground
  23. Go back to the developer.intuit.com tab
  24. Grab your app token (looks like b3197323bda36333b4b5fb17774440fe34d6)
  25. Go to the OAuth playground tab and put your app token in the proper field (called ‘App Token’). You’ll also want to have that later, so note it now.
  26. Click ‘Get Key and Secret using App Token’
  27. Note the consumer key and consumer secret, you’ll need them later.
  28. Click ‘Get Request Token Using Key and Secret’
  29. Click ‘Authorize Request Token’
  30. You should see a message like ‘testapp3 would like to access your Intuit company data’
  31. Click ‘Authorize’
  32. You should see a message like ‘You are securely connected to testapp3′
  33. Click ‘Return to TestApp3′
  34. Scroll down to the bottom, and you should see entries in the ‘Access Token’ and ‘Access Token Secret’ fields. Copy those, as you’ll need them later as well.
  35. Go to the SDKs page of developer.intuit.com
  36. Pick your language of choice, and follow the installation instructions.
  37. Follow the instructions in the ‘Data Service APIs’ section about setting up your environment. For Java, you’ll need to pull a few jar files into your classpath. Here’s my list of jar files in my Eclipse build path: ipp-java-qbapihelper-1.2.0-jar-with-dependencies.jar, ipp-v3-java-devkit-2.0.3-jar-with-dependencies.jar
  38. Write and run a class (like the one below) that runs a query, plugging in the six variables the values you captured above.
import static com.intuit.ipp.query.GenerateQuery.$;
import static com.intuit.ipp.query.GenerateQuery.select;

import com.intuit.ipp.core.Context;
import com.intuit.ipp.core.ServiceType;
import com.intuit.ipp.data.Customer;
import com.intuit.ipp.exception.FMSException;
import com.intuit.ipp.query.GenerateQuery;
import com.intuit.ipp.security.OAuthAuthorizer;
import com.intuit.ipp.services.DataService;
import com.intuit.ipp.services.QueryResult;

public class TestQBO {

	public static void main(String[] args) throws FMSException {
		String consumerKey = "...";
		String consumerSecret = "...";
		String accessToken = "...";
		String accessTokenSecret = "...";
		String appToken = "...";
		String companyId = "...";
		
		OAuthAuthorizer oauth = new OAuthAuthorizer(consumerKey, consumerSecret, accessToken, accessTokenSecret);           
		
		Context context = new Context(oauth, appToken, ServiceType.QBO, companyId);
		
		DataService service = new DataService(context);
		
		Customer customer = GenerateQuery.createQueryEntity(Customer.class);
		
		String query = select($(customer.getId()), $(customer.getGivenName())).generate();
		
		QueryResult queryResult = service.executeQuery(query);
	
		System.out.println("from query: "+((Customer)queryResult.getEntities().get(0)).getGivenName());      
	}
}

This code gets the first name and id of the first customer in your database, and prints it to stdout. Obviously just a starting point.

I am also not sure how long the accessToken and accessTokenSecret are good for, but this will actually give you command line access to your QBO data.

(Of course, I could have just used zapier, but that has its own limitations (limited ability to query data in an adhoc manner being the primary one).


Running multiple versions of Cordova 3.x

cordova, the command line interface to the Cordova development framework, is a node package. npm, the node package manager, has the concept of a ‘global’ install and ‘local’ installs. Typical installs of the cordova command line interface instruct you to perform a global installation. That is what the -g in npm install -g cordova (from the docs) means. Installing globally means that one version of cordova is available to every user on that computer, and will be installed in a system directory (for example, /usr/local/lib/node_modules, on linux). If you upgrade that one version, everyone and every project on the system will be upgraded (or forced to upgrade, depending on your perspective).

If you want to run different versions of cordova for different projects, you have to install locally. That means that all the cordova code will be installed in a local directory, rather than a system wide directory.

Why might you want do this? Any number of reasons, but primarily because the relationship between your application and a particular version of Cordova is important.

  • If your application depends on bugs or feature behavior that has changed between versions, you could need to freeze your application at a given version of Cordova, at least until you were able to update your application and test it against a newer version.
  • You could have multiple applications that were developed at different times, and the older ones could crash if upgraded (or perhaps you don’t have the time to test against the latest cordova version).
  • Applications under active development may be at the latest and greatest version of cordova, while the others may be sitting at what was the latest and greatest when they were last modified.
  • A local install will let you test your application against a new version of cordova without committing a global install, which might affect other team members, other projects, or necessitate a downgrade if the new version has issues.
  • You might be working on a platform that is not amenable to being place in a virtual machine (like iOS) and yet still want to run different versions of Cordova on the same machine.

Below, I outline steps to follow if you want to have one project using Cordova 3.3 and another using Cordova 3.1. npm view cordova lets you view the available versions–typically you want the latest minor revision (3.3.1-0.1.2 is better than 3.3.0-0.1.0). These are steps for Android, but it would be similar for any target platforms. And these steps assume you have successfully installed the target platform tools for Android previously.

mkdir cordova3.3 # or cordova3.1
cd cordova3.3 # or cordova3.1
npm install cordova@3.3.1-0.1.2 # or 3.1.0-0.2.0
mkdir project
cd project 
../node_modules/cordova/bin/cordova create test33 io.cordova.HelloWorld33 # or test31 io.cordova.HelloWorld31
cd test33 #(or test31)
../../node_modules/cordova/bin/cordova platform add android
../../node_modules/cordova/bin/cordova plugin add org.apache.cordova.device

Don’t forget to add the device feature to your config.xml

<feature name="Device">
    <param name="android-package" value="org.apache.cordova.device.Device" />
</feature>

You need to update the www/js/index.js file to display your Cordova version. This is only to prove you have different versions of Cordova running on the same machine

onDeviceReady: function() {
        alert(device.cordova);
        //app.receivedEvent('deviceready');
    },

Finally build the application and run it on your emulator:

emulator -avd avdname # start a previously created avd via the android command
../../node_modules/cordova/bin/cordova build android
../../node_modules/cordova/bin/cordova emulate android

Note that there should be no issues with running applications built with different versions of cordova on the same device–the applications are all sandboxed.

If you are often in this situation, you can create aliases that point to the correct version of the cordova command, since the results are indeterminate if you mix and match calls to different versions. For example:

alias cordova31='/path/to/cordova3.1/node_modules/cordova/bin/cordova'
alias cordova33='/path/to/cordova3.3/node_modules/cordova/bin/cordova'

Incidentally, I tried these steps out with the latest version of the android tools (version 19 of the Android build tools) with all four of the latest minor revisions of cordova: 3.0.10, 3.1.0-0.2.0, 3.2.0-0.4.0, and 3.3.1-0.1.2, and could only get the 3.1.x and 3.3.x versions to work. So another reason to do a local install is to perform a quick check to see if your platform installation is compatible with a given version of Cordova.

Do you have different versions of Cordova installed on the same machine?


Finding company phone numbers

I ran into a situation recently at work where I was trying to find the phone number of a company. Checked the company site, including the footer and contact us page, and no phone number was available.

Now, I can see why, as a small startup (which is what this company was), you would not want a phone number available. But we were doing due diligence and I thought a phone call or two would be appropriate.

So, here’s a list of places to go to find a company’s phone number, if it isn’t available on their site:

  • Dun and Bradstreet. Especially useful if the company has an iphone app, because Apple requires DNB registration for a corporate developer account.
  • Facebook often has different information than a corporate website, if they have a facebook page at all.
  • The whois database sometimes has address and phone numbers associated with a domain.
  • The Secretary of State office, for whatever state they are in. This is where business documents are file, and is worth checking.
  • Twitter. Depending on your situation, you can also just tweet them directly: “@foo, I’m looking to call you, do you have a phone number”.
  • LinkedIn, to look for people who know people who work there. This may be more or less useful based on the spheres you run in and where you live.
  • The Wayback Machine, which lets you see how websites appeared at various points in time. This is useful if the company at one point had a phone number on their website, but now does not.
  • Use the above methods on any other names you have turned up during your search, as well.

It is simply amazing what you can find on the internet with some digging. So, if you are looking to find the phone number of that company, because you need to talk to them, don’t give up.



© Moore Consulting, 2003-2010 +