There is a Hacker News meetup scheduled in Boulder one week from tonight (the 13th of July).
We are going to meetup at The Dark Horse in Boulder for happy hour. No cost, just geeks hanging out. Come on by.
If you want to let us know you are coming (or find out about future gatherings), sign up with the meetup group.
I’ve noticed a lot of comment spam lately, and had been dealing with it piecemeal. I have all comments moderated, so it wasn’t affecting the user experience, but I was getting email reminders to moderate lots of comments that were not English and/or advertising something.
WordPress lets you turn off comments for a post, but it is tedious through the GUI. So, I dove into the wordpress database, and ran this SQL query:
update wp_posts set comment_status = 'closed' where post_date < '2014-01-01';
This sets comments to closed for all posts from 2013 or earlier (over 1000 posts). The power of the command line is the ability to repeat yourself effortlessly. (The drawback of the command line is its obscurity.)
I should probably write a cron job to do that every year, as posts older than two years old aren’t commented on much. To be honest, the comments section of ‘Dan Moore!’ isn’t used much at all–I know some blogs (AVC) have vibrant comments areas, but I never tried to foster a community.
So, just finished up looking at authentication mechanisms for a REST service that runs on Camel for a client. I was using Camel 2.15.2 and the default Jetty component, which is Jetty 8. I was ready to use Oauth2 until I read this post which makes the very valid point that unless you have difficulty exchanging a shared secret with your users, SSL + basic auth may be enough. Since these APIs will be used by apps that are built for the same client, and there is no role based access, it seemed like SSL _ basic auth would be good enough.
This example is different from a lot of other Camel security posts, which are all about securing routes within camel. This example secures the external facing REST routes defined using the REST DSL. I’m pretty sure the spring security example could be modified to protect REST DSL routes as well.
So, easy enough. I looked at the Camel Jetty security configuration docs, and locking things down with basic auth seemed straightforward. But, they were out of date. Here’s an updated version. (I’ve applied for the ability to edit the Camel website and will update the docs when I can.)
First off, secure the rest configuration in your camel context:
<camelContext ...>
<restConfiguration component="jetty" ...>
<endpointProperty key="handlers" value="securityHandler"></endpointProperty>
</restConfiguration>
...
</camelContext>
Also, make sure you have the camel-jetty module imported your pom.xml (or build.gradle, etc).
Now, create the security context. The below is done in spring XML, but I’m sure it can be converted to Spring annotation configuration easily. This example uses a HashLoginService, which requires a properties file. I didn’t end up using it because this was POC code, but here’s how to configure the HashLoginService.
<!-- Jetty Security handling -->
<bean id="constraint" class="org.eclipse.jetty.util.security.Constraint"> <!-- this class's package changed -->
<property name="name" value="BASIC"/>
<property name="roles" value="XXXrolenameXXX"/>
<property name="authenticate" value="true"/>
</bean>
<bean id="constraintMapping" class="org.eclipse.jetty.security.ConstraintMapping">
<property name="constraint" ref="constraint"/>
<property name="pathSpec" value="/*"/>
</bean>
<bean id="securityHandler" class="org.eclipse.jetty.security.ConstraintSecurityHandler">
<property name="loginService">
<bean class="org.eclipse.jetty.security.HashLoginService" />
</property>
<property name="authenticator">
<bean class="org.eclipse.jetty.security.authentication.BasicAuthenticator"/>
</property>
<property name="constraintMappings">
<list>
<ref bean="constraintMapping"/>
</list>
</property>
</bean>
As mentioned above, I ended up writing a POC authentication class which had hardcoded values. This post was very helpful to me, as was looking through the jetty8 code on using grepcode.
public class HardcodedLoginService implements LoginService {
// matches what is in the constraint object in the spring config
private final String[] ACCESS_ROLE = new String[] { "rolename" };
...
@Override
public UserIdentity login(String username, Object creds) {
UserIdentity user = null;
// HERE IS THE HARDCODING
boolean validUser = "ralph".equals(username) && "s3cr3t".equals(creds);
if (validUser) {
Credential credential = (creds instanceof Credential)?(Credential)creds:Credential.getCredential(creds.toString());
Principal userPrincipal = new MappedLoginService.KnownUser(username,credential);
Subject subject = new Subject();
subject.getPrincipals().add(userPrincipal);
subject.getPrivateCredentials().add(creds);
subject.setReadOnly();
user=identityService.newUserIdentity(subject,userPrincipal, ACCESS_ROLE);
users.put(user.getUserPrincipal().getName(), true);
}
return (user != null) ? user : null;
}
...
}
How, when you retrieve your rest routes, you need to pass your username and password or the request will fail with a 401 error: curl -v --user ralph:s3cr3t http://localhost:8080/restroute
Note that the ACCESS_ROLE variable in the login class must match the roles property of the constraint object, or you’ll get this message:
Problem accessing /restroute. Reason: !role
You can find a working example on github (with the hardcoded login provider).
Thanks to my client Katasi for letting me publish this work.
Is it too early to write a takeaway post before a conference is over? I hope not!
I’m definitely not trying to write an exhaustive overview of Gluecon 2015–for that, check out the agenda. For a flavor of the conversations, check out the twitter stream:
Here are some of my longer term takeaways:
To conclude, I’m starting to see repeat folks at Gluecon and that’s exciting. It’s great to have such a thought provoking conference which looks at both the forest and the trees of large scale software development.
A few weeks ago, I engaged with a client who had a real issue. They sold a variety of goods via a website (if this was the 90s, they would have been called an ‘e-tailer’), and had been receiving intermittent double orders through their ecommerce system. Some customers were charged two times for one order. This led, as you can imagine, to very unhappy customers. This had been happening for a while and, unfortunately, due to some external obstacles, internal staff were not available to investigate the issue–they had their hands full with an existing higher priority project.
I was called in to see if I could solve this issue. I had absolutely no familiarity with the system. But in less than ten hours of time, I was able to find the issue and resolve it. How I approached the situation can be summed up in five rules:
Number one: define the problem. Ask questions, and capture the answers. What is the exact undesired behavior? When is the undesired behavior happening? What seems to trigger it? When did it start? Were there any changes that happened recently? Does the client have reproduction steps?
I gathered as much information as I could, but keep it high level. I asked for architecture and system diagrams. For the history of the application. For access to all systems that could possibly be relevant (this will save you time in the future). For locations of log files, source repositories, configuration files. For database credentials and credentials for third party systems like CC processors. It is important at this time to resist the temptation to dive in–at this point the job is to get a high level understanding so I can be efficient in the next steps.
You will get speculation about what the solution is when you are asking about the problem. Feel free to capture that, but don’t be influenced by it.
Number two–find the finish line. After getting a clear definition of the problem, I looked in the orders database and find out if the double orders were showing up there. They were, which was a clue as to which part of the system was malfunctioning, but more importantly let me see the effectiveness of any changes I was making. It also lets the customer know the objective end goal, which can be important if this is a t&m project, and it let me know the end state to which I was headed–important for morale. (BTW, don’t do fixed bids for this type of project–overruns will be unpleasant, and there will be overruns.)
I was able to write a SQL script to find double orders over a given time frame. I ended up writing a script which emailed the results of this query to myself and the client nightly, as an easy way to track progress. The results of this query were a quantifiable, objective measure of the problem.
Number three–start where you are familiar. I could have dove in and looked at the codebase, but due to my problem definition, I knew that there had been no changes to the checkout portion of the code base for years. I also was unfamiliar with the particular software that managed the ecommerce site and could have wasted a lot of time getting up to speed on the control flow. Instead, once I had the SQL query, I could find users that had been double charged, and look at their sessions in the web server logs. I’ve been looking at apache http logs for over a decade and was very familiar with this piece of the system.
Number four–follow your nose. I followed a few of the user sessions using grep and noticed some weirdness in the logs. There were an awful lot of messages that indicated the server had been restarted, and all the double orders I looked at had completed 5-6 seconds after the minute changed. (It’s hard to define weirdness explicitly, which is why it behooved me to start with a portion of the system that I was experienced with–it made the “weirdness” more obvious.) From here, I ended up looking at why or how the server was being restarted regularly. Ended up finding an errant cron job which was restarting the server often enough that the ecommerce system was getting confused and double booking orders–once before the restart and once after. This was easily fixed by commenting out the cron job.
Number five–know when to stop. This ecommerce system obviously had a logic flaw–after all, restarting the web server shouldn’t cause an order to be entered twice, whether you restart it every hour or once a year. I could have dug through the code to find that out. But instead, I commented out the cron job, let the system run for a week or so and waited for more double orders. There were none, indicating that the site was low traffic enough that whatever flaw was present didn’t get exercised often, if at all. I confirmed with the client that this situation met his expectations of completeness, and called it good.
Being thrown into a new system, especially when troubleshooting, is a difficult task. I am thankful the client was relatively responsive to my questions, and that pressure, while present, wasn’t intense. These five steps should help you, if you are put in any troubleshooting situation.
As a developer, my skills are applicable across a wide variety of domains. However, I re-engaged recently with a prospective client that I wrote about a while ago. They had used another booking solution over the past few months and still had the same pain. After thinking and doing some research, I can to a conclusion that hiring a developer was not the right answer for them.
This company had an interesting set of constraints:
In this situation, the best solution is to look for someone who can undertake the following steps:
So, what this client really needs is a technical project manager (TPM). While most freelancers have this skill to some degree, as it is hard to survive without it, making a full time living as a contract project manager is difficult. I know of only one person in 15 years who was making a living as a contract project manager (and she’s not doing it anymore). This particular project doesn’t seem like a full time effort, so the client should be able to get by with a moonlighter, at least until the results of step 4 are known.
Good technical project managers are hard to find. From a friend’s company’s job req (his company who is looking for a TPM, and the job desc captures the description of the skill set well), they have:
balance between hands-on technical knowledge, a ravenous appetite for order, and an understanding of humans and how they work.
Developers (or former developers) who want to project manage and have people skills aren’t quite the unicorn as someone who can both design and develop, but they are almost as rare. Unless the company can find a contract project manager who has technical chops, they’ll want to either source this internally or find an external contractor with another skill set (developer or designer) who wants to PM this project.
This is a great chance to for an employee to expand their skill set. Based on my experience managing vendors during my time at 8z (a 4 month website relaunch and a longer term, less time intensive data provisioning engagement), I would advise that this employee have technical skills. Challenging custom solution providers on technical grounds, or at least being able to follow along, ensures the company will get the best solution. This doesn’t work for “take it or leave it” SaaS apps, but this project appears custom enough that the TPM really needs to have both business and technical considerations in mind when managing the development shop. If they are looking for an outside contractor, engaging with a designer or developer who has PM experience would be an option.
As far as stretching budget, proposing a lower price to the development shop in exchange for shared ownership of the code might make sense. The partner could market this code to other clients and recoup some costs, while the client retains a perpetual license.
Sometimes, a generic developer isn’t the right answer for a software development problem. A technical project manager (or someone wearing that hat) can often stretch budget and leverage skill sets of focused development teams.
It’s only the beginning, lots more tech fun in the future:
This has caused me some grief over the past few weeks, so here’s some tips on calling bash scripts from java correctly.
Use ProcessBuilder. Please.
Make sure you set the redirectErrorStream to true, or make other provisions for handling stderr.
I’ve found that inheritIO is useful. Not calling this caused a failure in a bash script that called other java programs.
Make sure your shell scripts have are executable, or that you call them with ‘bash scriptname’. Tarballs preserve permissions, zipfiles do not.
Read the output of the shell script, even if you do nothing with it (but you probably want to log it, if nothing else). If you don’t, buffers fill up and the process will hang:
final InputStreamReader isr = new InputStreamReader(p.getInputStream());
final BufferedReader br = new BufferedReader(isr);
String line = null;
StringBuffer sb = new StringBuffer();
while ((line = br.readLine()) != null) {
sb.append(line).append("\n");
}
Check your exit values, and wait for your shell script to finish: (If looking for speed, or more complex process flow, write it in java.)
int exitValue = p.waitFor();
if (exitValue != 0) {
// error log
}
If you are a software engineer or developer right now, you are in demand. There’s a dearth of software developers, and so you are probably getting calls from recruiters. This is location dependent–my understanding is that the SF market is white hot, and the market for software developers in Yuma CO is softer.
When I first was approached by a recruiter, I was flattered. Here was a stranger trying to find me a job. Quickly I realized that recruiters were not working for me (they get paid to place a candidate, so they are working for the company) and that most recruiters didn’t have a technical background at all. This made me feel less charitable toward recruiters, and I have found that common amongst developers, especially now that they have so many job options.
Older and wiser, now I have a more balanced approach. Here’s my tips for dealing with tech recruiters:
Reviewing my list, I boil it down to:
Any tips you have on engaging with recruiters?
Think about decisions from all perspectives…
Follow me on Twitter
Follow me on Blue Sky