Hiring on oDesk: let's share experiences?

Discussions
#1

I’d like to ask for suggestions and to share my experience about hiring on oDesk.

I lack time but have excess funds in my consulting business. I also have a validated idea. So I decided to try and hire a contractor on oDesk to do a prototype.

I have worked with oDesk before. I’ve hired a guy from Philippines in past, and he was reasonably good at some $20/hr, except not very pro-active in communications. I found him by doing a search on oDesk and offering him the job, and he accepted. I recently heard on some podcast (SFROU?) a better process should be used:

  1. Post a job
  2. Wait until 10-15 contractors submit their applications
  3. Select 3
  4. Give those 3 an identical task to perform (paid task, this is only fair)
  5. Select 1 who provided best code quality/delivery time combination

Well, sounds reasonable, and so I decided to try this process.

Posting the job

That was simple. I described the business context, gave outline of what I want to achieve and explained the process I’m going to follow to hire.

I also requested submitters to answer a few questions regarding the project, i.e. “What part of the project you expect to take most time?” and a few others.

For a large Weblogic farm, I’m looking for a tool that walks over each
domain in the list and records the current configuration values into a
timestamped snapshot (XML).

Later the operator may select two of the snapshots and obtain the
difference between the two for troubleshooting or for propagating the
changes into other domains in DEV-DIT-SIT-UAT chain.

There is no satisfactory API to collect the configuration from a
Weblogic domain, and so the collecting should be done via a web
scrapping. Good news is that all the pages in Weblogic console are built
the same way. In fact, I now have a PoC plain Java utility that does
the collection of some of the configuration entries; that utility can
be used as a reference.

The end result should be a WAR-deployable UI application that allows to
add/update/delete the Weblogic domains to track, take snapshots on
demand and show the snapshots’ diff.

I prepared the screens mockups with Bootstrap (attached).

The main skill for the developer is Java Servlets + UI development.
JSoup is nice to have, Weblogic is nice to have, basic XML is good to
have.

The work is planned like this:

  1. I will collect a few of candidates and give each the same paid
    pilot to do: the minimal functionality that touches all the areas that
    needed for completion.
  1. I will evaluate the results and select one of the candidates for further development.
  1. The selected individual will be given tasks to continue the project
    to the next milestone. The rest would get their earned money (of course)
    and a honest feedback.

Contractors submit

Almost immediately contractors started to submit.

Few notable things.

First, the range of rates was very large. The minimum was around $9/hr, and the maximum was around $30/hr. oDesk doesn’t allow to specify the target rate, only a range “low, balanced, high”. My target was in 12-20 range, so immediately some of the contractors have to be dismissed (“Hide” button on the oDesk UI).

Second, some of the contractors clearly did not took time to read the job description. Their answers to the posted questions were not fitting at all. For instance, one guy was talking about a “site” while the job is about an intranet web application.

Some other worked around the posted questions by saying “Can be discussed over Skype” for each answer. Those folks I dismissed right away. If one cannot take a minute to answer a few simple questions to get a job, how can I expect a good communication with them during the job?

Third, some of the contractors came via “agency”. I’m cautions about those. One thing, I do not want to pay extra for “overheads”. Second, I noticed that some of the agency contractors have a much higher posted rate on profile than on the submission (e.g. 30/hr vs 16/hr). That tells me it could be a bite-and-switch type of operation.

It looks like the expected way of communication on oDesk is Skype. Almost all contractors immediately give you their skype id.

Nonetheless, in about a day I had 12 candidates within my rate range and matching the basic qualifications.

Select 3 Of Them

That’s where it got complicated. How exactly I go about selecting 3 “best fitting”?

Conventional wisdom is to do a face-to-face (over Skype) interviews. I had two reasons not to do it though:

  • I suck at doing interviews; did it in past and I know this is my weakness; I just cannot “read” people
  • The spelling language of some of the contractors was quite bad (worse than mine) and I only expected the spoken would be worse. But as long as they can deliver the code I did not want to discount them.
  • I simply did not have 12 hours to interview 12 people. Those hours I needed to bill my client.

And so I went with a more formal way. I made a spreadsheet with all the contractors where I listed their rates, ratings on oDesk, number of hours they did on oDesk, my subjective feeling if they “got” the task based on their answers, their familiarity with the technologies I needed, and so on.

I gave each of the factors a weight and calculated the overall rating. That gave me 3 best candidates. I then reviewed those candidates again to double-check my numbers against my gut feeling and they looked fine – within the right rate (not too cheap, not too expensive: all 3 was about $12/hr), with the right qualifications and with the right work history.

I hired all three for the test task, giving them the following direction:

The results will be evaluated both by code quality and the delivery time.

Code quality, in addition to obvious things like being bug-free, includes readability (i.e. self-documenting for future maintenance developers) and use of standard APIs vs DIY implementations.

“Hire” button in oDesk doesn’t close the job, so I could hire them one after another without creating a special test job for them. I set the weekly limit to 40 hours thinking this should be the maximum time a developer should spend on a basic CRUD UI, even with some extra functionality I throwed in.

Test Task

I figured in the test task I want to see them touching every area that will be required in the real application. So I requested to do:

  • Basic CRUD UI
  • Some basic XML work
  • Some use of JSoup library
  • Setting up the build structure

Early disappointments

Well, things did not go well right from the start.

One guy asked one clarification question, and then just went silent. No further questions, no progress reported. It’s only been 2 days, so I’m giving him a benefit of having a life.

Second guy was clearly confused and sent me a barrage of the questions which made me confused in turn – what exactly he doesn’t understand? After some 5-6 emails back and forth, we finally cleared his idea. He even has logged some progress. In oDesk the client can see screenshots of what the contractor is doing - the guy has spent 3 hours looking at some code examples. That’s not where I would start doing my task, but hey, everyone has their own style. Will be watching him.

Third contractor, who’s actually a young girl, asked much more pointed questions, and said she’d start working right away. Her estimate was some 40-60 hours. A bit high, I believe, but could be just a safe number for an unfamiliar technology (Weblogic). She seems skilled, I have some hopes for her.

Late entrants

I did not close the job when I hired those 3 guys, and some more applications were submitted. Most I hid right away (too cheap or too expensive) but one guy caught my attention. He had exactly the right qualifications – Weblogic experience, scrapping libraries – and his answers to my questions were much more reasonable than the rest. His rate was about $16, i.e. some 25% higher than other 3 guys.

I sent him a message asking what would be his estimate for the test task. He said 6-8 hours. Wut? Seriously? I have double-checked with him and decided to add him to the test as well. If he’s really that fast, it more than beats his rate increase.

To Be Continued

That was my notes so far. Should I continue to log the progress? Is it of interest to anyone?

2 Likes
#2

Thanks for sharing everything so far! I would be interressted in the process and the results!

#3

@rfctr This is excellent! Yes, I would be interested in hearing more about your experiment.

#4

I’ve seen posts that request a word at the beginning of an applicant’s reply. I tried it and it helped filter out many non-readers of my job description.

1 Like
#5

Interesting tactic!

A subtle request to include a keyword may help against “scanners” (vs “readers”).

My only concern is it may exclude good contractors who’ve already figured they are qualified somewhere mid-text and started scanning.

#6

Hired somebody for $22 an hour on odesk who’s been working out great. First time I’ve outsourced development.

I asked two question that only a developer who “knows his craft” could answer. I asked “name your favorite nuget package you’ve used recently” (nuget is like npm or ruby gems for .net land) and whats your favorite C# feature.

Out of 15 applicants only two answered the question “correctly” and his resume/portfolio was much better so he got the job. I think he’s a little slower than I’d like but he’s using some tech that’s rarer in the asp.net microsoft and our MVP isn’t perfect (as it should be) so I can’t really hold it against him.

I’ve ramped him on on tasks ranging from trivial fixes to full feature development and he’s asking the right questions and always committing code and making progress.

From what I understand this is rare and I really appreciate his finding a gem. He’s just cranking along on features that we need and are requested by customers but aren’t a priority for me. I started him on a small budget but am considering giving him as much work as he can handle.

Its a really huge timesaver. Although I have to spend time writing up tasks, reviewing code, merging code, etc the progress we are making is substantial enough that its money well spent.

1 Like
#7

I definitely would like to hear updates.

I don’t know if it’s a cultural thing, reticence to communicate in English or just lack of experience, but some of the people I’ve hired via oDesk from the Philippines aren’t great at status updates, even though overall I’m happy with them.

I think the strategy of including something, somewhere in the application that requires a bit of thought and something specific in the response is a sensible one.

#8

Ive hired three developers so far.

The 1st developer after giving him a couple tasks started using a mouse & cursor movement application. Really ticked me off. I got the money back, asked for him to delete the source and he did. He also lied to me about his father being in the hospital and took pictures to try and prove it. But he took them of an abandoned hospital instead. Really made me upset. Wrote a bad review on his job board to describing the situation. Paid him $5.00 an hour.

2nd developer did good work at a good rate and kept up communication. Paid him $2.50 an hour.

3rd developer is still with me. Currently paying him $2.85 an hour.

Before you might be upset with the low hourly, my start up is totally boot strap and just making enough to sustain costs. So I am forced to pay less than I would like.

Except for the 1st developer, I have had a good experience, but I should state that you really need to watch their automatic time screen shots after I was lied to by the 1st guy.

All developers are working in asp.net and C#…

#9

How you have noticed it? I mean, screenshots do not make it easy, do they?

Upset - no, but the rate is insanely low, and I’m surprised your were able to find not one, but 2 reasonably good developers at that rate.

#10

Update for Week 2

My contractors continue surprising me.

The guy who was asking a lot of questions, despite my skeptics about his abilities, suddenly rushed forward, and installed a brand-new Weblogic domain from 0 to fully configured in less than 4 hours. Keeping in mind he never have seen Weblogic before that was quite a feat. Not only that, but the guy started to do the actual coding and at this point is some 50% through. I already see the working pages on his screenshots.

He’s spent 16 hours so far, but has proven himself to be a fast learner. He’s worried how other 3 doing, but I told him to not worry about that, that he’s actually “keeping up”. The truth is, he’s the one who’s the closest to the completion. Overall he’s a nice surprise this week.

The girl started to do the coding too, and in just 2 hours she has the application at almost the same stage where the first guy had it. Granted, she has not started with Weblogic yet, so it may cost her some leading.

The guy who promised 6-8 hours to complete the task hasn’t started working yet. Hasn’t updated me when he plans to start either (all 3 others did). Could be a possible communication issue. Still, I gave them time till 10th, so I’ll wait.

The fourth guy sent me a message two days ago that he starts “tomorrow”. Today is “day after tomorrow”, still no activity, and no update on why he did not start. Not a good sign too.

More Applicants

The job is not closed yet, and I still get some submissions. Some are quite funny. One guy managed to raise all red flags I could think of in just a couple of paragraphs:

  • He set his rate to $5/hr
  • He refused to answer my job questions, saying “will talk on skype”
  • He stated he has “attention to details” and made 3 spelling errors right after that
  • He called himself “Senior Developer” with 1 year of C++ and 1 year of Java

I was considering to send him a message to point those out and advise to correct them, but… that wouldn’t be fair for other hiring folks, would it?

To Be Continued

Looks like next weekend I shall have at least two of the test tasks completed. Will keep you posted.

#11

Reviewing the oDesk screenshots, I noticed that one of my contractors uses Skype to IM with his other employers on my time. He’s not taking a call, just chats over Skype IM. Screenshot caught it 3 times already which means it is a common activity.

I have mixed feelings about it.

On one side, this is a time paid by me, and it shouldn’t be spent on anything else, no matter how short. Talking to anyone kills the context, the flow my developer was in, and it costs him time to get back on track.

On another side, chat is fast. The contractor may just reply in a single sentence and continue doing my job.

I decided to put up with it for the duration of the test task, but before starting the main job I’ll request that the oDesk logging be disabled before talking to Skype - or the time with such screenshots (some 10 minutes covered by one screenshot) will not be approved from that time on.

#12

Screen shots are great and a much required feature by me if I continue with ODesk. I look at them periodically, but when I found the first guy cheating them, I looked much deeper.

There are developers that work for cheap. Ones that apply too. But with my startup making the money it currently makes, Im sadly not able to pay much more per hour.

#13

OTOH, he’s got to always be looking for the next thing, and to try and keep everyone happy - maybe he answers you quickly on others’ time. I’d try and let him know you noticed it in a tactful way. If he’s smart, he’ll reduce that kind of thing to a minimum.

#14

I’ve been trying a similar approach, not overly much success so far.

Posted the job up using the ‘medium’ range on oDesk ($18-$28), and got about 15 candidates, narrowed it down to three. Gave all three a very small, simple task - simple if you know how to work with the tech (angularjs). Only one was able to complete it. His rate was $15/hour. He followed all the directions, quality was good, everything seemed great. So I hired him to work on a more complex, domain-specific problem. Absolute confusion. Communication was a real problem. After about a week, he started to make some progress. I could have kept him on at that point, but I just didn’t get a good vibe. He’d argue back when asked to do things in a particular way - for example, commit his code regularly.

Decided to try again, but this time tried with the ‘Experienced’ range ($28-50 I think). A lot more applications, but one really stood out. I liked the tone of his application, and he had relevant domain experience. The rate was high ($45/hr) but I thought it was worth a try. So I sent him the test task. Refused to do it, because it was too simple for him. Well, I can see his point, I’ll let that one through. On to the next task. Had some great chats, seems to really know his stuff, we get along fine. In the end, though, looks like he’s going to knock back the job because he feels he can’t be sufficiently productive because he doesn’t know one of the other libraries I’m using (breezejs, an ER mapper.) D’oh.

So, now back to the drawing board.

#15

Update After 2 Weeks

Well, I’ve got the test task results from all 4 of my contractors, and just completed the code review and testing.

Main Takeaway #1: Cheap Folks Are Expensive

The results prove something I knew (or thought I knew) - a good developer is 10x times better than an average one, while the rate is usually only moderately higher.

It certainly works this way for my own contracting practice, where I’m doing a particular standard task in about 1 hour, whereas an offshore vendor estimates it at 1 week; my rate though is only 5-7 times higher.

It worked this way for my test task too, where a 1.3x more expensive developer completed the work 6-8x faster, costing me 5x(!) less as a result.

Main Takeaway #2: The Shortlisting Is Not Working

I believe the process I have tried (shortlist developers, give them the same task) is not really helping.

Its weak point is the shortlisting, where one shall still be using one’s gut feeling to select the most fitting developers. If the gut feeling is involved, it could be just as well used for selecting one best developer and save the time and money.

Main Takeaway #3: Requirements Are Universally Misunderstood

Requirements are not understood by contractors even if spelled out in much details. It takes an experience in the particular business area to read the requirements the same way as you intended.

For the next hiring I’m going to do an experiment - I will request the contractors to write down the very same requirements in their own words to see how they understand them. Hopefully it may detect some of the mismatches in our understanding and close those gaps before the coding starts.

Events Of The Week

Developer A

Time spent: ~50h. Cost: $561

Even before the week started, I’ve got the first submission from the developer A. He’s the one who was initially confused, but then quickly started to develop pages, according to the screenshots. He’s sent me a package which I was able to start without any trouble.

Testing though revealed he did not understood some of the requirements explicitly expressed, and some of the requirements implicitly present (e.g. the tool is for enterprise, which means the application cannot expect it can save files into root of C:).

Main deficiency of his code was the lack of asynchronous processing. The task an operator going to trigger can be long, up to 30 minutes, and the application cannot block the UI for that time, but it was exactly what it did. (This was in fact a common error for all but developer D).

I have provided him a feedback, asking to implement the async processing to demonstrate how he’d do it. There were a few smaller issues, e.g. lack of support for HTTPS, which I explicitly asked not to fix as they were not critical for the test task.

He enthusiastically started to do the background threading, asking me a few clarification questions from which it was clear he did not do it before, but he was doing fine.

At the same time he went on and fixed some of the “do not fix” issues. I don’t get it – I clearly said to do it not; I was frowned upon it and was considering to ask him if he understood that this work (and thus pay) was not authorized? But I passed on it, because according to screenshots the overwhelming amount of time was spent on threading.

At the end of the week he has sent me the updated version. It is OK; the main issue I see after the testing is that the error messages (“connection refused”, …) are lost and never reported to the operator, making the troubleshooting hard. Later I learned it is the same for developers A, B and C.

Developer B

Time spent: ~48h. Cost: $578

This is the girl that I had quite high hopes for. I’m not sure why, but she seems to revisit the same screens and classes again and again, and was pretty late with the submission. On the last Friday (May 9) I have sent a reminder that the deadline is May 10, and she sent me the code on the morning of May 10.

She did not asked me any questions during the last week. One would think it means that the requirements were clear for her, but the testing shows they were not.

She made a few mistakes in the areas that were covered in the requirements. In addition, she seems not fully grasp the idea that a web application may be accessed in parallel by many users, and so the sufficient synchronization and protection should be put in place.

All in all, she did deliver, but her quality just wasn’t good enough for me.

Developer C

Time spent: ~39h. Cost: $468

This guy was late, just like developer B; I had to send a reminder too, and he had delivered late evening on May 10. I suspect he simply did not have time to polish his application as a result.

His app just did not start. After code review I have found the issue (a library wasn’t included into the package), corrected it and was able to do the testing, but it wasn’t a good start, eh?

His code issues were almost identical to developer B.

Same as with developer B, this quality is simply not cutting it.

Developer D

Time spent: ~6:30h. Cost: $108

Yes, it is not a mistake. It is 7 (seven) hours and $108 dollars. This guy rate is about $16, so he’s some 30% more expensive than A, B and C, but geez does he deliver!

Up to May 8th I though he’s not going to complete it at all. He was silent and no progress was reported in oDesk. Then on May 9th morning he suddenly sent me the package and a request to report 6 hours that he was working offline.

Huh? Why offline? The whole point of oDesk is that I can check on what my contractors are doing, right?

Anyway, I took a look at the code and was blown away. I do Java and recently SOA from 1996. That’s almost 20 years already, and I believe I know my platform pretty well. I though would never deliver that much functionality in such short time! Granted, most of it came from the right choice of frameworks, but that’s exactly where the experience shines, right?

Anyway, I gave him the permission to charge those 6 hours, have ran a test, found a couple of rather minor issues, asked if he could fix it before Monday when I’m going to review it with a stakeholder guy (yeah, I have my future customers right next to me in the office! :smile:) and he did fix those in another 30 minutes of work.

He has even provided a REST API – something I did not even think to ask (too complex for a test task) but it was on my todo list because it would be very useful in our heavily scripted environment. That IMHO shows that he understands the target environment better than anyone else from the contractors.

I have two concerns about him though:

  1. Why offline?! What if this all is a bait-and-switch again? The feedback though is all 5 stars.
  2. He’s from an area neighboring a freshly minted civil war, and the war may come to his town too. That I cannot control; even if he delivers me a solid application framework, and then will have to stop working due to becoming involved or just because the Internet cease to exist in that area… well, I cannot ask for more.

Summary

Tomorrow I’ll talk to the stakeholder’s team member, we’ll go through the test tasks. I do not expect him to give me an opinion different from what I have already formed: I’m going to hire the developer D.

Looking back, I realize that if I would just wait for 3-4 days after the job posting and then selected one and only one, that would be the developer D anyway, because of his rating, his previous experience with Weblogic and with other tools.

I just spent some 15 hundreds $ to learn that one shall not blindly follow whatever process one heard on Internet, even if it sounds right. What works for one person not necessary works for another.

Will keep you posted if any major events occur, but otherwise will just work with D. to push for the MVP.

Thank you for all the comments and shared experiences!

2 Likes
#16

$1.5k is a chunk of change, but less than getting the wrong dev. Unfortunately, not all of us have gut instincts worth listening to!

Do you think you could have got a useful idea of capability with a smaller, cheaper task? http://www.donnfelker.com/how-to-hire-outsourced-developers/ is talking about a 1hr task, or $50-200 to test 10 developers. Do you feel that would have given you enough insight to narrow down the field?

#17

In the hindsight, I could have simplified the task by some 20-30%. That though wouldn’t make a 1 hour task; the developer D. would probably do it in 3-4 hours instead. The most complexity was in the spawning a background thread for processing and getting its results safely - and I had to test that part because a developer doing good UI is not necessary good at multi-threading programming.

In a link above the guy is talking about 1-2 hours task; I reviewed the Rails one, and it is a heavily UI one; yes, it will show if the person can produce the Rails UIs fast, but it wouldn’t help in my case, because UI is the simplest part of it… I may have extracted the most complex part – the background thread work – as a task. Then again, it would take much longer than 1 hour anyway, but I wouldn’t be able to see all the tricky parts where the background worker has to be interfaced with UI code.

In short, I think the task was about the right size for my goals.

#18

@rfctr - Thanks for sharing your experiences.

Regarding the misunderstanding of requirements - did you try recording a screencast to explain the information? Rob walling’s VA course recommends recording a short video for complex tasks. In theory you can explain a lot more of the nuances of the task in a way that would be very difficult to explain by text alone. You could even walk them through portions of the code or apps they will be integrating with.

#19

What Rob does is tasks like “If the buyer wants his money back, login here, put his order # here, run a search, click here and confirm”.

Development of a new application is something different; I do not have screens to navigate yet.

I have done what is the closest thing to that – I made mockups in Bootstrap. Those mockups are working, allow to navigate the test UI but do not save the state, of course. Even with this help one of the developers made a pretty mess of navigation.

#20

Update

Now that I have to hire one guy out of 4, this turns out not such a straightforward thing as well.

See, I essentially need to tell Mr. D. “I want to hire you” and to the rest “Sorry, it did not work out”.

But I probably shouldn’t do it all in the same time, because what if Mr. D.'s circumstances change and he cannot accept it? If I already told the rest “Sorry…” and I come back with “Actually, the best guy just dropped, so I alas have to hire you” it may not be good for morale. And no matter how I word it, the fact that Mr. A was my second choice would poison the relationship.

So… if Mr. D for whatever reason declines… I think I will just make a pause and post the job again with the Mr. D’s test task as a foundation.

I have sent a message to Mr. D, but it is a deep night at his timezone, so I’ll have to wait for another 4-6 hours for the answer.