My perspective employer gave me a Codility test instead of an interview. There were 4 problems: fibonacci, stock market, monotonic and bits calculation. All problems had to be solved in 2 hours or less. In description Codility did not mention that they score not only correctness, but also program performance. They give option to verify your code and when I did it the test result was OK. I submitted the test and was surprised about poor results. For example, in fibonacci calculations 3 tests were marked as "time-out error". In bit count they did not mention that the value could be more than 32-bit, gave me successful test result verifying with 32-bit number and sent unsatisfactory report to my employer for tests with 1000000-bit nimber. As a result I lost this employment.

My opinion is that the current test is deceiving and illegal to use in hiring process. Test description must explicitly say that execution time is limited by 0.1s; that number of bits in a value could be indefinitely high, etc. Test must present verification results for the same test cases which will be given in the report, so user can improve his code before submission. Test time should not be limited by 2 hours. Instead time of completion could be entered in the report, so employer can use it in ranking candidates.

Current test is not valid for a hiring process.

Quote: Original post by mkuz
In description Codility did not mention that they score not only correctness, but also program performance.

Why would you assume that performance is unimportant? Right off the bat I know how you coded your Fibonacci routine (recursive, no memoization), and that indicates that you might not have a great handle on algorithmic complexity. Sounds like this situation was a success for the company, as well as a learning experience for you.

Quote: the current test is deceiving and illegal

heh :-)

You didn't get the job. Sorry to hear it. Practice up, and apply somewhere else.

I don't see how this would be illegal. You failed the tests they set you, and you didn't get the job. How is that unfair?

Honestly companies get thousands of applications a year, and many of the people applying aren't fit to work there. The bulk of the rest are about equal, with a handful being exceptional. The goal of HR departments is to cut out those who shouldn't have applied in the first place, and then find the exceptional ones. Failing any exceptional ones applying, it really becomes a game of luck for the bulk of them.

Quote: Original post by mkuz
For example, in fibonacci calculations 3 tests were marked as "time-out error".

Did you use recursion or correctly handle overlow?

Quote: In bit count they did not mention that the value could be more than 32-bit, gave me successful test result verifying with 32-bit number and sent unsatisfactory report to my employer for tests with 1000000-bit nimber. As a result I lost this employment.

Well, 32-bit integer is a bit simple already, since much of code today is 64-bit.

How did you read the input? Did you check for overflow there? Why not? What about reading strings in C code using scanf? How do you protect from buffer overflow? Can you write an overflow-safe input reader? Would use use malloc, calloc, realloc or alloca? What are the differences between them? If all you have are 32-bit registers, how would one multiple larger numbers? What about division? How does carry flag work? How would one detect overflow? What are half- and full-adders?

Those are all related questions. One simple way is to use languages which have large numbers built in. For coding competitions, unless there is a guaranteed upper bound, I simply use java's BigInteger, if only to cover potential overflow problem.

Quote: My opinion is that the current test is deceiving and illegal to use in hiring process.

"Illegal"? ...

Is this your first time doing the interview? At least you didn't need to take a day off, drive 2 hours, be grilled for 3 more, only to fail because you couldn't name all 67 parameters to ls.

Quote: Test time should not be limited by 2 hours.

In normal interview you would have to solve all 3 of those assignments in 15 minutes over a phone.

Quote: Current test is not valid for a hiring process.

Sure. And it's also not fair that bankers get $20 million in bonuses, that 200 million children will starve to death this year, that floods left 100 million homeless and that my neighbor insists on mowing lawn at 7 am on Sunday.


While it sucks not getting a job - speaking as someone from the other side, reading this post, I'd have much bigger concern about whiny attitude. No - life is not fair.

Look at it this way. Today you learned many valuable things:
- What codility is
- Hiring interviews are not necessarily fair
- Sometimes, problems require numbers that are larger than 32 bits
- It may have been good to have done some coding competitions before, even just for practice
- Position is a code monkey job - if employer can reject someone because of a trivial coding error in such conditions, they aren't interested into their 5 year project management experience (hence they have more than enough applicants and can throw them away like this)
- Unless codility publishes full test results to employer, it is a service to avoid by employers who aren't hiring in bulk
- When dealing with ASCII numbers, never assume upper bound (scanf overflow and similar)
- Life is not fair. There are people who get rejected because of their color, accent, wearing wrong brand of shoes, because the job has already been filled by CEO's brother, because interviewer has a bad day and doesn't want to spend time on you or because the job posting is a fake and merely used to justify using outsourced labor.


As a side note, google for codility+reddit. It was tested a while back, and many coders fell for exact same problem (timeout as well as overflow).

Quote:
My opinion is that the current test is deceiving and illegal...

Current test is not valid for a hiring process.

Unfortunately, the company hiring you disagrees. Oh well, that's life.

It's certainly not illegal.

Once you get out of college, life is nothing but problems that you have to solve correctly despite not having all of the constraints defined.

Nothing in those tests seems unreasonable to me. They didn't explicitly spell out some of the restrictions and requirements, but that means you're supposed to ask them. In fact, most good job interviewers will explicitly leave out important requirements or details to see if you do ask. A large part of software development work is making sure you have the right requirements and understand them.

"Would a naive recursive algorithm be OK or do you want to see a O(1) memory-space solution?"
"Whats the run-time restriction?"
"How many bits are valid for the input?"

Asking these sorts of things is a plus, because it shows that you're observant, thinking about the run-time/memory-space-complexity and inputs of your algorithm.

I don't think its unreasonable, but it is a little bit underhanded not to mention that these criteria are important. An attentive candidate should probably ask about additional constraints/requirements, but interviews can be stressful and people can forget such seemingly "obvious" steps.

Were I in an interview, I would tend towards writing simpler code rather than go for something more complex (which might be faster or better handle extreme boundaries) but more likely to contain a bug.

I believe many programmers would be the same. It is ultimately the company's call. I personally think they might be losing out on good applicants who would have done well if they were aware of the judging criteria in advance - and they might be hiring applicants who routinely present over-engineered solutions to simple problems. Not everything needs to be super optimised, and "int" has a good range for 99% of use.

Without stating additional requirements, I would find it hard to fault programmers with going for the simpler solution. It is what I do myself at work.

Codility does have a free demo. You can try it out and see how it works before you do it for real. One of the more interesting aspects of it is that just by choosing certain programming languages you pass some tests automatically for "free". Ex: using python gives you automatic bignum support with integers.

Quote: Original post by Rycross

Asking these sorts of things is a plus, because it shows that you're observant, thinking about the run-time/memory-space-complexity and inputs of your algorithm.

You can't ask during automated interview.

Fibonacci is *never* coded using recursion. The only exception is writing an introduction to algorithms and data structures, where it's used to demonstrate why recursion fails miserably. This is litmus test.

Counting bits is about reliable input parsing.

Quote: Not everything needs to be super optimised, and "int" has a good range for 99% of use.

Using C - read user's name. Same problem. How do you do it without being open to buffer overruns. Using any managed language - parse URL string. Can you ensure that the parsing algorithm will be properly bounded in time and space?

How do you count bits in a number (which comes from some input)? Read one char at a time, count bits. Unless input stated that sequence of ASCII characters will always parse into a number with range of less than 32-bits, assuming that is very flawed. Details depend on actual requirements, but I'm willing to bet that problem doesn't really require storing the number as binary or memory, or hint at that being a requirement.

Quote: I personally think they might be losing out on good applicants who would have done well if they were aware of the judging criteria in advance

Here is what the employer sees (not codility, but similar automated tests):

1. Candidate #1      100 14min2. Candidate #2      100 19min3. Candidate #3      100 24min4. Candidate #4      100 39min5. Candidate #3      98  36min...14. You              93  56min...55. Candidate #55    2   120min56. Candidate #56    0   120min...498. Candidate #498  0   0min499. Candidate #499  0   0min500. Candidate #500  0   0min

Top 5 are selected for phone interview. #5 will be grilled about that one failed test.

Numbers are made up, but quite realistic. Half of applicants won't even take the test (they got coaxed into the test by recruiter or weren't even told to take it, or are simply spamming resumes). The remaining half will take it, try the first assignment, then say screw that.

#14 might be a good developer. But since company will need to dedicate a full day to personally interviewing the 5 applicants, it makes sense to choose best-of-breed to start off.

Finally - if person's experience matters even in slightest, their first encounter will not be a dumb coding test, but they will be brought in because of past history, connections, references or related work. But for entry-level coding jobs - code is all that matters.


Sure, the person might learn quickly about Fib, they might learn about input parsing, or they might not. But the higher their initial level, the less burden they will be on others, and the more they can focus on business parts which actually make money. Especially since it's possible they won't be doing coding at all, but will instead be trained in more important business domain related skills, yet understand the details of actual programming.

I don't believe in "losing good applicants" anymore. If anyone feels they can't handle such tests, then they'll need to learn how to bypass HR, and get into company directly. That is how non-entry-level jobs are handled anyway. As long as you are asking someone to employ you, you have no bargaining position, and no matter how good, there are more than enough people better than you.

Now we can either throw around words like "illegal", or try to understand how the system works to deal with it effectively. Fortunately, hiring is messed up enough that simply trying 500 times will net a job.

But here is another perspective - assuming you passed that test with 100 points. How would you feel about your coworkers who only scored 74, 58 and 12, but were hired anyway?