DEVELOPER PRODUCTIVITY REPORT 2013

DEVELOPER 

PRODUCTIVITY 

REPORT 2013 

HOW ENGINEERING TOOLS & PRACTICES IMPACT 

SOFTWARE QUALITY & DELIVERY 

All rights reserved. 2013 © ZeroTurnaround OÜ 

1

TABLE OF CONTENTS 

INTRODUCTION 

WHO CARES ABOUT THE QUALITY AND 

PREDICTABILITY OF SOFTWARE RELEASES? 1-4 

PART I - METRICS 

HOW TO MEASURE QUALITY & PREDICTABILITY 5-8 

PART II - PRACTICES 

HOW THE THINGS YOU DO AFFECT QUALITY & PREDICTABILITY 9-26 

PART III - TOOLS 

HOW THE TOOLS YOU USE AFFECT QUALITY & PREDICTABILITY 27-37 

TL;DR - LET’S CLOSE THIS OUT! 

SUMMARY, CONCLUSIONS & OBSERVATIONS 

AND A GOODBYE COMIC ;-) 38-41 


2

INTRODUCTION: 

WHO CARES ABOUT QUALITY 

SOFTWARE AND PREDICTABLE 

DELIVERIES? 

In truth, we all should. You might be “Developer of 

the Year”, but if the team around you fails to deliver 

quality software on time, then it pays to review your 

team’s practices and tools to see if that is somehow 

related. So that’s what we did--and guess what: the 

things you do and the tools you use DO have an 

effect on your software quality and predictability 

to deliver. Let’s see how... 


3

What do we want to achieve with this report? 

How do the practices, tools and decisions 

of development teams have an effect on 

the quality and predictability of software 

releases? Seems a bit abstract, right? 

In fact, this is one of the most frustrating 

aspects of talking about the quality and 

predictability of software releases. We all 

hear a lot of noise about best practices 

in software engineering, but a lot of 

it is based on anecdotal evidence and 

abstractions. Do we ever see much data 

related to the impact of so-called best 

practices on teams? 

With our survey, the goal was to 

collect data to prove or disprove the 

effectiveness of these best practices-- 

including the methodologies, tools and 

company size & industry within the 

context of these practices. 


2

Our data and metrics 

In the end, we collected 1006 responses, which is reasonable for a survey 

where all questions are *required--last year over 1800 developers did at 

least half of our survey on tools and technologies. 

Note: It seems that getting good responses to surveys isn’t easy--most people find 

a 2-3 question survey palatable, but aside from just dumb numbers it’s not easy 

to learn much from someone in just a few seconds. We narrowed down our scope 

to 20 questions, which took our development team about 5 minutes to finish the 

one-page form. Still, we didn’t see a flood of respondent participation. 

So what metrics did we decide to track in order to understand how best 

practices actually work? 

After ascertaining that Quality and Predictability where two areas in which 

data could be gathered, we continued with further analysis based on tools 

used (e.g. VCS or CI servers), practices employed (i.e. testing, measuring, 

reviewing) and industry & company size. 

Quick note about bias: When analyzing the data, we discovered a couple 

of areas where bias was present. Compared to the software industry 

as a whole, our respondents represent a disproportionate bias towards 

Software/Technology companies as well as Java as a programming 

language. 

1. Quality of software - determined by the frequency of critical or 

blocker bugs discovered after release 

2. Predictability of delivery - determined by delays in releases, 

execution of planned requirements, and in-process changes (aka “scope 

creep”). 


3

A little history: ZeroTurnaround’s Java and Developer Productivity 

Reports from 2009 - Present 

If you’ve been following ZeroTurnaround and RebelLabs for a while, you’ll 

know that this is our fourth report in as many years. It started back in 2009, 

when we began our quest to understand developer productivity by looking 

at which Java EE application servers/containers that 1100+ developers were 

using and how much time drain from redeploys is associated with each one 

(we discovered that between 3-7 work weeks each year were lost to this 

process). 

So where does that leave us for 2013 and beyond? Issuing another report 

on the popularity of IDEs, Build Tools, CI servers, Web Frameworks and 

Application Servers was one idea--people loved our 2012 report. But 

would learning that Vaadin jumped 1% in popularity from 2012 to 2013, or 

confirming that Eclipse, Subversion, Jenkins, Spring MVC and Tomcat are 

still #1 in their respective fields truly be of value to the Java community as a 

whole? 

In 2011 we expanded our research efforts and this time asked approximately 

1000 developers about Build Tools, IDEs, Frameworks and Java EE standards 

in addition to App Servers, and again asked about how much of each hour 

was lost to restarts. We also asked ~1250 developers in India about tools and 

productivity, and saw some interesting differences between India and the 

rest of the world. 

By 2012, we wanted to go even further. Our Developer Productivity Report 

2012 focused on the vast array of tools & technologies that developers use 

each day, and looked deeper into what makes developers tick, asking about 

developers’ work week, stress and efficiency. Releasing this report was, in 

many ways, the unofficial birth of RebelLabs and the idea that high-quality, 

educational technical content is something we should continue to focus on. 

Instead, we looked to cover the more difficult areas, looking at how tools 

and practices affect organizations as a whole--namely with the Quality and 

Predictability of software releases. It’s our goal to be the premier provider 

of Java development productivity statistics outside of dedicated research 

agencies, and we’re completely transparent and honest about our data. We 

admit bias. We publish our raw data for your own analysis. So we set down 

some goals for how we would proceed. 

Moving forward, let’s go to Part I, where we discuss why it’s hard to measure 

quality and predictability, and what we did to quantify these metrics. 


4

PART I - METRICS: 

HOW TO MEASURE QUALITY 

AND PREDICTABILITY 

“How much quality goes into your app?” is a question 

we did not ask, since it’s ridiculous. But the answer to 

that question is what we were looking to learn. How 

do you track down abstract measures of quality and 

predictability and put numbers to them? Read on…. 


5

On Judging Quality 

“People forget how fast you did a job – but they remember how well you did it” 

- some guy named Howard Newton 

While an end user can quickly see whether your software is “good quality” 

or not, it’s not easy to gauge this at the development level. How can you tell 

if your app is high quality or low? 

We felt the most honest responses would be objective, self-reported metrics, 

so we decided that a good measure of quality is to ask “How often do you 

find critical or blocker bugs after release?” This seems like a good way to 

judge at least minimum requirements, and it’s the key metric of quality that 

is most likely to negatively impact the largest group of end users. 

It’s a pretty simple formula we’re looking at in this case as well: quality is 

100% if no critical/blocker bugs are found post-release. So we converted 

plain-language answers into percentages for this question, keeping the 

need for assumption as low as possible: 

Do you find critical or blocker bugs after a release? 

A. No - 100% 

B. Almost never - 75% 

C. Sometimes - 50% 

D. Often - 25% 

E. Always - 0% 

Based on this question alone, we were able to determine a bunch of yummy 

stats for you, and some interesting findings. This table below tells us how 

much of the time software is released into production without critical or 

blocker bugs: 

The % of time that software is released without any critical bugs 

Average 58% 

Median 50% 

Mode 50% 

Standard Deviation 19% 

Laggards (Bottom 10%) 25% 

Rock stars (Top 10%) 75% 

FINDINGS AND OBSERVATIONS 

1. Most respondents admitted to “Sometimes” releasing apps with 

bugs, so users have a 50-50 chance of the app they’re using containing 

a critical bug--maybe this is when you start calling it a “feature”? ;) 

2. On average, nearly 60% of releases go to production critical-bug free, 

so let’s say great job to those folks! 

3. The laggards of the group (bottom 10% of respondents) get apps out 

the door bug-free only 25% of the time. 

4. The rock stars of the group (top 10% of respondents) enjoy releasing 

their apps into production without critical bugs 75% of the time. 


6

On Judging Predictability 

“Do you miss me?” - Release Dates 

Not to beat a dead horse, but we already know software engineering 

productivity is hard to measure and impossible to compare. If quality of 

software is one half of the coin, then predictability of software delivery is the 

other. After all, you can build the world’s best software, but if you cannot get 

it to users--then who cares! 

We see predictability as a good associative measure of productivity in 

software, since if you can predict your delivery well it is likely that you 

understand it and can take steps to improve it. 

Our thinking is that predictability is 100% if releases are delivered on 

time and as planned. Again, we’re are looking for objective, self-reported 

answers for measuring predictability, so we asked in simple language three 

questions that could be quantified: 

How late are your releases (vs initial planned time)? 

A. On time or early 0% 

B. A bit 10% 

C. Moderately 25% 

D. A lot 50% 

How much of the original plans get done? 

A. Everything 

B. All but 10% 

C. All but 25% 

D. Only 50% 

E. Only 25% 

How much do plans change/expand during development? (scope creep) 

A. None 

B. 10% creep 

C. 25% creep 

D. 50% creep 

E. 75% or more creep 

In truth, predictability is more complicated to measure than simply 

frequency/number of bugs, like we did for quality. We overheard some 

people suggest that including change in scope in this is controversial, 

but this definitely affects your ability to predict your delivery, which is 

what we are trying to measure. Plus, we checked; omitting this does not 

change any trends! 

Of course, we are looking at trends here and trying to represent them 

with concrete numbers, so statistical analysis of this type may impact the 

accuracy of absolute numbers, but not the relative trends. 


7

Before we show you the stats, let’s see how we combined the three survey 

questions above into a mathematical representation for predictability: 

1 

( (1 + % late) ) 

PREDICTABILITY = x (% of plans delivered) x (1 - % scope creep) 

Example: Simon’s team is pretty good at delivering on time--they 

only release late 10% of the time. On average, they get 75% of their 

development plans done, and they’ve been able to limit scope creep 

to just 10% as well. Based on that, we can calculate that Simon’s team 

releases software with 61% predictability. 

1 

( (1 + 0.10 ) ) 

MATH! x (0.75) x (1- 0.10) = 0.61 = 61% 

Note: This number isn’t quite a probability (it’s not normalized to 100%, but very 

close to that). We could normalize over delivery, but chose not to as it didn’t 

affect trends, but made the number harder to interpret. 

Based on the replies to these questions, we generated some data that tells 

us how predictable releases are for development teams. 

How predictable are software releases? 

Average 56% 

Median 61% 

Mode 61% 




FINDINGS AND OBSERVATIONS 

1. It looks like the industry can predict deliveries within a margin of 

60%. This is in line with anecdotal data on late releases, cut features 

and unplanned work creeping up. 

2. The rock stars reach the 80% margin, which is probably the 

reasonable limit in being able to predict your delivery. So, shoot 

for 80% predictability and you won’t be disappointed. 

3. There is a large spread between the rock stars and laggards, which 

gives high hopes that the worst and the average can improve a lot. 

4. When we looked into Predictability by Industry, we saw no significant 

relationship, and in fact Predictability by Company Size increases 

slightly (3%) for larger companies. We theorize that this is due to 

a greater level of organizational requirements, thus more nondevelopment 

staff are available to coordinate projects and release 

plans as teams scale. 


8

PART II - PRACTICES: 

HOW THE THINGS YOU DO AFFECT 

QUALITY & PREDICTABILITY 

Ok, you’re probably ready for some numbers and goodlooking 

graphs now, so let’s get into it. In this section, 

we look at how the methodologies (i.e. these so-called 

“best practices”) affect the quality and predictability of 

software releases. 


9

When looking at Practices, we want to 

know what methods and team activities 

work best to improve productivity for 

the team as a whole. So we’ll be looking 

into things like communication methods, 

testing and reviews, branching etc. We 

asked questions to give us a good view 

on each of the following. 

Next you’ll find the questions as the 

respondents saw them, with our analysis 

of the responses. So let’s go! 

. Technical debt 

. Code quality 

. Test coverage 

. Pairing up 

. Code reviews 

. Task estimation 

. Branching 

. Code testing 

. Communication methods 


10

Do you work on solving technical debt? 

(refactoring, performance, builds, etc) 

As we wrote in a blog post 

showing some early results of 

this survey, technical debt is a key 

metric in determining developer 

productivity. In the same way that 

the visible part of an iceberg is 

often concealing a massive, hidden 

underbelly, technical debt cannot 

be measured at first glance, but 

has long-term, negative impacts on 

your software. Here we can see the 

breakdown of how the respondents 

react to technical debt. 


11

If we compare these findings based on 

the measurements of 1) Predictability 

and 2) Quality of software releases, we 

can see some trends. 

The effects of solving technical debt 

on predictability and quality 

There isn’t a lot of relation with either 

Quality or Predictability. The only thing 

that can be said with any certainty, 

is that not doing any technical debt 

maintenance does impact them 

negatively. But there isn’t a lot of 

difference from that point on. This is 

slightly controversial, and could be 

unrelated altogether. 


12

Do you monitor and fix code quality problems? 

(e.g. with Sonar) 

Code quality coverage is often 

a hot-debated topic among 

developers. Based on the 

responses of over 1000 engineers, 

we see half of those asked don’t 

fix code quality issues, and most of 

those people don’t even monitor 

code quality. 


13

When applying this question to differences in 

predictability and quality, we see a clear increase 

in both with teams that fix code quality issues. 

A nearly 10-point increase in predictability 

comes with fixing all code quality issues--seems 

kind of like a no-brainer now, huh? 

The effects of fixing code quality issues 



14

How much functionality (not code) 

is covered by automated tests? 

Automating functional tests are a good way to add another layer 

of stability to your software and make sure that what works in dev 

also works in production! Here is what the respondents told us 

about how much automated testing is used to verify functionality. 

In more plain numbers, we can see an average of one-third (33%) 

of software functionality is covered by automated tests--but the 

standard deviation is huge, and most often respondents utilized 

only 10% of automated testing for their software. 

The laggards literally did no automated testing (0%), and the rock 

stars covered 75% of functionality with automated tests. Way to go, 

rock stars! 

How much of your functional testing is automated? 

Average 33% 

Median 25% 

Mode 10% 





15

When relating functional test automation with 

predictability and quality, we see a significant gain 

in both metrics as the % of (self-reported) test 

automation increases. It’s interesting that folks who 

don’t do any automated testing (0%) are better off 

than folks who do a little (10%). We theorize that 

teams who do no automated testing are probably 

doing more manual testing and those who do just 

a little trust them more than they should. Once 

going further with it, we see a steady increase. 

The effects of automated functional testing 



16

Do you pair up? 

(programming, debugging, support, etc; ad hoc or policy) 

For a page on wikiHow, this explains the 

benefits of pair programming remarkably 

fully: 

“Some benefits you can expect: better 

code (simpler design, fewer bugs, more 

maintainable), higher morale (more 

fun!), shared knowledge throughout 

your team (both specific knowledge 

of your codebase and general 

programming knowledge), better time 

management, higher productivity.” 

http://www.wikihow.com/Pair- 

Program 

While two-thirds (66%) of respondents 

pair up at least sometimes, one in three 

teams don’t. 


17

Well, we see a significant increase in quality and 

a slight increase in predictability with pairing 

up. This is one place where we do wish we 

could measure productivity as well, as the main 

argument against pairing up is that two people 

separately can do more work. We can’t disprove 

that, but we can put numbers behind the 

increase in quality. 

The effects of pairing up on quality and predictability 


18

Do you do code reviews? 

A solid majority (76%) of respondents 

review code for at least “Some 

commits”. Reviewing code has 

become more or less a standard 

practice for development teams. 

Only a tiny sliver of about 2% of 

teams do multiple code reviews 

for all commits. 


19

The impact on predictability of releases is very 

high! However, the low effect on software 

quality is surprising. Looks like programmers 

are bad at spotting bugs in code, but good at 

spotting software design issues and code smells. 

The effects of code reviews on quality and predictability 


20

Who is involved in the time estimation for the tasks? 

When it comes to estimating the type 

and timeline of development tasks, we 

discovered a direct relationship between 

quality & predictability and the job role of 

the individual or people assigning the tasks. 

The results for ‘job role’ were pretty evenly 

distributed, so we decided to relate each job 

role with our metrics for predictability and 

quality. Check this out. 

Put simply, when the estimating person or 

group creating the task is less-technical, there 

is a decrease in the predictability and quality 

of the software release. Interestingly, team 

leads or architects have a null effect on both, 

whereas estimating with the team as a whole, 

along with the healthy discussion, generates 

the most gains in the key metrics. If 6% 

and 4% gains in predictability and quality 

respectively are powerful enough drivers to 

involve the team as a whole, then here is the 

data to back your move. 


21

How do you branch? 

The different styles of branching can 

always generate a discussion both inhouse 

and in public forums. We asked 

to check off the different styles, and 

surprisingly, they were pretty evenly 

represented. Looking at the relations 

to key metrics we see little strong 

association either way. 

The only clear trend is a slightly negative 

one for the single trunk and a positive 

one for feature branches. Go figure. 


22

Who tests your code? 

Are developers or team leads testing your 

code? If you have a QA/testing department, 

they are most likely doing it, but perhaps you 

should consider automating your tests based 

on these results. 

If there was even a good case for automation, 

it would be this. Automated tests showed the 

largest overall improvements both in the 

predictability and quality of software deliveries. 

Quality goes up most when Developers are 

testing the code (also discussed in Sven Peter’s 

talk “How to do Kickass Software Development” 

at GeekOut 2013), which means that you 

shouldn’t just leave testing to QA team, but 

bake it into the development process as 

well. The rest of the measurements were 

more or less insignificant, although we don’t 

recommend letting your customers/users test 

your software for you. 


23

How do you communicate? 

We asked respondents how they communicate 

most often, based on a list of these choices: 

. Ad-hoc conversations 

. Daily standup 

. Meeting several times a week 

. Weekly meeting 

. Email 

. Chat 

. Phone/Voice chat 

. Teleconferencing 

. Forums 

. Trackers 

. Wikis 

The effects of different communication methods 

on predictability of software delivery 

The choices Email and Ad hoc conversations were 

the most common selections, but there was a 

fairly even distribution among the rest of the 

of the choices. However, when we relate these 

findings with Predictability and Quality of software 

delivery, we see a some interesting trends. 

With regards to Predictability, having Daily 

standups and Team chat seem to be the best 

ways to communicate, and relying on Email to 

communicate is even slightly detrimental to 

predictability. 


24

When looking at software quality with regards 

to communication methods, the positive trends 

are not very strong--Ad hoc conversations and 

Weekly meetings both score a 2% increase in 

quality. However, the negative trends shed a 

little light on how teams communicate: 

The effects of different communication methods 

on software quality 

Teleconferencing naturally implies working with 

remote teams, which is expected to decrease 

quality slightly. And if you’ve ever heard the 

phrase “Too much of a good thing”, then the 

law of diminishing returns should be applied to 

Meeting several times a week. Whereas a Weekly 

meeting is positive, doing it too often appears 

to cause “meeting fatigue” and impacts quality 

worse than any other communication method, 

according to these results. 


25

Summary of main takeaways 

. Technical debt - Most teams work on technical debt at least 

sometimes, but we saw no significant increases in quality and 

predictability of software releases; however, a negative trend 

appears by not doing any technical debt maintenance. 

. Code quality - over 50% of respondents do not fix code quality 

issues. We saw a significant increase in both predictability and 

quality in teams that regularly fix code quality issues. 

. Test coverage - More than half of respondents have less than 

25% test coverage. There is a significant increase in both 

predictability and quality as test coverage increases. 

. Pairing up - 2/3 of respondents team up at least sometimes, 

and there is a significant increase in software quality associated with 

this, with a slight increase in predictability. No data about increased/ 

decreased productivity from pairing up. 

. Code reviews - We see that 76% of respondents do code reviews 

for at least some commits, which significantly increases 

predictability of releases, but does little for software quality. 

More is better. 

. Task estimation - Simply put, keep managers out of task 

estimation. Estimations by the team as a whole significantly 

increases software quality and predictability of releases, but even 

having individual developers doing it doesn't hurt anything. 

. Branching - There was no significant effect on predictability or 

quality for branching, but there was a slight negative trend for the 

single trunk and a slight positive trend for feature branches. 

. Testing code - Of all the choices, automated tests showed the 

largest overall improvements both in the predictability and quality 

of software deliveries. Quality goes up most when Developers are 

testing the code. 

. Communication methods - the results here show that daily 

standups, team group chat online, and weekly meetings all improve 

predictability, while meeting multiple times a week and having a 

remote team (teleconferencing) associates with decreased quality. 


26

PART III - TOOLS: 

HOW THE TOOLS YOU USE AFFECT 


The previous section showed some pretty interesting 

results related to how practices influence the quality and 

predictability of software delivery. Now, let’s check out the 

relationship between the tools development teams use 

and how this affects quality and predictability. Gonna be 

fun on a bun! 


27

Different teams, different motivations 

In past Developer Productivity and Java EE productivity reports, we became 

well-known by our reporting of tool and technology usage/popularity. But 

we covered a lot of this in 2012 and decided that one year wasn’t enough 

time to wait before reporting these statistics again. So in this section, we not 

only show tool and technology popularity, but also how they correspond to 

increases or decreases in predictability and quality. It’s possible that this has 

never been done before, so strap on your geek helmets and get ready for 

some cool stats. 

Here is what we will look at: 

. Popularity of tool types in use 

. Increase in Predictability based on tool type usage 

. Popularity of Version Control Systems (VCS) used by respondents 

For the sake of comparison, we will check out tool/technology popularity 

numbers from our previous Developer Productivity Report 2012 -- (SPOILER: 

If you don’t already, start using Continuous Integration and Version Control 

ASAP!) 

. Popularity of Continuous Integration (CI) servers used by respondents 

. Popularity of Issue Trackers used by respondents 

. Popularity of Communication tools used by respondents 

. The effect of tools on predictability and quality (only tools with 10%+ 

popular shown) 

. Tool usage by Rock Stars (Top 10%) compared to All Respondents 


28

What kind of tools do you use in development? 

Nothing truly groundbreaking in this chart, but it is surprising that less than half of respondents are using text editors, since only a decade ago 

the question was “Vi/Vim or Emacs?”, not “Should a text editor be used?” We’ve come a long way. We can see that Infrastructure as a Service tools 

(e.g. such as Amazon Web Services, AWS) are just breaking the 10% usage area. The biggest mystery to us is that there are still people out there 

not using IDEs or Version Control. What the heck are you developing? Cloned hamburger meat? ;-) 

Tool type usage 


29

How does predictability change 

based on tool type usage? 

Which technologies and tool types influence 

how predictable your releases can be? Believe it 

or not, the same relation set against quality 

measurements showed no significant trends- 

-it looks like quality is affected by development 

practices, but not development tools. 

Increase in predictability per tool type 

Using Version Control and IDE will significantly 

improve the predictability of your deliveries, 

and we see a reasonable increase in 

predictability for users of Code Quality Analysis, 

Continuous Integration, Issue Tracker, Profiler 

and IaaS solutions. Use of a Text editor and 

Debugger (surprisingly) has little or no effect on 

predictability. 


30

Popularity of Version Control Systems (VCS) used by respondents 

Subversion (58%) is being threatened by Git (47%) for de facto leadership of the Version Control space. Compared to our Developer 

Productivity Report 2012, it ranked as follows: Subversion - 66%, Git - 33%, CVS - 12%, Mercurial - 10%, so we can see a definite trend 

moving to distributed VCS. 

In this report, we also gathered some numbers on smaller players like Perforce (3.8%), TFS (3.7%), Clearcase (3.4%) and a few others. 


31

Popularity of Continuous Integration (CI) servers used by respondents 

The last time we asked about Continuous Integration servers, we saw the following numbers: Jenkins (Hudson) - 49%, Bamboo - 7%, 

TeamCity - 5%, CruiseControl - 4%. Jenkins (Hudson) remains firmly in the leading position, and we’ve seen growth for both Bamboo 

and TeamCity plus the introduction of Travis CI. 


32

Popularity of Issue Trackers used by respondents 

This is the first time we asked about Issue 

Trackers, but we can see that they have a 

reasonable effect on the predictability of 

software releases. Atlassian’s JIRA (57%) is 

the most used tool, but it’s interesting to 

see GitHub (21%), who is better known as 

a very cool VCS tools provider, coming in 

at #2 as an Issue Tracker. Other players 

that popped on to this radar include 

Redmine, Bugzilla, BitBucket and Mantis, 

along with a half dozen smaller providers. 


33

Popularity of communication tools used by respondents 

We wanted to know which 

communication tools were most used 

by respondents--after all, we saw 

that communication methods can 

significantly affect both predictability 

and quality. These tools span the realm 

of collaboration, from household video/ 

audio conferencing tools to tools made 

for software teams and grouchy coders 

rockin’ out on IRC. 

Skype takes the lead (39.3%), but it’s good 

to see that Confluence (29.7%), a tool 

made for technology professionals, is 

coming in at #2. The Google consortium 

Google Docs, Google Hangout & 

Google+ take a nice chunk of in the 

communication tool space as well. 


34

The effect of tools on predictability 

(only tools with 10%+ popular shown) 

Probably one of the coolest parts of this analysis is that we were able to 

connect specific tools with predictability of releases. In general, we didn’t 

see many significant trends, however it would seem that if you pick a 

CI server, go for Bamboo or Jenkins over TeamCity. 

In the main competitors for the Version Control space, Git clearly takes 

out Subversion once again. Users of Confluence show a positive trend in 

the area of predictability, and JRebel, that time-saving tool we’ve heard 

about before, shows a statistically significant increase of 8% 

in predictability of software delivery. 

*Note: In order to maintain objectivity, we didn’t originally include JRebel in 

the list of tools. However, we couldn’t help ourselves and matched the emails 

provided by 47% of all respondents against our client list. We identified that onethird 

of respondents with email address used JRebel, two-thirds didn’t, and used 

their data for comparison. The results were too cool to omit :) 

TOOL 

EFFECT ON 

PREDICTABILITY 

TeamCity -2% 

Google Docs -1% 

Subversion -1% 

Skype -1% 

Git 0% 

GitHub 0% 

Google Hangout 0% 

CVS 2% 

JIRA 2% 

Confluence 3% 

Bamboo 4% 

Jenkins (Hudson) 4% 

JRebel* 8% 


35

Tool usage by rock stars (top 10%) 

compared to all respondents 

As our final analysis, we wanted to see 

what the rock stars of this group are 

using for tools. As a reminder, rock stars 

are respondents in the top 10% for both 

predictability and quality of software 

delivery. By seeing what these geek gurus 

are using in terms of specific, we might 

be able to figure out what we can do to 

improve. We only choose those tools that 

had a large enough base of users to serve 

for analysis. 

In terms of statistically significant figures, 

we can only really say that the rock stars 

of this sample population vouches for 

Jenkins and doesn’t like Google Drive very 

much (we have firsthand experience of the 

diminishing return of a bazillion Google 

Docs!). 

Other trends represented include a definite 

move towards Git over Subversion, a 

preference for Google Hangouts over Skype 

conversations. Atlassian products are hit 

or miss here as well--where Confluence 

is preferred by the rock star group, JIRA is 

shunned and Bamboo comes in with no 

discernable change. 

TOOL ROCK STARS ALL RESPONDENTS DIFFERENCE 

Google Drive 22.31% 26.94% -4.63% 

Subversion 54.55 58.15% -3.60% 

JIRA 54.55% 56.96% -2.41% 

Skype 38.84% 39.26% -0.42% 

Github 21.49% 21.67% -0.18% 

Bamboo 10.74% 10.44% 0.30% 

TeamCity 9.09% 7.75% 1.34% 

Confluence 32.23% 29.72% 2.51% 

Git 50.41% 47.02% 3.39% 

Google Hangout 20.66% 16.70% 3.96% 

Jenkins 66.94% 56.06% 10.88% 


36

Summary of main takeaways 

. Tool type usage - Some people still aren't using IDEs or Version 

Control, however the all-powerful text editor dropped to below 

50% usage. 

. Increase in predictability of release per tool type - 

The biggest winners are Version Control and IDEs, but predictability 

increases with the use of Code Quality Analysis, Continuous 

Integration, Issue Tracker, Profiler and IaaS tools 

. VCS popularity - The domination of Subversion (58%) is being 

threatened by Git (47%) for de facto leadership of the Version 

Control space. Mercurial seems to be losing ground to Git as well. 

. CI popularity - Jenkins (56%) remains firmly in the leading position, 

followed by Bamboo (10%) and TeamCity (8%). 

. Issue Tracker popularity - The space is effectively controlled by 

JIRA (57%) as the most used tool, but it’s interesting to see 

GitHub (21%) at #2. 

. Communication tools popularity - Skype (39%) is the most 

used communication tool, but Confluence (30%), the tool for 

software companies, is #2. Google Docs, Google Hangout and 

Google+ comprise the rest of the top 5. 

. How specific tools affect release predictability - In CI, take 

Jenkins or Bamboo over TeamCity, in VCS select Git over Subversion, 

and use tools like Confluence and JRebel. 

. What the rock stars do differently from the average 

respondent - The only trends to speak of here are that this group 

likes Jenkins (a lot!), doesn’t like Google Drive, prefers Git over 

Subversion and Google Hangout over Skype. 


37

TL;DR - LET’S CLOSE THIS OUT! 

For those of you too lazy to read the entire report, you 

can come to this section to see all the juiciest statistical 

morsels from the entire report, and the observations 

& conclusions we made based on the data we collected 

from over 1000 engineers. 


38

Summary of overall findings and a goodbye comic :-) 

PART I - METRICS: HOW TO MEASURE QUALITY 

& PREDICTABILITY 

PART II - PRACTICES: HOW THE THINGS YOU DO AFFECT 


Neither quality nor predictability was significantly affected by industry or 

company size, which is good to see. 

In terms of Quality, nearly 60% of releases on average go to production 

free of critical bugs, so those teams can feel proud. The bottom 10% get 

apps out the door bug-free only 25% of the time, whereas the rock stars of 

the group enjoy releasing their apps into production without critical bugs 

75% of the time. 

There are certain practices that significantly influence the quality and 

predictability of software releases. Fixing code quality issues (up to 

+7% and +9% respectively) and automating tests (up to +9% and 12% 

respectively) are excellent practices to utilize across the board. 

In terms of improving software quality alone, having developers pair up 

(up to +7%), allowing developers to test code (+5%) and avoiding too many 

meetings each week (-4%) does the most to increase quality. 

When it comes to Predictability, the industry can predict deliveries within a 

margin of 60%, which matches up with the anecdotal data on late releases, 

features that needed to be cut and unplanned scope creep. The rock stars 

get to that enviable 80% for predictability, which as we said is probably the 

reasonable limit in being able to predict your delivery. 

In terms of predictability of releases alone, doing code reviews for 

commits (up to +11%) is the largest single beneficial practice, along with 

estimating tasks as a team (+6%); however, involving the management 

drops it by -6%. 


39

PART III - TOOLS: HOW THE TOOLS YOU USE AFFECT 


We can also see a relationship between the tools respondents use and how 

these tools affect predictability (quality was not significantly affected by 

any tool sets, prompting us to remember that quality is based mainly on 

practices, not tools). 

The tool types that increase the predictability of releases most are 

Version Control (+9%) and IDEs (+8%), but Code Quality Monitoring, CI, 

Profiler, Issue Tracker, and IaaS solutions (up to +5% for the group) also 

improve release predictability. The top 3 individual tools that enhance 

predictability are JRebel (+8%), Jenkins / Bamboo (+4%) and Confluence 

(+3%). 

In terms of popularity, here are the technologies used by over 50% 

of respondents: IDE (97%), Version Control (91%), Issue Tracker (79%), 

Debugger (71%) and Continuous Integration (68%) 

The Top 10 tools/technologies used by respondents: Subversion (58%), 

JIRA (57%), Jenkins (56%), Git (47%), Skype (39%), Confluence (30%), Google 

Drive (27%), GitHub (22%), Google Hangout (17%) and Bamboo (10%). 

Thanks XKCD! http://xkcd.com/678/ 

Finally, we display what the rock stars of the group (those in the top 10% 

for quality and predictability metrics) are using. Basically, the message from 

these folks is: use Jenkins, look for something other than Google Drive, 

choose Git instead of Subversion, and opt for Google Hangout over Skype. 


40

Conclusions and Observations 

“To improve is to change; to be perfect is to change often.” 

- Winston Churchill 

STILL NO SILVER BULLET 

Sorry to break it to you, but there isn’t any single practice or tool that will 

be a true game changer for you when it comes to improving your software 

quality or predictability of delivery. Teams in general should focus on a) 

improving various work practices, b) using the best tools available and c) 

continuously improving the nature of their organizational culture so that 

they can meet up with the rock stars hanging out in the top 10%. 

RELATION != CAUSATION 

We feel that it’s dangerous to think that some practice makes an 

organization better...it’s quite possible that it is the other way around: good 

organisations pick particular practices to follow! Still it’s a good reason to 

try it out and see for yourselves. Looking for direction? Trying following the 

rock stars and see if relation equals causation for you! 

TEST AND MEASURE 

Trying different things is the key to improving your organization. However 

you need to pick your metrics carefully and measure the improvement 

or lack of such. Treat it as a scientific experiment and record results. The 

framework of quality and predictability is a great way to compare even 

different projects, and if you try one tool/practice at a time you can compare 

to the rest of the organization to see improvement or lack of such. 

TOOLS MATTER 

If you’re a toolmaker yourself, then you know this to be the case. Not only 

do good tools make you more productive, but they also serve as focal point 

on which you can enforce the practices that further improve your ability to 

predictably deliver quality software. If you’re on a tight budget, using free 

stuff is much better than using nothing, but there is a tendency among 

developers who purchase tools to fall in love with their new toys. 

More specifically, it’s time to use Continuous Integration & Version Control 

if you haven’t started already. You’re waaaay behind the curve. And for you 

Java developers out there, check out JRebel and see if it makes such a big 

difference for you too :) 


41

Contact Us 

Twitter: @RebelLabs 

Web: http://zeroturnaround.com/rebellabs 

Email: labs@zeroturnaround.com 

Estonia 

USA 

Ülikooli 2, 5th floor 399 Boylston Street, 

Tartu, Estonia, 51003 Suite 300, Boston, 

Phone: +372 740 4533 MA, USA, 02116 

All rights reserved. 2013 © ZeroTurnaround Phone: OÜ 1(857)277-1199 

Czech Republic 

Osadní 35 - Building B 

Prague, Czech Republic 170 00 

Phone: +372 740 4533 

This report is brought to you by: 

Jevgeni Kabanov, Oliver White, Toomas Römer 

Ladislava Bohacova 

42

DEVELOPER PRODUCTIVITY REPORT 2013

Create successful ePaper yourself

Delete template?

Save as template?