Best practices for robust enterprise level software development (using python Django) [closed]

Question

We are building a software project (using python Django and a bunch of other Open Source technologies). The project is fairly big. To give an idea, it has over 50 templates.

Our team is composed of people inexperienced in Django. Its the first django project for all of us. As such, whenever a feature is wanted, everyone just follows the first approach that comes to their mind, with hardly any idea of its consequences on what has been done earlier. Nobody looks at the big picture. As a result, things that were working in the past break down often.

We have two major problems.

1. First, as mentioned above, what worked earlier breaks later.
2. Second, when the developers are given features to implement, they do that on their own computers and show them to be working in the weekly demo to the client. But when these components of the project are integrated into the main part, something or the other breaks.

At present, we have somehow managed to finsh half of our project (also, the project has been delayed by 1 year). Half of it still remains. Also, a new project has come up. I have been given the task to identify the root causes of the above two issues and suggest how to minimise them. If we do that, the new project would be developed in a better way.

Integration testing with automated tests is one way that I can think of to catch the bugs. But my aim here is to somehow prevent them from being created in the first place, at least minimise them.

How can I increase robustness of our code. In other words, what are best practices for enterprise level Django projects ?

Can someone, specially people experienced in managing projects please suggest me the right approaches or point in the right direction ?

Answer 1

1. set up Continuous Integration and (as you noted) automated unit and integration tests
2. get the team agree on a common Definition of Done, including that no feature is completed until it is integrated back into the whole system and is proven to work there
3. get better Django developers by one or more of
   • arranging training for your existing developers,
   • hiring some experienced devs and using them at least partly as mentors / coaches,
   • introducing pair programming,
   • promoting and supporting self-learning and sharing knowledge with each other.
4. start regular retrospectives to ruthlessly analyse every single problem and failure, get to the root causes and eliminate them

The order of these can be debated, ideally you should start doing all of them as soon as possible :-) There probably are a lot more potential improvements, but I think these are the key actions that may get you a long way ahead.

If you want to understand better the principles / phrases used above, you may want to read up on agile and lean software development.

You may miss suggestions specific to Python or Django; other answerers may (and hopefully will) give you such advice. My perception is that your problems are not specific to Python or Django, rather systemic problems, so solutions must be systemic too.

Answer 2

I see two problems, and you've identified one of them yourself: nobody has technical oversight of the system as a whole. Things are breaking in integration because nobody is thinking about the whole system until that point. The other is that your people appear to be functioning as a collection of solo developers, rather than as a team.

Unfamiliarity with Django is a bit of a red herring. A bigger problem is that your team doesn't have enough familiarity with the code you yourselves are writing.

Your developers don't know the APIs that will break things if they are not respected, or the assumptions in the design that will break things if they are violated, or the approaches that other people have already taken to solve similar or related problems.

Programmer's Documentation

The first two can be documented, in the code itself and/or in a developers' guide. Use a tool like Sphinx that can auto-generate documentation from properly-formatted docstrings.

Writing tests that focus on the APIs is a simple practical exercise that will both start to develop that knowledge (if you have everybody do some of it) and will help prevent some of the integration problems you're having (if you mandate running these tests before commit).

Code Reviews

For the third, you need to get everybody's eyes on more code than just their own. Start requiring code reviews for any change that is non-trivial. Use an online tool like ReviewBoard or GitHub's pull requests that let reviewers make comments inline, as well as general comments. Make sure people post code for review when it is in "first draft" stage rather than "polished and finished", and focus discussion on how and why things were done a certain way, pros & cons of different approaches, rather than on style and adherence to coding standards. Set expectations that the outcome of the review may include significant rewrites, if a better approach is identified during review, and that blocks of code that confuse reviewers should be commented with explanations.

Hopefully your team realizes they are inexperienced with the technology they are using, and want to improve their skills. Pitch the code reviews as an efficient way to build up and share knowledge (and avoid like the plague anything that smacks of competition or shaming). Make sure they understand that regular reviews are part of their job as developers, not an optional addon. If your people are charging or tracking work for each feature, they should charge the review work for the feature under review: that will also help with a sense of team ownership of the whole product. Eventually, you should start to see some common approaches to problems emerge, as your devs start to recognize that this problem is similar to that one that was implemented in the last build, and can be worked in a similar way.

Planning Poker

I'd recommend you do something similar up front, too. You didn't say anything about the process by which features are assigned to developers, but bring that into a team process as well.

My team has found that Planning Poker is even more valuable as a structure for technical discussion than it is for its nominal purpose of estimating effort. It gets us all talking about the scope and complexity of each task before we try to estimate it; then when estimates vary, it is often because of differences in approach that get discussed. This will help with your problem of individual devs implementing the first approach that comes into their heads. It will also help with the code reviews, because all your devs will have been in on the prior technical discussion.

Given that your people are still inexperienced with the tech you're using, make sure you circulate the list of tasks to be estimated a week or two in advance, to give people a chance to get familiar with the feature request and read up on possible approaches in advance of the game.

Once all the tasks have been estimated, I typically let my team divide the work up among themselves, as long as the effort and the high priority tasks are evenly spread. (Of course we may reallocate as the development cycle continues, if it turns out that some work took more or less effort than estimated.) Again, avoid anything that smacks of competition or shaming.

System Oversight

These two practices will gradually help your devs be aware of the big picture and collectively perform the necessary system oversight, but it will take time.

In the meantime, if possible, I would consider identifying one or two people in the team who are good at thinking systemically and whose expertise is generally respected, and task them with providing system oversight: understanding the architecture & APIs, and identifying and curating common approaches to common problems. This person should participate in all code reviews, if possible; not as the lead reviewer (that would undermine the goal of spreading knowledge and responsibility), but at least as a final reviewer to raise any system-related issues that weren't already identified.

The team lead can do this if s/he has time; the people who were involved in the initial design would also be logical candidates.

If you engage in code reviews and planning games on the new project from the getgo, there will likely be less need to explicitly task someone as system architect/curator. On the other hand, given the problems you've had in the past, it might be worth doing anyway.

Tests and Retrospectives

Definitely build your team a standard set of tests, including integration tests, that are run every night and that every developer can run routinely while developing. Set expectations that every new feature should be delivered together with new tests, and that developers should test before committing.

Retrospectives after each build/delivery/sprint are an essential element of continuous process improvement. I like to bring goodies to these, to show appreciation and set a relaxed atmosphere. As team lead, I start off by asking the general questions, What worked well? What didn't? What do we want to change about our process next time? Then I mostly just take notes (on the projector so everyone can see) and facilitate the discussion and decisions. It's important to identify the good stuff as well as the bad, and it's okay to have some griping about stuff that didn't go well as long as you also then get some proposed solutions.

There are more structured approaches to retrospectives out there that I haven't tried, though they look interesting. But this very basic approach has worked well for my team.

(BTW, I lead a small team that developed and maintains a medium/small Python Django web application. We were inexperienced with Django when we started. We have used everything I've recommended here.)