The codes are a-changin’!

by Abhishek

FYI

via Digg the Blog by Joe Stump on 10/16/08


Hi all –

Later today Digg will be undergoing some significant updates, but upon first glance, it may seem that nothing has changed. That’s because the changes included in the coming release are 100% under the hood. For the last few months, Digg’s developers have been heads down scrubbing, cleaning, porting and improving the core code of Digg. Before I get into the details, I want to give a big thanks to everyone on Digg’s development team. They’ve done a great job with what, technically, is one of the largest pushes Digg’s released.

Digg’s development team has grown tremendously over the last few years to over twenty developers (we’re hiring!). At the same time, Digg’s code has been maturing from a monolithic ad-hoc approach to a set of consistent frameworks. Our goal was to standardize the way applications at Digg are written, managed, and deployed. Sounds fun, but how did we do it?

– During the profile redesign, we also rewrote our login and registration system using an internal event driven framework we call App. It’s a small, simple framework for quickly creating applications. App implements the ‘V’ and ‘C’ in ‘MVC’ along with input sanitization and authentication. Basically, it allows Digg developers to rapidly develop applications without worrying about the basics. As of this latest push we’ve ported the majority of Digg’s applications into App. Every developer at Digg has had a hand in porting thousands and thousands of lines of code to App.

– We also have another framework, called AJAX (I’m not very creative at naming my frameworks), that manages all of our AJAX endpoints (the PHP code that processes and responds to AJAX calls from jQuery). AJAX allows our developers to create a simple PHP class to process requests while the AJAX framework handles JSON and AHAH encoding, token checking, authentication, XSS/CSRF checks, input sanitization, HTTP error codes, exceptions, etc. Before this push about 70% of our endpoints were in the AJAX framework. As of this forthcoming push we’ve managed to port 100% of our endpoints to AJAX.

– One of the major problems at Digg, with regards to development processes, was that our code was monolithic. With three developers this isn’t a problem, but with 20+ it can get hairy. It can make merging difficult, interdependencies impossible, and make it impossible to promote/enforce ownership within SVN. In response to this, all of our core frameworks, applications, etc. are now PEAR packages. For those who don’t know, PEAR is a package management system for PHP, along with being a world class repository of PHP libraries. This allows us to break our code up into dozens of smaller projects which are owned by individuals and teams. PEAR also allows teams to define specific PEAR, PHP and PECL code that it depends on. What this means is that App_Login can enforce dependencies on App, Message, Mail, etc. and deployed separately from App_PermaLink or App_StoryList.

– Since our VP of Engineering John Quinn has joined the team, he’s been spoon feeding us the Agile pixie dust. Part of the Agile philosophy is unit testing, and the result is that all code at Digg must now have unit tests which are fed into our continuous integration environment. Our CI environment automagically checks out code every ten minutes, runs unit tests and verifies the code conforms to our coding standards, and then emails developers of any problems.

So, despite the appearance of nothing changing, big changes have been afoot in Digg’s development department. Tens of thousands of lines of code have been rewritten, massaged, ported, unit tested and modularized. But, why? So we can continue to develop quality new features at a quick pace even as we continue to grow our development team.

Thanks!

– Joe