Ramblings from a Software Developer Half Way through their Career

As I reach the 1/2 point in my career developing software, I started thinking about where we are at in the industry of Software Development and how much progress (or lack thereof) as well as some observations and things I’ve learned.  If you would have asked me 10 years ago if I would be at the half way point in 2009, I would say, heck no, I’ll be retired and doing this for fun…so much for the internet bubble and stock options.

   

The following are some observations and thoughts on some things I've learned at the different positions and work I’ve done with the last 19 years.  If you make it to the end of my "ramblings" drop me a note and let me know what you think.  

  

After graduating from Minnesota State University in 1990 with a degree in Computer Science and Electrical Engineering Technology, my first job was as a Software Engineer responsible for designing and programming embedded systems for weight and force measurement devices.  We had about 5-6 Software Engineers on staff, each week we had meetings to discuss ways we could improve the process of developing software.  This was a long and painful process where our primary objective was to build consensus on development standards.  We started with coding conventions for FORTH, C & C++.  What made this extremely painful is that as we all had opinions (although only a couple of us actually programmed in C/C++ at the time) and at one point we spent weeks discussing putting curly brackets around single lines after an if statement.  Anyway, my point is that although it was important to build consensus this approach wasted way too much time, and although I think we did make some modest gains, these meetings were probably better held in an academic setting.  I had an excellent mentor and the company probably still doesn’t want to know the number of hours the two of us spent shooting the sh*t discussing software engineering and software architectures instead of programming devices.  Luckily the two of use seemed to be the only ones to ship on time so I guess they didn’t mind too much.  I considered those years as almost a “Masters” level education.  Some of the observations/lessons from this position:

• Designing the hardware, writing the low-level drivers and then the software that made the microcontroller do something meaningful is truly an enlightening experience.  It still influences how I program in high level languages such as C# today.

• Another invaluable lesson learned at this job was the difference between sales and marketing.  And the good, the bad and the ugly of working with other areas within a company.

• Get your developers in front of the customer and in the real world as much as reasonably possible.  It puts a different perspective on things when you see your software run a huge rock smasher or rail-road train scale instead of turning on $0.05 LEDs in the right sequence that simulates the actual hardware.  This is also true if your developers experience is limited to only seeing a 1 with a bunch of zero’s that represents a decimal data type on a sales order instead of a real order for a gazillion dollars that represents real money.  It's an eye opener to see how real users actually use your software.

• Keep your developers coding, and then have them code some more and when they are done doing that, have them code a little more.  As with any craft you get better with experience.

• If you don't know where you are going a map is no help, developers shouldn't be the ones making up the requirements, if they are get them in front of the customer as much as possible (note the conflict with the previous item)
• Software estimating even on small projects is very difficult.  Estimates from developers should be doubled.
• When marketing drives certification efforts such as ISO9000, the results are very shallow.

Next I went to work for a Sales Force Automation company, starting out as a software engineer and ending up with the title of “Director of Tools and Technology”.  I had two primary roles as a director, first was to dispatch fire fighting teams get projects out of the ditch and the second was to put in a set of processes and tools to help keep those projects out of the ditch in the first place.  I had the luxury of hiring my own senior staff.  I’m a firm believer that right people are single most influential factor in the success for failure of a software project.  I actually found that people with engineering degrees in some respect were better suited for this type of work than ones with a pure comp-sci degree.  In my previous job I was introduced to the classic Managing the Software Process by Watts Humphrey this is a classic must read book.  A number of concepts seemed to make considerable sense when working with large projects with a duration of 1+ years and teams of 6-12 developers.  Much more so than the single person 3-4 month projects at my previous position.  One of the lessons learned here is that different types of software require different set of processes and talent.  Some other observations from this position:

• Developing large systems is just plain hard, there is no silver bullet, but with the right people it is possible and can even be fun.
• Team building is important to the success of your project.  Finding the right chemistry is worth the effort, bad apples or “negative producers” must be swiftly removed from the team.
• It’s really all about data, for the type of systems we were building (and even the ones I build today) getting the data model right will make or break your project.
• While the business analysts are figuring out what to build, it's a good time to have your senior and lead developers plan out your architecture and figure out what common functionality needs to be organized into the application frameworks.
• It's worth the investment to build scaffolding to test your application.  At the time we weren’t doing any formal unit testing, but had all sorts of little tools and utilities to help make our life easier.  Really a necessity when the MFC compiler took 45 minutes to do a full build.
• I was formally introduced to “Design Patterns” and the GoF.  Coming from an engineering background, I had always thought Software Development needed to mature and “Design Patterns” were going to be our building blocks.
• Platform Independence sounds good when on the marketing cut-sheet’s but in reality…well…you know the story here.
• Senior, top level management buy-in and participation is absolutely critical to the success of any software process improvement effort.  The primary reason for this is there are more departments in your organization other than development and everyone needs to play together nicely to ensure successful projects.
• When spec’ing and selling multi-million dollar projects, make sure you have representatives from the dev staff involved with pre-sales.  Without that I’ve found there is usually an expectation mismatch, this is true in smaller projects but critical in larger ones.
• Our company was organized into Product and Professional Services divisions, using real world requirements from Professional Services goes a long way to help product development planning.
• There should be an inverse proportion of the amount of code you write with respect to your level within the management food chain.  When I started as a software engineer, I spent 90% of my time coding.  As a director, I spent about 5% coding if I was lucky.  The challenge here is to keep your skill sharp, you need to code, period.  To be a good manager you need to manage period, if you are coding you are doing your employees a disservice.
• In our industry, a manager works for their employees.  That is to say a managers first and foremost responsibility is to equip her coders with everything they need to succeed and aggressively remove road blocks.
• Although difficult, setting and managing expectations is critical to the success of delivering software.
• Although I’m sure the agile folk’s won’t agree, I’ve found the longer I can stay out of the “problem domain” and keep working on my application frameworks (factoring out common/abstract functionality) the sooner I’ll ship my software.  This deserves another post to explain in more detail.
• Technical solutions generally don’t work to fix process problems
• Towards the end of my time there I had a keen understanding of why executives “don’t like surprises”.
• If you get a stock option grant (especially one big enough to retire with) it's a good idea to get a lawyer involved early on so you don't end up with Toilet Paper.
• It’s all really about People, Process and Technology

 

PPT Solutions, LLC

Three of us left the Sales Force Automation company and started our own venture it was called PPT Solutions (People Process & Technology, in that order as we liked to say).  We put together our business plans, cash flow projections and since this was in the internet bubble we were convinced there is no way we could fail.  Although I wouldn’t consider this business a success, we did manage to ship a product, and I did learn a lot about going into business for yourself.

• We were told going into a business partnership with friends was a bad idea, we thought we were different.  We weren’t.  The reality is that someone needs to be in charge and equity needs to be based upon contribution to the company.  Friendships complicate this, especially in our industry where we think we are all “rock-stars”.  Eight years later the wounds from this are finally healed.
• Sales is critical, sales is not easy, sales requires a different skill set than developers, we sold our product into Imation (3M Spin off and electronic media company) I think we landed the sale because I was sleeping with the global sales training manager (who I've been married to for the last nine years and still madly in love with).
• Although it sounds good to get a sale and then use that money to develop a product and then market it that way, in reality the product will contain all sorts of very specialized features for the company paying for the majority of the effort.  Care should be taken so that specialized requirements don't contaminate the core product.
• Make your estimate than double it, both time and dollars
• It’s a double edged sword working out of your house.  It's nice, but it requires discipline to stay focused.  Having weekly goals is a good idea.  You will probably also be working on weekends to a certain extent.  Do whatever possible to separate your work week from your weekend, this helps burn-out.
• If you partner with other people to start a business, clearly define expectations and responsibility up front.

Next up was a small startup, let's just say I don't have a lot of good memories from that place.  This wasn't really a technology learning experience, however I did learn how to successfully get what I needed in a difficult environment to ship software.  The company came in at the tail end of the internet bubble and had visions of making a pile of cash with very little effort and a lot of 0's and 1's in the right order.  Enough said…moving On….

 

That takes me up to what I'm doing today, Software Logistics, LLC.  My company.  I don't have any employees, but use contractors as necessary.  I seemed to have found a nice niche.  I spend about 90% of my week coding.  Although I do miss the thrill of leading a team into battle, I guess the pure joy of carving out excellent software makes up for it.  I've worked with very small companies and very large companies during the past eight years.  Each has their strengths and weaknesses and enjoy working with both.  This has been a great learning experience and I’ve shipped a number of projects in the process.

• To get started, networking and word of mouth is very important.  Even more so is networking with the right people.  There is time for friends and time for business.  As with my previous venture, these don't often mix very well.  That doesn't mean you can't have fun with the people you work with, however since this is going to be your primary source of income it's important to stay objective.
• Even though you are out on your own, that doesn't give you an excuse to cut corners or skip basic SCM principles such as source control, issue tracking, build environments etc…  What's nice about being on your own is you don't have to reach consensus and you can experiment with different processes and technologies.  Although still in evolution, my CI and deployment environment rivals if not exceeds most of companies I work with.
• Invest in yourself.  I attend at a minimum two conferences a year.  Even those these are on my own nickel this is a must.  Going to conferences allows you the time to focus on what's new and get a quick overview of a wide variety of technologies.  Don't expect to become a master at any of these, but you should get an idea of what's real and what's worth investing more time in.  Even more important is getting a sense of that latest flavor of the month so you don't waste your time or sabotage your project.
• If done right, it's generally important to come off strong in the first 15% of the project and finish strong in the last 25%.  This doesn't mean you go "dark" but during the development phase it's a balance between keeping your client comfortable with the progress and staying out of the details enough to keep on your schedule.
• Communications is critical, if I had to say there is one common denominator across all the clients I've worked with is that they all need to improve their communications.  There needs to be a balance between using tools such as a defect tracking and verbal communications.  SCM work tracking tools are good for capturing and managing a large amount of details, however even with a "priority" ranking, they just don't convey the sense of urgency and importance that verbal communications can provide.
• When in a meeting (even in sunny Tampa) no-one likes to see your arms, wear a nice button down dress shirt and khaki's.  Remember you are a professional, impressions matter, especially when negotiating the yearly rate increase.  I don't think it's important to drive a BMW, but you should project a feeling of success.
• You can't generally rely on the QA of your clients.  If you can't do it yourself (which is very difficult if not impossible) hire someone and get a good issue tracking system.  I really like Visual Studio Team System, if you have a premium subscription, you should be able to leverage the workgroup license.
• Hold frequent status meetings, you should run these meetings, always have an agenda.  Set the expectations as to what you will cover in your agenda and ask for feedback if they have any concerns.
• Respect your clients time, most people have more work than time.  At the start of the meeting ask for how much time you have and adjust your schedule accordingly.
• You are viewed to be the expert, and to be successful, you should be a jack of all trades and a masters of a lot of them.  However it’s impossible to know everything.  If you don't know something don't BS your client.  Most of the time the people you are working with can tell, you need to keep your credibility and trust.  If you don't know something, find out and make sure you follow up promptly.
• Your demos will almost never go as planned.  I remember one demo that I practiced a number of times and it worked flawlessly, I got to the client site and they had custom DNS settings on their internal network.  It broke my demo.  Think quick, think on your feet, and adapt, in my case a simple tweak of the HOSTS file was all that was necessary.
• At all costs do not get involved in company politics.  All companies have them.
• Don't expect the same level of support/services from other people at your client as you did when you worked as an employee within a company.  People are generally overworked and requests from the consultant come after requests from their internal managers.
• If you are billing hourly, make sure you have a good system for capturing and tracking your time, even if your client doesn't ask for it.  Never over bill your client, if your client isn't paying you enough to live on, ask for more money find an additional or a different client.  Remember it's your name on the line.
• If a client pays you to build software, they own the source code (unless you work out a different arrangement).  Don't use that source code in a different project or for a different client without their written approval.  That isn't however saying that you can't re-implement the same algorithm, they own the source, not the concept with certain obvious exceptions such as industry or trade secrets.
• I've found that there is no right answer to the right amount of documentation you should provide.  I had one client that complained about the level of documentation I produced.  When it came down to detail and specific documents with senior management, I could always produce the documents in their repository and the response was "I didn't know we had that".  I don't think the documents were even looked at.  I guess my point is for a developer, documentation is no fun.  If no one is going to read it it's your call on how much to deliver.  It's good for CYA.  Another client I have uses a WIKI for documentation, nice and simple, yet effective.  If your client doesn't have one, set one up for them on your server.  If it gets traction install on their server.
• Don't skimp on your computer setup and hardware.  Time is money, would you rather spend your time installing and configure software, or would you rather cut code and get paid.
• I've found that using a tool like VMware Workstation works great for doing client work.  I have a base setup that has all generic development tools.  I can than copy that base VM and configure it with the specific tools and source to work on client projects.  An added benefit here is that at the end of the project, you can always just give them the VM and they can pick up where you left off.  My main dev boxes generally are multi-core and very fast, when I'm working in a VM, I just can't tell a difference.
• If the environment is right, offer more than you ask for, I really enjoy mentoring young developers, and showing them new tools, technologies and techniques, be careful here though not to step on other internal developers toes.

If I haven't bored you too much and you made it this far (I'm a coder not a writer), I can offer some general thoughts of where we are as an industry:

• Every project is different, yet there are many similarities amongst different projects.  If anyone says they have the one true way to develop software, you can tell them, ok, whatever, let me buy you beer and let’s poke holes in the process for what I’m working on.  There are also differences in techniques required when working on the same project.  An example is a non-trivial LOB application, there is the application frameworks, data layer, business objects, UI library, each of these is different and requires a different approach and the investment in time and design.  My point here is that there seems to be a lot of great work done developing tools, technologies and processes, we just need to start doing a better job of discussing where they should be used.  And general sweeping statements such as any code with out tests is considered “legacy” (read crap) don’t do much for helping us bring the right tool to the right problem.
• I'm not sure there are any metrics out there, but I would suspect the majority of developers out there work on what I would consider Line of Business (LOB) type of applications.  Generally the most difficult part of this development is the ability to track and manage changes to the business rules and requirements.  I've seem a few implementations, that probably were much more complicated than need be.  Sometimes it's fun to pick up the latest technology or pattern when all you really needs is some simple OO principles and a nice Forms over Data application.  Some of us need to build Jaguar's and some of us need to build F-150's.
• I mentioned it above, but one of the core problems I see in our industry today is the lack of effective communications, this was true back in 1990 and is still true today.  This may not be true of all companies, but I would suspect it is for the majorities of the companies out there.  I think Software Development can be described as managing thousands of little details, understanding how they act together and how to effectively communicate those details with others.  We do this by a series of translations; users translate their needs to a user advocate, the user advocate translates those needs to a business analyst, the business analysts turns those into requirements and prioritizes, the developer picks up those requirements, the develop translates the requirements to a design, then the design into code and finally the compiler turns the code into something that can run on a computer.  Even if you effectively implement agile techniques (which I suspect most companies that say they are agile really aren't) this is just plain difficult.  Take into account Fredrick's Brooks law of adding man power to a late software project makes it later and you get an idea of the scope of the challenge.  So, you say, what's the answer?  I'm not sure there really is one.  What I do know is some people are better at this than others, I think this is probably an important attribute that is overlooked when hiring.  Sure you may be a rock star coder, but if you don't code the right thing, oops…what's the point?
• Everyone in our industry thinks they are a "Rock Star" in reality very few people are.  Another theory from the class Mythical Man Month is that there is an order of magnitude between an average developer and ones that are truly "Rock Stars".  I’ve found this to be the case.  On some big projects I’ve worked on I’ve found that 20% of the team completes about 80% of the work.  The trick that I've learned is accepting that I'm probably not one of the “Rock Stars” and as the Verizon quote goes try to "Make Progress Every Day" into becoming a better developer.  This includes reading all the classics, writing tons of code that gets into production.  Coding for fun generally glazes over the details, details are what makes software hard.  I've also found it to be beneficial to work with really smart people, they push you, you push them.
• I still keep in contact with the ninja's that worked for me to setup what we had thought was a decent development organization.  Since then, we went down two different roads, they went to work at a huge enterprise, where the basic idea is to manage change and make sure the gazillion lines of code compile and bugs don't cost too many millions of dollars per year.  I'm cuttin' code pretty much every day.  I don't know which path is better, I'm not sure there is a better path, I know I love what I'm doing now.  When we get together a couple times a year, it sounds like progress is being made in managing change, however the systems are getting more and more complex.  The net result is that things are just as screwed up as ever and the same fires we put out back then are still flaring up however they are handled much better.  Kudos to them, I have a lot of respect for what they do.
• It appears that more and more companies tend to rely on libraries and third party and open source technologies.  I think this is both good and bad, if you are working on a small 1-2 month project go ahead and use the 3rd party building blocks.  If you are working on a large system, it may be better to re-invent the wheel.  I think a good example of this is your Data Access Layer, if you are doing a small one off type of project absolutely use something like Linq 2 SQL or NHibernate.  If you are working on a fairly large project with a number of developers, it may be worth it to carve out your own DAL.  This is a non-trivial process, but 6-8 months into the project, you should recoup the investment by having exactly the feature set you need and understanding the inner workings of the critical piece of technology.  Spend the time up front, think through your architecture and requirements.  Realize that those 1/2 hour demos you saw where someone created a CRM site with just 5 lines of code, probably leaves out a considerable number of details.  Sometimes just dropping in a library introduces problems as noted by Joel Spolsky's Law of leaky abstractions. 

Well I guess I should probably close as I could ramble on for another few pages, I think in summary I would have to say that our industry is maturing with new processes, technologies and techniques but at the same time the systems are developing are getting much more complex so I would say our overall progress as an industry is probably flat.  At one time I had thought we would mature to a more engineering approach to delivering software but I think with the dynamic nature of the problem space and technologies, I'm not convinced that will happen anytime soon.  As to being at the 1/2 way point in my career, I’m still trying to build the “Killer App” who knows….

-ec