Development and IT in the Era of Agile need to evolve.
I am reading LinkedIn, following up on various companies, and often come across the term DevOps in various flavors. The term is popular but there is a great difference in what it means for different people as goals are changing.
For many years we knew very well the role of the IT department and of its important responsibilities. On the other side of the “fence”, we had software development teams.
Time of Agile
Software development went through drastic changes in recent years – waterfall, XP, and now Agile. The main goal of these changes was to improve the effectiveness of the software development teams. It involved the proactive discovery of requirements and developing solutions through the collaboration of self-organizing, cross-functional teams and being closer to customers and end-users. Practice includes adaptive planning, incremental development, early delivery, and continual improvement. Agile encourages a quick reaction to changes in requirements and resource availability.
Development has been already absorbed and incorporated the Quality Assurance team. Code delivered must be stable and a rapid development cycle requires close collaboration between two teams. We are going through a new “merger” now.
The software does not exist in a vacuum. It depends on infrastructure both for development in-house and in production. Here is where the IT team becomes essential in supporting dev activities or becomes a roadblock that slows momentum.
Practice shows that the best operating and effective team must blur boundaries between Development, Quality Assurance, and IT. This is where the DevOps term is born.
IT is not DevOps, neither DevOps is IT. As we will discuss below, DevOps is a robust process that incorporates an entire stack of teams in the software company and elevates it to the next level.
IBM defines DevOps process as “a software development process and an organizational culture shift that speeds the delivery of higher quality software by automating and integrating the efforts of development and IT operations teams – two groups that traditionally practiced separately from each other, or in silo”. It does not however stops there and must extend across other essential teams – Quality Assurance, security, compliance, product management and more.
DevOps is a compound term for Development (Dev) and Operations (Ops). It is not the siloed team responsible for specific “it”, but the versatile aggregate of various roles in your organization allowing bring everyone together with one goal – the ability to better respond to customer needs, streamline the development process, and achieve business goals faster. The end goal is to build better products faster without jeopardizing customer satisfaction.
It is essential to understand that you cannot simply rename the IT team as DevOps, nor can you make the dev team responsible for DevOps activities. As with the Agile process in development, there must be a mental shift from a linear approach (transition from Waterfall to Agile in software development) in the IT department. For the dev team to move fast, the IT team must have buy-in and be able to move fast as well.
There are a few considerations that must not be forgotten or overlooked – IT teams traditionally are responsible for IT infrastructure security and compliance. This is a very regulated and tedious area that requires careful orchestration and practices to ensure the stability and security of the software and client’s data. Said that it is not prohibitive to structure IT activities to incorporate agility. If anything, it is easier, in my opinion, for IT to be agile in the era of cloud computing and infrastructure than it is for some software engineering teams.
Continuous Integration and Continuous Delivery become cornerstones of any successful development team build. It ensures that repetitive tasks are automated – from continuous automated testing to the ability to certify and deploy new versions.
Corner Stones of DevOps
The same IBM white paper defines the following steps integral for any development:
- Planning – identify and scope new features for the current dev cycle based on company’s product goals and client’s feedback. Rapid iterations allow for some flexibility and “catch up” approach in requirements as next cycle will bring more feedback and more clarity of what need to happen.
- Development – each team can implement their own steps of getting from A to B in implementation and choose between varieties of approaches – test driven development, behavior-driven development, contract programming, etc.
Main thing style must accommodate ability fast deliver quality code in each iteration.
- Integration – CI or Continuous Integration – You cannot ensure rapid development without clearly defined and automated way to test and ensure that software is not broken in each iteration. Yes, you can beef up your QA team to allow for full regression testing in reasonable time, but to ensure that testing fatigue would not set in, automation is a key. It is a goal for QA team to spend reasonable time on scripting test where and as deep as possible so unit and UX testing can be plugged into Build process.
- Deployment – CD or Continuous Deployment – all above steps are meaningless if software cannot be deployed properly ensuring all setting and prerequisites are met. Deployment downtime only can be limited if and when there is reasonable amount of automation in deployment and deployment validation. While humans are perfect and we can do many things over many cycles, assurance that all deployment steps are accounted for is possible only through automation. If you have to provide SOC compliance evidence, this step becomes even more important and evidence collection is automated and repeatable.
- Operations – After production delivery development team’s role could phase out. Performance and Security monitoring, along with audit, and client support becomes routine day-to-day activity. It is important to mention that in agile cycle developers must be very close to client’s pain, be able to live and feel it every day. Only that way software will evolve in organic way. This leads us to …
- Learning – CF or Continuous Feedback – teams must talk. You must demolish walls, you must have buy-in from all teams involved. It is essential to ensure that there is no “broken phone line” between clients, development, IT and QA teams. Exchanging information in most effective way is crucial. While learning from client’s feedback and any discovered errors we are going back to step 1 – Planning – above. New cycle starts.
Agility of Agile Process
It is important to recognize that the Agile process is not formal to the same degree as the Waterfall models of the past. Your team must be flexible and adopt practices best suited for your product, your organization structure, and your budget. This is the beauty of the Agile approach – you can scale up, scale down, expand and shrink your iterations to gain as much as possible from it – be flexible.
Agile Process and Tools
None of the above is possible without proper tools which will help to streamline and automate DevOps process adoption. Any team needs all or most of the tools to be able to automate steps in the above list – Source control tools, test and build automation tools, team management, communication, workflow, and visualization.
I am not going to spend much time here discussing available options because tomorrow there will be new and better tools as the software stack constantly changing. Once you decide to adopt the Agile approach, please spend proper time deciding on tools that will fit your budget and your requirements, but do not overlook the importance of tools that will make your Agile and DevOps journey easier. Looks at GCP, AWS, and Azure offerings if you are in the cloud, or even if you are not. But do not stop there, there is plenty of good open-source and proprietary tools on the market and with each day new products would appear. The main thing to remember is to ensure that what tools you would settle on should not restrict your ability to continue to evolve.
As DevOps methodology evolves new variances start to appear. One of the recent extensions represents growing pain associated with security – DevSecOps – in addition to focusing on the collaboration of Development and Operations teams, the Security team is folded in to ensure that software is designed and proactively addresses growing concerns associated with data breaches.
Who is DevOps Engineer
While DevOps is more of a business philosophy than anything else, DevOps Engineer is still a real thing in the job market today. Companies would target these positions to cover the necessity to program for automation and monitoring tools. If in the past bash script knowledge was enough, with the complexity of cloud environments and integrated systems, the person in this role becomes closer and closer aligned with the software team than with IT in classical terms. Not only knowledge of CI/CD tools is required, but knowledge of the cloud stack, various APIs, and things like Lambda, Python, Java, Ruby, Git, Jira, and even languages and IDEs used by supported dev teams. The boundary between IT and dev is further blurred to the point where there would be very little distinction between knowledge required.
Transformation is complete
When adopting the DevOps methodology software organizations must be ready and accepting of the fact that everyone is working toward one goal, there are fewer and fewer inter-team boundaries and the distinction between IT and development disappears. For the organization to be effective in today’s world we must now focus more on clear communications, better requirements, and eliminating “org silos”.