Welcome!

Recurring Revenue Authors: Zakia Bouachraoui, Yeshim Deniz, Elizabeth White, Pat Romanski, Xenia von Wedel

Related Topics: Recurring Revenue, Microservices Expo, Containers Expo Blog, @DevOpsSummit

Recurring Revenue: Article

Incidents Happen with Microservices | @DevOpsSummit #DevOps #Microservices

No matter how resiliently you build your application, incidents will happen

Incidents Happen, Even with Microservices
by Eric Jeanes

No matter how resiliently you build your application, incidents will happen. And as your application grows, the complexity of troubleshooting and resolving an incident also grows. With prior virtualization technologies, this might involve a virtualization admin, a network admin, a storage admin, and an applications developer to sort through the issue, put a resolution in place, and determine a root cause. With microservices architectures and Docker containers, this complexity is taken to the next level.

Before an incident occurs, it is important to remember that the customer experience is paramount. If the infrastructure underlying the application can recover from a fault, or if the infrastructure can recover the portion of the application at fault, and this recovery is transparent and seamless to the user, then it is a job well done, and will result in less middle-of-the-night calls for the team. These kinds of failures can wait until morning. Resilience is a key first step in any incident management regime.

Step two is discoverability. Inherent in microservices architectures is the ability for each service to discover the other services it needs to find. It is important for the operations team to be able to find services that are underperforming or in a failed state. This is done with tooling, but also with strong documentation. Inline comments are very helpful for troubleshooting at a microlevel, but at a macrolevel, configuration documentation is invaluable in decreasing  time to resolution.

When designing a microservices solution, it is also very important to limit non-isolation patterns (or eliminate them entirely. Microservices should only interact with each other through a strictly controlled, and well documented RESTful API. If the service is not self-contained, or shares a database with another service, it can exponentially increase the amount of time required to resolve the incident.

Sharing a database is a problem waiting to happen, an incident waiting to begin.
An application leveraging microservices should be relatively easy for a developer who has worked on the service to troubleshoot and resolve. If the code is well documented, it should also be relatively easy for other developers to review the service and discover the issues. This is another perk of having hard boundaries and single processes contained in one service.

At this point, the issue has been determined not be internal to the service. The next step, following an incident management process, is an escalation to the engineering team to review the state of the environment and see if there is an issue that underlies the service itself. This can happen for a number of reasons, but is most likely an issue with the most recent deployment of the service, or an underlying configuration change that was not captured by the change management process.

After the incident has been resolved, it is always important to debrief. In the heat of the moment, people forget to document a solution to the issue, and the priority is usually simply to resolve the issue. A regular debriefing after any incident helps to ensure there is information on how the issue first can be triggered, and if it occurs again, how it can be resolved is captured. A mandatory debriefing also helps inform a continual improvement process to further harden and refine development and deployment of the application.

The post Incidents Happen, Even with Microservices appeared first on PagerDuty.

More Stories By PagerDuty Blog

PagerDuty’s operations performance platform helps companies increase reliability. By connecting people, systems and data in a single view, PagerDuty delivers visibility and actionable intelligence across global operations for effective incident resolution management. PagerDuty has over 100 platform partners, and is trusted by Fortune 500 companies and startups alike, including Microsoft, National Instruments, Electronic Arts, Adobe, Rackspace, Etsy, Square and Github.

IoT & Smart Cities Stories
Darktrace is the world's leading AI company for cyber security. Created by mathematicians from the University of Cambridge, Darktrace's Enterprise Immune System is the first non-consumer application of machine learning to work at scale, across all network types, from physical, virtualized, and cloud, through to IoT and industrial control systems. Installed as a self-configuring cyber defense platform, Darktrace continuously learns what is ‘normal' for all devices and users, updating its understa...
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throug...
CloudEXPO has been the M&A capital for Cloud companies for more than a decade with memorable acquisition news stories which came out of CloudEXPO expo floor. DevOpsSUMMIT New York faculty member Greg Bledsoe shared his views on IBM's Red Hat acquisition live from NASDAQ floor. Acquisition news was announced during CloudEXPO New York which took place November 12-13, 2019 in New York City.
OpsRamp is an enterprise IT operation platform provided by US-based OpsRamp, Inc. It provides SaaS services through support for increasingly complex cloud and hybrid computing environments from system operation to service management. The OpsRamp platform is a SaaS-based, multi-tenant solution that enables enterprise IT organizations and cloud service providers like JBS the flexibility and control they need to manage and monitor today's hybrid, multi-cloud infrastructure, applications, and wor...
The Master of Science in Artificial Intelligence (MSAI) provides a comprehensive framework of theory and practice in the emerging field of AI. The program delivers the foundational knowledge needed to explore both key contextual areas and complex technical applications of AI systems. Curriculum incorporates elements of data science, robotics, and machine learning-enabling you to pursue a holistic and interdisciplinary course of study while preparing for a position in AI research, operations, ...
Codete accelerates their clients growth through technological expertise and experience. Codite team works with organizations to meet the challenges that digitalization presents. Their clients include digital start-ups as well as established enterprises in the IT industry. To stay competitive in a highly innovative IT industry, strong R&D departments and bold spin-off initiatives is a must. Codete Data Science and Software Architects teams help corporate clients to stay up to date with the mod...
Tapping into blockchain revolution early enough translates into a substantial business competitiveness advantage. Codete comprehensively develops custom, blockchain-based business solutions, founded on the most advanced cryptographic innovations, and striking a balance point between complexity of the technologies used in quickly-changing stack building, business impact, and cost-effectiveness. Codete researches and provides business consultancy in the field of single most thrilling innovative te...
Atmosera delivers modern cloud services that maximize the advantages of cloud-based infrastructures. Offering private, hybrid, and public cloud solutions, Atmosera works closely with customers to engineer, deploy, and operate cloud architectures with advanced services that deliver strategic business outcomes. Atmosera's expertise simplifies the process of cloud transformation and our 20+ years of experience managing complex IT environments provides our customers with the confidence and trust tha...
With the introduction of IoT and Smart Living in every aspect of our lives, one question has become relevant: What are the security implications? To answer this, first we have to look and explore the security models of the technologies that IoT is founded upon. In his session at @ThingsExpo, Nevi Kaja, a Research Engineer at Ford Motor Company, discussed some of the security challenges of the IoT infrastructure and related how these aspects impact Smart Living. The material was delivered interac...
Intel is an American multinational corporation and technology company headquartered in Santa Clara, California, in the Silicon Valley. It is the world's second largest and second highest valued semiconductor chip maker based on revenue after being overtaken by Samsung, and is the inventor of the x86 series of microprocessors, the processors found in most personal computers (PCs). Intel supplies processors for computer system manufacturers such as Apple, Lenovo, HP, and Dell. Intel also manufactu...