Welcome!

Recurring Revenue Authors: Zakia Bouachraoui, Carmen Gonzalez, Yeshim Deniz, Elizabeth White, Pat Romanski

Related Topics: @DevOpsSummit, Recurring Revenue, Artificial Intelligence

@DevOpsSummit: Blog Post

How to Ditch Scheduled Maintenance By @PagerDuty | @DevOpsSummit [#DevOps]

Scheduled maintenance tends to take place late at night on weekends—a tough proposition for operations engineers and admins

How to Ditch Scheduled Maintenance

By Julie Arsenault

You like sleep and weekends. Customers hate losing access to your system due to maintenance. PagerDuty operations engineer Doug Barth has the solution:

Ditch scheduled maintenance altogether.

That sounds like a bold proposition. But as Doug explained at DevOps Days Chicago, it actually makes a lot of sense.

Scheduled maintenance tends to take place late at night on weekends-a tough proposition for operations engineers and admins. Customers require access at all hours, not just daylight ones. And scheduled maintenance implies your system is less reliable than you think, because you're afraid to change it during the workday.

The solution? Avoid it altogether, and replace it with fast, iterative maintenance strategies that don't compromise your entire system.

That might sound a bit ‘out there.' But shelving scheduled maintenance is easier than you think. In his talk, Doug offered four ways to do it.

Deploy in stages

First thing's first: if you discard scheduled maintenance, your deployments need to be rock-solid. They should be scripted, fast and rolled back quickly, as well as tested periodically to ensure rollbacks don't lag.

They also need to be forward and backward compatible by one version. It's not an option to stop the presses when you push out a new version. Red-blue-green deployments are crucial here, as they ensure only a third of your infrastructure undergoes changes at any given time.

Lastly, stateless apps must be the norm. You should be able to reboot an app without any effect on the customer (like forced logouts or lost shopping carts).

Send canaries into the coal mine

Use canary deploys judiciously to test rollouts, judge their integrity and compare results. These test deployments affect only a small segment of your system, so bad code or an unexpected error doesn't spell disaster for your entire service.

Doug suggested a few practical ways to accomplish this:

  • Gate features so you can put out code dark and slowly apply new features to a subset of customers.
  • Find ways to slowly bleed traffic over from one system to another, to reduce risk from misconfiguration or cold infrastructure.
  • Run critical path code on the side. Execute it and log errors, but don't depend on it right away.

As Doug summed it up for the DevOps Days crowd: "Avoid knife-edge changes like the plague."

Make retries your new best friend

Your system should be loaded with retries. Build them into all service layer hops, and use exponential backoffs to avoid overwhelming the downstream system. The requests between service layers must be idempotent, Doug emphasized. When they are, you'll be able to reissue requests to new servers without double-applying changes.

Use queues where you don't care about the response to decouple the client from the server. If you're stuck with a request/response flow, use a circuit breaker approach, where your client library delivers back minimal results if a service is down-reducing front-end latency and errors.

Don't put all of your eggs in one basket

Distribute your data to many servers, so that no one server is so important you can't safely work on it.

At PagerDuty, the team uses multi-master clusters, which help with operations and vertical scaling. They also use multiple database servers like Cassandra: No one server is that special, which means operational work can happen during the day.

Put together, these strategies help admins and operational engineers sleep more, worry less and maintain better-all ahead of schedule.

Questions? Share your thoughts in the comments below.

The post How to Ditch Scheduled Maintenance appeared first on PagerDuty Blog.

Read the original blog entry...

More Stories By PagerDuty Blog

PagerDuty’s operations performance platform helps companies increase reliability. By connecting people, systems and data in a single view, PagerDuty delivers visibility and actionable intelligence across global operations for effective incident resolution management. PagerDuty has over 100 platform partners, and is trusted by Fortune 500 companies and startups alike, including Microsoft, National Instruments, Electronic Arts, Adobe, Rackspace, Etsy, Square and Github.

IoT & Smart Cities Stories
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throug...
History of how we got here. What IoT devices are most vulnerable? This presentation will demonstrate where hacks are most successful, through hardware, software, firmware or the radio connected to the network. The hacking of IoT devices and systems explained in 6 basic steps. On the other side, protecting devices continue to be a challenging effort. Product vendors/developers and customers are all responsible for improving IoT device security. The top 10 vulnerabilities will be presented a...
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throug...
Every organization is facing their own Digital Transformation as they attempt to stay ahead of the competition, or worse, just keep up. Each new opportunity, whether embracing machine learning, IoT, or a cloud migration, seems to bring new development, deployment, and management models. The results are more diverse and federated computing models than any time in our history.
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throug...
Atmosera delivers modern cloud services that maximize the advantages of cloud-based infrastructures. Offering private, hybrid, and public cloud solutions, Atmosera works closely with customers to engineer, deploy, and operate cloud architectures with advanced services that deliver strategic business outcomes. Atmosera's expertise simplifies the process of cloud transformation and our 20+ years of experience managing complex IT environments provides our customers with the confidence and trust tha...
Where many organizations get into trouble, however, is that they try to have a broad and deep knowledge in each of these areas. This is a huge blow to an organization's productivity. By automating or outsourcing some of these pieces, such as databases, infrastructure, and networks, your team can instead focus on development, testing, and deployment. Further, organizations that focus their attention on these areas can eventually move to a test-driven development structure that condenses several l...
The graph represents a network of 1,329 Twitter users whose recent tweets contained "#DevOps", or who were replied to or mentioned in those tweets, taken from a data set limited to a maximum of 18,000 tweets. The network was obtained from Twitter on Thursday, 10 January 2019 at 23:50 UTC. The tweets in the network were tweeted over the 7-hour, 6-minute period from Thursday, 10 January 2019 at 16:29 UTC to Thursday, 10 January 2019 at 23:36 UTC. Additional tweets that were mentioned in this...
Over the course of two days, in addition to insightful conversations and presentations delving into the industry's current pressing challenges, there was considerable buzz about digital transformation and how it is enabling global enterprises to accelerate business growth. Blockchain has been a term that people hear but don't quite understand. The most common myths about blockchain include the assumption that it is private, or that there is only one blockchain, and the idea that blockchain is...
Never mind that we might not know what the future holds for cryptocurrencies and how much values will fluctuate or even how the process of mining a coin could cost as much as the value of the coin itself - cryptocurrency mining is a hot industry and shows no signs of slowing down. However, energy consumption to mine cryptocurrency is one of the biggest issues facing this industry. Burning huge amounts of electricity isn't incidental to cryptocurrency, it's basically embedded in the core of "mini...