New open source solutions library from Shoreline.io aims to deliver self-healing infrastructure

The pre-built op packs are geared at addressing common production operations incidents to increase teams’ productivity.

Incident automation company Shoreline.io has unveiled its open source solutions library, a collection of op packs designed to make it easier to diagnose and repair the most common infrastructure incidents in production cloud environments.

The solutions library focuses on addressing issues including JVM memory leaks, filling disks, rogue processes and stuck Kubernetes pods. It is launching with over 35 op packs that are available for free to the Shoreline community.

“If the issue is a filling disk, for example, the solution might involve deleting temp files, archiving old files and possibly also allocating additional resources, depending on which combination of these is most appropriate for the disk that’s been identified as an issue,’’ said Anurag Gupta, co-founder and CEO of Shoreline.

SEE: Hiring kit: Back-end Developer (TechRepublic Premium)

If there is a JVM memory leak, the op pack automatically captures heap dump, thread dump, garbage collection stats and other debug data for use by engineers that will need to debug and eliminate the root cause, Gupta said. Customers can choose to push this data to Amazon S3, Google storage or any other object store before optionally restarting the JVM.

The Shoreline op packs are built to work with infrastructure hosted by AWS, Azure and Google Cloud.

Pre-built automations and diagnostic notebooks

On-call teams understand that self-healing infrastructure drives higher availability, fewer tickets and better customer satisfaction, the company said. Before now, the path to incident automation was challenging. Developers can now create and share open source op packs built in hours, as opposed to months, with Shoreline. The pre-built automations and diagnostic notebooks are designed to save time and accelerate the path to increased reliability.

Each op pack is published and provisioned as open source Terraform modules and contains everything necessary to solve a specific issue, including pre-defined metrics, alarms, actions, bots, scripts and tests. With Shoreline’s Op Pack library, the community identifies what to monitor, what alarms to set and what scripts to run to complete the repair.

All op packs are completely configurable and allow cloud operations teams to decide whether to use full automation or an interactive notebook for human-in-the-loop repair, according to Shoreline. They were co-developed with Shoreline customers and are based on real world on-call experience at large enterprises, rapidly growing unicorns and the largest hyperscale production environments, the company said.

“Companies can no longer afford to write their own runbooks or custom code automations from scratch,” Gupta said. “With Shoreline, every time someone in our community fixes a problem, everyone else benefits.”

Shoreline’s free solutions

The following op pack solutions are available now and free to Shoreline customers:

Streamline Kubernetes operations

• Kubernetes node retirement: Gracefully terminate nodes when marked for retirement by the cloud provider.
• Kubernetes pod out of memory (OOM): Generate diagnostic information and restart pods that ran out of memory.
• Kubernetes pods stuck in terminating: Identify, safely drain and restart stuck pods.
• Kubernetes pods restarting too often: Detect pod restart loops and capture diagnostics to identify the root cause.
• IP exhaustion: Clear away failed jobs or pods that are consuming too many IP addresses.
• Stuck Argo workflows: Argo makes declaratively managing workflows easy, but it can leave behind many stale pods after workflow execution that should be deleted.

 Reduce toil on both VMs or Kubernetes

• Disk resize / disk clean: Disk full incidents can lead to wide-spread outages and data loss that can damage customer experiences and lose revenue.
• Networking issues: Network related issues are often hard to diagnose and can lead to a very bad experience for customers.
• Intermittent JVM issues: Capture diagnostic information for intermittent issues that are hard to reproduce and debug.
• Server drift: Restore uniformity when configuration files, databases and data sources on your VMs and containers differ.
• Config drift: Ensure observed state matches desired state on your system configuration.
• Memory exhaustion: Running out of memory rapidly degrades customer experience and must be preempted.
• Disk failures in kern.log: Detect when a disk has errors or has entirely failed by inspecting the OS’s kern.log. Automatically capture these events and kick off fixes such as recycling the VM.
• Network failures in kern.log: Detect when a network interface has errors or has entirely failed by inspecting the OS’s kern.log. Automatically capture these events and initiate fixes such as recycling the VM.
• Endpoints unreachable: Determine when there are no endpoints behind your Kubernetes service or these endpoints have become unreachable.
• Elastic sharding replica management: Determine when your elastic search clusters have too few replicas per shard, and automatically kick off healing.
• Log processing at the edge: Analyze log files on the box to identify issues that cause production incidents and eliminate costs of centralized logging.
• Kafka data Processing Lag: Restart slow or broken consumers when systems are falling behind in processing messages through a queue.
• Kafka topic management: When the length of your Kafka topic is too long, applications may begin to break.
• Processes consuming too many resources: Determine if the system is using too much memory or CPU at the process level.
• Restart CoreDNS service: CoreDNS, the default Kubernetes DNS service, can degrade in performance with too many calls causing massive latency.

Optimize cloud spend

• Rightsize pod CPU and memory allocations: Automatically reduce pod CPU and/or memory limits that are set too high.
• Reclaim idle hosts: Mark low utilization virtual machine instances for inactivity, then terminate them.
• Delete unused EBS volumes / snapshots: Eliminate costs from unused resources.
• Manage data transfer costs: Detect increased data transfer volumes, and pinpoint the reasons.
• Excessive use of on-demand hosts: Determine if converting on-demand VMs to reserved instances would create substantial savings.

Increase security

• Privileged container check: Flag any container or pod running in privileged mode.
• Users with root access check: Flag any VM or container which has server processes running as a user with root permissions.
• Open port check: Ports can easily be opened unintentionally in a development environment, especially port 22 for SSH and port 3389 for remote login.
• Connections from unexpected ports: Detect network connections on ports that are not found on an approved list.
• Process list check: Ensure the correct server processes are running, since processes sometimes die silently or old versions are left running.
• Detect cryptocurrency mining operations: Unauthorized cryptocurrency miners must be stopped from abusing free tiers of cloud service providers.

Avoid major outages

• Certificate rotation: Sooner or later every company gets bitten by expired certificates and when they do, it can cause a catastrophic outage.
• DNS lag: Trigger rolling restarts of the DNS servers when they are responding slowly and causing widespread system issues.