Cloud Engineering Archives - Software Engineering Daily

Cloud computing started to become popular in 2006 with the release of Amazon EC2, a system for deploying applications to virtual machines sitting on remote data center infrastructure . With cloud computing, application developers no longer needed to purchase expensive server hardware. Creating an application for the Internet became easier, cheaper, and simpler.

As the cloud has become popular, new ways of deploying applications have emerged. A developer with a web app has so many different options.

You can host your app on an Amazon EC2 server, which will require you to manage cloud infrastructure in case your server crashes. You can deploy your app to Heroku, which gives your cloud deployment better uptime guarantees for a higher price than Amazon EC2. Or you can use Linode, or Microsoft Azure, or Google Cloud.

There is such a large market for cloud computing that the world of cloud providers serves more niches every year. In past episodes we have explored a variety of different cloud providers, and the markets they target.

Pivotal Cloud Foundry is for managing complex distributed systems applications, typically with large teams. Firebase is a cloud provider that simplifies the developer experience for applications with small teams. Spotinst is a cloud provider that emphasizes low cost. Zeit is a cloud provider that is built to manage applications through serverless “functions-as-a-service” like AWS Lambda.

In today’s episode, Mathias Biilman Christensen, CEO of Netlify, joins the show. Netlify is a cloud provider that was built for modern web projects. Netlify represents the convergence of several trends in software development converging: static site deployment, serverless functions, a desire to have a “no-ops” deployment with minimal management, and the rise of newer tools like GraphQL and Gatsby.

Mathias explores these trends in detail, and explores the technical challenges of building Netlify. He was a great guest, capable of talking about difficult backend problems that require writing C++, as well as the frontend world of JavaScript frameworks.

One announcement before we begin: we are having a $5000 hackathon. The $5000 hackathon is for a new product we’ve been working on: FindCollabs. FindCollabs is a platform for finding collaborators and building projects. Whether you are an engineer, a musician, a designer, a videographer, or an artist, FindCollabs lets you find people and collaborate. To try out FindCollabs, just go to FindCollabs.com, you can make a project or you can join someone else’s project. And it’s very easy to make these projects–you don’t need to have anything built yet–you need to have a vision for what you want to build. And to find out about the hackathon, go to findcollabs.com/hackathon. We are giving away $5000 in cash to the coolest projects that get built before Sunday April 14th. So I recommend getting started early, finding some people to collaborate with, and building some cool stuff!

The post Netlify with Mathias Biilmann Christensen appeared first on Software Engineering Daily.

Kubernetes has become the standard way of deploying new distributed applications.

Most new internet businesses started in the foreseeable future will leverage Kubernetes (whether they realize it or not). Many old applications are migrating to Kubernetes too.

Before Kubernetes, there was no standardization around a specific distributed systems platform. Just like Linux became the standard server-side operating system for a single node, Kubernetes has become the standard way to orchestrate all of the nodes in your application.

With Kubernetes, distributed systems tools can have network effects. Every time someone builds a new tool for Kubernetes, it makes all the other tools better. And it further cements Kubernetes as the standard.

Google, Microsoft, Amazon, and IBM each have a Kubernetes-as-a-service offering, making it easier to shift infrastructure between the major cloud providers. We are likely to see Digital Ocean, Heroku, and longer tail cloud providers offer a managed, hosted Kubernetes eventually.

In this editorial, I explore the following questions:

• • Why is this happening?
• • What are the implications for developers?
• • How are cloud providers affected?
• What are the new business models that are possible in a Kubernetes-standardized world?

Software Standards

Standardized software platforms are both good and bad.

Standards allow developers to have expectations around how their software will run. If a developer builds something for a standardized platform, they can make smart estimations about the total addressable market for that piece of software.

If you write a program in JavaScript, you know that it will run in everyone’s browser. If you create a game for iOS, you know that everyone with an iPhone will be able to download it. If you build a tool for profiling garbage collection in .NET, you know that there is a large community of Windows developers with memory issues who can buy your software.

Standardized proprietary platforms can lead to massive profit returns for the platform provider. In 1995, Windows was such a good platform that Microsoft could sell a CD in a cardboard box for $100. In 2018, the iPhone is so good that Apple can take 30% from all app sales on its platform.

Proprietary standards lead to fragmentation.

Your iPhone app does not run on my Kindle Fire. I can’t use your Snapchat augmented reality sticker on Facebook Messenger. My favorite digital audio workstation only runs on Windows–so I have to keep a Windows computer around just to make music.

When developers see this fragmentation, they groan. They imagine the greedy capitalists, who are making money at the expense of software quality. Developers think, “why can’t we all just get along? Why can’t we have things be open and free?”

Developers think, “we don’t need proprietary standards. We can have open standards.”

Growth of Apache, part of the LAMP (Linux, Apache, MySQL, PHP) Stack

This happened with Linux. These days, new server side applications are mostly in Linux. There was a time when that was controversial (see the far left hand side of chart above).

More recently, we saw a newer open standard with Docker. Docker gave us an open, standardized way of packaging, deploying, and distributing a single node. This was hugely valuable! But for all of the big problems that Docker solved, it highlighted a new problem–how should we be orchestrating these nodes together?

After all–your application is not just a single node. You know you want to be deploying a Docker container–but how are your containers communicating with each other? How are you scaling up instances of these containers? How are you routing traffic between container instances?

Container Orchestration

After Docker became popular, a scramble of open source projects and proprietary platforms emerged to solve the problem of container orchestration.

Mesos, Docker Swarm, and Kubernetes each offered a different set of abstractions for managing containers. Amazon ECS offered a proprietary managed service that took care of installation and scaling of Docker containers for AWS users.

Some developers did not adopt any orchestration platform, and used BASH, Puppet, Chef, and other tools to script their deployments.

Whether a developer was deploying their container by using an orchestration platform or a script, Docker sped up development, and made things more harmonious between developers and ope...

Steve Herrod led engineering at VMWare as the company scaled from 30 engineers to 3,000 engineers. After 11 years, he left to become a managing director for General Catalyst, a venture capital firm. Since he has both operating experience and a wide view of the technology landscape as an investor, he is well-equipped to discuss a topic that we have been covering on Software Engineering Daily: the integration of cloud and edge computing.

Today, we think of the cloud as a network of large data centers operated by big players like Google, Amazon, and Microsoft. The cloud is where most of the computation across the world takes place. My smartphone and laptop are “edge” devices. They are lightweight computers that don’t perform much complex processing. I would not be able to run a large production database or a 3 terabyte MapReduce job on my laptop.

The current division of labor makes sense in this world of smart clouds and low-power, low-bandwidth devices. But the devices are getting cheaper, smarter, and more proliferate. Cars, drones, security cameras, sensors, and other devices can serve as points of computation that are geographically between the edge devices and the cloud. With more devices between you and the cloud, there is an opportunity to put computation on those devices.

Everyone knows that cloud and edge computing will become intermingled in the coming years. But predicting just how it will play out is nearly impossible. And as an investor, if you bet on something too early, you get the same result as someone who was wrong altogether.

A good analogy for the “cloud and edge” space of investments might be the “smart home.” Everyone knows the smart home is coming eventually, but it’s very hard to tell how long it will be before smart home systems are in widespread use–so it is an open question of how to invest in the space.

Summer internship applications to Software Engineering Daily are also being accepted. If you are interested in working with us on the Software Engineering Daily open source project full-time this Summer, send an application to [email protected]. We’d love to hear from you.

If you haven’t seen what we are building, check out softwaredaily.com, or download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily.

The post Cloud and Edge with Steve Herrod appeared first on Software Engineering Daily.

Incident management is the process of responding to unplanned events or service interruptions, and then restoring service to an operational state. Having robust incident management is vital to many software teams. Blameless is a framework designed to help software companies manage their production incidents effectively. It provides a workflow for managing the incident response as well as the post-incident retrospective process. In this way, Blameless aims to give a unified learning and process feedback framework that focuses on actionable insights. Ken Gavranovic is the Chief Operating Officer at Blameless and he’s our guest today.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Sponsorship inquiries: [email protected]

Please click here to view this show’s transcript.

The post Blameless with Ken Gavranovic appeared first on Software Engineering Daily.

Observability is becoming an increasingly competitive space in the software world. Many developers have heard of Datadog and New Relic, but there are a seemingly countless number of observability products out there. Costa Tsaousis (he/him) is the Founder and CEO of Netdata. His goal was to build an open-source platform that was high-resolution, real-time, and easily scalable. Netdata is the result. It’s relatively new to the crowded observability space, but it’s grown into a major presence. Costa joins the show to talk about the design philosophy of Netdata, and how it inverts a common observability design pattern. This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments. Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com. Please click here to view this show’s transcript. Sponsorship inquiries: [email protected]

The post A Different Monitoring Philosophy with Costa Tsaousis appeared first on Software Engineering Daily.