Machine Learning Archives - Software Engineering Daily

There are many types of early stage funding available from friends and family to seed to series A. Some firms invest across a wide set of technologies and seek only to provide capital. Others are in it for the long haul – they focus on specific areas of technology and develop both long term relationships and deep expertise over time.

Today, we are interviewing Matt Turck of First Mark Capital, who is in it for the long haul and whose portfolio companies include Dataiku, Crossbeam, Ada, Cockroach Labs, Clickhouse and more. Today we will talk about Matt’s career, investment point of view, founding the Data-driven NYC community and the recent release of the 20234 MAD – an industry resource for understanding the Machine Learning, AI and Data Landscape

Be sure to check out the show notes for links to the MAD

This epsiode is hosted by Jocelyn Houle. Follow Jocelyn on Linked or on Twitter @jocelynbyrne.

Show notes –

In today’s show we referenced a couple things you may want to check out.

Matt’s blog and MAD Landscape

The interactive MAD Landscape

The picture in Matt’s Office was The Son of Man by Rene Magritte

Matt’s full bio

FirstMark Capital Site

Sponsorship inquiries: [email protected]

The post Data Investing and the MAD with Matt Turck appeared first on Software Engineering Daily.

If you’re a developer, there’s a good chance you’ve experimented with coding assistants like GitHub Copilot. Many developers have even fully integrated these tools into their workflows. One way these tools accelerate development is by autocompleting entire blocks of code. The AI achieves this by having awareness of the surrounding code. It understands context. However, in many cases the context available to an AI is limited. This restricts the AI’s ability to suggest more sweeping changes to a codebase, or even to refactor an entire application.

Quinn Slack is the CEO of Sourcegraph. He is now hard at work on the challenge of giving more context to AI – to make it aware of entire codebases, dependencies, error logs, and other data. Quinn joins the show today to talk about what it takes to move beyond code autocomplete, how to develop the next generation of coding AI, and what the future looks like for software engineers and programming languages.

Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript. Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development. You can find Josh on: Bluesky, Fosstodon, Twitter, Twitch, YouTube, and joshuakgoldberg.com.

Please click here to view this show’s transcript.

Sponsorship inquiries: [email protected]

The post Sourcegraph with Quinn Slack appeared first on Software Engineering Daily.

Automation is changing the labor market.

To automate a task, someone needs to put in the work to describe the task correctly to a computer. For some tasks, the reward for automating a task is tremendous–for example, putting together mobile phones. In China, companies like FOXCONN are investing time and money into programming the instructions for how to assemble your phone. Robots execute those instructions.

FOXCONN spends millions of dollars deploying these robots, but it is a worthwhile expense. Once FOXCONN pays off the capital investment in those robots, they have a tireless workforce that can build phones all day long. Humans require training, rest, and psychological considerations. And with robots, the error rate is lower. Your smart phone runs your life, and you do not want the liability of human imperfection involved in constructing that phone.

As we race towards an automated future, the manual tasks that get automated first depend on their economic value. The manual labor costs of smartphone construction is a massive expense for corporations. This is also true for truck driving, food service, and package delivery. The savings that will be reaped from automating these tasks are tremendous–regardless of how we automate them.

There two ways of building automated systems: rule-based systems and machine learning.

With rule-based systems, we can describe to the computer exactly what we want it to do–like following a recipe. With machine learning, we can train the computer by giving it examples and let the computer derive its own understanding of how to automate a task.

Both approaches to automation have difficulties. A rule-based approach requires us to enumerate every single detail to the machine. This might work well in a highly controlled environment like a manufacturing facility. But rule-based systems don’t work well in the real world, where there are so many unexpected events, like snowstorms.

As we reported in a previous episode about how to build self-driving cars, engineers still don’t quite know what the right mix of rule-based systems and machine learning techniques are for autonomous vehicles. But we will continue to pour money into solving this problem, because the investment is worth figuring out how to train the machine.

The routine tasks in our world will be automated given enough time. How soon something will be automated depends on how expensive that task is when it is performed by a human, and how hard it is to design an artificial narrow intelligence to perform the task instead of a human.

Manual software testing is another type of work that is being automated today.

If I am building a mobile app to play podcast episodes, and I make a change to the user interface, I want to have manual quality assurance (QA) testers run through tests that I describe to them, to make sure my change did not break anything. QA tests describe high level application functionality. Can the user register and log in? Can the user press the play button and listen to a podcast episode on my app?

Unit tests are not good enough, because unit tests only verify the logic and the application state from the point of view of the computer itself. Manual QA tests ensure that the quality of the user experience was not impacted.

With so many different device types, operating systems, and browsers, I need my QA test to be executed in all of the different target QA environments. This requires lots of manual testers. If I want manual testing for every deployment I push, that manual testing can get expensive.

RainforestQA is a platform for QA testing that turns manual testing into automated testing. The manual test procedures are recorded, processed by computer vision, and turned into automated tests. RainforestQA hires human workers from Amazon Mechanical Turk to execute the well-defined manual tests, and the recorded manual procedure is used to train the machines that can execute the same task in the future.

Russell Smith is the CTO and co-founder of RainforestQA, and he joins the show to explain how RainforestQA works: the engineering infrastructure, the process of recruiting workers from mechanical turk, and the machine learning system for taking manual tasks and automating them.

Show Notes: Andrej Karpathy Turk Story

The post Training the Machines with Russell Smith appeared first on Software Engineering Daily.

Machine learning applications are widely deployed across the software industry.

Most of these applications used supervised learning, a process in which labeled data sets are used to find correlations between the labels and the trends in that underlying data. But supervised learning is only one application of machine learning. Another broad set of machine learning methods is described by the term “reinforcement learning.”

Reinforcement learning involves an agent interacting with its environment. As the model interacts with the environment, it learns to make better decisions over time based on a reward function. Newer AI applications will need to operate in increasingly dynamic environments, and react to changes in those environments, which makes reinforcement learning a useful technique.

Reinforcement learning has several attributes that make it a distinctly different engineering problem than supervised learning. Reinforcement learning relies on simulation and distributed training to rapidly examine how different model parameters could affect the performance of a model in different scenarios.

Ray is an open source project for distributed applications. Although Ray was designed with reinforcement learning in mind, the potential use cases go beyond machine learning, and could be as influential and broadly applicable as distributed systems projects like Apache Spark or Kubernetes. Ray is a project from the Berkeley RISE Lab, the same place that gave rise to Spark, Mesos, and Alluxio.

The RISE Lab is led by Ion Stoica, a professor of computer science at Berkeley. He is also the co-founder of Anyscale, a company started to commercialize Ray by offering tools and services for enterprises looking to adopt Ray. Ion Stoica returns to the show to discuss reinforcement learning, distributed computing, and the Ray project.

If you enjoy the show, you can find all of our past episodes about machine learning, data, and the RISE Lab by going to SoftwareDaily.com and searching for the technologies or companies you are curious about . And if there is a subject that you want to hear covered, feel free to leave a comment on the episode, or send us a tweet @software_daily.

Sponsorship inquiries: [email protected]

The post Anyscale with Ion Stoica appeared first on Software Engineering Daily.

Google Search allows humans to find and access information across the web. A human enters an unstructured query into the search box, the search engine provides several links as a result, and the human clicks on one of those links. That link brings up a web page, which is a set of unstructured data. Humans can read and understand news articles, videos, and Wikipedia pages.

Google Search solves the problem of organizing and distributing all of the unstructured data across the web, for humans to consume. Diffbot is a company with a goal of solving a related, but distinctly different problem: how to derive structure from the unstructured web, understand relationships within that structure, and allow machines to utilize those relationships through APIs.

Mike Tung is the founder of Diffbot. He joins the show to talk about the last decade that he has spent building artificial intelligence applications, from his research at Stanford to a mature, widely used product in Diffbot. I have built a few applications with Diffbot, and I encourage anyone who is a tinkerer or prototype builder to play around with it. It’s an API for accessing web pages as structured data.

Diffbot crawls the entire web, parsing websites, using NLP and NLU to comprehend those pages, and using probabilistic estimations to draw relationships between entities. It’s an ambitious product, and Mike has been working on it for a long time. I enjoyed our conversation.

Show Notes

• Knowledge Graph Search Widget
• Knowledge Graph, AI Web Data Extraction and Crawling | Diffbot
• Knowledge Graph | Diffbot
• Automatic APIs: automatically extract content from web pages
• Web Crawler API: Crawlbot
• Diffblog | Structured Data Blog
• Introducing the Diffbot Knowledge Graph | Diffblog
• Diffbot launches AI-powered knowledge graph of 1 trillion facts about people, places, and things | VentureBeat
• Web crawler – Wikipedia
• Natural Language Processing (NLP) Techniques for Extracting Information | Search Technologies

We recently launched a new podcast: Fintech Daily! Fintech Daily is about payments, cryptocurrencies, trading, and the intersection between finance and technology. You can find it on fintechdaily.co or Apple and Google podcasts. We are looking for other hosts who want to participate. If you are interested in becoming a host, send us an email: [email protected]

The post Diffbot: Knowledge Graph API with Mike Tung appeared first on Software Engineering Daily.