We’ve got a new / old opening...Allen goes off / on script? Michael denies Joe the “swing” vote, and Joe is all in on AI assistance Testing for concurrency issues is hard because it’s non-deterministic – basically you get unlucky due to the timing of things Serializability Common Implementations Actual Serial Execution Encapsulating transactions in [...]

We finally start talking about Apache Kafka! Also, Allen is getting acquainted with Aesop, Outlaw is killing clusters, and Joe was paying attention in drama class.

The full show notes are available on the website at https://www.codingblocks.net/episode235

News

• Atlanta Dev Con is coming up, on September 7th, 2024 (www.atldevcon.com)

Intro to Apache Kafka

What is it?

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Core capabilities

• High throughput – Deliver messages at network-limited throughput using a cluster of machines with latencies as low as 2ms.
• Scalable – Scale production clusters up to a thousand brokers, trillions of messages per day, petabytes of data, and hundreds of thousands of partitions. Elastically expand and contract storage and processing
• Permanent storage – Store streams of data safely in a distributed, durable, fault-tolerant cluster.
• High availability – Stretch clusters efficiently over availability zones or connect separate clusters across geographic regions.

Ecosystem

• Built-in stream processing – Process streams of events with joins, aggregations, filters, transformations, and more, using event-time and exactly-once processing.
• Connect to almost anything – Kafka’s out-of-the-box Connect interface integrates with hundreds of event sources and event sinks including Postgres, JMS, Elasticsearch, AWS S3, and more.
• Client libraries – Read, write, and process streams of events in a vast array of programming languages
• Large ecosystem of open source tools – Large ecosystem of open source tools: Leverage a vast array of community-driven tooling.

Trust and Ease of Use

• Mission critical – Support mission-critical use cases with guaranteed ordering, zero message loss, and efficient exactly-once processing.
• Trusted by thousands of organizations – Thousands of organizations use Kafka, from internet giants to car manufacturers to stock exchanges. More than 5 million unique lifetime downloads.
• Vast user community – Kafka is one of the five most active projects of the Apache Software Foundation, with hundreds of meetups around the world.

What is it?

• Getting data in real-time from event sources like databases, sensors, mobile devices, cloud services, applications, etc. in the form of streams of events. Those events are stored “durably” (in Kafka) for processing, either in real-time or retrospectively, and then routed to various destinations depending on your needs. It’s this continuous flow and processing of data that is known as “streaming data”
  How can it be used? (some examples)
• Processing payments and financial transactions in real-time
• Tracking automobiles and shipments in real time for logistical purposes
• Capture and analyze sensor data from IoT devices or other equipment
• To connect and share data from different divisions in a company

Apache Kafka as an event streaming platform?

• It contains three key capabilities that make it a complete streaming platform
  • Can publish and subscribe to streams of events
  • Can store streams of events durably and reliably for as long as necessary (infinitely if you have the storage)
  • To process streams of events in real-time or retrospectively
• Can be deployed to bare metal, virtual machines or to containers on-prem or in the cloud
• Can be run self-managed or via various cloud providers as a managed service

How does Kafka work?

• A distributed system that’s composed of servers and clients that communicate using a highly performant TCP protocol

Servers

• Kafka runs as a cluster of one or more servers that can span multiple data centers or cloud regions
• Brokers – these are a portion of the servers that are the storage layer
• Kafka Connect – these are servers that constantly import and export data from existing systems in your infrastructure such as relational databases
• Kafka clusters are highly scalable and fault-tolerant

Clients

• Allows you to write distributed applications that allow to read, write and process streams of events in parallel that are fault-tolerant and scale
  • These clients are available in many programming languages – both the ones provided by the core platform as well as 3rd party clients

Concepts

Events

• It’s a record of something that happened – also called a “record” in the...

It’s time to talk about the types of things you should consider when diving into multi-tenant applications and things to be especially aware of if you attempt to make things faster by going multi-threaded at the same time. Join us as Allen continues is non-winning / non-losing streak, JZ is taken out by a storm, and Outlaw continues to ignore the notes for the intro to the show...

News

Reviews

• Spotify – Colum Ferry

Upcoming Events

• Atlanta Dev Con
  September 7th, 2024
  https://www.atldevcon.com/
• DevFest Central Florida on September 28th, 2024
  https://devfestflorida.com/

Episode

• Single vs multi-tenanted applications and multi-threaded programming
  • Be very careful about sharing state when doing anything in multi-threaded AND multi-tenanted applications

What are some important things to consider when writing multi-tenanted applications – per Claude

As mentioned in the episode, Claude’s answers seem to be very database heavy

• Data isolation: Ensure that each tenant’s data is securely separated and inaccessible to other tenants.
• Authentication and authorization: Implement robust systems to verify tenant identity and control access to resources.
• Scalability: Design the architecture to handle growth in both the number of tenants and data volume per tenant.
• Performance: Optimize database queries and resource allocation to maintain speed as tenant count increases.
• Customization: Allow for tenant-specific configurations and features without compromising the core codebase.
• Monitoring and logging: Implement comprehensive logging and monitoring to track usage, errors, and performance per tenant.
• Maintenance and updates: Design the system to allow for updates and maintenance with minimal disruption to tenants.
• Data backup and recovery: Implement tenant-specific backup and recovery processes.
• Compliance and data regulations: Ensure the system can meet various regulatory requirements that may differ by tenant.
• Tenant onboarding and offboarding: Create efficient processes for adding new tenants and removing old ones.
• Good feature flagging support?
• Tracing / OpenTelemetry

Can you elaborate on the data isolation?
Data isolation is crucial in multi-tenant systems to prevent unauthorized access and ensure data privacy. Here are some key strategies to ensure data isolation:

• Separate databases
  • Use a separate database for each tenant
    • Pros: Strong isolation, easier compliance
    • Cons: Higher maintenance, potentially higher costs
• Shared database with separate schemas
  • Each tenant gets their own schema within a shared database
    • Pros: Good isolation, more cost-effective than separate databases
    • Cons: Requires careful access control, potential for errors
• Shared schema with tenant ID
  • Use a single schema but include a tenant ID column in each table
    • Pros: Most cost-effective, easiest to maintain
    • Cons: Highest risk of data leaks, requires vigilant query filtering
• Row-level security
  • Implement database-level security policies to restrict access based on tenant ID
  • Particularly useful with shared schema approach
• Encryption
  • Encrypt sensitive data, potentially using tenant-specific keys
  • Adds an extra layer of protection even if other measures fail
• Connection pooling
  • Use separate connection pools for each tenant to prevent session mixups
• Query filtering
  • Always include tenant ID in queries, preferably at the database level
  • Use ORMs or query builders that automatically add tenant filters
• API and application-level checks
  • Validate tenant ID in all API calls and application logic
  • Implement middleware to enforce tenant-specific access

Resources / Mentions

• Salable Postgres product – Citus
  https://www.citusdata.com/
• Envelope encryption if you’re wanting to use tenant specific keys for multi-tenanted encryption – one approach is envelope encryption
  https://en.wikipedia.org/wiki/Hybrid_cryptosystem#Envelope_encryption
  https://cloud.google.com/kms/docs/envelope-encryption
• OWASP multi-tenant or “Cloud Tenant” Isolation