News Aggregator

Database Keys: A Comprehensive Guide

Aggregated on: 2024-09-17 14:21:14

In the world of database management, keys play a crucial role in organizing, accessing, and maintaining data integrity. Whether you're a seasoned database administrator or just starting your journey in data management, understanding the various types of database keys is essential. In this post, we'll explore all the different types of keys, their purposes, and how they contribute to effective database design. To illustrate these concepts, we'll use a simple university database with the following tables:

A Hands-On Guide to OpenTelemetry: Manual Instrumentation for Developers

Aggregated on: 2024-09-17 13:21:14

Are you ready to start your journey on the road to collecting telemetry data from your applications? In this series, you'll explore how to adopt OpenTelemetry (OTel) and how to instrument an application to collect tracing telemetry. You'll learn how to leverage out-of-the-box automatic instrumentation tools and understand when it's necessary to explore more advanced manual instrumentation for your applications. By the end of this series, you'll have an understanding of how telemetry travels from your applications to the OpenTelemetry Collector, and be ready to bring OpenTelemetry to your future projects. Everything discussed here is supported by a hands-on, self-paced workshop authored by Paige Cruz.

Commonly Occurring Errors in Microsoft Graph Integrations and How To Troubleshoot Them (Part 6)

Aggregated on: 2024-09-16 23:21:13

This article discusses the following example: changing a column value for a list item of a “user-defined” SharePoint list, which represents a PDF file within a sub-folder of the SharePoint drive, using a DriveItem, which also represents the PDF file. SharePoint lists are central data structures that make it possible to store, organize, and manage information in tabular form. The test aims to demonstrate and explain how the MS-Graph API can be successfully used to change values in a list and correctly transfer this change to SharePoint. This ensures that the integrity of the file is maintained after the change and that no unexpected errors occur during API communication. Microsoft Graph API The Microsoft Graph API is a REST-based programming interface that provides access to a variety of services and data within the Microsoft 365 platform. It offers developers the ability to access a central interface to read, write, and manage data from services such as Azure Active Directory, Outlook, OneDrive, SharePoint, Microsoft Teams, and many more.

Observability Agent Architecture

Aggregated on: 2024-09-16 22:21:13

Observability agents are essential components in modern software development and operations. These software entities act as data collectors, processors, and transmitters, gathering critical telemetry data from applications, infrastructure, and network devices. This data is then sent to centralized observability platforms where it can be analyzed to gain valuable insights into system performance, identify issues, and optimize operations. By efficiently capturing, processing, and transmitting logs, metrics, and traces, observability agents provide a comprehensive view of system health and behavior. This enables organizations to make informed decisions, improve application reliability, and ensure compliance with relevant regulations.

Low-Level Optimizations in ClickHouse: Utilizing Branch Prediction and SIMD To Speed Up Query Execution

Aggregated on: 2024-09-16 21:21:13

In data analysis, the need for fast query execution and data retrieval is paramount. Among numerous database management systems, ClickHouse stands out for its originality and, one could say, a specific niche, which, in my opinion, complicates its expansion in the database market. I’ll probably write a series of articles on different features of ClickHouse, and this article will be a general introduction with some interesting points that few people think about when using various databases.

Build an Advanced RAG App: Query Routing

Aggregated on: 2024-09-16 20:21:13

In previous articles, we built a basic RAG application. We also learned to introduce more advanced techniques to improve a RAG application. Today, we will explore how to tie those advanced techniques together. Those techniques might do different — sometimes opposite — things. Still, sometimes we need to use all of them, to cover all possibilities. So let's see how we can link different techniques together. In this article, we will take a look at a technique called Query Routing. The Problem With Advanced RAG Applications When our Generative AI application receives a query, we have to decide what to do with it. For simple Generative AI applications, we send the query directly to the LLM. For simple RAG applications, we use the query to retrieve context from a single data source and then query the LLM. But, if our case is more complex, we can have multiple data sources or different queries that need different types of context. So do we build a one-size-fits-all solution, or do we make the application adapt to take different actions depending on the query?

The Ultimate Database Scaling Cheatsheet: Strategies for Optimizing Performance and Scalability

Aggregated on: 2024-09-16 19:21:13

As applications grow in complexity and user base, the demands on their underlying databases increase significantly. Efficient database scaling becomes crucial to maintain performance, ensure reliability, and manage large volumes of data. Scaling a database effectively involves a combination of strategies that optimize both hardware and software resources to handle increasing loads. This cheatsheet provides an overview of essential techniques for database scaling. From optimizing query performance with indexing to distributing data across multiple servers with horizontal scaling, each section covers a critical aspect of database management. Whether you're dealing with a rapidly growing application or preparing for future growth, understanding these strategies will help you make informed decisions to ensure your database remains robust and responsive.

Leveling Up My GraphQL Skills: Real-Time Subscriptions

Aggregated on: 2024-09-16 18:21:13

For a few years now, I’ve tried to identify frameworks, products, and services that allow technologists to maintain their focus on extending the value of their intellectual property. This continues to be a wonderful journey for me, filled with unique learning opportunities. The engineer in me recently wondered if there was a situation where I could find a secondary benefit for an existing concept that I’ve talked about before. In other words, could I identify another benefit with the same level of impact as the original parent solution previously recognized?

Selenium Grid Tutorial: Essential Tips and How To Set It Up

Aggregated on: 2024-09-16 17:21:13

Being a tester or a skilled web developer, you need to test your applications for bugs and performance in all the available browsers and operating systems. But with so many dependencies in hand, having not just different browsers, but different versions, too, it surely becomes a hefty task. Most importantly, all these processes must be automated to the greatest extent because in major companies, individually creating test cases and pipelines would be expensive and the most obvious, the least preferable method ever. Now this is where Selenium Grid, an extensively used server-based test automation tool comes into the picture.

Retrieval-Augmented Generation (RAG): Enhancing AI-Language Models With Real-World Knowledge

Aggregated on: 2024-09-16 16:21:13

In recent years, AI has made big leaps forward, mainly because of large language models (LLMs). LLMs are really good at understanding and generating text that’s human-like, and they led to the creation of several new tools like advanced chatbots and AI writers. While LLMs are great at generating text that’s fluent and human-like, they sometimes struggle with getting facts right. This can be a huge problem when accuracy is really important

Obfuscation vs Encryption: How To Protect Your .NET Code the Right Way

Aggregated on: 2024-09-16 15:21:13

When working on .NET applications, one main concern is safeguarding your code from unauthorized access, intellectual property theft, and reverse engineering. This can be achieved by implementing data and code protection techniques to protect the application. There are two main techniques used to secure .NET code: obfuscation and encryption. But do you know which technique is right? And when to use each? Let’s learn from this guide.

How to Merge Excel XLSX Files in Java

Aggregated on: 2024-09-16 14:21:13

In this article, we’re going to learn how to increase the efficiency of a common file merging workflow with the help of a web API solution. Specifically, we are going to learn how to merge Excel XLSX files – one of the most common document types we can expect to work within a file-processing automation environment. Context for Programmatic XLSX File Merging If we’re writing code to solve problems related to file processing efficiency and automation, there’s a good chance we’re creating and/or expanding applications that deal with large volumes of Excel files. We may, for example, find that there’s no automated workflow currently in place (or, perhaps, an inefficient one) to combine the various unique Excel reports created by each individual department in our organization.

Zero To AI Hero, Part 3: Unleashing the Power of Agents in Semantic Kernel

Aggregated on: 2024-09-16 13:21:13

As I promised in Part 2, it is time to build something substantial with our Semantic Kernel so far. If you are new to Semantic Kernel and must dive into code/head first, I highly recommend starting with Part 1 of this series. There is a lot of theory out there, but we explore these articles with a GitHub sample you can easily download and play with to understand the core concepts. I wanted to use Agent Smith from The Matrix, but I can't seem to find one without copyrights. So, DALL-E 3 to the rescue.

Releasing MuleSoft API

Aggregated on: 2024-09-13 23:06:12

MuleSoft Anypoint is an enterprise platform of an API event-driven architecture that allows developers to create design and build an API, and to name a few capabilities where you can share API templates and application assets. It also has a central web interface to manage, integrate, secure, and monitor API performance. Open API Specification MuleSoft API development supports Open API specification.

Setting Up Secure Data Lakes for Starlight Financial: A Guide to AWS Implementation

Aggregated on: 2024-09-13 21:51:12

Continuing on our fictitious financial company, Starlight, series of posts, here is how to set up a data lake on AWS with security as the primary thought. Introduction In the fast-moving financial industry, data is a core asset. Starlight Financial needs to use vast amounts of data for decision-making, improving customer experience, and keeping ahead of its rivals. Consider a data lake: it's a vital part of modern data architectures, letting enterprises store both structured and unstructured data in large quantities of any kind whatsoever. Tony Hoare famously observed that with great data comes great responsibility — and so it is. Eventually, it will be some comfort to know that one of the most important steps for consultancy in validating big data architectures using AWS services has been elucidated. That is to say: test them just like any other system you might use. This is a guide to establishing a highly secure data lake using AWS services, specifically focused on the needs of financial institutions, written by us using a blog structure.

Synchronizing Files from Maximo Application Suite 8 To a Shared Drive Without OpenShift Integration

Aggregated on: 2024-09-13 20:51:12

In IBM Maximo 7.6 and earlier versions, it was common to have Maximo drop files directly onto a network shared drive connected to your administrative server. This setup allowed for seamless file handling and sharing across your network. With the advent of Maximo Application Suite (MAS) 8, you can achieve similar functionality if the shared drive is integrated as part of your OpenShift storage class (e.g., NFS, CSI). However, there are scenarios where you might not want the shared drive to be part of OpenShift. In this article, we explore how to implement file synchronization from MAS8 to a shared drive without involving OpenShift storage. Design and Solution Implementation In this solution, we provide a script that runs as a part of your Orchestrator server, using a kubeconfig file for authentication. Additionally, ensure that the shared drive is mounted to the Orchestrator server. This setup allows you to copy files from the MAXINST pod to the shared drive without integrating the drive into OpenShift.

A Short History of AJAX and SSR

Aggregated on: 2024-09-13 19:21:12

My journey in programming began over two decades ago, a time when JavaScript was a far cry from its current state, and developers were primarily focused on Microsoft Internet Explorer. One of my proudest achievements back then was writing a few lines of code that allowed users to add and remove table rows entirely on the client side. We called it DHTML. Many developers today have forgotten about it — or never knew it existed. A few years later, AJAX emerged, revolutionizing the way we approached web development. The emergence of AJAX marked a significant shift in web development, transferring more logic from the server to the client, and this shift was not without reason.

The Critical Role of Data at Rest Encryption in Cybersecurity

Aggregated on: 2024-09-13 17:51:12

Securing information is crucial as cyber-attacks are getting more sophisticated. Data residing in an unprotected state at rest (databases, stored files, and backups) pose one of the most significant risks. Data at rest encryption is necessary to guarantee that information is safe and unreadable even in the event of illegal access. This discussion highlights why encrypting data is seen as an indispensable part of bolstering overall cyber security and keeping assets safe from unwanted entry.

How To Solve OutOfMemoryError: Metaspace

Aggregated on: 2024-09-13 16:21:12

There are 9 types of java.lang.OutOfMemoryErrors, each signaling a unique memory-related issue within Java applications. Among these, java.lang.OutOfMemoryError: Metaspace is a challenging error to diagnose. In this post, we’ll delve into the root causes behind this error, explore potential solutions, and discuss effective diagnostic methods to troubleshoot this problem. Let’s equip ourselves with the knowledge and tools to conquer this common adversary. JVM Memory Regions To better understand OutOfMemoryError, we first need to understand different JVM Memory regions. Here is a video clip that gives a good introduction to different JVM memory regions. But in a nutshell, JVM has the following memory regions: Figure 1: JVM memory regions

Understanding the Purposes of Key Terraform Files and How to Structure Their Folders

Aggregated on: 2024-09-12 19:21:11

Terraform is the go-to platform for deploying and managing infrastructure as code (IaC). But for Terraform to function, it needs a set of configuration files that specify the resources, settings, and structure of your infrastructure. How you create and organize IaC files will have a big impact on how effective and efficient your IaC deployment becomes, especially as your environment scales and evolves. The goal of this article is to help you understand the role and function of some of the most critical Terraform files, and how you can structure their folders to optimize the efficiency and scalability of your cloud resources. Learn about the importance of keeping Terraform files in order, and explore some tips for making sure your setup serves your IaC operations well.

Designing a Secure Architecture for Distributed Systems

Aggregated on: 2024-09-12 17:21:11

Securing distributed systems is a complex challenge due to the diversity and scale of components involved. With multiple services interacting across potentially unsecured networks, the risk of unauthorized access and data breaches increases significantly. This article explores a practical approach to securing distributed systems using an open-source project. The project demonstrates how to integrate several security mechanisms and technologies to tackle common security challenges such as authentication, authorization, and secure communication. Understanding Security Challenges in Distributed Systems Distributed systems involve multiple services or microservices that must communicate securely across a network. Key security challenges in such architectures include:

How To Build an OpenAI Custom GPT With a Third-Party API

Aggregated on: 2024-09-12 16:21:11

With the advent of Large Language Models (LLM), the conversational user interface has become very popular. You simply ask what you're seeking, and ChatGPT produces an output. In addition, the model can perform actions in response to your command or question and present you with the result. This opened up a huge opportunity to build a new kind of interface where the only input is the user's words — written or spoken. This is fantastic, as you can imagine the limitless possibilities. What we need is for LLMs to have access to as many complex, real-world actions as possible, which they can then invoke appropriately. This is where third-party APIs come into the picture.

Open-Source Data Management Practices and Patterns

Aggregated on: 2024-09-12 15:36:11

Open-source data architectures offer organizations and teams significant benefits, ranging from reduced vendor lock-in and improved cost effectiveness to greater scalability, availability, and flexibility. In this Refcard, you'll learn core practices for building an open-source data architecture stack, along with design patterns for infrastructure components, high availability, scalability, security, and more.

10 Things To Avoid in Domain-Driven Design (DDD)

Aggregated on: 2024-09-12 15:21:11

Domain-Driven Design (DDD) is an important strategic approach to software development. It involves deeply understanding and modeling a business domain, particularly beneficial in complex domains with intricate business rules, processes, and interactions. However, effectively implementing DDD requires discipline, a strong grasp of the domain, and the avoidance of common pitfalls that can lead to suboptimal designs and technical debt. In this article, we'll explore 10 things to avoid in DDD and examples to illustrate these pitfalls. 1. Focusing Too Much on Technical Patterns Sample Scenario A team begins a project by excessively creating repositories, aggregates, and value objects without fully grasping the business domain. For example, they develop a complicated repository for managing Customer entities without understanding how customers are represented and utilized within the business. Consequently, the repository contains numerous unnecessary methods that do not align with the domain's actual use cases and requirements.

Modern Authentication on .NET: OpenID Connect, BFF, SPA

Aggregated on: 2024-09-12 14:21:11

As web technologies continue to advance, so do the methods and protocols designed to secure them. The OAuth 2.0 and OpenID Connect protocols have significantly evolved in response to emerging security threats and the growing complexity of web applications. Traditional authentication methods, once effective, are now becoming outdated for modern Single Page Applications (SPAs), which face new security challenges. In this context, the Backend-For-Frontend (BFF) architectural pattern has emerged as a recommended solution for organizing interactions between SPAs and their backend systems, offering a more secure and manageable approach to authentication and session management. This article explores the BFF pattern in depth, demonstrating its practical application through a minimal solution implemented with .NET and React. By the end, you'll have a clear understanding of how to leverage the BFF pattern to enhance the security and functionality of your web applications. Historical Context The history of OAuth 2.0 and OpenID Connect reflects the ongoing evolution of Internet technologies. Let’s take a closer look at these protocols and their impact on modern web applications.

Understanding Floating-Point Precision Issues in Java

Aggregated on: 2024-09-12 13:21:11

Java Floating Numbers Look Familiar In Java, we have two types of floating-point numbers: float and double. All Java developers know them but can't answer a simple question described in the following meme: Are you robot enough?

Uncovering a Prototype Pollution Regression in the Core Node.js Project

Aggregated on: 2024-09-11 23:21:11

As a Node.js developer and security researcher, I recently stumbled upon an interesting security regression in the Node.js core project related to prototype pollution. This happened to be found while I was conducting an independent security research for my Node.js Secure Coding books and yet the discovery highlights the complex nature of security in open-source projects and the challenges of maintaining consistent security measures across a large codebase. Even at the scale of a project like Node.js, regressions can occur, potentially leaving parts of the codebase vulnerable to attack.

Optimizing Data Management for AI Success: Industry Insights and Best Practices

Aggregated on: 2024-09-11 22:21:11

As artificial intelligence (AI) continues transforming industries, organizations face increasing challenges in managing and utilizing data for AI initiatives. Recent industry surveys and expert insights highlight the critical role of effective data management in AI success. This article explores key trends, challenges, and best practices in data management for AI projects, providing valuable insights for developers, engineers, and architects based on a recent discussion with Adrian Estala, VP and Field Chief Data Officer at Starburst. The Imperative of Real-Time Data Access in AI Real-time data access has emerged as a crucial factor for AI success. Implementing real-time analytics poses several challenges for organizations:

How To Set Up and Configure the Debugger in Visual Studio Code for React.js [Video]

Aggregated on: 2024-09-11 21:36:10

In this article, we’re diving into a critical aspect of React development: setting up and configuring the debugger in Visual Studio Code. Debugging is a crucial part of development that allows you to step through your code, inspect variables, and quickly fix issues, without relying heavily on console logs. This makes the debugging process more efficient and enjoyable. Getting Started Before we begin, make sure you have the following tools installed:

How Stalactite ORM Implements Its Fluent DSL

Aggregated on: 2024-09-11 19:36:11

“DX”, aka Developer Experience One of the goals of Stalactite is to make developers aware of the impact of the mapping of their entities onto the database, and, as a consequence, onto performances. To fulfill this goal, the developer's experience, as a user of the Mapping API, is key to helping him express his intention. The idea is to guide the user-developer in the choices he can make while he describes its persistence. As you may already know, Stalactite doesn’t use annotation or XML files for that. It proposes a fluent API that constrains user choices according to the context. To clarify: available methods after a call to mapOneToOne(..) are not the same as the ones after mapOneToMany(..). This capacity can be done in different ways. Stalactite chose to leverage Java proxies for it and combines it with the multiple-inheritance capability of interfaces.

How To Implement Specific Distributed System Patterns Using Spring Boot: Introduction

Aggregated on: 2024-09-11 18:36:10

Regarding contemporary software architecture, distributed systems have been widely recognized for quite some time as the foundation for applications with high availability, scalability, and reliability goals. When systems shifted from a centralized structure, it became increasingly important to focus on the components and architectures that support a distributed structure. Regarding the choice of frameworks, Spring Boot is a widely adopted framework encompassing many tools, libraries, and components to support these patterns. This article will focus on the specific recommendations for implementing various distributed system patterns regarding Spring Boot, backed by sample code and professional advice. Spring Boot Overview One of the most popular Java EE frameworks for creating apps is Spring. The Spring framework offers a comprehensive programming and configuration mechanism for the Java platform. It seeks to make Java EE programming easier and increase developers' productivity in the workplace. Any type of deployment platform can use it. It tries to meet modern industry demands by making application development rapid and straightforward. While the Spring framework focuses on giving you flexibility, the goal of Spring Boot is to reduce the amount of code and give developers the most straightforward approach possible to create web applications. Spring Boot's default codes and annotation setup lessen the time it takes to design an application. It facilitates the creation of stand-alone applications with minimal, if any, configuration. It is constructed on top of a module of the Spring framework.

Optimizing Container Synchronization for Frequent Writes

Aggregated on: 2024-09-11 17:36:10

Efficient data synchronization is crucial in high-performance computing and multi-threaded applications. This article explores an optimization technique for scenarios where frequent writes to a container occur in a multi-threaded environment. We’ll examine the challenges of traditional synchronization methods and present an advanced approach that significantly improves performance for write-heavy environments. The method in question is beneficial because it is easy to implement and versatile, unlike pre-optimized containers that may be platform-specific, require special data types, or bring additional library dependencies. Traditional Approaches and Their Limitations Imagine a scenario where we have a cache of user transactions:

8 Practices Software Engineers Should Adopt and Champion for Cybersecurity

Aggregated on: 2024-09-11 16:21:10

The Importance of Cybersecurity I firmly believe that in today's cybersecurity expectations, software engineers should prioritize the security of their computer systems and internal IT networks. I would consider it to be a mistake to rely heavily on technology due to the simple fact that a lot of risks are non-technical in nature. The Internet can leave businesses vulnerable to data breaches and ransomware attacks that can cause lasting reputational damage. These non-technical risks can indeed be addressed using technical solutions. These incidents can additionally result in significant financial losses and potential legal issues. In my opinion, implementing strong cybersecurity measures is not just a technical requirement but a critical initiative for any business looking to strengthen its defenses and resilience.

The Significance of Complex Event Processing (CEP) With RisingWave for Delivering Accurate Business Decisions

Aggregated on: 2024-09-11 15:21:10

About Complex Event Processing (CEP) Complex event processing (CEP) is a highly effective and optimized mechanism that combines several sources of information and instantly determines and evaluates the relationships among events in real time. It is a real-time data and event identification, processing, and analysis approach. By gathering and combining across various IoT sensor feeds, CEP has a transformative effect by collecting IoT sensor streams for real-time monitoring, analytics, and troubleshooting. CEP provides insight into what's happening by continuously comparing incoming events to patterns. This enables us to operate proactively and effectively. Although event stream processing (ESP) and CEP are often used interchangeably, they are not exactly the same. Traditional ESP applications typically handle a single stream of data that arrives in the correct time sequence. For instance, in algorithmic trading, an ESP application might analyze a stream of pricing data to decide whether to buy or sell a stock. However, ESP generally doesn't account for event causality or hierarchies. This limitation led to the development of CEP, which is essentially a more advanced and sophisticated version of ESP.

Real-Time GenAI With RAG Using Apache Kafka and Flink To Prevent Hallucinations

Aggregated on: 2024-09-11 14:21:10

How do you prevent hallucinations from large language models (LLMs) in GenAI applications? LLMs need real-time, contextualized, and trustworthy data to generate the most reliable outputs. This blog post explains how RAG and a data streaming platform with Apache Kafka and Flink make that possible. A lightboard video shows how to build a context-specific real-time RAG architecture. Also, learn how the travel agency Expedia leverages data streaming with Generative AI using conversational chatbots to improve the customer experience and reduce the cost of service agents. What Is Retrieval Augmented Generation (RAG) in GenAI? Generative AI (GenAI) refers to artificial intelligence (AI) systems that can create new content, such as text, images, music, or code, often mimicking human creativity. These systems use advanced machine learning techniques, particularly deep learning models like neural networks, to generate data that resembles the training data they were fed. Popular examples include language models like GPT-3 for text generation and DALL-E for image creation.

Modes and Modality in Performance Testing

Aggregated on: 2024-09-11 13:21:10

When reporting summary statistics for performance test results, there's a common assumption that the data follows a central tendency. But is this always true? What if the data does not exhibit a single central tendency? In such cases, the traditional metrics like averages might be misleading. This article explores why it's crucial to check for modality of performance test results, how to detect multimodal distributions, and how to handle them. For users of JMeter, a new plugin is introduced to help automate this process.

Automatic 1111: Custom Sketch-To-Image API

Aggregated on: 2024-09-10 22:21:10

In this article, we will develop a custom Sketch-to-Image API for converting hand-drawn or digital sketches into photorealistic images using stable diffusion models powered by a ControlNet model. We will extend the Automatic 1111's txt2img API to develop this custom workflow. Prerequisites Stable Diffusion Web UI (Automatic 1111) running on your local machine. Follow the instructions here if you are starting from scratch. SD APIs Enabled. Follow the instructions on this page (scroll down to the Enabling APIs section) to enable the APIs if you haven't already done so. ControlNet extension installed: Click on the Extensions tab on Stable Diffusion Web UI. Navigate to the Install from URL tab. Paste the following link in URL for extension's git repository input field and click Install. After the successful installation, restart the application by closing and reopening the run.bat file if you're a PC user; Mac users may need to run ./webui.sh instead. After restarting the application, the ControlNet dropdown will become visible under the Generation tab in the txt2img screen. Download and add the following models to Automatic 1111: RealVisXL_V4.0_Lightning: HuggingFace: SG161222/RealVisXL_V4.0_Lightning. Copy this model to the Stable Diffusion models folder which is under the project root directory:/models/Stable-diffusion diffusers_xl_canny_full - HuggingFace: lllyasviel/sd_control_collection Copy the downloaded model to /extensions/sd-webui-controlnet Payload Now that we have all our prerequisites in place, let's build the payload for the/sdapi/v1/txt2img API.

Exploring Real-Time Data Ingestion Into Snowflake Using CockroachDB, Redpanda, and Kafka Connect

Aggregated on: 2024-09-10 21:21:10

Previous Articles on Snowflake Tour of Snowflake ingestion using CockroachDB and Redpanda Connect Integrating Snowflake with Trino Previous Articles on CockroachDB CDC Emitting Protocol Buffers with CockroachDB CDC Queries Using CockroachDB CDC with Apache Pulsar Using CockroachDB CDC with Azure Event Hubs SaaS Galore: Integrating CockroachDB with Confluent Kafka, FiveTran, and Snowflake Using CockroachDB CDC with Confluent Cloud Kafka and Schema Registry CockroachDB CDC using Minio as cloud storage sink CockroachDB CDC using Hadoop Ozone S3 Gateway as cloud storage sink Motivation This article builds upon the previous discussion in "Tour of Snowflake ingestion using CockroachDB and Redpanda Connect," where we investigated the process of streaming changefeeds from CockroachDB to Snowflake using Redpanda Connect and Snowpipe in batch mode. Here, we will shift our focus to Kafka Connect and demonstrate how both batch and streaming modes can be utilized for data ingestion into Snowflake. Overview Deploy a CockroachDB cluster with enterprise changefeeds Deploy Snowflake Deploy Kafka Connect Verify Conclusion Detailed Instructions Deploy a CockroachDB Cluster With Enterprise Changefeeds Start by either launching a CockroachDB instance or utilizing a managed service.

REST and HTTP Semantics

Aggregated on: 2024-09-10 20:21:10

Roy Fielding created REST as his doctorate dissertation. After reading it, I would boil it down to three basic elements: A document that describes object state A transport mechanism to transmit the object state back and forth between systems A set of operations to perform on the state While Roy was focused solely on HTTP, I don't see why another transport could not be used. Here are some examples:

Explainable AI (XAI): How Developers Build Trust and Transparency in AI Systems

Aggregated on: 2024-09-10 19:21:10

Developers working on Explainable AI (XAI) must address several key aspects, such as the problem, boundary scope, and potential solutions of XAI, as well as some specific use cases and benefits that can enhance an organization's credibility when implementing or leveraging this technology. The more AI is incorporated into different sectors, developers play a critical role in making those systems interpretable and transparent. XAI is crucial in making AI models highly interpretable and debuggable; it also guarantees responsible use of highly complex AI technologies, which should be fair, transparent, and accountable to society's users and stakeholders.

Transforming Continuous Delivery With Feature Flags

Aggregated on: 2024-09-10 18:21:10

Continuous Delivery is a practice and methodology that helps you build and deploy your software faster so that it can be released to production systems at any time. It facilitates shortening the lifecycle times of various development and operations processes. Effectively applying the concepts of Continuous Integration (CI) and Continuous Deployment (CD) helps achieve the benefits of the continuous delivery principles, also enabling faster software releases. We explore the challenges encountered by software teams implementing CI/CD and demonstrate how feature flags can help mitigate these risks.

Accelerate Your Journey to a Modern Data Platform Using Coalesce

Aggregated on: 2024-09-10 17:06:10

Most organizations face challenges while adapting to data platform modernization. The critical challenge that data platforms have faced is improving the scalability and performance of data processing due to the increased volume, variety, and velocity of data used for analytics. This article aims to summarize answers to the challenging questions of data platform modernization, and here are a few questions:

Digital Garden

Aggregated on: 2024-09-10 16:06:10

The topic of note-taking remains relevant today. We know the benefits it provides to the author. We are familiar with various approaches to note-taking and tools that can be used, and we have choices. Imagine you have found your approach, your tool, and your note base is growing and pleasing to the eye. What next? I want to discuss one path of development in this area. This article is dedicated to the concept of a digital garden — the philosophy of publicly maintaining personal notes.

How Can the Scalability of MySQL Be Improved for BenchmarkSQL TPC-C Testing?

Aggregated on: 2024-09-10 15:06:10

Current State of MySQL 5.7 MySQL 5.7 is not ideal in terms of scalability. The following figure illustrates the relationship between TPC-C throughput and concurrency in MySQL 5.7.39 under a specific configuration. This includes setting the transaction isolation level to Read Committed and adjusting the innodb_spin_wait_delay parameter to mitigate throughput degradation. Figure 1: Scalability problems in MySQL 5.7.39 during BenchmarkSQL testing

Mastering Multi-Cloud and Edge Data Synchronization: A Retail Use Case With KubeMQ’s Java SDK

Aggregated on: 2024-09-10 14:06:10

In today’s rapidly evolving enterprise landscape, managing and synchronizing data across complex environments is a significant challenge. As businesses increasingly adopt multi-cloud strategies to enhance resilience and avoid vendor lock-in, they are also turning to edge computing to process data closer to the source. This combination of multi-cloud and edge computing offers significant advantages, but it also presents unique challenges, particularly in ensuring seamless and reliable data synchronization across diverse environments. In this post, we’ll explore how the open-source KubeMQ’s Java SDK provides an ideal solution for these challenges. We’ll focus on a real-life use case involving a global retail chain that uses KubeMQ to manage inventory data across its multi-cloud and edge infrastructure. Through this example, we’ll demonstrate how the solution enables enterprises to achieve reliable, high-performance data synchronization, transforming their operations.

Top Security Flaws Hiding in Your Code Right Now and How To Fix Them

Aggregated on: 2024-09-10 13:51:10

In 2019, a famous breach in Fortnite, the famous game, reportedly put millions of players at risk of malware. The incident highlighted the importance of properly securing SQL databases. But this is not an isolated issue.

Zero To AI Hero, Part 2: Understanding Plugins in Semantic Kernel, A Deep Dive With Examples

Aggregated on: 2024-09-10 13:06:10

I talked a little about semantic kernel in Part 1 but decided to leave out the juicy part. There are a few gears in SK. They spin and connect like the inside of the watch to make it do the magic. One of those gears is plugins. Plugins are the robotic arms of Semantic Kernel, capable of doing some work beyond chit-chatting. Why Do We Need Plugins? We all know GenAI is about — surprise, surprise: "Generative AI". What do they train on? Tokens. These tokens are then mapped onto vector databases and can be used to predict the next token, etc, and the story goes on and on. What are these models capable of? Predicting subsequent tokens, and that's about it (at least as of now). So, for those who think GAI is equal to AGI, no! Not yet!

Spring Boot GoT: Game of Trace!

Aggregated on: 2024-09-09 23:06:09

When we, developers, find some bugs in our logs, this sometimes is worse than a dragon fight! Let's start with the basics. We have this order of severity of logs, from most detailed to no detail at all:

Working With Multiple testng.xml Files

Aggregated on: 2024-09-09 20:06:09

While working on an open source GitHub project that was created to showcase the working of Selenium WebDriver framework with Java, as the project grew, there was a need to create multiple testng.xml files for running different tests. These multiple files were created to segregate the tests and place all the tests related to a respective website in a single testng.xml (I have used different demo websites to demo different actions that can be automated using Selenium WebDriver).

Model-Driven Development and Testing

Aggregated on: 2024-09-09 19:06:09

The Meta of Design With several decades of experience, I love building enterprise applications for companies. Each solution requires a set of models: an SQL database, an API (Application Programming Interface), declarative rules, declarative security (role-based access control), test-driven scenarios, workflows, and user interfaces. The "meta" approach to design requires thinking of how each of these components interacts with the other. We also need to understand how changes in the scope of the project impact each of these meta-components. While I have worked in many different languages (APL, Revelation/PICK, BASIC, Smalltalk, Object/1, Java, JavaScript, Node.js, Python) these models are always the foundation that influences the final integrated solution. Models are meta abstractions that describe how the shape, content, and ability of the object will behave in the running environment regardless of language, platform, or operating system (OS). Model First Approach Starting with an existing SQL Schema and a good ORM allows the abstraction of the database and the generation of an API. I have been working with ApiLogicServer (a GenAI-powered Python open-source platform) which has a command line interface to connect the major SQL databases and create an SQLAlchemy ORM (Object-Relational Model). From this model, an Open API (aka Swagger) for JSON API is created, and a YAML file (model) drives a react-admin runtime. The YAML file is also used to build an Ontimize (Angular) user interface. Note that the GenAI part of ApiLogicServer lets me use a prompt-driven approach to get this entire running stack using just a few keywords.