News Aggregator

Unit Integration Testing With Testcontainers Docker Compose

Aggregated on: 2024-10-02 13:21:22

Is your test dependent on multiple other applications and do you want to create an integration test using Testcontainers? Then the Testcontainers Docker Compose Module is the solution. In this blog, you will learn how convenient it is to create an integration test using multiple Testcontainers. Enjoy! Introduction Using Testcontainers is a very convenient way to write a unit integration test. Most of the time, you will use it in order to test the integration with a database, a message bus, etc. But what if your application interacts with an application that consists of multiple containers? How can you add these containers to your unit integration test? The answer is quite simple: you should use the Docker Compose Module.

Kubernetes Observability: Lessons Learned From Running Kubernetes in Production

Aggregated on: 2024-10-02 00:21:22

Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Kubernetes in the Enterprise: Once Decade-Defining, Now Forging a Future in the SDLC. In recent years, observability has re-emerged as a critical aspect of DevOps and software engineering in general, driven by the growing complexity and scale of modern, cloud-native applications. The transition toward microservices architecture as well as complex cloud deployments — ranging from multi-region to multi-cloud, or even hybrid-cloud, environments — has highlighted the shortcomings of traditional methods of monitoring.

Leveraging Artificial Intelligence for Software Testing To Test Healthcare Applications

Aggregated on: 2024-10-01 22:21:23

The healthcare domain is complex, and Medicare often undervalues the potential of patients to adapt to modern technology. Despite this, technology can significantly enhance healthcare software products. However, the acquisition of these advancements is not easy for patients, particularly those on Medicare or with chronic conditions, due to unique obstacles. To offer top-notch software products, it's necessary to stay up-to-date with technology. AI-driven solutions are on the verge of rapid evolution, offering transformative benefits. For instance, predictive analytics that utilizes artificial intelligence can predict health risks based on patient data, which can lead to timely interventions. Furthermore, administrative tasks are streamlined through automated systems, resulting in improved efficiency and cost reduction.

Balance Trade-Offs in Microservices Architecture

Aggregated on: 2024-10-01 21:21:22

When architecting microservices, we deal with different non-functional requirements and their trade-offs. It's important to keep the right balance among different architectural aspects. Every architectural decision has its advantages and disadvantages, the challenge is finding a balance among them as much as possible according to what each system needs. We must balance the trade-offs in microservices architecture between various functional and non-functional elements to maximize the system efficiently. Following are the key trade-offs you can deal with in implementing the microservices, which are discussed with different scenario-based strategies:

How To Improve the Developer vs. AI Relationship

Aggregated on: 2024-10-01 20:21:22

Developers are often skeptical of artificial intelligence. Count me as one of them. But the potential of AI tools — in the right context, of course — should outweigh our skepticism, regardless of how justified it may be. Like it or not, the path to faster releases and better products runs through AI. It’s on the development community to adapt.

Workarounds for Oracle Restrictions on the Size of Expression Lists

Aggregated on: 2024-10-01 19:21:22

When developing an enterprise system — whether it is a completely new system or simply the addition of a new feature — it is not uncommon to need to retrieve a significant volume of records (a few hundred or even thousands) from an underlying relational database. A usual scenario is having a list of identifiers (user IDs, for example) and needing to retrieve a set of data from each of them. Disregarding all the ease provided by Object-Relational Mappers, among which Hibernate can be underlined as a very reliable option, a straightforward way to address such a situation is to build a SQL SELECT query and bind all identifiers as a comma-delimited list of expressions in combination with a SQL IN operator. The snippet below shows the query structure.

Best Practices for Market Data Replay

Aggregated on: 2024-10-01 18:21:22

This tutorial introduces the practical implementation of the DolphinDB’s replay feature. Based on DolphinDB’s streaming replay, distributed database architecture, and APIs, you can create a powerful tool for historical insight and model back-testing, which allows you to review situations of interest and improve future performance. By following the instructions in this tutorial, you will learn: The workflow of constructing a market replay solution The optimal data storage plan for data sources The methods to process and analyze replayed data via APIs The performance of the replay functionality Note: For detailed instructions on how to replay market data, refer to Tutorial: Market Data Replay.

Using Lombok Library With JDK 23

Aggregated on: 2024-10-01 17:21:22

Java 23 is finally out, and we can start migrating our project to it. The very first pitfall comes quickly when switching to the latest JDK 23 with compilation issues when using the Lombok library in your project. Let's begin with the symptom description first. Description The Lombok library heavily relies on annotations. It's used for removing a lot of boilerplate code; e.g., getters, setters, toString, loggers, etc.

Using AWS WAF Efficiently To Secure Your CDN, Load Balancers, and API Servers

Aggregated on: 2024-10-01 16:21:22

The introduction of software has made remarkable changes to how business is conducted. "Back then," people would meet in person, and most companies used manual methods, which were not scalable. Software has changed the game, and web applications are essential for a business's success. Software is how customers interact with businesses, share their data, and receive goods and services. Software-as-a-service (SaaS) has become a giant industry, taking care of hosting services used by customers by upgrading, scaling, and securing customer data. With the massive proliferation of SaaS services, many are using AWS, and security is a big concern. Malicious actors seek to steal customer data or DDoS-ing the service to prevent legitimate customers from accessing the website.

Guarding Kubernetes From the Threat Landscape: Effective Practices for Container Security

Aggregated on: 2024-10-01 15:21:22

Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Kubernetes in the Enterprise: Once Decade-Defining, Now Forging a Future in the SDLC. Kubernetes is driving the future of cloud computing, but its security challenges require us to adopt a full-scale approach to ensure the safety of our environments. Security is not a one-size-fits-all solution; security is a spectrum, influenced by the specific context in which it is applied. Security professionals in the field rarely declare anything as entirely secure, but always as more or less secure than alternatives. In this article, we are going to present various methods to brace the security of your containers.

Azure Deployment Using FileZilla

Aggregated on: 2024-10-01 14:21:22

In today's digital landscape, deploying web applications to the cloud is a common practice. Azure provides various deployment options, including GitHub, Azure DevOps, Bitbucket, FTP, or a local Git repository. In this step-by-step guide, we will focus on the FileZilla FTP client as a means to publish your Angular UI application to Azure. Follow these steps to make your Angular app accessible to the world.

A Hands-On Guide to OpenTelemetry: Linking Metrics to Traces With Exemplars, Part 2

Aggregated on: 2024-10-01 13:21:22

Are you ready to start your journey on the road to collecting telemetry data from your applications? Great observability begins with great instrumentation! In this series, you'll explore how to adopt OpenTelemetry (OTel) and how to instrument an application to collect tracing telemetry. You'll learn how to leverage out-of-the-box automatic instrumentation tools and understand when it's necessary to explore more advanced manual instrumentation for your applications. By the end of this series, you'll have an understanding of how telemetry travels from your applications to the OpenTelemetry Collector, and be ready to bring OpenTelemetry to your future projects. Everything discussed here is supported by a hands-on, self-paced workshop authored by Paige Cruz.

Building a CI/CD Pipeline With Kubernetes: A Development Guide With Deployment Considerations for Practitioners

Aggregated on: 2024-09-30 23:51:21

Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Kubernetes in the Enterprise: Once Decade-Defining, Now Forging a Future in the SDLC. In the past, before CI/CD and Kubernetes came along, deploying software to Kubernetes was a real headache. Developers would build stuff on their own machines, then package it and pass it to the operations team to deploy it on production. This approach would frequently lead to delays, miscommunications, and inconsistencies between environments. Operations teams had to set up the deployments themselves, which increased the risk of human errors and configuration issues. When things went wrong, rollbacks were time consuming and disruptive. Also, without automated feedback and central monitoring, it was tough to keep an eye on how builds and deployments were progressing or to identify production issues.

Building a Food Inventory Management App With Next.js, Material-UI, Firebase, Flask, and Hugging Face

Aggregated on: 2024-09-30 23:21:21

These days, restaurants, food banks, home kitchens, and any other business that deals with products and foods that go bad quickly need to have good food inventory management. Kitchens stay organized and waste is kept to a minimum by keeping track of stock, checking expiration dates, and managing usage well. I will show you how to make a Food Inventory Management App in this guide. With this app, users can:

Network Sniffing: A Critical Concept in Network Security

Aggregated on: 2024-09-30 22:21:21

What Is Network Sniffing? Sniffing includes the passive interception of data packets crossing a network with further analysis. Initially, sniffing was developed to help network administrators troubleshoot connectivity problems, and since then, it has evolved into an important technique of network management and security testing. How Does Network Sniffing Work? Basically, network sniffing is based on the fact that a network interface card is set in "promiscuous mode:" all network traffic crossing its path is captured, not just packets addressed to the device. Then sniffing software analyzes the packets captured, allowing a user to read and interpret the data crossing the network.

A Decade of Excellence: The Journey, Impact, and Future of Kubernetes

Aggregated on: 2024-09-30 20:51:21

Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Kubernetes in the Enterprise: Once Decade-Defining, Now Forging a Future in the SDLC. A decade ago, Google introduced Kubernetes to simplify the management of containerized applications. Since then, it has fundamentally transformed the software development and operations landscape. Today, Kubernetes has seen numerous enhancements and integrations, becoming the de facto standard for container orchestration.

Using Spring AI With LLMs To Generate Java Tests

Aggregated on: 2024-09-30 20:21:21

The AIDocumentLibraryChat project has been extended to generate test code (Java code has been tested). The project can generate test code for publicly available GitHub projects. The URL of the class to test can be provided then the class is loaded, the imports are analyzed and the dependent classes in the project are also loaded. That gives the LLM the opportunity to consider the imported source classes while generating mocks for tests. The testUrl can be provided to give an example to the LLM to base the generated test. The granite-code and deepseek-coder-v2 models have been tested with Ollama. The goal is to test how well the LLMs can help developers create tests.

Geo-Location Redirects With AWS CloudFront

Aggregated on: 2024-09-30 19:06:21

In the age of global digital services, geo-location-based content is critical for improving user experience and engagement, especially if you implement any shops or subscription services that should be adopted by the local market. AWS CloudFront is one of the most widely used content delivery network (CDN) systems. It includes certain necessary features to implement geo-location-based redirection out of the box, but not in a single click.

How To Improve RAG Quality by Storing Knowledge Graph in Vector Database

Aggregated on: 2024-09-30 18:06:21

Retrieval-Augmented Generation (RAG) techniques, by integrating external knowledge bases, provide additional contextual information for LLMs, effectively alleviating issues such as hallucination and insufficient domain knowledge of LLM. However, relying solely on general knowledge bases has its limitations, especially when dealing with complex entity relationships and multi-hop questions, where the model often struggles to provide accurate answers. Introducing Knowledge Graphs (KGs) into the RAG system provides a new solution to this problem. KGs present entities and their relationships in a structured manner, offering more refined contextual information during retrieval. By leveraging the abundant relational data of KGs, RAG can not only pinpoint relevant knowledge more accurately but also better handle complex question-answering scenarios, such as comparing entity relationships or answering multi-hop questions.

Why Haven’t You Upgraded to HTTP/2?

Aggregated on: 2024-09-30 17:06:21

From a Java perspective, I’ve been the beneficiary of some pretty amazing features over the years: Generics (Java 5) Streams and Lambda Expressions (Java 8) Enhanced Collection Functionality (Java 9) Sealed Classes (Java 17) As key features become available, I’ve been able to reduce development time as I implement features, while also seeing benefits in performance and supportability.

Transformers: Age of Attention

Aggregated on: 2024-09-30 16:06:21

In 2017, a group of researchers (from Google and the University of Toronto) introduced a new way to handle natural language processing (NLP) tasks. Their revolutionary paper “Attention is All You Need” presented the Transformer model, an architecture that has since become the basis of many advanced AI systems today. The model's performance, scalability, and versatility have led to its widespread adoption, forming the backbone of state-of-the-art models like BERT (Bidirectional Encoder Representations) and GPT (Generative Pre-trained Transformers). Before the Transformer model, most AI models that processed language relied heavily on a type of neural network called a Recurrent Neural Network (RNN) or its improved version, the Long Short-Term Memory Network (LSTM). In particular, problems like language modeling and machine translation (also called sequence transduction). These models processed words in a sequence, one by one, from left to right (or vice versa). While this approach made sense because words in a sentence often depend on the previous words, it had some significant drawbacks:

Agile Primitives

Aggregated on: 2024-09-30 15:06:21

TL;DR: Agile Primitives Are we losing sight of what truly matters in Agile? The future isn’t about rigidly adhering to or outright dismissing frameworks like Scrum or SAFe. Instead, it’s about returning to the fundamental principles — the Agile Primitives — that genuinely empower teams to adapt, innovate, and deliver real value. By refocusing on these core elements, organizations can move beyond mere methodological compliance and embrace authentic agility. Dive in to explore how the Agile community is coming full circle, rediscovering the essence that made Agile transformative in the first place.

Stream Processing in the Serverless World

Aggregated on: 2024-09-30 14:06:21

It’s a very dynamic world today. Information moves fast. Businesses generate data constantly. Real-time analysis is now essential. Stream processing in the serverless cloud solves this. Gartner predicts that by 2025, over 75% of enterprise data will be processed outside traditional data centers. Confluent states that stream processing lets companies act on data as it's created. This gives them an edge. Real-time processing reduces delays. It scales easily and adapts to changing needs. With a serverless cloud, businesses can focus on data insights without worrying about managing infrastructure.

Maximizing Cloud Network Security With Next-Generation Firewalls (NGFWs): Key Strategies for Performance and Protection

Aggregated on: 2024-09-30 13:06:21

As cloud networks continue to expand, security concerns become increasingly complex, making it critical to ensure robust protection without sacrificing performance. One key solution organizations use to achieve this balance is the deployment of Next-Generation Firewalls (NGFWs), which play an essential role in securing cloud environments. These advanced firewalls are integral to cloud security strategies, combining multiple layers of defense with optimized performance to tackle evolving threats. Understanding NGFWs in Cloud Networks To fully appreciate the scope of cloud network security, it's essential to understand both the capabilities of NGFWs and how they integrate into broader cloud security approaches. NGFWs go beyond traditional firewalls, offering advanced features such as deep packet inspection, application-level filtering, intrusion prevention systems (IPS), SSL/TLS inspection, and threat intelligence integration. Together, these tools help NGFWs safeguard dynamic cloud environments from sophisticated attacks, ensuring a higher level of protection.

Using AI in Your IDE To Work With Open-Source Code Bases

Aggregated on: 2024-09-28 13:06:20

Thanks to langchaingo, it's possible to build composable generative AI applications using Go. I will walk you through how I used the code generation (and software development in general) capabilities in Amazon Q Developer using VS Code to enhance langchaingo. Let's get right to it!

Leveraging IBM WatsonX Data With Milvus To Build an Intelligent Slack Bot for Knowledge Retrieval

Aggregated on: 2024-09-27 22:21:20

In today's fast-paced work environment, quick and easy access to information is crucial for maintaining productivity and efficiency. Whether it's finding specific instructions in a runbook or accessing key knowledge transfer (KT) documents, the ability to retrieve relevant information swiftly can make a significant difference. This tutorial will guide you through building an intelligent Slack bot that leverages IBM WatsonX.data and Milvus for efficient knowledge retrieval. By integrating these tools, you'll create a bot that can search and provide answers to queries based on your organization's knowledge sources. We will use IBM WatsonX.data to populate and query relevant documents and IBM WatsonX.ai to answer questions from the fetched documents.

Indexes Under The Hood

Aggregated on: 2024-09-27 20:21:20

Many things can improve database performance. One of the most obvious solutions is to use indexes. This blog post will examine them in detail to understand how they work in PostgreSQL and what makes them useful. Data Storage Basics Before getting to indexes, we need to understand the basics of data storage in PostgreSQL. We will explain how things are represented internally, but we won’t cover all the aspects of the data storage like shared memory and buffer pools.

Creating MVPs on a Budget: A Guide for Tech Startups

Aggregated on: 2024-09-27 18:51:20

You must have used Snapchat and Uber, but do you know how they started? These widely used applications began as minimum viable products (MVPs). Uber was initially launched as “UberCab,” a basic, on-demand ride-hailing app with one main feature: allowing users in San Francisco to book black car services through their smartphones. Similarly, Snapchat started as a single-feature application called “Picaboo,” enabling users to send photos that would disappear after a few seconds.

Hands-On: Assigning Pods to Nodes Using Affinity Rules

Aggregated on: 2024-09-27 17:06:20

This article describes how to configure your Pods to run in specific nodes based on affinity and anti-affinity rules. Affinity and anti-affinity allow you to inform the Kubernetes Scheduler whether to assign or not assign your Pods, which can help optimize performance, reliability, and compliance. There are two types of affinity and anti-affinity, as per the Kubernetes documentation:

Is Spring AI Strong Enough for AI?

Aggregated on: 2024-09-27 15:51:20

In recent years, there has been a significant surge in the adoption of artificial intelligence (AI) and machine learning (ML) technologies across a wide range of industries. Frameworks such as TensorFlow, PyTorch, and Scikit-learn have emerged as popular choices for AI development due to their versatility and robustness. However, the seamless integration of AI into enterprise-grade, production-ready applications poses unique challenges that need to be addressed. Spring, a widely recognized enterprise-level framework, is celebrated for its exceptional robustness, scalability, and flexibility in crafting sophisticated applications. Nevertheless, the question arises: can Spring effectively cater to the intricate demands of AI/ML-based applications? This article aims to explore the depths of Spring's capabilities within the AI domain, its potential integration with AI libraries, and its ability to effectively manage AI workflows within production environments.

Low-Code AI Agent for Classifying User Support Tickets With OpenAI and Kumologica

Aggregated on: 2024-09-27 13:06:20

This article will demonstrate how Kumologica and OpenAI can assist in developing an AI agent API that efficiently classifies cases within an enterprise using user data, without the need for customer support agent intervention. An ideal case management solution should be able to automatically categorize and prioritize cases from various channels, identify high-priority cases, apply labels, and assign them to the appropriate groups without requiring agents to manually perform these tasks. There are several case management products currently available on the market, such as ServiceNow, JIRA, and Salesforce Case Management. While some of these products include built-in solutions for ticket classification, others offer no such functionality. Even for those with built-in solutions, there are limitations in their ability to integrate with third-party systems.

Advanced Strategies for Building Modern Data Pipelines

Aggregated on: 2024-09-26 23:51:19

In today's data-driven world, organizations increasingly rely on sophisticated data pipelines to manage vast volumes of data generated daily. A data pipeline is more than just a conduit for data — it is a complex system that involves the extraction, transformation, and loading (ETL) of data from various sources to ensure that it is clean, consistent, and ready for analysis. This article explores advanced strategies, the process of building data pipelines, and the pillars of a successful modern data strategy, with a focus on both real-time and batch data processing. Steps for Building Data Pipelines Building modern data pipelines that are efficient, scalable, and resilient requires advanced strategies and a deep understanding of the underlying technologies. Let’s dive into the key steps to building out your data pipelines.

Integrating Data Engineering Into Artificial Intelligence

Aggregated on: 2024-09-26 23:06:19

Data is often referred to as the lifeblood of artificial intelligence and machine learning (ML). We know the value of data: it fuels insights, provides input for predictions, and is foundational for data engineers' and data scientists' decisions. The journey from raw data to meaningful information is lengthy which is where data engineering fits into play. Incorporating data engineering and dealing with the messiness of the dataset is crucial to having a smooth flow of data through the ML pipeline, leading us to better models and consistent results. In this article, I’ll take a deep dive into how data engineering and Machine Learning intersect and why it is important in this era of modern data-driven applications. Importance of Data Engineering in Artificial Intelligence Machine learning relies on data and its accuracy. Raw data transformation involves landing data, cleaning, transformation as per requirements, and loading to the target to make it meaningful. These activities take from extracting the source systems up to a point where you analyze using some tool/configuration. The ML lifecycle has these stages, and data engineering is the focus of each step.

Maturing an Engineering Organization From DevOps to Platform Team

Aggregated on: 2024-09-26 22:06:19

The DevOps model broke down the wall between development and production by assigning deployment and production management responsibilities to the application engineers and providing them with infrastructure management tools. This approach expanded engineers' competencies beyond their initial skill sets. This model helped companies gain velocity as applications weren't passed around from team to team, and owners became responsible from ideation to production. It shortened the development lifecycle and time to deployment, making companies more agile and responsive.

Java for AI

Aggregated on: 2024-09-26 21:06:19

Artificial Intelligence (AI) is modernizing industries by enabling machines to perform tasks that typically require no human intervention; tasks such as problem-solving, natural language understanding, and image processing. For AI-related software development, Python is often used. However, Java is also a powerful option, as many organizations are using it in enterprise applications due to its robustness and scalability. In this article, we explore how the Java programming language can be used for AI development, along with supporting libraries and tools. Java Programming Language for AI Java offers several features that make Java suitable for AI-related task development:

Angular RxJS Unleashed: Supercharge Your App With Reactive Operators

Aggregated on: 2024-09-26 20:06:19

Angular is one of the most popular frameworks for building dynamic web applications, and at its core lies RxJS, is a library designed for composing asynchronous and event-based programs by using observable sequences. RxJS operators are powerful tools that allow developers to manipulate and transform data streams efficiently. But how exactly do you use them in Angular? This article will provide a detailed, step-by-step guide on how to leverage RxJS operators in Angular, complete with examples to help you master this essential skill. What Are RxJS Operators? In Angular, RxJS operators play a key role in managing HTTP requests, user actions, and other asynchronous tasks, making it easier to chain operations and work with data streams. These operators are essentially functions that take an Observable as input and produce a new Observable as output. This allows developers to efficiently handle asynchronous workflows, event handling, and state management. RxJS operators are grouped into categories like creation, transformation, filtering, and combination, offering a powerful toolkit for composing and managing complex operations.

Using Flink, Kafka, and NiFi for Real-Time Airport Arrivals and Departures

Aggregated on: 2024-09-26 19:06:19

In this stage of development of our real-time data pipeline, we are starting to build up all of the feeds we need to be able to make smart decisions quickly and provide all the necessary data to AI and ML models to things like answer LLM/NLP chat questions on how should I go somewhere if I am leaving tomorrow, now, or soon. This will incorporate weather, air quality, roads, buses, light rail, rail, planes, social media, travel advisories, and more. As part of this, we will provide real-time notifications to users via Slack, Discord, Email, Web socket front-ends, and other dashboards. I am open to working with collaborators in open source or suggestions for end-user applications and other data processors like my friends at RisingWave, Timeplus, StarTree Pinot, LLM/Vector Database collaborators like Zilliz Milvus, IBM watsonx.ai, and others. openskyairport — nifi — kafka — flink SQL Photo by Jue Huang on Unsplash REST API To Obtain Airport Information https://opensky-network.org/api/flights/arrival?airport=${airport} &begin=${now():toNumber():divide(1000):minus(604800)} &end=${now():toNumber():divide(1000)}

Backpressure in Distributed Systems

Aggregated on: 2024-09-26 17:06:20

An unchecked flood will sweep away even the strongest dam. – Ancient Proverb

Augmenting the Client With Vue.js

Aggregated on: 2024-09-26 16:06:19

In my previous post, I laid the ground to build upon. Now is the time to start "for real". I heard a lot of Vue.js. Additionally, a friend who transitioned from developer to manager told me good things about Vue, which further piqued my interest. I decided to take a look at it. It will be the first "lightweight" JavaScript framework I'll study — from the point of view of a newbie, which I am.

Building a Zero Trust API With ASP.NET Core: A Developer’s Guide

Aggregated on: 2024-09-26 01:21:19

In a world where cyber threats are just one click away (or just one QR code scan away), the old-school “castle and moat” security approach isn’t enough. Enter Zero Trust — a security model that flips the script, requiring every request to be verified, authenticated, and monitored, no matter where it comes from. For developers working with ASP.NET Core APIs that handle sensitive data, this isn’t just a trend; it’s a necessity. Let’s dive into the why, what, and how of implementing Zero Trust in your ASP.NET Core API.

Concurrency Issues in SQL and Distributed Systems

Aggregated on: 2024-09-26 00:36:19

An SQL database must handle multiple incoming connections simultaneously to maintain optimal system performance. The expectation is that the database can accept and process numerous requests in parallel. It's straightforward when different requests target separate data, such as one request reading from Table 1 and another from Table 2. However, complications arise when multiple requests involve reading from and writing to the same table. How do we keep the performance high and yet avoid consistency issues? Let’s read on to understand how things work in SQL databases and distributed systems. Transactions and Issues An SQL transaction is a collection of queries (such as SELECT, INSERT, UPDATE, DELETE) sent to the database to be executed as a single unit of work. This means that either all queries in the transaction must be executed, or none should be. Executing transactions is not atomic and takes time. For example, a single UPDATE statement might modify multiple rows, and the database system needs time to update each row. During this update process, another transaction might start and attempt to read the modified rows. This raises the question: should the other transaction read the new values of the rows (even if not all rows are updated yet), the old values of the rows (even if some rows have been updated), or should it wait? What happens if the first transaction must be canceled later for any reason? How should this affect the other transaction?

Automating Data Pipelines With Snowflake: Leveraging DBT and Airflow Orchestration Frameworks for ETL/ELT Processes

Aggregated on: 2024-09-25 22:51:19

In the era of digitization and data landscape, automating data pipelines is crucial for enhanced efficiency, consistency, and scalability of the lake house. Snowflake is a leading cloud data platform that integrates seamlessly with various tools to facilitate the automation of ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) processes. This article delves into automating data pipelines with Snowflake by leveraging dbt (data build tool) and orchestration frameworks and the best practices for streamlining data workflows to ensure reliable data processing.

Feature Engineering Transforming Predictive Models

Aggregated on: 2024-09-25 21:51:19

Imagine you’re building a model to predict house prices: two models, identical in every aspect except one; one uses raw data, and the other leverages thoughtfully engineered features like the age of the house, proximity to schools, and seasonal price trends. Which model do you think performs better? The answer is intuitive: the latter. Feature engineering is the process of using domain knowledge to create features that make machine learning algorithms work more effectively. It bridges the gap between raw data and the insights needed to drive decision-making. In this article, we’ll explore how feature engineering can significantly impact the performance of your predictive models.

Securing Your Azure Kubernetes Services Cluster

Aggregated on: 2024-09-25 20:51:19

In this article, I will present my perspective on securing an Azure Kubernetes cluster with the principle of least privilege as a top priority. I will explain the available built-in Azure Kubernetes Roles, the function of the Microsoft Entra (formerly Azure Active Directory) groups, and how to utilize Kubernetes RBAC to manage access to the workloads. Photo by "ArminH" on Freeimages.com

Probabilistic Graphical Models: A Gentle Introduction

Aggregated on: 2024-09-25 17:21:19

What Are Probabilistic Graphical Models (PGMs)? Probabilistic models represent complex systems by defining a joint probability distribution over multiple random variables, effectively capturing the uncertainty and dependencies within the system. However, as the number of variables increases, the joint distribution grows exponentially, making it computationally infeasible to handle directly. Probabilistic Graphical Models (PGMs) address this challenge by leveraging the conditional independence properties among variables and representing them using graph structures. These graphs allow for a more compact representation of the joint distribution, enabling the use of efficient graph-based algorithms for both learning and inference. This approach significantly reduces computational complexity, making PGMs a powerful tool for modeling complex, high-dimensional systems. PGMs are extensively used in diverse domains such as medical diagnosis, natural language processing, causal inference, computer vision, and the development of digital twins. These fields require precise modeling of systems with many interacting variables, where uncertainty plays a significant role [1-3].

The Future of Application Security: Empowering Developers in the AI Era

Aggregated on: 2024-09-25 16:21:19

In an era where software vulnerabilities can lead to catastrophic breaches, application security has never been more critical. Yet, for many developers, security remains a complex and often frustrating aspect of the development process. At Black Hat 2024, I sat down with Randall Degges, Head of Developer Relations and Community at Snyk, to explore how the landscape of application security is evolving, particularly in light of cloud-native architectures, serverless computing, and the rise of AI-assisted development.

How To Convert Files to Thumbnail Images in Java

Aggregated on: 2024-09-25 15:21:19

In this article, we’ll discuss the benefits of programmatically generating thumbnail images for a wide range of file types. We’ll take a high-level look at resolving this workflow with open-source Java solutions, and we’ll learn how to take advantage of an efficient web API to get the job done. Why Convert Documents to Thumbnail Images? "Converting" documents to thumbnail images — i.e., programmatically generating thumbnail images from document contents — creates lightweight content previews that can be viewed quickly by document recipients with minimal effort.

Simplifying Multi-Cloud Observability With Open Source

Aggregated on: 2024-09-25 14:21:18

Gartner predicts by 2028, 50% of enterprises will utilize the cloud. The growth has also seen an increase in different strategies for organizations to use the cloud. Initially, organizations were completely on-prem, then they were hybrid where some workloads were still on-prem but some were migrated to the cloud. Eventually, companies started moving to multi-cloud where they use more than one cloud provider to host their workloads. A recent Oracle survey indicates that 98% of enterprises are either considering or already implementing a multi-cloud strategy. So what are the motivations for these enterprises to move towards multi-cloud?

Optimizing External Secrets Operator Traffic

Aggregated on: 2024-09-25 13:21:19

In Kubernetes, a Secret is an object that stores sensitive information like a password, token, key, etc. One of the several good practices for Kubernetes secret management is making use of a third-party secrets store provider solution to manage secrets outside of the clusters and configuring pods to access those secrets. There are plenty of such third-party solutions available in the market, such as: HashiCorp Vault Google Cloud Secret Manager AWS Secrets Manager Azure Key Vault These third-party solutions, a.k.a External Secrets Managers (ESM), implement secure storage, secret versioning, fine-grain access control, audit and logging.

Oracle NoSQL Database: A Comprehensive Guide for Developers

Aggregated on: 2024-09-24 20:21:18

About Oracle NoSQL Database Oracle NoSQL Database is a robust solution offering a blend of SQL-like interfaces for working with NoSQL data models. This database system enables developers to handle various data types without impedance mismatch issues, including flat relational data, hierarchical typed data, and schema-less JSON data. Impedance mismatch occurs when differences between the database and the programming language models create data handling challenges. Still, Oracle NoSQL eliminates this by providing a seamless experience across multiple data structures. Oracle NoSQL Database Cloud Service simplifies the development process, allowing developers to focus on building applications rather than managing infrastructure. It supports document, fixed schema, and key-value database models, providing low-latency response times, active-active regional replication, and elastic scalability. The service is designed to scale effortlessly with dynamic workloads, offering features such as: