How We Migrated Our Data Warehouse from Snowflake to DuckDB

Snowflake is great but sometimes a team needs to roll their own solution and this piece does a great job presenting an open source alternative in DuckDB. It presents the classic tradeoff of ease of use versus costs (at least in the open source software space). In either situation I'm sure I'd find a way to give our CTO heart attacks sending him fake screenshots of dropped database commands. - John Forstmeier

AI Copilots Are Changing How Coding Is Taught

It is interesting to consider the effect of AI tools like Copilot on students learning to program. I agree that in theory, these tools take the focus off of syntax and free up capacity to focus on problem solving instead. That being said, I am skeptical about relying heavily on things like Copilot, particularly when you are trying to learn a new language. I think these tools are great for once you are beyond the point of trying to learn the language, but should not be used a crutch or substitute for learning in the first place - Wes Poulsen

Amazon, Meta-Backed Scale AI Raises $1B, Boosting Value to $13.8B

The article on Scale AI raising $1 billion and reaching a valuation of $13.8 billion is significant for the data science community as it underscores the industry's growth and investor confidence, particularly from major players like Amazon and Meta. Scale’s advancements in data annotation and AI training tools directly impact the capabilities available to data scientists. Such funding boosts often lead to strategic partnerships, technological innovations, and new career opportunities, influencing the broader AI and data science landscape. - Johnathan Rodriguez

Data Engineering, Redefined

Data titles have always been interesting to me since I joined the data space. Bernd offers an interesting perspective in this article on why he thinks data engineering should be redefined. - Steven Johnson

Podcast Recommendations

Data Skeptic - June 28, 2024 - Analysis of Unstructured Data

Robbie Moon from the Georgia Tech Scheller College of Business discusses using natural language processing to analyze financial disclosures and predict stock performance. He covers techniques like sentiment analysis and machine learning models to classify text.

DataFramed - July 02, 2024 - The Future of Programming: Accelerating Coding Workflows with LLMs

This podcast episode covers everything you need to know about getting a job in AI in 2024, from the skills required to practical strategies for success. With perspectives from AI experts in both technical and business roles, it's a must-listen for anyone interested in joining the AI job market.

Quickly Generate Time-Series Synthetic Data with OpenAI’s Fine-Tuning API

Synthetic data is increasingly important in the context of training data as it can help avoid privacy concerns or fill in the gaps for limited available raw data. The post does a nice little recap, with helpful code snippets, of how to accomplish synthetic data generation using OpenAI fine-tuned models. Now I just need to figure out how to use this to trade meme stocks... - John Forstmeier

How Can Data Science Accelerate Drug Discovery Processes?

Miller asserts that data science is not just a supplementary tool but a crucial component in modern drug discovery and development. From accelerating drug discover and repurposing, to tailoring treatments, and enhancing drug safety, data science touches on a multitude of aspects of the drug industry. - Katt Baum

How behavioral biometrics reduces online banking fraud

We've implemented biometrics in our lives for awhile now - scanning a fingerprint, using our face to unlock our phones, etc. Behavioral Biometrics use machine learning to analyze movement, online interactions, and more. We use biometrics in order to ensure that the user attempting to access something is the correct person. With almost everything being online, bad actors have increased banking fraud with $3.3 billion in losses being reported in 2020. Implementing behavioral biometrics is the next step in alternative biometrics in order to prevent this. - Reed Cowan

Is your Data Platform Sabotaging your AI Dreams?

Data teams are under immense pressure - costs are soaring, value justification is a nightmare, and new AI projects keep piling up. How can data teams move quickly while keeping costs down in an incredibly fragmented data landscape? The answer lies in DATA READINESS, reimagined. It goes beyond traditional data readiness for BI, reporting, and basic analytics. It powers today's GenAI and LLM world. This article shares actionable steps to get your data ready for AI. - Jagdish Purohit from 5X

Podcast Recommendations

The Analytics Power Hour - June 24, 2024 - #248: The Fundamentally Fascinating World of APIs with Marco Palladino

This podcast episode discusses APIs - what they are, why they're important, and how infrastructure plays a key role in enabling developers to build reliable and scalable APIs. The hosts interview Marco Paladino, an API expert, who provides insightful perspectives.

Data Engineering Podcast - June 23, 2024 - Stitching Together Enterprise Analytics With Microsoft Fabric

Dipti Borkar discusses Microsoft's new Fabric product for building data platforms. She covers how Fabric simplifies data management, supports open formats like Delta Lake, leverages AI with Microsoft Co-Pilot, and opens up extensibility so other vendors can build on the platform.

Apache Iceberg - What is it?

If you've been hearing about Apache Iceberg, but you're not quite sure what it is or why it matters, Julien breaks it down in the most understandable way possible. It reminds me a lot of what AWS Athena was aiming for, but without the vendor lock-in. Store your files in any storage system and gain the ability to query those files easily with SQL. - Blake Burch

Data exploration using VIM

To all of you Vim connoisseurs (nerds) out there comes a nice little piece demonstrating how you can use Vim to dynamically explore data. It's actually pretty slick and is a neat alternative to the goto Jupyter notebook or something similar. - John Forstmeier

Data detective work: An anti-money laundering example

Ever wonder about the effectiveness of sanctions? This articles dives into the dynamic connections between economic sanctions, international finance, and the enforcement of human rights in such a rich manner! The article goes into how sanctions can be used as strategic and diplomatic tools while also bringing into like the complexities of tracking down financial flows using a unified knowledge graph. Using past examples, Morrison shows how efficient financial crime prevention can be! - Reed Cowan

12 Companies Hiring Data Engineers

With a lot of companies going through significant changes, it's always important to stay up to date on what the job market looks like. This article has a wide variety of open data roles across fintech, health tech and more! - Angel Catalan

Podcast Recommendations

The Analytics Power Hour - June 10, 2024 - Professional Development, Analytically Speaking with Helen Crossley

This episode features an experienced data leader sharing insights on goal-setting, providing effective feedback, transitioning from individual contributor to manager, and more. The guest gives direct, actionable advice to help analysts and managers develop their skills.

The Gradient: Perspectives on AI - June 13, 2024 - Values, Legibility, and Gamification

Philosopher Christopher Thi Nguyen discusses how metrics and scoring systems shape values. He argues they simplify complex realities to enable large-scale coordination but risk excluding nuance and individual well-being. Still, some tradeoffs between scalability and sensitivity may be worthwhile.

How to develop an AI-enabled SaaS product

Software as a Service grew due to Cloud computing, adding cost, scalability, and ease of use advantages. With AI on the rise, a new sect of Services have started to come about - AISaaS or Artificial Intelligence Software as a Service. These are AI tools that are made to help businesses grow at a larger rate by streamlining processes that were manual before hand. This article deconstructs what it takes to build your own AISaaS, and who knows, maybe inspire you to build something of your own! - Reed Cowan

The democratization and commoditization of AI/ML

While machine learning engineers will benefit from the AI boom, machine learning scientists could feel a pinch, according to the author. These scientists will need to move into more specialized fields of focus in order to continue providing value while generally-trained models dominate the low-hanging-fruit problems. I for one am excited to get back to fundamental maths. - John Forstmeier

The Ultimate Guide to Making Sense of Data

Walbaum argues that few companies are actually data driven; most rely on judgement calls before data. In order to steer the ship with data, you need to have a handle on what drives your business (and there's a helpful link in there to get you started). Overall, this is a lovely breakdown of how to decide which data metrics to track, how to monitor them, and what insights to extract. - Katt Baum

Boosting Efficiency and Performance with Innovative Data Engineering Strategies

Ashmin is a consummate data engineering professional and has an extensive background with a wide toolset. With talks on how automation can potentially displace jobs, Ashmin does a great job at talking through how automation will just allow engineers to focus on engineers, not admins. He also believes that seamlessly integrating systems is the future of successful data teams, a mission we strongly believe in as well. - Angel Catalan

Podcast Recommendations

Super Data Science: ML & AI Podcast with Jon Krohn - June 04, 2024 - 789: ML for Wind-Powered Energy Generation

Dr. Jason Yosinski shares how his startup Windscape AI uses machine learning to help wind turbines generate more clean energy. He discusses visualizing neural networks, making wind power predictable to stabilize grids, and enabling global ML research collaboration. 

Gradient Dissent: Conversations on AI - June 06, 2024 - AI in electronics: Quilter’s journey in PCB design

Sergiy Nesterenko, founder of Quilter AI, talks about how his company uses reinforcement learning to automate the complex process of designing printed circuit boards. He discusses the challenges of solving this spatial reasoning task with AI and adapting models for real-world use.

Building a data pipeline using Snowflake primitives

I love simplifying things (that's why I love Shipyard) and the author makes the case that using features provided by Snowflake, you can effectively "flatten" your data architecture. It's super appealing, particularly for a pretty simple/straightforward pipeline, to have most of the architecture all live within the same application. No code (architecture) is good code (architecture). - John Forstmeier

Day 1 Summit Overview

Starting a new data job can feel like drinking from a fire hose of data. Figuring out a new company's infra, building relationships, and proving your worth all while jumping onto projects with barely any info? New hires need resources that skip the fancy talk and give them real-world help. 5X's recently concluded Day1 Summit offers valuable resources for new data hires. - Jagdish Purohit from 5X

Data Libraries - the Secret Sauce to Regulatory Environments

This article explores how well-managed data libraries play a crucial role in navigating regulatory compliance challenges and driving efficiencies. It underscores the significance of centralizing data for informed decision-making and streamlined compliance processes, offering practical tips for optimizing resource allocation and reducing redundancy. Whether grappling with compliance issues or aiming to enhance operational efficiency, this article provides valuable strategies for leveraging data organization effectively in a data-driven landscape. - Jon Davidson

Data by Design

While this strays from the typical articles we share, Data by Design is series of in-depth interactive articles that walk through the history of data visualization through the lens of colonialism. While I have yet to read through all of the material, it is extremely well researched and provides engaging perspectives that are worth digging into. - Blake Burch

Podcast Recommendations

Vanishing Gradients - May 30, 2024 - Episode 27: How to Build Terrible AI Systems

Hugo speaks with Jason Liu, an independent consultant who uses his expertise in recommendation systems to help fast-growing startups build out their RAG applications. Jason talks about leveraging large language models to build AI apps, consulting for startups on building production LLM systems, avoiding common pitfalls, and taking an inverted thinking approach to build robust low latency AI. He discusses the future of AI engineering roles and shares lessons on maintaining simplicity.

The Analytics Power Hour - May 27, 2024 - I've Got 99 Analytical Methodologies and Need to Pick Just One

In this lively discussion, data experts share their struggles with picking the right analytical methodology for any given business question. With humor and humility, they admit there are no easy answers, but offer great tips like start simple, check assumptions, explain uncertainty, and learn by doing.

Alex Garcia-Outlier 2022-Command Line Data Visualizations

I'm breaking all the rules and including a YouTube video but I think it's a cool one. The presenter does a neat, short rundown of a number of command line-based tools for visualizing data. Super cool if you want to whip up a small script to impress your coworkers. - John Forstmeier

Data-Driven Sustainability: Beyond Numbers, Towards Action

Climate change is a serious matter that requires everyone's attention. Taking action towards a more sustainable future is crucial. The concept of data-driven sustainability is promising and should be embraced by companies sooner rather than later. - Jon Davidson

AI Helps Nestle Develop Drought, Disease Resistant Coffee

This showcases a practical application of data science in agriculture, highlighting how AI-driven methods are being used to improve crop resilience and sustainability. - Johnathan Rodriguez

Algorithmic Thinking for Data Scientists

Chinmay argues that algorithmic thinking is crucial for data science projects as organizations push to streamline their data operations. The recommendation I found most interesting is that one should develop an intuition around algorithmic thinking by solving example coding problems. By practicing, you not only can find solutions faster, but you can more easily identify problems that you might not have noticed otherwise. - Katt Baum

Podcast Recommendations

The TWIML AI Podcast - May 20, 2024 - Chronos: Learning the Language of Time Series

In this podcast, Abdul Fatir Ansari discusses Chronos, a new approach to time series forecasting using language models. He explains how Chronos tokenizes time series data so it can leverage transformer models like T5, allowing it to achieve strong performance even on unseen datasets. 

Data Skeptic - May 20, 2024 - HMMs for Behavior

This podcast explores how statisticians and ecologists collaborate to understand animal behavior using tracking data and time series models like hidden Markov models. It gives an overview of the field and discusses specific techniques for modeling movement patterns to infer behavioral states.

5 Ways AI Could Be Putting Your Business Under Threat

When I first read the title, I thought it was how AI might put businesses, well, out of business, but that is not what the article is about at all! The authors outline the various risks associated with integrating AI into business operations. Some of the more obvious risks are accuracy of information and the privacy concerns around using public LLMs. The one I found most interesting was the possible legal implications of using AI: who is held accountable if the AI breaks the law? - Katt Baum

How to ensure that your data science team isn't the bloated cost center?

In this piece the author makes the argument that it's important to integrate your data science team into multiple areas of the business rather than treating them like a walled off resources. While their mastery of algorithms is important, their first-hand understanding of their data is a competitive advantage - bringing them directly into the decision making process is a way of vastly improving your organizations effectiveness. Get those guys out of their cubicles for once. - John Forstmeier

Metadata management in data lakes

Metadata is exactly that- data about data.So why is it important to manage metadata effectively in data lakes? Metadata that is well-manages allows data management professionals to quickly discover relevant data within their data lakes, seamlessly translate and transform data for their users, allow innovation across their enterprise business, and much more! - Reed Cowan

How AI and Business Analytics Foster Better Collaboration

With the recent development surges in the AI space, it's no mistake that we're seeing AI work closely with businesses and their data teams. This article does a great job at painting the landscape for how AI is working with BI teams, specifically, and what to expect as data and AI develop together. - Angel Catalan

Podcast Recommendations

Talk Python To Me - May 15, 2024 - Pandas and Beyond with Wes McKinney

West McKinney, creator of Pandas, discusses the history and impact of Pandas, his work on other popular Python data science libraries like Apache Arrow, and the future of data analysis in Python.

DataFramed - May 16, 2024 - #207 Data Driven Venture Capital with Andre Retterath, Partner at Earlybird VC

This podcast discusses how data and AI can revolutionize venture capital investing. Andre shares how his firm implements data-driven techniques to source promising startups earlier and make better investment decisions. He provides tangible examples of models outperforming top human investors

Why did Golang lose to Rust for Data Engineering?

I think this is an interesting discussion for a few reasons, first of all I think Rust and Go are both excellent languages but it is true that Rust has a bigger footprint in the DE space. That being said, neither language are used as the optimal choice for standard tasks in data engineering. The fact remains that SQL and Python are still king, and not going to be directly replaced by Rust. I think the main effect of Rust will be felt in new libraries that are improvements on existing solutions in the Python ecosystem (i.e. Polars) - Wes Poulsen

Where did we come from? Exploring the explosion of interest in data and data tooling

I started my data journey when the "Modern" Data Stack was the goal for all companies. Not only was there a desire to have the best in breed tool, but the "explosion" of a lot of tools that accomplished niche tasks was highly sought out by companies looking to be data-forward. With the rise in AI, what is now considered to be "best in breed" has shifted, and this article does a great job at discussing the ushering in of the AI era in the data tooling space. - Angel Catalan

Stack Overflow signs deal with OpenAI to supply data to its models

OpenAI has made an agreement with StackOverflow to access its data. It's a reversal from their prior stance against their data being used to train models but makes sense since generative AI models have taken a chunk of their traffic. I for one am excited because I can now do CTRL+C, CTRL+V from ChatGPT instead of from both it and StackOverflow. - John Forstmeier

The Limits of Data

This article provides a deep dive into why you should always be incredulous of the data you're presented. This data likely contains biases and a lack of context that's inherent with the very nature of data collection. This was a fantastic read to be aware of in a world where it seems like data dominates every decision we make. - Blake Burch

Podcast Recommendations

Microsoft Research Podcast - Abstracts: May 6, 2024

This episode discusses Math Vista, a new benchmark for evaluating how well large language models can reason about math problems presented with both text and images. Dr. Michelle Galley explains why multimodal reasoning is an important but largely unexplored capability, how her team collected visual math reasoning data sets, and what they found when testing models like GPT-4.

Machine Learning Street Talk (MLST) - Can Machines Replace Us? (AI vs Humanity)

AI expert Maria Santacaterina discusses the promise and perils of AI, arguing we must ensure it serves humanity not replaces it. She provides an insightful critique of AI's failings and advocates for a 'human-centered' approach focused on empowering peopl

From Space to Story in Data Journalism

This is a super interesting overview of several recent examples of how satellite imagery has been used as a core source in journalism. Instead of just being visually supportive, the author argues investigative journalists should lean heavily into this data, as well as learn skills like programming and data analysis, to drive their work. - John Forstmeier

3 Reasons Data Engineers Are the Unsung Heroes of GenAI

AI is the hot new object on the technology scene, but seldom do people realize the important data is to having fundamentally useful LLM's. This article does a great job of discussing how exactly data engineers play a crucial role in accelerating the growth of AI and the interdependencies between AI and data. - Angel Catalan

A Comprehensive Guide to Essential Tools for Data Analysts

Rosidi is getting back to basics with this high-level guide for Data Analysts. Mostly for those new to the field, it doesn't hurt for an expert to take a look to see if they are missing out on something considered essential. My favorite part is the quick and tidy Tools Overview table... are you using any tools or languages in ways that aren't ideal? - Katt Baum

Machine Unlearning in 2024

The concept of Machine Unlearning was completely new to me, but in this in-depth article, Ken makes strong points in favor of adjusting existing ML models to remove training bias, copyright influence, and improve data privacy. - Blake Burch

Podcasts Recommendations

The PolicyViz Podcast - Charting Success: Nick Desbarats' Practical Approach to Data Visualization

In this episode, Nick Desbarats discusses his new book, Practical Charts, which provides specific guidelines for choosing chart types based on the data and goal. The episode debates controversial rules like never using box plots, aiming to clarify rather than simply simplify data visualization. The conversation highlights strategies to teach beginners while allowing flexibility for experienced designers.

Data Skeptic - Behavioral Genetics

This podcast discusses the genetics behind dog behavior and breeding. It explores how we can use science and research to better understand and improve outcomes in breeding programs for guide dogs as well as family pets.

How data impacts the digitalization of industries

Digitalization reaches far beyond just converting paper documents to documents stored on a computer. Digitalization shapes and molds a company's business strategy, their future projects, and the lessons they learn from past insights. However, how a company goes about digitalization defers on a case by case basis, and those cases rely heavily on data. Different sects of industry rely on data differently, and this article touches base on aa few different realms and how those industries incorporate data differently to drive their digitalization. - Reed Cowan

How Data Engineering Helps Health Systems Reach Their Most Critical Goals

As we've deepened our work within the healthcare segment, it's evident that a lot of these companies have been victims to legacy data platforms. This article outlines how healthcare organizations are pivoting to bring on data talent and the positive implications that brings for the teams making these investments. - Angel Catalan

DuckDB in Action

For non-technical users, like myself, and with DuckDB as a partner, understanding DuckDB and its applications is important for the following reasons: Data Analysis: DuckDB can be used for analyzing large datasets efficiently, allowing non-technical users to perform complex data analysis tasks without relying heavily on technical support. Data Processing: It offers capabilities for processing and querying data in real-time, enabling non-technical users to extract insights and make data-driven decisions more rapidly. Tool Familiarity: Familiarizing oneself with DuckDB and similar tools enhances data literacy among non-technical users, enabling them to adapt to evolving technologies and contribute effectively to data-related projects within their organizations. Collaboration: Understanding DuckDB facilitates better collaboration between technical and non-technical teams, as it enables non-technical users to communicate their data needs and requirements more effectively to technical counterparts. -Jack Ryan

Caveats

For those of us spending our day-to-day in the land of data, visualizations are an important way we communicate information to stakeholders. This is a GREAT resource for improving that presentation mode. It basically boils down to: keep it simple, stupid. - John Forstmeier

Podcast Recommendations

DataFramed - 50 Years of SQL with Don Chamberlin, Computer Scientist and Co-Inventor of SQL

Don Chamberlin, co-inventor of SQL, shares behind-the-scenes stories about developing the revolutionary database language 50 years ago. He explains SQL's origin, the challenges faced, its standardization, and why it succeeded despite competition.

Practical AI: Machine Learning, Data Science - Mamba & Jamba

Yoav Shoham, co-founder of AI company AI21, joins to discuss how enterprises can unlock value from unstructured text data using large language models. He explains AI21's focus on reliability and efficiency to make models practical for business use cases like summarization and product descriptions.

Microsoft, OpenAI plan $100 billion data-center project, media report says

In the continuing data war between the tech giants, Microsoft and OpenAI are planning a massive $100 billion data center project. Bigger, better, faster, stronger. - John Forstmeier

From Data Scientist to ML / AI Product Manager

What are the differences between an ML/AI Product Manager vs a software Product Manager? Both have to communicate complex technical concepts to stakeholders and translate stakeholders' desires into viable product decisions that engineers can easily act on. However, the unique challenges an ML/AI PM must face are interesting: maintaining a deep understanding of machine learning and data science in an ever-changing landscape, continuously measuring model performance through rapid growth, and ensuring the ethical use of AI. That last one is curial as the ethical considerations of AI are far-reaching and nuanced. - Katt Baum

Why pandas feels clunky when coming from R

I completely agree with this article that R's data manipulation libraries are superior to pandas. Of course you can accomplish the same task in either, but I have always felt that the dplyr/tidyverse syntax is more straightforward, primarily because it resembles SQL. I use Python everyday and am not an R user anymore (and haven't been for about 2.5 years), but if something comes where I need to analyze tabular data or make a few charts, I would much rather do that in R than in pandas/matplotlib. For literally any other task, Python all the way - Wes Poulsen

State of Analytics Engineering by DBT

For non-technical users, understanding the state of analytics engineering is crucial for several reasons: Business Alignment: It helps in understanding how analytics is being used to drive business decisions and strategies, enabling non-technical users to align their efforts with broader organizational goals. Data Literacy: Being aware of trends and challenges in analytics engineering enhances data literacy among non-technical users, empowering them to interpret and utilize data effectively in their roles. Collaboration: It facilitates better collaboration between technical and non-technical teams by providing insights into the tools, processes, and methodologies used in analytics engineering. Informed Decision-Making: Non-technical users can make more informed decisions when they understand the capabilities and limitations of analytics systems, leading to better outcomes for their projects and initiatives. - Jack Ryan

Podcast Recommendations

R for the Rest of Us

Statistician Will Landau discusses how he uses R at Eli Lilly and as an open source developer. He provides an overview of his R package targets which helps streamline data analysis workflows and ensures reproducibility.

R Weekly Highlights

This R Weekly podcast episode discusses integrating AI chatbots into RStudio, writing tests to fail fast, and checking R packages across platforms with the updated Rhub package. The hosts review useful tools for R developers.

YAML developers and the declarative data platforms

The author does a great job laying out how languages like YAML have made a significant home in the data engineering space. Declarative paradigms, inspired by SQL, are something that can be rapidly intuited and implemented by old and new engineers alike. Now only if Shipyard had something like that... (https://www.shipyardapp.com/docs/blueprint-library/google-bigquery/google-bigquery-execute-query/#yaml) - John Forstmeier

People and Python in AI

This article touches on industry trends, programming literacy, and the practical use of common tools to enhance data-driven decision-making. The article also addresses the cultural challenges organizations face in becoming truly data-driven, an important consideration for anyone in the field. - Johnathan Rodriguez

Unlocking The Power of Analytics: How To Adopt A Data-Driven Mindset

We always talk about data tools and best practices but one of the biggest hurdles a lot of companies (even people) face is having a data-driven mindset. You can have the best tools in the industry but if the way you tackle problems relies more on "gut feel" than data, these tools are essentially useless. This article walks through how to change your mindset to ensuring data is at the forefront of every problem being solved. - Angel Catalan

Data-as-a-Product and Data-Contract: An evolutionary approach to data maturity

Olivier took a novel approach to finding the tipping point of when an organization should actually start caring about data mesh and data contracts by relating the stages of data maturity to Wardley's evolution model. He posits that once a dataset drives more than one use case internally, the need to apply product thinking to the data increases greatly, especially if the data needs to be trusted by multiple stakeholders. - Blake Burch

Podcast Recommendations

Machine Learning Street Talk (MLST) - Prof. Chris Bishop's NEW Deep Learning Textbook!

In this fascinating episode, pioneering machine learning researcher Chris Bishop discusses the past, present, and future of AI and deep learning. He shares insights on inductive bias, scientific applications, and the continuing mystery of why deep learning works so well.

R Weekly Highlights

This week's R Weekly podcast discusses troubleshooting CRAN build failures, new features in the WebR platform like better graphics and error handling, and an excellent blog post with animated visuals explaining tidyverse data wrangling functions.

State of Analytics Engineering 2024

dbt Labs recently published a survey of 456 data practitioners to understand what concerns were facing data teams over the course of the next 12 months. I always enjoy reading these types of studies to make sure we're understanding and aligning with the top pain points in the data space. Does your experience align with these results? - Blake Burch

Digital transformation in finance: Challenges and benefits

Digital transformation is not just a fancy add-on anymore for financial institutions; it's necessary. Implementing advanced analytics, AI, blockchain, and more, allow customers to get what they want before they even want it. Despite what you might think, this allows more personalized services for your customer- giving them a smoother, more tailored experience with more efficiency. This comes with challenges for older institutions to maintain security with their data, but the pros outweigh these cons. - Reed Cowan

AI Companies Running Out Of Training Data After Burning Through Entire Internet

If data is the new oil, then the big AI companies are coming close to sucking the well dry, according to the author. While big models require big data, alternatives like small, tailored models or synthetic data could be the answer to the problem. Nothing is ever boring in the world of ML! - John Forstmeier

Top Data Science Trends Reshaping the Industry in 2024

In a rapidly evolving field like data science, staying abreast of the latest terminology can be daunting. This article effectively demystifies current buzzwords and concepts, ensuring they're more than just industry fluff. - Johnathan Rodriguez

Podcast Recommendations

The Analytics Power Hour, Ep 242 - The Rise and Fall of Data Communities with Pedram Navid - Insightful discussion about the evolution of data analytics communities over time. They reflect on the pros and cons of growth, provide advice to newcomers, and share thoughts on staying engaged.

The Gradient: Perspectives on AI - Andrew Lee: How AI will Shape the Future of Email - This episode discusses how Shortwave, an AI-powered email app, was built to help users be more productive. Founder Andrew Lee explains how they leverage natural language models to provide features like automatic email summarization, drafting assistants, and search across a user's entire inbox.

Data Science vs Data Engineering: An In-Depth Comparison

As the data world evolves, so does the nature of the roles that serve organizations with data. While the lines are often blurred at many organizations, this article makes some very important and clear distinctions between Data Engineering and Data Science to help folks understand the skillsets required to be successful in each role. - Angel Catalan

Data Center Tour & Technical Deep Dive into the Power, Data and Cooling Infrastructure!

There is no cloud, only someone else's server. This is a really cool breakdown, that I only partially understood, of the physical infrastructure that goes into a data center. Worth a watch! - John Forstmeier

How to design resilient and large scale data systems

Excellent article on how to think about data pipelines from an architectural design standpoint. When discussing data system design, most people want to jump right into the implementation details (the code), but before you can come to that step you need to consider the architecture of the system. This article provides a great summary of some of the considerations to account for - Wes Poulsen

Data Science to Implement a Circular Economy

Developing a circular economy in the context of fast fashion can be good for the Earth and a company's bottom line, but there is a lot to consider moving from vision to reality. From ML algorithms to optimize the sorting of used clothes, to analyzing customer behavior and tailoring incentives to motivate sustainable actions, Saci argues that data science plays a crucial role in building the infrastructure needed to support such a goal. - Katt Baum

Podcast Recommendations

Practical AI: Machine Learning, Data Science - AI vs software devs

This special episode samples talks about AI's impact on software developers. Clips discuss the Devon coding assistant, using AI tools to augment skills instead of replace jobs, and how creators of programming languages view AI's influence.

Gradient Dissent: Conversations on AI - Navigating the Vector Database Landscape

Edo Liberty, founder of leading vector database Pinecone, shares his expertise on building vector databases, their role in powering AI applications, and overcoming challenges to get AI into production

What is the significance of color in data visualization?

Red means stop. Green means go. Yellow is happy. We associate color with feelings, information, and decision making more than we think in our day to day lives. So do we do the same with data? Color speaks to the audience and allows businesses to communicate messages with a greater effect. It evokes emotion, builds connections, creates a story, and provides depth. So why not begin thinking a bit more about its use in your data visualization? - Reed Cowan

A beginner's guide to building a Retrieval Augmented Generation (RAG) application from scratch

As someone trying to get more involved with AI, I had constantly heard about RAG but had yet to explore it myself. Bill simplifies the concept, the code, and provides visuals to make sure everything is well understood. It's one of the best introductions I've had to a concept. Once you finish this tutorial, there's 4 more excellent tutorials to go through! - Blake Burch

Enabling near real-time data analytics on the data lake

The post provides a really neat and moderately technical overview of how to improve an often overlooked aspect of data lakes: writing latency. If you're frequently updating the data lake, it can have negative downstream impact on query performance. The authors address this by breaking down the data source frequency and format and building a system designed to scale with demand. - John Forstmeier

Reproducible data science with Nix, part 6 -- CI/CD has never been easier

I'm trying to get into Nix and learn more about it. This definitely wets the appetite and shows a good example use case with a language and use case I'm unfamiliar with. Without a data background, it's interesting to see other disciplines use reproducibility. - Eric Elsken

Podcast Recommendations

DataFramed, Ep 188 - Scaling Enterprise Analytics with Libby Duane Adams, Chief Advocacy Officer and Co-Founder of Alteryx

This podcast episode explores how organizations can scale their data analytics capabilities to get more value from their data. Experienced analytics leader Libby Duane Adams provides advice on overcoming common challenges like data governance, ensuring data privacy and security, developing employees' data skills, and calculating the business impact of analytics initiatives.

Data Skeptic - What You Know About Intelligence is Wrong

In this insightful interview, behavior analyst Hank Schlinger reveals how intelligence is more of an adjective describing behavior, not an inherent trait. He unpacks why psychology struggles to define vague concepts like intelligence and consciousness, preferring instead to focus on measurable actions.

Although I may be a bit biased, I think this conversation with Brian Toole was really insightful. If you are thinking about a career as a Data Engineer, this conversation gives you a full look into the job to make sure it is for you. - Steven Johnson

Healthcare orgs face significant data modernization challenges

With our focus on the health tech space and all the recent data breaches in the healthcare space specifically, this article does a great job of walking through the challenges healthcare companies face when trying to establish a modern data architecture. - Angel Catalan

Google Intros Gemini 1.5

The AI space and race to be the leading AI continues to move at blazing speed, Google is innovating at lightning speed w Gemini 1.5; particularly interesting is the safety and ethics portion in the documentation as well as the speed of the architecture. - Jack Ryan

Why Your Clients Are Resistant To Data Literacy

If you're in the data consultancy field, this is a must-read article. It raises some points that I hadn't been aware of before, specifically data literacy and the resistance to it. This post puts together a short and clear strategy for overcoming it and helping improve your clients' data posture. - John Forstmeier

DataTalks.Club - Accelerating The Job Hunt for The Perfect Job in Tech

Sarah Mestiri, a data scientist and career coach, joins Alex to share tips on finding your ideal tech job, including researching roles, networking strategically, tailoring your resume, and preparing for interviews.

Data Engineering Podcast - Tackling Real Time Streaming Data With SQL Using RisingWave

Yingzhen Wu discusses Rising Wave, an open source streaming database built on S3 that makes stream processing more accessible. It offers SQL support, change data capture from Postgres and MySQL, and integrates with data lakes.

Data operations are notoriously energy-demanding and AWS is taking the step to leverage nuclear to solve that problem. It's really cool to see giants like AWS acknowledging the need to incorporate clean and consistent energy via nuclear power plants in order to continue providing data-intensive services to their customers. Plus it's a cool PA win I'm always happy to see. : ) - John Forstmeier

Solving a Tennis Refactoring Challenge in Python using SOLID

I completely agree with the first point that these types of exercises are becoming highly beneficial for data scientists, with the increasing popularity of their work. Whether intended or not, the best point for me is the test driven development out of the box in the repo. No matter what refactoring you do, the Python section of the repo provides tests to increase confidence in your refactoring. - Eric Elsken

Exploring the Potential of Transfer Learning in Small Data Scenarios

Many teams don't have access to or the resources to get the massive amounts of data needed to train models well. Enter transfer learning: taking pre-trained models and using them on smaller datasets for specialization. Davies talks about some of the hurdles to using pre-trained models, but ultimately concludes that there is huge potential in transfer learning, especially as more effort is put into automating the selection and fine-tuning of pre-trained models. - Katt Baum

Best Data Engineering Projects for Beginners in 2024

As we look to help data teams make the most out of their engineering resources, we also offer an easy platform for folks to get started and sharpen up their data engineering skills. This article is a great starting point for folks looking to pick up a new data skill or put some recently used knowledge to good use. - Angel Catalan

Snowflake Lowers Cost from 1.5x Usage to 1.2x

Namely to highlight usage cost going down currently while other tools (DBT, DB, etc) are going up exponentially. - Jack Ryan

Share All Hands on Data

Using Databricks Notebooks for Production Data Pipelines

This was an interesting read because I find myself in the "notebooks should never see the light of day in a production pipeline." That being said, the author makes a good case for notebooks, emphasizing their strengths and why data scientists are drawn to them in the first place. Additionally, it seems that Databricks notebooks in particular may have some additional safe guards in place to address some of the common pitfalls (like versioning) - Wes Poulsen

Although the latter half the piece is pitching the company's product, the author does highlight a pet peeve of mine: unused stored data. Whenever a software product's usage grows, so does the data stored and collected but does not necessarily translate into productive use. I'd go a step further than the article to even suggest finding data you no longer need to collect and cut it out with a machete. - John Forstmeier

Design Principles in Software

the basic of software design principles to me is wildly foreign. just as there is process in sales, standard questions, techniques, etc. so to is in software engineering. my favorite point particularly is how SOLID refers to having the development so simplified and that you either solve for X or you solve for Y (no alts). - Jack Ryan

SQL Server: Powering your data warehouse with insights and efficiency

As someone who works heavily in the data space, it's always interesting to read about different ways to consolidate and read data. This article provides an interesting read on how SQL Server can help model data, transform that data, secure that data, and analyze that data. A lot of businesses have data in various places, spread all throughout different spreadsheets saved in different services, and I think it's worth diving into a tool that can place all the different data into one centralized hub. - Reed Cowan

The Future of Data Science

Teamcubate discusses emerging trends and the increasing role of AI and machine learning in data science, which are crucial for interpreting vast amounts of data and assisting businesses in making informed decisions. It also covers the importance of data ethics and privacy, the impact of data science across various industries like retail, healthcare, and finance, and the growing need for skilled data scientists. - Johnathan Rodriguez

Data will not tell you what to do

When people try to be data driven, they oftentimes get themselves into trouble ending up with "decision-by-spreadsheet". Mikkel recommends focusing more on using intuition, guiding by ancillary data, to help move the business forward. I really agree with the approach of not being too rigid with your use of data. - Blake Burch

Seeing Our Reflection in LLMs

This was a really thought-provoking philosophical read. Something I love to see when talking science (data or otherwise). How should we respond the reflection of society that LLMs (or as Kirmer notes, multimodals) produce? Do we tweak the model's predictions based on our societal aspirations (which often aren't unanimous) or employ zero content moderation? Kirmer believes a middle ground is needed, but as most middle grounds go, that will require a fair amount of communal effort. So, do we direct that effort and fixing the reflection? Or fixing the subject of the reflection? - Katt Baum

The Modern Data Stack is a Mess

I love the term the author uses for the modern data stack: "Frankenstacks" - large, unwieldy, cobbled-together resources that unnecessarily increase costs and complexity without providing value. The author points to a number of potential factors causing this and it's definitely worth a quick read. - John Forstmeier

Aspiring Analytics Engineers, Start Here.

Starting out in data can be quite a daunting task. Following my chat from last week's All Hands on Data, I think Analytics Engineering is one of the most exciting roles in data. If you are looking to get started in that role, this article gives you some great tips to get rolling. - Steven Johnson

You should stop writing Dockerfiles today - Do this instead

I actually like writing Dockerfiles, they're a fun way to build up a system, and I love the containerization technology. However, another cool piece of AI is coming for our containers. I'm excited to try this feature for more robust and secure Dockerfiles. - Eric Elsken

Visualisation 101: Choosing the Best Visualisation Type

Data visualizations are crucial for revealing insights that might be hidden in tables of numbers. As Mansurova puts it, we can use them for exploring or for explaining data. In addition to an extensive discussion about the tools and chart types, Mansurova provides a few questions we can ask ourselves before choosing a visualization method. Here are some that were most interesting to me: What context does my audience already have? What concerns/questions might they have? Am I showing the whole picture? - Katt Baum

Why Orchestration matters

I wanted to crowd source info from the anonymous world of reddit specifically as to why orchestration matters. To my non technical mind, management of multiple tables (dimension, transactional, fact tables) to expedite time and insight in real time of data running, seemed like a great use case we are seeing at shipyard with small to mid sized data teams. - Jack Ryan

Building the Modern Data Team: Why Diversity Matters

We always talk about the modern data stack, but this article focuses on the crux of any good data team, the people. This article chats through the importance and benefits of diversity in data teams and highlights the value these unique perspectives bring. - Angel Catalan

New approach to teaching computer science could broaden the subject's appeal

The article is intriguing as it presents a forward-looking method for integrating computer science education into K-12 curriculums. This approach not only addresses current educational challenges but also emphasizes preparing children, who are the future, for a rapidly evolving technological landscape. It suggests that by weaving computer science into various subjects, we can foster a generation that's adept and adaptable to the future changes in technology. - Johnathan Rodriguez

Learn Julia Salinas' Tips for a Career in Analytics Engineering

The field of Analytics Engineering has always been interesting to me. Honestly, I thought it was a made up job title. I was able to dig into the role more with Julia and see what a Analytics Engineer actually does on a day-to-day basis. - Steven Johnson

Is the "Modern Data Stack" Still a Useful Idea?

The term Modern Data Stack has been floating around pretty heavily for the past 5 years but is starting to lose it's allure. Does anyone actually know what it means? Is it actually modern? Or is there a better way to describe the way Data teams get work done? Tristan Handy, CEO of dbt Labs, explores what the term was supposed to mean and what it became. - Blake Burch

The Essential Guide to SQL's Execution Order

Unlike other languages, SQL is not executed line by line. Understanding the execution order can lead to eliminating common bugs (particularly in the WHERE or HAVING clause) as well as improving query performance - Wes Poulsen

How edge computing is transforming data management

Edge computing involves processing data where it's generated and not relying on data processing. There are many different advantages to this approach suches as reduced latency, enhanced processing capabilities, and decreased bandwidth constraints. This article goes through what edge computing is, how it impacts data management, and even shows a real world example through smart traffic management system. It makes a strong case on how your organization can harness its full potential to improve your data management strategies. - Reed Cowan

New data shows Waymo crashes a lot less than human drivers

In an article highlighting the importance of data over headlines, research indicates that Waymo's self-driving cars are multiples safer than human-driven cars. Despite anecdotal claims of danger, the data does not bear out that narrative. While the self-driving cars still need to be fully tested on open highways and fatal crash data has yet to be collected, it's certainly a step in a positive direction for the industry. - John Forstmeier