The Curious Engineer

All blogs / AI / ML

Win Vector LLC

win-vector.com

Advising, Consulting, Training About Win Vector LLC Win Vector LLC is a San Francisco based company specializing in data science. We work in a variety of industries, with organizations both large and small. Our Win Vector blog is followed by data scientists worldwide. Read More About Win Vector LLC

AI / MLMathPython

’10’15’20’25
Posts
929 from 19 Jun 2007 to 27 Oct 2026
Posting
monthly (23 in the last year)
Kind
company blog
Feed
https://win-vector.com/feed/

2026 20

  1. “Boy Born on Tuesday” A Thornier Problem Than The Monte Hall Problem?
  2. Speaking at the Bayesian Interest Group November 2026 Virtual Meeting
  3. How to Critically Read LLM Texts [Video]
  4. A Simplified Mental Model of LLMs
  5. Expected Log-Return as a Risk Control
  6. Choose by Probability, not by Expectation
  7. LLMs Will Benefit from Scratch Workspaces
  8. Maybe Jacobian Conjecture Counterexamples are not Rare
  9. Is it the box or the “yes” that does not fit?
  10. Why You Are Not Getting Full Value From Your Analytics
  11. Who’s the Best Batter? Estimating Probabilities from Unevenly Collected Data
  12. Squeezing the Most Utility From Your Models
  13. What is a Centipawn Advantage?
  14. Exploring Chess Positions and Counts
  15. There are a lot of Stockfish Chess Games
  16. How Many Chess Games are Possible?
  17. Uncovering Hidden Price Elasticity
  18. Even citing NSF funded research is an ACM premium feature
  19. Modeling Censored Data with Tobit
  20. I am Tired of Praise of Byproduct

2025 24

  1. How Logistic Regression Changes with Prevalence
  2. Text as a “Market for Lemons”
  3. The Fermi Paradox Just From Rarity
  4. A Fairly Hostile Subtext
  5. High Dimensional Optimization Remains Hard
  6. Escape the Over Tuning Trap
  7. Reducing Optimization Risk through Portfolio Methods
  8. Modeling Uncertainty For Better Outcomes
  9. Reverse Chess
  10. Toying with the Lambda Calculus
  11. They did use a calculator!
  12. How About Pi ~ 31/32?
  13. Don’t Let a Data Leak Sink Your Project
  14. Working Through A Trivial Algorithm Whose Analysis Isn’t
  15. Demonstrating Kelly Betting with Chips
  16. Is There a Difference Between Calculation and Computation?
  17. Changing Forecasts for Python Questions on Stack Overflow
  18. Is GitHub Lying Here?
  19. Don’t think of a basketball player, or search engines don’t even support “not”
  20. Best Before Dates by Bass
  21. The Statistics of Drawing Cards
  22. Trying to Describe 2024 AI Premises
  23. Hat Color Puzzles
  24. Reposting Partial Pooling

2024 46

  1. What Fraction of Containerization is Just Working Around Dynamic Linking?
  2. Kelly Betting With Discrete Stakes
  3. Kelly Can’t Fail
  4. Examining Meta-Analysis
  5. Calculating at Pencil and Paper Scale
  6. ODSC West 2024
  7. The Monkey and the Coconuts: An Introduction to the Extended Euclidean Algorithm
  8. Dyson’s Algorithm for the Twelve Coins Problem
  9. Come Learn About Time Series with Me at ODSC West Wednesday October 30th, 2024
  10. The Twelve Coins Puzzle
  11. Dudeney’s Catching The Mice Puzzle
  12. Dudeney’s Remainder Problem
  13. An Easy Puzzle: The Perplexed Banker
  14. Bachet’s Four Weights Problem
  15. Solving 100 Bushels Using Matrix Factorization
  16. The 100 Bushels Puzzle
  17. Bay Area use R Meetup is Back!!!!
  18. Machine Learning Model Homotopy
  19. ChatGPT is Getting Scary Better at Math
  20. Please Version Data
  21. More on Adjusting Saturated Multivariate Linear Models
  22. Post-hoc Adjustment for Zero-Thresholded Linear Models
  23. Scoping Data Science Projects
  24. Time Series Forecasting and Independence
  25. Pulling a Loose Thread on Pull()
  26. I Remain Astounded by How Bad AT&T Customer Service Is
  27. ARMAX Offerings Remain a Muddle
  28. Transferring Coefficients of Linear Models
  29. Yet Another Way Corporations Outrank People
  30. Popularizers Earned the 1*1=2 Egg in Face
  31. Good Models do not Match Variance
  32. Book Review: Bernoulli’s Fallacy
  33. Algorithmic Art
  34. I Want Flexible Queries, Not RAG
  35. Solving Recurrence Relations
  36. The AI Problem is not Alignment
  37. The m = n Machine Learning Anomaly
  38. How Data Quantity Drives Model Quality
  39. Exact Sums Over a Group Action
  40. Illustrating the F-test in Action
  41. What Good is Analysis of Variance?
  42. New Nina Zumel article: Clarity, Not Magic
  43. Use Jupyter Notebooks Inside For-Loops
  44. Disposing of People Appears to be Priced Cheaper Than Parting With Cash
  45. Tools for Jupyter in (and near) Production
  46. What You Should Know About Linear Markov Chains

2023 32

  1. Conditioning on the Future
  2. The Biased Drunkard’s Walk
  3. A Deep Dive on The Drunkard’s Walk
  4. Consider Evaluating LLMs as Search Engines
  5. Yet Another ChatGPT Winge
  6. A Slightly Unfair Game
  7. Unrolling the ROC Curve
  8. A/B Tests for Engineers
  9. Record Shaping with CData
  10. Including ggplot2 Plots in Python Notebooks
  11. Schemas for Python Data Frames
  12. Solving for Hidden Data
  13. Detecting Data Differences Using the Sphering Transform
  14. Omitted Variable Effects in Logistic Regression
  15. More on Parameterized Jupyter
  16. List Coloring Latin Squares
  17. The Expected L1 Norm of Unit L2 Vectors
  18. Tilting at Sign
  19. How often do the L1 and L2 norms agree?
  20. Vector Packing Vacation
  21. Some of the Perils of Time Series Forecasting
  22. A Time Series Apologia
  23. It Isn’t Just the AIs Hallucinating
  24. The Sell ∀ ∃ as ∃ ∀ Scam
  25. A Pandas/Polars Rosetta Stone
  26. Fun Introduction to the Ideas of A/B testing
  27. Doing Better than the Average
  28. How Much Data Do You Need?
  29. Bounding Excess Generalization Error for Linear Regression Models
  30. Thinking About Linear Regression
  31. Short Data Science Video: Parameterized Juypter Notebooks
  32. Yet Another Data Transform Tutorial

2022 19

  1. Data Algebra over Polars Ready for Production Use
  2. Touching the 3rd Rail of Data Science: “R or Python?”
  3. Experimenting with Polars for Data in Python
  4. Data Science: Street Fighting Statistics
  5. What a Data Engineer Needs to Know About Bitemporal Modeling
  6. Y-Aware PCA
  7. wvpy Clean Up
  8. An Effective Personal Jupyter Data Science Workflow
  9. Worry Over Columns, not Rows
  10. I am recommending a Data Science book I just started reading
  11. What does machine learning fitting actually do?
  12. Separating Code from Presentation in Jupyter Notebooks
  13. Wondering How To Think About Data Science
  14. Just For Fun: Computing the Probability of Winning a Tournament
  15. Working in CRAN’s World
  16. An appreciation of Cover’s universal portfolio in Python
  17. Method Warnings
  18. How to Re-Map Many Columns in a Database
  19. xicor for Confusion Matrices

2021 35

  1. Exploring the XI Correlation Coefficient
  2. How to Read Sourav Chatterjee’s Basic XICOR Definition
  3. Data Algebra Method Catalog
  4. Machine Learning Data Preparation in the Database with vtreat
  5. Composing Queries in the Data Algebra
  6. What Shape Minimizes Cable Tension?
  7. Effect Sizes Say More Than Standard Scores
  8. Data Algebra 0.9.0 Release
  9. How to Compare Two Tables Using the Data Algebra
  10. Using the data algebra for Statistics and Data Science
  11. I think Pandas may have “lost the plot.”
  12. Back to Teaching
  13. My opinion on “5” == 5
  14. Using WITH For Neater SQL
  15. data_algebra 0.7.0 What is New
  16. Data Science Bite: What is Statistics?
  17. Data Science Bite: A/B Testing
  18. Upcoming Talk on Probability Models
  19. Data Science Bite: “What is Supervised Machine Learning?”
  20. When Profitable Betting Systems are not Possible
  21. Kelly Thorp Betting
  22. Plotting Multiple Curves in Python
  23. New Math Chalk Talk: The Game of Infinity Questions
  24. It Has Always Been Wrong to Call order on a data.frame
  25. Introducing wrapr::bc()
  26. Please share! Math for Programmers Humble Bumble!!
  27. What is a Good Test Set Size?
  28. Rule 42 Software
  29. Bilingual Data Science
  30. Code for the “Variable Utility is not Intrinsic” Article
  31. Variable Utility is not Intrinsic
  32. The Dual Numbers
  33. The Hyper Dance
  34. Smoothing isn’t Always Safe
  35. The Nature of Overfitting

2020 106

  1. Against Accuracy
  2. The Data Scientist as The Bus Driver
  3. abs and relu are not Mercer Kernels
  4. Bounding Excess Generalization Error
  5. PAC Learning and VC Dimension
  6. What Every Data Scientist Should Know About Floating Point
  7. New Math Chalk Talk: The Real Numbers
  8. Overfit and Reversion to Mediocrity, the Bane of Data Science
  9. Some Fun With User/Package Level Pipes/Anonymous-Functions
  10. My Opinion on R’s Upcoming Pipe
  11. Some Thoughts on Numbers
  12. R is Getting an Official Pipe Operator
  13. Happy Anniversary Practical Data Science with R 2nd Edition!
  14. Evaluating Probability Models
  15. Evaluating Regression Models
  16. Measure Theoretic Probability Video
  17. Some Ideas on How to Think About Vectors
  18. The Purpose of our Data Science Chalk Talk Series
  19. Classification as Censored Regression
  20. A Classic Math Joke
  21. What is Statistics?
  22. The Science of Data Analysis
  23. New Light-board Lecture: wrapr::unpack
  24. How and Why am I Making These Light-board Video Lectures
  25. New Free Video Lecture: Estimating the Odds with Bayes’ Law
  26. New Free Mini-Lecture: Simpson’s Paradox in A/B Testing
  27. A Single Parameter Family Characterizing Probability Model Performance
  28. An Example of a Calibrated Model that is not Fully Calibrated
  29. The Double Density Plot Contains a Lot of Useful Information
  30. Your Lopsided Model is Out to Get You
  31. The Shift and Balance Fallacies
  32. Surgery on ROC Plots
  33. Model Homotopies in the Wild
  34. Tailored Models are Not The Same as Simple Corrections
  35. Upcoming Series: Probability Model Homotopy
  36. How to Pick an Optimal Utility Threshold Using the ROC Plot
  37. Estimating Uncertainty of Utility Curves
  38. Squeezing the Most Utility from Your Models
  39. Maybe Businesses Don’t Need To Worry So Much About Inference
  40. Site Maintenance
  41. “Statistics to English Translation”
  42. On The Decomposition of Variance
  43. Kolmogorov’s Axioms of Probability: Even Smarter Than You Have Been Told
  44. Clearly The Author Does Not Know What The Natural Numbers Are
  45. Why Working With AUC is More Powerful Than One Might Think
  46. A Gruesome Example of Bayes’ Law
  47. Data Science is a Science (Just Not the One You May Think)
  48. San Francisco Wednesday September 9th
  49. The Intercept Fallacy
  50. 0.83 is a Special AUC
  51. New WVPlot: ROCPlotPairList
  52. An Example Where Square Loss of a Sigmoid Prediction is not Convex in the Parameters
  53. Why not Square Error for Classification?
  54. Unrolling the ROC
  55. Don’t Use Classification Rules for Classification Problems
  56. We recommend this book! Deep Learning for Coders with fastai and PyTorch
  57. Let A be a Pedant and Let B be a Pedant
  58. Free vtreat Tutorial Videos
  59. Looking for Feedback on an Article or Series Proposal
  60. Data Science is not “Statistics Done Wrong”
  61. Data Science Intensive for Software Engineers
  62. vtreat for KNIME!
  63. Penguins Data for Python
  64. Genetic Art Introduction Video
  65. knotEd: Knot Diagram Editing Software
  66. What is Chapter 8 of Practical Data Science with R?
  67. Site re-Org
  68. Linear and Logistic Regression in Practical Data Science with R 2nd Edition
  69. Don’t Feel Guilty About Selecting Variables
  70. Data engineering and data shaping in Practical Data Science with R 2nd Edition
  71. General Data Science Means Cross-Language Tools, Training, and Documentation
  72. Deal of the Day May 10: Half off Practical Data Science with R, Second Edition
  73. Thank you “Why R?” for Being Awesome Hosts
  74. Nina and John Speaking at Why R? Webinar Thursday, May 7, 2020
  75. Some Applications of The Spicy Soup Test
  76. Imputing Out of Mixtures, or Un-Stirring Spicy Soup
  77. Discount on Manning Books, Including our own Practical Data Science with R 2nd Edition
  78. Y-Conditionally Regularized Neural Nets
  79. R Tip: How To Look Up Matrix Values Quickly
  80. wrapr 2.0.0 up on CRAN
  81. Re-Share: vtreat Data Preparation Documentation and Video
  82. Version Control is a Time Machine That Translates Common Hindsight Into Valuable Foresight
  83. Free Coupon for our R Video Course: Introduction to Data Science
  84. A Little Something From Practical Data Science with R Chapter 1
  85. Use the Same Cross-Plan Between Steps
  86. Keep Calm and Use vtreat (in R and in Python)
  87. Cross-Methods are a Leak/Variance Trade-Off
  88. Python Data Science Tip: Don’t use Default Cross Validation Settings
  89. Nifty Upcoming Enhancements to unpack/to
  90. Monitoring for Changes in Distribution with Resampling Tests
  91. What is New For vtreat 1.5.2?
  92. New improved cdata instructional video
  93. New Data Scientist Stickers
  94. wrapr Update: Removing Some Under-Used Functions and Classes
  95. R Tip: Check What Repos You are Using
  96. Data re-Shaping in R and in Python
  97. wrapr 1.9.6 is now up on CRAN
  98. Off topic: Horror Translations by Nina Zumel
  99. Why we wrote wrapr to/unpack
  100. Using unpack to Manage Your R Environment
  101. unpack Your Values in R
  102. sklearn Pipe Step Interface for vtreat
  103. New vtreat Feature: Nested Model Bias Warning
  104. New Year’s Resolution 2020: Work on more R Data Science Projects
  105. Manning Deal of the Day January 3, 2020 : Half off Practical Data Science with R, Second Edition
  106. New Timings for a Grouped In-Place Aggregation Task

2019 107

  1. Introduction to Data Science in R, Free for 3 days
  2. PyData Los Angeles 2019 talk: Preparing Messy Real World Data for Supervised Machine Learning
  3. What is a Second Edition?
  4. Why to try Practical Data Science with R, 2nd Edition
  5. A Richer Category for Data Wrangling
  6. Better SQL Generation via the data_algebra
  7. New rquery Vignette: Working with Many Columns
  8. data_algebra/rquery as a Category Over Table Descriptions
  9. What is new for rquery December 2019
  10. Slides for PyData LA 2019 vtreat Talk
  11. Practical Data Science with R 2nd Edition now in-stock at Amazon.com!
  12. Slides from the PyData2019 data_algebra lightning talk
  13. Practical Data Science with R, 2nd Edition: Introduction Video
  14. Nina Zumel and John Mount speaking on vtreat at PyData LA 2019
  15. Practical Data Science with R, 2nd Edition, IS OUT!!!!!!!
  16. When Cross-Validation is More Powerful than Regularization
  17. New Introduction to rquery
  18. Practical Data Science with R 2nd Edition update
  19. Free R/datascience Extract: Evaluating a Classification Model with a Spam Filter
  20. AI for Engineers
  21. vtreat Cross Validation
  22. You Can Override Just About Anything in R
  23. New vtreat Documentation (Starting with Multinomial Classification)
  24. Why Do We Plot Predictions on the x-axis?
  25. How to Prepare Data
  26. Preparing Data for Supervised Classification
  27. The Advantages of Record Transform Specifications
  28. Practical Data Science with R update
  29. WVPlots 1.1.2 on CRAN
  30. Advanced Data Reshaping in Python and R
  31. New Getting Started with vtreat Documentation
  32. Why R?
  33. It is Time for CRAN to Ban Package Ads
  34. Introducing data_algebra
  35. Eliminating Tail Calls in Python Using Exceptions
  36. What is vtreat?
  37. Speaking at BARUG
  38. vtreat up on PyPi
  39. Returning to Tides
  40. Florence Nightingale, Data Scientist
  41. Lord Kelvin, Data Scientist
  42. PyCharm Video Review
  43. Some Notes on GNU Licenses in R Packages
  44. A Comment on Data Science Integrated Development Environments
  45. A Kind Note That We Really Appreciate
  46. R Books Discount!
  47. Big News: Porting vtreat to Python
  48. An Ad-hoc Method for Calibrating Uncalibrated Models
  49. Some Details on Running xgboost
  50. Common Ensemble Models can be Biased
  51. Link Functions versus Data Transforms
  52. Programming Over lm() in R
  53. Replicating a Linear Model
  54. My Favorite data.table Feature
  55. data.table is Much Better Than You Have Been Told
  56. Cohen’s D for Experimental Planning
  57. Minimal Key Set is NP hard
  58. Estimating Rates using Probability Theory: Chalk Talk
  59. Technical books are amazing opportunities
  60. Practical Data Science with R, half off sale!
  61. Free Video Lecture: Vectors for Programmers and Data Scientists
  62. Timing Working With a Row or a Column from a data.frame
  63. What is “Tidy Data”?
  64. Could not Resist
  65. Data Layout Exercises
  66. Practical Data Science with R Book Update (April 2019)
  67. Controlling Data Layout With cdata
  68. Piping is Method Chaining
  69. R Photo
  70. Practical Data Science with R Book Update
  71. Not Always C++’s Fault
  72. Why RcppDynProg is Written in C++
  73. What are the Popular R Packages?
  74. C++ is Often Used in R Packages
  75. Standard Evaluation Versus Non-Standard Evaluation in R
  76. Sunny day at the office, and new mocha cup!
  77. Operator Notation for Data Transforms
  78. How cdata Control Table Data Transforms Work
  79. Why we Did Not Name the cdata Transforms wide/tall/long/short
  80. Support Rotary to Support our World
  81. Tidyverse users: gather/spread are on the way out
  82. Quantifying R Package Dependency Risk
  83. wrapr::let()
  84. Software Dependencies and Risk
  85. Unit Tests in R
  86. Data Manipulation Corner Cases
  87. Starting With Data Science: A Rigorous Hands-On Introduction to Data Science for Software Engineers
  88. rquery Substitution
  89. Binning Data in a Database
  90. “If You Were an R Function, What Function Would You Be?”
  91. R Journal Volume 10/2, December 2018 is out!
  92. More on Macros in R
  93. Getting Started With rquery
  94. Playing With Pipe Notations
  95. Query Generation in R
  96. PDSwR2 Free Excerpt and New Discount Code
  97. cdata Control Table Keys
  98. PDSwR2: New Chapters!
  99. Function Objects and Pipelines in R
  100. Fully General Record Transforms with cdata
  101. Make Teaching R Quasi-Quotation Easier
  102. R Tip: Use Inline Operators For Legibility
  103. Practical Data Science with R, 2nd Edition discount!
  104. R Tip: Use seqi() For Indexes
  105. A Beautiful 2 by 2 Matrix Identity
  106. Timing the Same Algorithm in R, Python, and C++
  107. What does it mean to write “vectorized” code in R?

2018 102

  1. Introducing RcppDynProg
  2. Rotary
  3. vtreat Variable Importance
  4. Quoting Concatenate
  5. Reusable Pipelines in R
  6. Sharing Modeling Pipelines in R
  7. Timing Grouped Mean Calculation in R
  8. Very Non-Standard Calling in R
  9. Quoting in R
  10. More on Bias Corrected Standard Deviation Estimates
  11. How to de-Bias Standard Deviation Estimates
  12. More on sigr
  13. R tip: Make Your Results Clear with sigr
  14. coalesce with wrapr
  15. The blocks and rows theory of data shaping
  16. Use Pseudo-Aggregators to Add Safety Checks to Your Data-Wrangling Workflow
  17. Conway’s Game of Life in R: Or On the Importance of Vectorizing Your R Code
  18. Scatterplot matrices (pair plots) with cdata and ggplot2
  19. Designing Transforms for Data Reshaping with cdata
  20. Faceted Graphs with cdata and ggplot2
  21. Quasiquotation in R via bquote()
  22. Piping into ggplot2
  23. Some R Guides: tidyverse and data.table Versions
  24. Running the Same Task in Python and R
  25. Quick Significance Calculations for A/B Tests in R
  26. Modeling multi-category Outcomes With vtreat
  27. A Better Example of the Confused By The Environment Issue
  28. A Subtle Flaw in Some Popular R NSE Interfaces
  29. Timing Column Indexing in R
  30. Using a Column as a Column Index
  31. Dot-Pipe Paper Accepted by the R Journal!!!
  32. Parameterizing with bquote
  33. On “Competition” in the R Ecosystem
  34. Better R Code with wrapr Dot Arrow
  35. Announcing wrapr 1.6.2
  36. Practical Data Science with R2
  37. A Quick Appreciation of the R transform Function
  38. R Tip: Give data.table a Try
  39. R Tip: How to Pass a formula to lm
  40. R Tip: Put Your Values in Columns
  41. Timings of a Grouped Rank Filter Task
  42. R Tip: Consider radix Sort
  43. More Practical Data Science with R Book News
  44. Announcing Practical Data Science with R, 2nd Edition
  45. data.table is Really Good at Sorting
  46. Meta-packages, nails in CRAN’s coffin
  47. Collecting Expressions in R
  48. How to use rquery with Apache Spark on Databricks
  49. John Mount speaking on rquery and rqdatatable
  50. Speed up your R Work
  51. seplyr 0.5.8 Now Available on CRAN
  52. Big News: vtreat 1.2.0 is Available on CRAN, and it is now Big Data Capable
  53. R Tip: Be Wary of “…”
  54. wrapr 1.5.0 available on CRAN
  55. R Tip: use isTRUE()
  56. rqdatatable: rquery Powered by data.table
  57. Talking about clinical significance
  58. WVPlots now at version 1.0.0 on CRAN!
  59. wrapr 1.4.1 now up on CRAN
  60. Ready Made Plots make Work Easier
  61. rquery: SQL from R
  62. Upcoming speaking engagments
  63. R Tip: Use Slices
  64. cdata Update
  65. Neglected R Super Functions
  66. R Tip: Use match_order() to Align Data
  67. magrittr and wrapr Pipes in R, an Examination
  68. Four Years of Practical Data Science with R
  69. R Tip: Think in Terms of Values
  70. Hangul/Korean edition of Practical Data Science with R!
  71. R Tip: Use Named Vectors to Re-Map Values
  72. R Tip: Use let() to Re-Map Names
  73. R Tip: Break up Function Nesting for Legibility
  74. R Tip: Use stringsAsFactors = FALSE
  75. Take Care If Trying the RPostgres Package
  76. The Many Faces of R
  77. R Tip: Use the vtreat Package For Data Preparation
  78. R Tip: Introduce Indices to Avoid for() Class Loss Issues
  79. R Tip: Use vector(mode = “list”) to Pre-Allocate Lists
  80. R Tip: Get Out of the Habit of Calling View() Directly
  81. Speaking on New Tools for R at Big Data Scale
  82. R Tip: Make Arguments Explicit in magrittr/dplyr Pipelines
  83. R Tip: Use drop = FALSE with data.frames
  84. Wanted: cdata Test Pilots
  85. Is R base::subset() really that bad?
  86. R Tip: Force Named Arguments
  87. R Tip: Use [[ ]] Wherever You Can
  88. R Tip: Use seq_len() to Avoid The Backwards Sequence Trap
  89. R Tip: Use qc() For Fast Legible Quoting
  90. We Want to be Playing with a Moderate Number of Powerful Blocks
  91. Is 10,000 Cells Big?
  92. Why No Exact Permutation Tests at Scale?
  93. Supercharge your R code with wrapr
  94. Latest vtreat up on CRAN
  95. A break from our regular programming
  96. Advisory on Multiple Assignment dplyr::mutate() on Databases
  97. Data Reshaping with cdata
  98. Base R can be Fast
  99. Setting up RStudio Server quickly on Amazon EC2
  100. rquery: Fast Data Manipulation in R
  101. New wrapr R pipeline feature: wrapr_applicable
  102. Big cdata News

2017 90

  1. Kudos to Professor Andrew Gelman
  2. Announcing rquery
  3. Plotting Deep Learning Model Performance Trajectories
  4. How to Greatly Speed Up Your Spark Queries
  5. More Pipes in R
  6. Getting started with seplyr
  7. How to Avoid the dplyr Dependency Driven Result Corruption
  8. Please inspect your dplyr+database code
  9. Win-Vector LLC announces new “big data in R” tools
  10. Vectorized Block ifelse in R
  11. Arbitrary Data Transforms Using cdata
  12. RStudio Keyboard Shortcuts for Pipes
  13. Data Wrangling at Scale
  14. Update on coordinatized or fluid data
  15. Let X=X in R
  16. Big Data Transforms
  17. Some Announcements
  18. Partial Pooling for Lower Variance Variable Encoding
  19. Custom Level Coding in vtreat
  20. Upcoming data preparation and modeling article series
  21. On debugging
  22. My advice on dplyr::mutate()
  23. Remember: p-values Are Not Effect Sizes
  24. It is Needlessly Difficult to Count Rows Using dplyr
  25. Permutation Theory In Action
  26. Why to use the replyr R package
  27. Neat New seplyr Feature: String Interpolation
  28. wrapr: R Code Sweeteners
  29. Some Neat New R Notations
  30. Is dplyr Easily Comprehensible?
  31. Thank You For The Very Nice Comment
  32. Supervised Learning in R: Regression
  33. More on “The Part-Time R-User”
  34. Let’s Have Some Sympathy For The Part-time R User
  35. More documentation for Win-Vector R packages
  36. Tutorial: Using seplyr to Program Over dplyr
  37. seplyr update
  38. dplyr 0.7 Made Simpler
  39. Better Grouped Summaries in dplyr
  40. What is magrittr’s future in the tidyverse?
  41. In praise of syntactic sugar
  42. Working With R and Big Data: Use Replyr
  43. Join Dependency Sorting
  44. Using wrapr::let() with tidyeval
  45. Please Consider Using wrapr::let() for Replacement Tasks
  46. wrapr Implementation Update
  47. Non-Standard Evaluation and Function Composition in R
  48. An easy way to accidentally inflate reported R-squared in linear regression models
  49. Use a Join Controller to Document Your Work
  50. Managing intermediate results when using R/sparklyr
  51. Campaign Response Testing no longer published on Udemy
  52. More on safe substitution in R
  53. There is usually more than one way in R
  54. R summary() got better!
  55. In defense of wrapr::let()
  56. Summarizing big data in R
  57. Managing Spark data handles in R
  58. New series: R and big data (concentrating on Spark and sparklyr)
  59. On indexing operators and composition
  60. dplyr in Context
  61. Why to use wrapr::let()
  62. Programming over R
  63. Encoding categorical variables: one-hot and beyond
  64. Teaching pivot / un-pivot
  65. You can’t do that in statistics
  66. Visualizing relational joins
  67. Coordinatized Data: A Fluid Data Specification
  68. Debugging Pipelines in R with Bizarro Pipe and Eager Assignment
  69. Datashader is a big deal
  70. Practical Data Science with R: ACM SIGACT News Book Review and Discount!
  71. Another R [Non-]Standard Evaluation Idea
  72. New screencast: using R and RStudio to install and experiment with Apache Spark
  73. Practical Data Science with R errata update: Java SQLScrewdriver replaced by R procedures and article
  74. Some Win-Vector R packages
  75. sigr: Simple Significance Reporting
  76. Step-Debugging magrittr/dplyr Pipelines in R with wrapr and replyr
  77. replyr: Get a Grip on Big Data in R
  78. vtreat: prepare data
  79. wrapr: for sweet R code
  80. Iteration and closures in R
  81. The Zero Bug
  82. Announcing the wrapr packge for R
  83. My recent BARUG talk: Parametric Programming in R with replyr
  84. Evolving R Tools and Practices
  85. Going to Strata / Hadoop World 2017 San Jose?
  86. Using the Bizarro Pipe to Debug magrittr Pipelines in R
  87. Upcoming Win-Vector LLC public speaking engagements
  88. Upgrading to macOS Sierra (nee OSX) for R users
  89. Why do Decision Trees Work?
  90. A Theory of Nested Cross Simulation

2016 86

  1. Data Preparation, Long Form and tl;dr Form
  2. Does replyr::let work with data.table?
  3. Comparative examples using replyr::let
  4. A Simple Example of Using replyr::gapply
  5. A bit on the formatting of code on this site and HTML/RSS
  6. help(let, package=’replyr’)
  7. Organize your data manipulation in terms of “grouped ordered apply”
  8. magrittr’s Doppelgänger
  9. The Case For Using -> In R
  10. The case for index-free data manipulation
  11. Using replyr::let to Parameterize dplyr Expressions
  12. Parametric variable names and dplyr
  13. Be careful evaluating model predictions
  14. vtreat data cleaning and preparation article now available on arXiv
  15. New R package: replyr (get a grip on remote dplyr data services)
  16. MySql in a container
  17. Teaching Practical Data Science with R
  18. You should re-encode high cardinality categorical variables
  19. Laplace noising versus simulated out of sample methods (cross frames)
  20. Some vtreat design principles
  21. A quick look at RStudio’s R notebooks
  22. Data science for executives and managers
  23. Upcoming Talks
  24. The unfortunate one-sided logic of empirical hypothesis testing
  25. On calculating AUC
  26. Adding polished significance summaries to papers using R
  27. Proofing statistics in papers
  28. Relative error distributions, without the heavy tail theatrics
  29. Adversarial machine learning
  30. Did she know we were writing a book?
  31. Variables can synergize, even in a linear model
  32. The R community is awesome (and fast)
  33. Variable pruning is NP hard
  34. vtreat 0.5.27 released on CRAN
  35. My criticism of R numeric summary
  36. The Win-Vector parallel computing in R series
  37. Can you nest parallel operations in R?
  38. The magrittr monad
  39. On accuracy
  40. A budget of classifier evaluation measures
  41. vtreat version 0.5.26 released on CRAN
  42. y-aware scaling in context
  43. Another note on differential privacy
  44. Why you should read Nina Zumel’s 3 part series on principal components analysis and regression
  45. Free e-book: Exploring Data Science
  46. Using geom_step
  47. A demonstration of vtreat data preparation
  48. Principal Components Regression, Pt. 3: Picking the Number of Components
  49. On ranger respect.unordered.factors
  50. Principal Components Regression, Pt. 2: Y-Aware Methods
  51. Installing WVPlots and “knitting R markdown”
  52. Principal Components Regression, Pt.1: The Standard Method
  53. For a short time: Half Off Some Manning Data Science Books
  54. Coming up: principal components analysis
  55. vtreat cross frames
  56. On Nested Models
  57. Improved vtreat documentation
  58. Free data science video lecture: debugging in R
  59. Half off Win-Vector data science books and video training!
  60. A bit on the F1 score floor
  61. WVPlots: example plots in R using ggplot2
  62. For loops in R can lose class information
  63. More on “npm” leftpad
  64. Upcoming Win-Vector LLC appearances
  65. sample(): “Monkey’s Paw” style programming in R
  66. More on preparing data
  67. Bend or break: strings in R
  68. Win-Vector video courses: price/status changes
  69. Reading and writing proofs
  70. More Shiny user showcase demonstrations
  71. What does a “slow news day” look like at Win-Vector LLC?
  72. Finding the K in K-means by Parametric Bootstrap
  73. Databases in containers
  74. Neglected optimization topic: set diversity
  75. Free video course: applied Bayesian A/B testing in R
  76. Using PostgreSQL in R: A quick how-to
  77. “Introduction to Data Science” video course contest is closed
  78. Shiny Developer Conference
  79. Running R jobs quickly on many machines
  80. Win-Vector data science mailing list (and a give-away!)
  81. Prepping Data for Analysis using R
  82. A gentle introduction to parallel computing in R
  83. Nina Zumel and John Mount part of R Day at Strata + Hadoop World in San Jose 2016
  84. Using Excel versus using R
  85. Nothing to do with data science (just a little drama)
  86. Some programming language theory in R

2015 65

  1. An R function return and assignment puzzle
  2. Win-Vector news
  3. Practical Data Science with R examples
  4. Sequential Analysis
  5. Wald’s sequential analysis technique
  6. What was data science before it was called data science?
  7. Free gradient boosting lecture
  8. Fluid use of data
  9. Upcoming Win-Vector Appearances
  10. Fast food, fast publication
  11. Bitcoin’s status isn’t as simple as ruling if it is more a private token or a public ledger
  12. Our Differential Privacy Mini-series
  13. Don’t use stats::aggregate()
  14. Who is allowed to call themselves a data scientist?
  15. Thank you Joseph Rickert!
  16. A simple differentially private-ish procedure
  17. Baking priors
  18. Thumbs up for Anaconda
  19. Some key Win-Vector serial data science articles
  20. Using differential privacy to reuse training data
  21. A Simpler Explanation of Differential Privacy
  22. A deeper theory of testing
  23. Win-Vector site improvements
  24. How do you know if your model is going to work?
  25. How do you know if your model is going to work? Part 4: Cross-validation techniques
  26. How do you know if your model is going to work? Part 3: Out of sample procedures
  27. What does an announcement like “vtreat now up on CRAN!” mean?
  28. How do you know if your model is going to work? Part 2: In-training set measures
  29. vtreat up on CRAN!
  30. Bootstrap Evaluation of Clusters
  31. How do you know if your model is going to work? Part 1: The problem
  32. How Do You Know if Your Data Has Signal?
  33. I was wrong about statistics
  34. Efficient accumulation in R
  35. Text encoding is a convoluted mess
  36. Working with Sessionized Data 2: Variable Selection
  37. Working with Sessionized Data 1: Evaluating Hazard Models
  38. A dynamic programming solution to A/B test design
  39. What is a good Sharpe ratio?
  40. A bit about Win-Vector LLC
  41. Why does designing a simple A/B test seem so complicated?
  42. Neural net image salad again (with code)
  43. Betting with their money
  44. I do not believe Google invented the term A/B test
  45. Wanted: A Perfect Scatterplot (with Marginals)
  46. R in a 64 bit world
  47. My favorite R bug
  48. What is new in the vtreat library?
  49. Proof style in the Erdős-Ko-Rado theorem
  50. Erdős writing like a programmer
  51. I still think you can manufacture an unfair coin
  52. What can be in an R data.frame column?
  53. New video course: Campaign Response Testing
  54. How and why to return functions in R
  55. Using closures as objects in R
  56. One place not to use the Sharpe ratio
  57. The Win-Vector R data science value pack
  58. Does Balancing Classes Improve Classifier Performance?
  59. Announcing: Introduction to Data Science video course
  60. How sure are you that large margin implies low VC dimension?
  61. Check your return types when modeling in R
  62. Soft margin is not as good as hard margin
  63. R bracket is a bit irregular
  64. A bit more on testing
  65. Random Test/Train Split is not Always Enough

2014 45

  1. Let’s try to motivate schemes
  2. Is there a Kindle edition of Practical Data Science with R?
  3. The Geometry of Classifiers
  4. A comment on preparing data for classifiers
  5. Great new post by Win-Vector’s Nina Zumel
  6. Excel spreadsheets are hard to get right
  7. Can we try to make an adjustment?
  8. Bias/variance tradeoff as gamesmanship
  9. Win-Vector LLC’s John Mount at Strata + Hadoop World October 2014
  10. Estimating Generalization Error with the PRESS statistic
  11. Factors are not first-class citizens in R
  12. What is a win vector?
  13. Reading the Gauss-Markov theorem
  14. Vtreat: designing a package for variable treatment
  15. Diversion: Win-Vector LLC’s Nina Zumel takes time off to publish a literary book review
  16. Automatic bias correction doesn’t fix omitted variable bias
  17. Frequentist inference only seems easy
  18. R minitip: don’t use data.matrix when you mean model.matrix
  19. R style tip: prefer functions that return data frames
  20. Skimming statistics papers for the ideas (instead of the complete procedures)
  21. How does Practical Data Science with R stand out?
  22. Trimming the Fat from glm() Models in R
  23. Save 50% on Practical Data Science with R (and other titles) at Manning through May 30, 2014
  24. Save 45% on Practical Data Science with R (expires May 21, 2014)
  25. R has some sharp corners
  26. Great book discount from Manning (and more about one of our authors)
  27. A clear picture of power and significance in A/B tests
  28. A bit of the agenda of Practical Data Science with R
  29. Bandit Formulations for A/B Tests: Some Intuition
  30. What is meant by regression modeling?
  31. Old tails: a crude power law fit on ebook sales
  32. You don’t need to understand pointers to program using R
  33. Practical Data Science with R: Release date announced
  34. Can a classifier that never says “yes” be useful?
  35. Some statistics about the book
  36. The Statistics behind “Verification by Multiplicity”
  37. Drowning in insignificance
  38. One day discount on Practical Data Science with R
  39. The gap between data mining and predictive models
  40. Unprincipled Component Analysis
  41. Bad Bayes: an example of why you need hold-out testing
  42. Use standard deviation (not mad about MAD)
  43. The Extra Step: Graphs for Communication versus Exploration
  44. Unspeakable bets: take small steps
  45. Generalized linear models for predicting rates

2013 33

  1. Big News! Practical Data Science with R is content complete!
  2. Unit tests as penance
  3. On writing a technical book
  4. Sample size and power for rare events
  5. Yet another bit on bitcoin: is it a semi closed-end index fund on electricity and hardware?
  6. Practical Data Science with R: Manning Deal of the Day November 19th 2013
  7. Resolving git “pseudo conflicts”
  8. Practical Data Science with R October 2013 update
  9. Estimating rates from a single occurrence of a rare event
  10. Some puzzles about boxes
  11. Just spoke at The Berkeley R Language Beginner Study Group
  12. Speaking at BARUG Wednesday, August 21, 2013
  13. Practical Data Science with R, deal of the day Aug 1 2013
  14. What is “Practical Data Science with R”?
  15. On Writing Our Book: A Little Philosophy
  16. Practical Data Science with R news
  17. Big News! “Practical Data Science with R” MEAP launched!
  18. Bayesian and Frequentist Approaches: Ask the Right Question
  19. A pathological glm() problem that doesn’t issue a warning
  20. Prefer = for assignment in R
  21. Data Science, Machine Learning, and Statistics: what is in a name?
  22. Checking claims in published statistics papers
  23. Spring 2013 Win Vector LLC marketing drive
  24. Worry about correctness and repeatability, not p-values
  25. Win Vector LLC now tweets
  26. A non-technical post and ask
  27. A bit more on sample size
  28. Don’t use correlation to track prediction performance
  29. Revisiting Cleveland’s The Elements of Graphing Data in ggplot2
  30. A randomized algorithm that fails with near certainty
  31. Make it more effort to do the wrong thing
  32. Data science project planning
  33. More on ROC/AUC

2012 35

  1. New article on NinaZumel.com : on balance
  2. How to test XCOM “dice rolls” for fairness
  3. Please stop using Excel-like formats to exchange data
  4. Yet Another Java Linear Programming Library
  5. Working an example of von Neumann and Morgenstern utility
  6. Added worked example to logistic regression project
  7. Win-Vector’s Nina Zumel: “I Write, Therefore I Think”
  8. Error Handling in R
  9. Rudie can’t fail (if majorized)
  10. Level fit summaries can be tricky in R
  11. I am done with 32 bit machines
  12. On Being a Data Scientist
  13. On Writing Technical Articles for the Nonspecialist
  14. The Mathematician’s Dilemma
  15. Newton-Raphson can compute an average
  16. How robust is logistic regression?
  17. What does a generalized linear model do?
  18. A bit more on impact coding
  19. Minimal Version Control Lesson: Use It
  20. Modeling Trick: Impact Coding of Categorical Variables with Many Levels
  21. Modeling Trick: Masked Variables
  22. How to outrun a crashing alien spaceship
  23. Selection in R
  24. Pragmatic Machine Learning
  25. Enhance OSX Finder
  26. The differing perspectives of statistics and machine learning
  27. How to remember point shape codes in R
  28. Congratulations to both Dr. Nina Zumel and EMC- great job
  29. Setting expectations in data science projects
  30. Small github reorginization
  31. Modeling Trick: the Signed Pseudo Logarithm
  32. Why I don’t like Dynamic Typing
  33. Ergodic Theory for Interested Computer Scientists
  34. Six Fundamental Methods to Generate a Random Variable
  35. Importance Sampling

2011 22

  1. Why you can not to use statistics to dispute magic
  2. What to do when you run out of memory
  3. My Favorite Graphs
  4. Correlation and R-Squared
  5. An Appreciation of Locality Sensitive Hashing
  6. “The Mythical Man Month” is still a good read
  7. Kernel Methods and Support Vector Machines de-Mystified
  8. Increase your productivity
  9. The equivalence of logistic regression and maximum entropy models
  10. The Simpler Derivation of Logistic Regression
  11. Win-Vector starts submitting content to r-bloggers.com
  12. Programmers Should Know R
  13. Book Review: Ensemble Methods in Data Mining (Seni & Elder)
  14. Your Data is Never the Right Shape
  15. Gerty, a character in Duncan Jones’ “Moon.”
  16. What is a large enough random sample?
  17. Automatic Detection of Potential Deadlock
  18. Brevity is a Virtue
  19. Do your tools support production or complexity?
  20. Blog Slowdown
  21. The cranky guide to trying R packages
  22. SQL Screwdriver

2010 20

  1. Your credit card as the supercomputer
  2. Large Data Logistic Regression (with example Hadoop code)
  3. Learn Logistic Regression (and beyond)
  4. A Personal Perspective on Machine Learning
  5. Lanchester’s Law: why small advantages swell in StarCraft
  6. Statsmanship: Failure Through Analytics Sabotage
  7. Fast Portfolio re-Balancing as a Fractional Linear Program
  8. What Did Theorists Do Before The Age Of Big Data?
  9. Gradients via Reverse Accumulation
  10. Automatic Differentiation with Scala
  11. Must Have Software
  12. Algorithmic Movie (with texture)
  13. SIGACT Review of: Combinatorics the Rota Way
  14. Deming, Wald and Boyd: cutting through the fog of analytics
  15. R annoyances
  16. Postel’s Law: Not Sure Who To Be Angry With
  17. Living in A Lognormal World
  18. Winter 2010 Subscription Campaign
  19. “Easy” Portfolio Allocation
  20. Relative returns: a banker versus trader paradox

2009 28

  1. Statistics to English Translation, Part 2b: Calculating Significance
  2. CRU graph yet again (with R)
  3. Statistics to English Translation, Part 2a: ’Significant’ Doesn’t Always Mean ’Important’
  4. R examine objects tutorial
  5. The Local to Global Principle
  6. “I don’t think that means what you think it means;” Statistics to English Translation, Part 1: Accuracy Measures
  7. Google AdSense Channels IDs and the Cramer Rao Inequality
  8. What is the gambler’s equivalent of Amdahl’s Law?
  9. Survive R
  10. A Discrete Model Gauging Market Efficiency
  11. Good Graphs: Graphical Perception and Data Visualization
  12. A Demonstration of Data Mining
  13. On The Hysteria Over “The Cloud”
  14. Should your mom use Google search?
  15. Microsoft Store Again
  16. Thievery considered harmful
  17. Public Service Article: JSTOR and other Useful Research Archives
  18. Public Service Article: Back Up
  19. What is “Genetic Art?”
  20. Programs reduced to statistics
  21. The Joy of Calculation
  22. The Data Enrichment Method
  23. What does the Market Think?
  24. It is not all the quants’ fault.
  25. Volunteers in Large Clubs: The Theorist’s View
  26. Map Reduce: A Good Idea
  27. Exciting Technique #1: The “R” language.
  28. New “exciting techniques” series of articles.

2008 12

  1. The Purpose of this Blog
  2. Something I don’t get about business and bailouts
  3. A Quick Appreciation of the Sharpe Ratio
  4. What is Mathematics, Really?
  5. YAYGDA (Yet Another Yahoo Google Deal Article)
  6. How Market Designs Set Prices
  7. Betting Best-Of Series
  8. Is Search Advertising a Market for Lemons?
  9. I know, I am the one being a jerk
  10. Sorting Used in Anger
  11. Do Not Let Your Medical Records Be Used Against You
  12. Hello World: An Instance Of Rhetoric in Computer Science

2007 2

  1. Paper on stock trading
  2. New Paper