After releasing a tile calculator app (500+ downloads), I applied the same formula to paint calculation.
Tech stack:

Flutter + Riverpod (state management)
Hive (calculation history)
SharedPreferences (settings persistence)
Material 3, green #4CAF50
flutter_localizations for i18n

Key challenges:
Units system — storing metric/imperial per saved calculation was tricky. Each history record saves its own unitSystem so when you load it back, units don’t reset to current settings.
Opening deductions — supporting both standard sizes (door 2.0×0.9m, window 1.2×1.0m) and custom sizes with separate counters for each type.
Paint brand database — auto-filling coverage when a brand is selected while still allowing manual override.
The app:
Paint Calculator: Room & Wall
https://play.google.com/store/apps/details?id=com.paintcalculator.squarefeet.wallroom
Happy to answer questions about the Flutter implementation.

I spent several weeks reading the wagtail/news-template codebase for my Google Summer of Code application. This post shares what I found – what works, what is broken, and what could be better.

The wagtail/news-template homepage – dark mode enabled by default

What the starter kit is

The news-template is a Wagtail starter kit for news and media websites. It gives you article pages, topic filtering, pagination, search, and dark mode. Everything works out of the box. The design uses Tailwind CSS and is responsive.

What it does well

The content model is solid. ArticlePage supports authors, topics, publication dates, and hero images. The body is built from StreamField blocks. Editors can add sections, quotes, stats, and cards – no code needed.

The block system is flexible. FeaturedArticleBlock lets editors override an article’s image, description, and CTA text. Topic filtering works via query parameters. It is a small but thoughtful detail.

What is broken or missing

Reading the code revealed several bugs.

Newsletter signup never submitted. The email input had no name attribute. The button was an <a> tag with no connection to the input. Every visitor who tried to sign up got no result. I fixed this in PR #107.

Mobile navigation links were broken. A regression left mobile nav links with empty href="" attributes. Users were redirected back to the current page. Fixed in PR #101.

FeaturedArticleBlock had an empty template. The block defined five fields but its template was a stub. It rendered raw block data instead of HTML. Fixed in PR #103.

Missing imports caused runtime errors. slugify was not imported in utils/models.py. Pagination exceptions were missing in news/models.py. Both caused NameError crashes.

Form fields lacked label elements. Inputs had no <label> elements. This is a WCAG 2.1 AA accessibility violation. Fixed in PR #99.

Test coverage was minimal. The project had three tests before my contribution. Core models, template tags, and views had no coverage at all.

What could be added

• Author profile pages with bio and social links

• RSS feed for articles

• Open Graph meta tags for social sharing

• Multilingual support

Final thoughts

The starter kit is a strong foundation. The content model and block system are well thought out. But several core features were broken. Contributing fixes taught me more about Wagtail than any tutorial could.

If you care about statistics, machine learning, or what everyone now calls “AI” — learning linear regression well is the single most important investment you can make. Not transformers. Not diffusion models. Linear regression.

You can go as shallow as calling model.fit(X, y) and declaring you know regression. Or you can go deep — the linear algebra behind the closed-form solution, the calculus behind gradient descent, the numerical analysis behind SVD, the Gauss-Markov assumptions that tell you when least squares is optimal, Cook’s distance that reveals when one rogue data point is hijacking your entire model, and the Bayesian interpretation that turns regularisation into prior beliefs. People still write PhD theses on this subject. It runs that deep.

This post takes you through five fundamentally different algorithms for solving the same regression problem — each from a different branch of mathematics — and they all produce the exact same answer. Five roads, one destination. Along the way, you’ll build intuitions that transfer directly to neural networks, Gaussian processes, and Bayesian inference.

By the end, you’ll implement all five from scratch and understand why they agree.

Let’s Build It

Click the badge to open the interactive notebook:

Watch gradient descent iteratively converge on the least-squares solution:

Here’s the setup — a synthetic dataset with a clear linear trend plus Gaussian noise:

import numpy as np
import matplotlib.pyplot as plt

np.random.seed(42)
n = 50
X_raw = np.random.uniform(0, 10, n)
y = 2.5 * X_raw + 7 + np.random.normal(0, 2.5, n)

# Design matrix: add a column of ones for the intercept
X = np.column_stack([np.ones(n), X_raw])  # shape (50, 2)

Our goal: find $boldsymbol{beta} = [beta_0, beta_1]$ that minimises the sum of squared residuals:

The true values are $beta_0 = 7$ (intercept) and $beta_1 = 2.5$ (slope). Let’s recover them five ways.

Method 1: Normal Equation

The closed-form solution from linear algebra:

beta_normal = np.linalg.inv(X.T @ X) @ X.T @ y
print(f'Normal equation: y = {beta_normal[0]:.2f} + {beta_normal[1]:.2f}x')

Method 2: Gradient Descent

Iteratively update $boldsymbol{beta}$ in the direction that reduces the error:

def gradient_descent(X, y, lr=0.001, n_iter=500):
    beta = np.zeros(X.shape[1])
    history = [beta.copy()]

    for _ in range(n_iter):
        residuals = X @ beta - y
        gradient = (2 / len(y)) * X.T @ residuals
        beta -= lr * gradient
        history.append(beta.copy())

    return beta, history

beta_gd, gd_history = gradient_descent(X, y, lr=0.02, n_iter=1000)
print(f'Gradient descent: y = {beta_gd[0]:.2f} + {beta_gd[1]:.2f}x')

Method 3: SVD (Pseudo-Inverse)

The numerically stable route via singular value decomposition:

U, s, Vt = np.linalg.svd(X, full_matrices=False)
beta_svd = Vt.T @ np.diag(1 / s) @ U.T @ y
print(f'SVD:             y = {beta_svd[0]:.2f} + {beta_svd[1]:.2f}x')

Method 4: scipy.optimize.minimize

Treat regression as a generic optimisation problem:

from scipy.optimize import minimize

def mse_loss(beta, X, y):
    return np.mean((y - X @ beta) ** 2)

result = minimize(mse_loss, x0=[0, 0], args=(X, y), method='Nelder-Mead')
beta_scipy = result.x
print(f'scipy.optimize:  y = {beta_scipy[0]:.2f} + {beta_scipy[1]:.2f}x')

Method 5: sklearn

The one-liner you’ll use in practice:

from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit(X_raw.reshape(-1, 1), y)
beta_sklearn = np.array([model.intercept_, model.coef_[0]])
print(f'sklearn:         y = {beta_sklearn[0]:.2f} + {beta_sklearn[1]:.2f}x')

All five produce the same answer:

The lines overlap perfectly. Every method converges to the same $boldsymbol{beta}$ because they’re all minimising the same objective — the sum of squared residuals.

What Just Happened?

Five algorithms, one answer. Let’s understand why, and where they differ.

The Normal Equation: Where the Gradient Is Zero

The mean squared error is:

Setting the gradient to zero:

Solving for $boldsymbol{beta}$:

This is exact — no iteration, no approximation. But it requires $X^top X$ to be invertible. If your features are collinear (or nearly so), the determinant is close to zero and the solution explodes numerically.

Gradient Descent: Follow the Slope Downhill

Instead of solving the equation analytically, gradient descent takes small steps in the direction of steepest descent:

where $eta$ is the learning rate. The gradient $nabla L = -frac{2}{n}X^top(mathbf{y} - Xboldsymbol{beta})$ points uphill, so we subtract it.

This is the same algorithm that trains neural networks — just applied to a simpler problem. If you understand gradient descent here, you understand SGD everywhere.

The learning rate matters. Too large and the iterations overshoot and diverge. Too small and convergence is painfully slow. For linear regression, the optimal learning rate relates to the eigenvalues of $X^top X$.

SVD: The Numerically Stable Path

The SVD decomposes $X = U Sigma V^top$. Substituting into the normal equation:

This avoids computing $X^top X$ entirely. When $X^top X$ is ill-conditioned (near-singular), the normal equation gives garbage, but SVD handles it gracefully by working with the singular values directly. This is what numpy’s lstsq and sklearn use internally.

scipy.optimize: The Black-Box Approach

The minimize function treats regression as a generic optimisation problem. We pass the loss function, initial guess, and algorithm — it handles the rest. Nelder-Mead (a derivative-free simplex method, like in our GP hyperparameter optimisation) doesn’t even compute gradients. This approach generalises to any loss function — L1, Huber, custom losses — where closed-form solutions don’t exist.

sklearn: What Happens Inside

sklearn’s LinearRegression uses the SVD internally via numpy.linalg.lstsq. It handles the design matrix construction (adding the intercept column), feature scaling, and edge cases. For production code, always use sklearn — it’s tested, optimised, and handles corner cases you’d forget.

Going Deeper

OLS Is Maximum Likelihood Under Gaussian Noise

If we assume the noise is Gaussian — $y_i = mathbf{x}_i^topboldsymbol{beta} + epsilon_i$ where $epsilon_i sim mathcal{N}(0, sigma^2)$ — then the likelihood is:

Taking the log and maximising gives exactly the least-squares objective. So OLS is maximum likelihood estimation under Gaussian noise — the same MLE framework that appears throughout our tutorial series.

Ridge Regression: When the Normal Equation Breaks

If $X^top X$ is singular or nearly so, the normal equation is unstable. Ridge regression adds a penalty $lambda|boldsymbol{beta}|^2$ to the objective:

The $lambda I$ term makes the matrix invertible regardless of collinearity. Larger $lambda$ shrinks coefficients toward zero — a bias-variance tradeoff you can explore hands-on in the Overfitting Explorer.

def ridge_regression(X, y, lam):
    """Ridge regression: closed-form solution."""
    n_features = X.shape[1]
    return np.linalg.inv(X.T @ X + lam * np.eye(n_features)) @ X.T @ y

lambdas = [0, 1, 10, 100]
for lam in lambdas:
    beta_r = ridge_regression(X, y, lam)
    print(f'lambda={lam:>3d}: y = {beta_r[0]:.2f} + {beta_r[1]:.2f}x')

The Bayesian interpretation: Ridge regression is equivalent to placing a Gaussian prior $boldsymbol{beta} sim mathcal{N}(mathbf{0}, sigma^2/lambda cdot I)$ on the weights and computing the MAP estimate. The connection to Bayesian inference runs deep — Ridge is OLS with a prior, and GP regression generalises this to infinite-dimensional feature spaces.

Influential Observations: When One Point Rules the Model

OLS minimises squared errors, which means a single outlier far from the data cloud can drag the entire regression line toward itself. This isn’t a bug — it’s a mathematical consequence of the $L^2$ loss.

Cook’s distance (Cook, 1977) quantifies this: it measures how much all fitted values change when observation $i$ is removed. The formula combines the residual size with the observation’s leverage — how extreme its $x$ value is:

A common rule of thumb flags points with $D_i > 4/n$ as potentially influential. The danger is subtle: a high-leverage point (extreme $x$) doesn’t always distort the fit — it might sit right on the true line. But when a high-leverage point is also an outlier in $y$, it can single-handedly determine the slope.

This is why residual analysis is not optional. Plotting residuals against fitted values, checking for heteroscedasticity (fan-shaped patterns), and inspecting Q-Q plots for normality are how you verify that your model’s assumptions actually hold on your data. A clean-looking $R^2$ can mask serious problems that only residual plots reveal.

When influential points are genuine (not data errors), consider robust alternatives: Huber loss, L1 regression (which minimises absolute errors instead of squared), or RANSAC.

When NOT to Use Linear Regression

1. Non-linear relationships — if the true relationship is curved, linear regression gives a systematically wrong answer. Try polynomial features or non-parametric methods.
2. Outliers — OLS is sensitive to outliers because it minimises squared errors. A single outlier can tilt the line dramatically. Use Cook’s distance to diagnose, and consider Huber loss or L1 regression as robust alternatives.
3. Collinear features — when features are highly correlated, the normal equation becomes unstable. Use Ridge, Lasso, or PCA first.

Computational Complexity Comparison

Where $n$ = samples, $p$ = features, $T$ = iterations. For most tabular ML, sklearn’s SVD-based approach is the right choice.

Where This Comes From

Legendre (1805) and Gauss (1809)

The method of least squares has one of the most famous priority disputes in science. Adrien-Marie Legendre published it first in 1805, in Nouvelles méthodes pour la détermination des orbites des comètes. But Carl Friedrich Gauss claimed he had been using it since 1795 — and in his 1809 Theoria motus corporum coelestium, he provided the theoretical justification by connecting least squares to maximum likelihood under Gaussian errors.

The original application was astronomy: predicting the orbit of the asteroid Ceres from a handful of noisy observations. The same problem structure — fitting a model to noisy data — appears in every regression task today.

The Normal Equation

Gauss showed that the least-squares estimator satisfies the normal equations:

The name “normal” comes from the geometric interpretation: the residual vector $mathbf{y} - Xboldsymbol{beta}$ is normal (perpendicular) to the column space of $X$. The OLS solution is the orthogonal projection of $mathbf{y}$ onto the space spanned by the features.

The Gauss-Markov Theorem

OLS is optimal — but only under specific assumptions. The classical linear model requires:

1. Linearity: $y = Xboldsymbol{beta} + boldsymbol{epsilon}$ — the true relationship is linear in the parameters
2. Exogeneity: $mathbb{E}[epsilon_i mid X] = 0$ — errors are unrelated to the features
3. Homoscedasticity: $text{Var}(epsilon_i) = sigma^2$ — constant error variance across all observations
4. No autocorrelation: $text{Cov}(epsilon_i, epsilon_j) = 0$ for $i neq j$ — errors are independent
5. Full rank: $X$ has full column rank — no feature is a perfect linear combination of others

When these hold, the Gauss-Markov theorem guarantees that OLS is the Best Linear Unbiased Estimator (BLUE):

Among all linear unbiased estimators, OLS has the smallest variance.

The key word is unbiased. The theorem doesn’t say OLS is the best estimator period — biased estimators like Ridge can achieve lower mean squared error by trading a small bias for a large reduction in variance.

When the assumptions fail, OLS can mislead. Heteroscedastic errors inflate standard errors. Autocorrelated residuals (common in time series) make confidence intervals too narrow. And multicollinearity makes the normal equation numerically unstable — exactly the scenario Ridge regression was designed for.

Further Reading

• Hastie, Tibshirani & Friedman (2009) The Elements of Statistical Learning — Chapter 3 covers linear regression comprehensively. Free online.
• Bishop (2006) Pattern Recognition and Machine Learning — Chapter 3 connects linear regression to Bayesian inference
• Strang (2019) Linear Algebra and Learning from Data — the linear algebra perspective on regression and SVD
• Cook (1977) Detection of Influential Observation in Linear Regression — the original Cook’s distance paper
• Our GP regression post — see how the normal equation generalises to non-parametric regression

Try It Yourself

The interactive notebook includes exercises:

1. Learning rate experiments — Try learning rates of 0.0001, 0.005, and 0.05. Plot the loss curve for each. At what rate does gradient descent diverge?
2. L1 vs L2 regression — Generate data with 3 outliers. Compare the OLS line (L2) with the minimum absolute deviation line (L1, using scipy.optimize with the min_abs_dev_error loss from the original code). Which is more robust?
3. Polynomial features — Generate data from $y = x^2 + epsilon$ and fit linear regression with features $[1, x, x^2]$. How does the design matrix change?
4. Condition number — Create two nearly collinear features ($x_2 = x_1 + epsilon$). Compare the normal equation vs SVD solutions. What happens to np.linalg.cond(X.T @ X)?

Interactive Tools

• Regression Playground — Fit linear, polynomial, and regularised models to your own data in the browser
• Overfitting Explorer — See the bias-variance tradeoff in action as you increase polynomial degree

Related Posts

• Maximum Likelihood Estimation from Scratch — OLS is MLE under Gaussian noise
• Gaussian Process Regression: The Bayesian Approach to Curve Fitting — GP regression generalises the normal equation to infinite dimensions
• Backpropagation: Neural Nets from First Principles — Gradient descent on a much harder problem

Frequently Asked Questions

What are the different ways to solve linear regression?

Linear regression can be solved with the normal equation (closed-form matrix algebra), gradient descent (iterative optimisation), QR decomposition (numerically stable matrix factorisation), singular value decomposition (SVD), and regularised variants like Ridge and Lasso. Each method has different trade-offs in speed, stability, and applicability.

When should I use gradient descent instead of the normal equation?

Use gradient descent when the number of features is very large (more than about 10,000), because the normal equation requires inverting a matrix that is quadratic in the number of features. Gradient descent scales linearly with the number of features and can also handle datasets too large to fit in memory via mini-batch processing.

What is the difference between Ridge and Lasso regression?

Both add a penalty to prevent overfitting, but Ridge (L2) shrinks coefficients towards zero without eliminating them, while Lasso (L1) can set coefficients exactly to zero, effectively performing feature selection. Use Lasso when you expect many features to be irrelevant; use Ridge when most features contribute to the prediction.

Why does the normal equation sometimes fail?

The normal equation requires inverting the matrix X^T X, which fails when features are perfectly collinear (the matrix is singular). Even near-collinearity causes numerical instability, amplifying rounding errors. QR or SVD-based methods handle these cases more gracefully. Ridge regression also solves this by regularising the matrix.

Is linear regression still useful in the age of deep learning?

Absolutely. Linear regression is interpretable, fast, requires no hyperparameter tuning (for the basic form), and works surprisingly well when the relationship is approximately linear. It serves as a strong baseline, and understanding it deeply is essential for understanding more complex models that build on the same principles.

Hi, this post was initially written for Caché. In June 2023, I finally updated it for IRIS. If you are revisiting the post since then, the only real change is substituting Caché for IRIS! I also updated the links for IRIS documentation and fixed a few typos and grammatical errors. Enjoy 🙂

In this post, I show strategies for backing up InterSystems IRIS using External Backup with examples of integrating with snapshot-based solutions. Most solutions I see today are deployed on Linux on VMware, so a lot of the post shows how solutions integrate VMware snapshot technology as examples.

IRIS backup – batteries included?

IRIS online backup is included with an IRIS install for uninterrupted backup of IRIS databases. But there are more efficient backup solutions you should consider as systems scale up. External Backup integrated with snapshot technologies is the recommended solution for backing up systems, including IRIS databases.

Are there any special considerations for external backup?

Online documentation for External Backup has all the details. A key consideration is:

“To ensure the integrity of the snapshot, IRIS provides methods to freeze writes to databases while the snapshot is created. Only physical writes to the database files are frozen during the snapshot creation, allowing user processes to continue performing updates in memory uninterrupted.”

It is also important to note that part of the snapshot process on virtualised systems causes a short pause on a VM being backed up, often called stun time. Usually less than a second, so not noticed by users or impacting system operation; however, in some circumstances, the stun can last longer. If the stun is longer than the quality of service (QoS) timeout for IRIS database mirroring, then the backup node will think there has been a failure on the primary and will failover. Later in this post, I explain how you can review stun times in case you need to change the mirroring QoS timeout.

A list of other InterSystems Data Platforms and performance series posts is here.

You should also review IRIS online documentation Backup and Restore Guide for this post.

Backup choices

Minimal Backup Solution – IRIS Online Backup

If you have nothing else, this comes in the box with the InterSystems data platform for zero downtime backups. Remember, IRIS online backup only backs up IRIS database files, capturing all blocks in the databases that are allocated for data with the output written to a sequential file. IRIS Online Backup supports cumulative and incremental backups.

In the context of VMware, an IRIS Online Backup is an in-guest backup solution. Like other in-guest solutions, IRIS Online Backup operations are essentially the same whether the application is virtualised or runs directly on a host. IRIS Online Backup must be coordinated with a system backup to copy the IRIS online backup output file to backup media and all other file systems used by your application. At a minimum, system backup must include the installation directory, journal and alternate journal directories, application files, and any directory containing external files the application uses.

IRIS Online Backup should be considered as an entry-level approach for smaller sites wishing to implement a low-cost solution to back up only IRIS databases or ad-hoc backups; for example, it is helpful in the set-up of mirroring. However, as databases increase in size and as IRIS is typically only part of a customer’s data landscape, External Backups combined with snapshot technology and third-party utilities are recommended as best practice with advantages such as including the backup of non-database files, faster restore times, enterprise-wide view of data and better catalogue and management tools.

Recommended Backup Solution – External backup

Using VMware as an example, Virtualising on VMware adds functionality and choices for protecting entire VMs. Once you have virtualised a solution, you have effectively encapsulated your system — including the operating system, the application and the data — all within .vmdk (and some other) files. When required, these files can be straightforward to manage and used to recover a whole system, which is very different from the same situation on a physical system where you must recover and configure the components separately — operating system, drivers, third-party applications, database and database files, etc.

VMware snapshot

VMware’s vSphere Data Protection (VDP) and other third-party backup solutions for VM backup, such as Veeam or Commvault, take advantage of the functionality of VMware virtual machine snapshots to create backups. A high-level explanation of VMware snapshots follows; see the VMware documentation for more details.

It is important to remember that snapshots are applied to the whole VM and that the operating system and any applications or the database engine are unaware that the snapshot is happening. Also, remember:

By themselves, VMware snapshots are not backups!

Snapshots enable backup software to make backups, but they are not backups by themselves.

VDP and third-party backup solutions use the VMware snapshot process in conjunction with the backup application to manage the creation and, very importantly, deletion of snapshots. At a high level, the process and sequence of events for an external backup using VMware snapshots are as follows:

• Third-party backup software requests the ESXi host to trigger a VMware snapshot.
• A VM’s .vmdk files are put into a read-only state, and a child vmdk delta file is created for each of the VM’s .vmdk files.
• Copy on write is used with all changes to the VM written to the delta files. Any reads are from the delta file first.
• The backup software manages copying the read-only parent .vmdk files to the backup target.
• When the backup is complete, the snapshot is committed (VM disks resume writes and updated blocks in delta files written to parent).
• The VMware snapshot is now removed.

Backup solutions also use other features such as Change Block Tracking (CBT) to allow incremental or cumulative backups for speed and efficiency (especially important for space saving), and typically also add other important functions such as data deduplication and compression, scheduling, mounting VMs with changed IP addresses for integrity checks etc., full VM and file level restores, and catalogue management.

VMware snapshots that are not appropriately managed or left to run for a long time can use excessive storage (as more and more data is changed, delta files continue to grow) and also slow down your VMs.

You should think carefully before running a manual snapshot on a production instance. Why are you doing this? What will happen if you revert back in time to when the snapshot was created? What happens to all the application transactions between creation and rollback?

It is OK if your backup software creates and deletes a snapshot. The snapshot should only be around for a short time. And a crucial part of your backup strategy will be to choose a time when the system has low usage to minimise any further impact on users and performance.

IRIS database considerations for snapshots

Before the snapshot is taken, the database must be quiesced so that all pending writes are committed, and the database is in a consistent state. IRIS provides methods and an API to commit and then freeze (stop) writes to databases for a short period while the snapshot is created. This way, only physical writes to the database files are frozen during the creation of the snapshot, allowing user processes to continue performing updates in memory uninterrupted. Once the snapshot has been triggered, database writes are thawed, and the backup continues copying data to backup media. The time between freeze and thaw should be quick (a few seconds).

In addition to pausing writes, the IRIS freeze also handles switching journal files and writing a backup marker to the journal. The journal file continues to be written normally while physical database writes are frozen. If the system were to crash while the physical database writes are frozen, data would be recovered from the journal as usual during start-up.

The following diagram shows freeze and thaw with VMware snapshot steps to create a backup with a consistent database image.

VMware snapshot + IRIS freeze/thaw timeline (not to scale)

Note the short time between Freeze and Thaw — only the time to create the snapshot, not the time to copy the read-only parent to the backup target.

Summary – Why do I need to freeze and thaw the IRIS database when VMware is taking a snapshot?

The process of freezing and thawing the database is crucial to ensure data consistency and integrity. This is because:

Data Consistency: IRIS can be writing journals, or the WIJ or doing random writes to the database at any time. A snapshot captures the state of the VM at a specific point in time. If the database is actively being written during the snapshot, it can lead to a snapshot that contains partial or inconsistent data. Freezing the database ensures that all transactions are completed and no new transactions start during the snapshot, leading to a consistent disk state.

Quiescing the File System: VMware’s snapshot technology can quiesce the file system to ensure file system consistency. However, this does not account for the application or database level consistency. Freezing the database ensures that the database is in a consistent state at the application level, complementing VMware’s quiescing.

Reducing Recovery Time: Restoring from a snapshot that was taken without freezing the database might require additional steps like database repair or consistency checks, which can significantly increase recovery time. Freezing and thawing ensure the database is immediately usable upon restoration, reducing downtime.

Integrating IRIS Freeze and Thaw

vSphere allows a script to be automatically called on either side of snapshot creation; this is when IRIS Freeze and Thaw are called. Note: For this functionality to work correctly, the ESXi host requests the guest operating system to quiesce the disks via VMware Tools.

VMware tools must be installed in the guest operating system.

The scripts must adhere to strict name and location rules. File permissions must also be set. For VMware on Linux, the script names are:

# /usr/sbin/pre-freeze-script
# /usr/sbin/post-thaw-script

Below are examples of freeze and thaw scripts our team use with Veeam backup for our internal test lab instances, but these scripts should also work with other solutions. These examples have been tested and used on vSphere 6 and Red Hat 7.

While these scripts can be used as examples and illustrate the method, you must validate them for your environments!

Example pre-freeze-script:

#!/bin/sh
#
# Script called by VMWare immediately prior to snapshot for backup.
# Tested on Red Hat 7.2
#

LOGDIR=/var/log
SNAPLOG=$LOGDIR/snapshot.log

echo >> $SNAPLOG
echo "`date`: Pre freeze script started" >> $SNAPLOG
exit_code=0

# Only for running instances
for INST in `iris qall 2>/dev/null | tail -n +3 | grep '^up' | cut -c5-  | awk '{print $1}'`; do

    echo "`date`: Attempting to freeze $INST" >> $SNAPLOG

    # Detailed instances specific log    
    LOGFILE=$LOGDIR/$INST-pre_post.log

    # Freeze
    irissession $INST -U '%SYS' "##Class(Backup.General).ExternalFreeze("$LOGFILE",,,,,,1800)" >> $SNAPLOG $
    status=$?

    case $status in
        5) echo "`date`:   $INST IS FROZEN" >> $SNAPLOG
           ;;
        3) echo "`date`:   $INST FREEZE FAILED" >> $SNAPLOG
           logger -p user.err "freeze of $INST failed"
           exit_code=1
           ;;
        *) echo "`date`:   ERROR: Unknown status code: $status" >> $SNAPLOG
           logger -p user.err "ERROR when freezing $INST"
           exit_code=1
           ;;
    esac
    echo "`date`:   Completed freeze of $INST" >> $SNAPLOG
done

echo "`date`: Pre freeze script finished" >> $SNAPLOG
exit $exit_code

Example thaw script:

#!/bin/sh
#
# Script called by VMWare immediately after backup snapshot has been created
# Tested on Red Hat 7.2
#

LOGDIR=/var/log
SNAPLOG=$LOGDIR/snapshot.log

echo >> $SNAPLOG
echo "`date`: Post thaw script started" >> $SNAPLOG
exit_code=0

if [ -d "$LOGDIR" ]; then

    # Only for running instances    
    for INST in `iris qall 2>/dev/null | tail -n +3 | grep '^up' | cut -c5-  | awk '{print $1}'`; do

        echo "`date`: Attempting to thaw $INST" >> $SNAPLOG

        # Detailed instances specific log
        LOGFILE=$LOGDIR/$INST-pre_post.log

        # Thaw
        irissession $INST -U%SYS "##Class(Backup.General).ExternalThaw("$LOGFILE")" >> $SNAPLOG 2>&1
        status=$?

        case $status in
            5) echo "`date`:   $INST IS THAWED" >> $SNAPLOG
               irissession $INST -U%SYS "##Class(Backup.General).ExternalSetHistory("$LOGFILE")" >> $SNAPLOG$
               ;;
            3) echo "`date`:   $INST THAW FAILED" >> $SNAPLOG
               logger -p user.err "thaw of $INST failed"
               exit_code=1
               ;;
            *) echo "`date`:   ERROR: Unknown status code: $status" >> $SNAPLOG
               logger -p user.err "ERROR when thawing $INST"
               exit_code=1
               ;;
        esac
        echo "`date`:   Completed thaw of $INST" >> $SNAPLOG
    done
fi

echo "`date`: Post thaw script finished" >> $SNAPLOG
exit $exit_code

Remember to set permissions:

# sudo chown root.root /usr/sbin/pre-freeze-script /usr/sbin/post-thaw-script
# sudo chmod 0700 /usr/sbin/pre-freeze-script /usr/sbin/post-thaw-script

Testing Freeze and Thaw

To test the scripts are running correctly, you can manually run a snapshot on a VM and check the script output. The following screenshot shows the “Take VM Snapshot” dialogue and options.

Deselect– “Snapshot the virtual machine’s memory”.

Select – the “Quiesce guest file system (Needs VMware Tools installed)” check box to pause running processes on the guest operating system so that file system contents are in a known consistent state when you take the snapshot.

Important! After your test, remember to delete the snapshot!!!!

If the quiesce flag is true, and the virtual machine is powered on when the snapshot is taken, VMware Tools is used to quiesce the file system in the virtual machine. Quiescing a file system is a process of bringing the on-disk data into a state suitable for backups. This process might include such operations as flushing dirty buffers from the operating system’s in-memory cache to disk.

The following output shows the contents of the $SNAPSHOT log file set in the example freeze/thaw scripts above after running a backup that includes a snapshot as part of its operation.

Wed Jan  4 16:30:35 EST 2017: Pre freeze script started
Wed Jan  4 16:30:35 EST 2017: Attempting to freeze H20152
Wed Jan  4 16:30:36 EST 2017:   H20152 IS FROZEN
Wed Jan  4 16:30:36 EST 2017:   Completed freeze of H20152
Wed Jan  4 16:30:36 EST 2017: Pre freeze script finished

Wed Jan  4 16:30:41 EST 2017: Post thaw script started
Wed Jan  4 16:30:41 EST 2017: Attempting to thaw H20152
Wed Jan  4 16:30:42 EST 2017:   H20152 IS THAWED
Wed Jan  4 16:30:42 EST 2017:   Completed thaw of H20152
Wed Jan  4 16:30:42 EST 2017: Post thaw script finished

This example shows 6 seconds of elapsed time between freeze and thaw (16:30:36-16:30:42). User operations are NOT interrupted during this period. You will have to gather metrics from your own systems, but for some context, this example is from a system running an application benchmark on a VM with no IO bottlenecks and an average of more than 2 million Glorefs/sec, 170,000 Gloupds/sec, and an average 1,100 physical reads/sec and 3,000 writes per write daemon cycle.

Remember that memory is not part of the snapshot, so on restarting, the VM will reboot and recover. Database files will be consistent. You don’t want to “resume” a backup; you want the files at a known point in time. You can then roll forward journals and whatever other recovery steps are needed for the application and transactional consistency once the files are recovered.

For additional data protection, a journal switch can be done by itself, and journals can be backed up or replicated to another location, for example, hourly.

Below is the output of the $LOGFILE in the example freeze/thaw scripts above, showing journal details for the snapshot.

01/04/2017 16:30:35: Backup.General.ExternalFreeze: Suspending system

Journal file switched to:
/trak/jnl/jrnpri/h20152/H20152_20170104.011
01/04/2017 16:30:35: Backup.General.ExternalFreeze: Start a journal restore for this backup with journal file: /trak/jnl/jrnpri/h20152/H20152_20170104.011

Journal marker set at
offset 197192 of /trak/jnl/jrnpri/h20152/H20152_20170104.011
01/04/2017 16:30:36: Backup.General.ExternalFreeze: System suspended
01/04/2017 16:30:41: Backup.General.ExternalThaw: Resuming system
01/04/2017 16:30:42: Backup.General.ExternalThaw: System resumed

VM Stun Times

At the creation point of a VM snapshot and after the backup is complete and the snapshot is committed, the VM needs to be frozen for a short period. This short freeze is often referred to as stunning the VM. A good blog post on stun times is here. I summarise the details below and put them in the context of IRIS database considerations.

From the post on stun times: “To create a VM snapshot, the VM is “stunned” in order to (i) serialize device state to disk, and (ii) close the current running disk and create a snapshot point.…When consolidating, the VM is “stunned” in order to close the disks and put them in a state that is appropriate for consolidation.”

Stun time is typically a few 100 milliseconds; however, if there is a very high disk write activity during the commit phase, stun time could be several seconds.

If the VM is a Primary or Backup member participating in IRIS Database Mirroring and the stun time is longer than the mirror Quality of Service (QoS) timeout, the mirror will report the Primary VM as failed and initiate a mirror takeover.

Update March 2018:
My colleague, Peter Greskoff, pointed out that a backup mirror member could initiate failover in as short a time as just over half QoS timeout during a VM stun or any other time the primary mirror member is unavailable.

For a detailed description of QoS considerations and failover scenarios, see this great post: Quality of Service Timeout Guide for Mirroring, however the short story regarding VM stun times and QoS is:

If the backup mirror does not receive any messages from the primary mirror within half of the QoS timeout, it will send a message to ensure the primary is still alive. The backup then waits an additional half QoS time for a response from the primary machine. If there is no response from the primary, it is assumed to be down, and the backup will take over.

On a busy system, journals are continuously sent from the primary to the backup mirror, and the backup would not need to check if the primary is still alive. However, during a quiet time — when backups are more likely to happen — if the application is idle, there may be no messages between the primary and backup mirror for more than half the QoS time.

Here is Peter’s example; Think about this time frame for an idle system with a QoS timeout of:08 seconds and a VM stun time of:07 seconds:

• :00 Primary pings the arbiter with a keepalive, arbiter responds immediately
• :01 backup member sends keepalive to the primary, primary responds immediately
• :02
• :03 VM stun begins
• :04 primary tries to send keepalive to the arbiter, but it doesn’t get through until stun is complete
• :05 backup member sends a ping to primary, as half of QoS has expired
• :06
• :07
• :08 arbiter hasn’t heard from the primary in a full QoS timeout, so it closes the connection
• :09 The backup hasn’t gotten a response from the primary and confirms with the arbiter that it also lost connection, so it takes over
• :10 VM stun ends, too late!!

Please also read the section, Pitfalls and Concerns when Configuring your Quality of Service Timeout, in the linked post above to understand the balance to have QoS only as long as necessary. Having QoS too long, especially more than 30 seconds, can also cause problems.

End update March 2018:

For more information on Mirroring QoS, also see the documentation.

Strategies to keep stun time to a minimum include running backups when database activity is low and having well-set-up storage.

As noted above, when creating a snapshot, there are several options you can specify; one of the options is to include the memory state in the snapshot – Remember, memory state is NOT needed for IRIS database backups. If the memory flag is set, a dump of the internal state of the virtual machine is included in the snapshot. Memory snapshots take much longer to create. Memory snapshots are used to allow reversion to a running virtual machine state as it was when the snapshot was taken. This is NOT required for a database file backup.

When taking a memory snapshot, the entire state of the virtual machine will be stunned, stun time is variable.

As noted previously, for backups, the quiesce flag must be set to true for manual snapshots or by the backup software to guarantee a consistent and usable backup.

Reviewing VMware logs for stun times

Starting from ESXi 5.0, snapshot stun times are logged in each virtual machine’s log file (vmware.log) with messages similar to:

2017-01-04T22:15:58.846Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 38123 us

Stun times are in microseconds, so in the above example, 38123 us is 38123/1,000,000 seconds or 0.038 seconds.

To be sure that stun times are within acceptable limits or to troubleshoot if you suspect long stun times are causing problems, you can download and review the vmware.log files from the folder of the VM that you are interested in. Once downloaded, you can extract and sort the log using the example Linux commands below.

Example downloading vmware.log files

There are several ways to download support logs, including creating a VMware support bundle through the vSphere management console or from the ESXi host command line. Consult the VMware documentation for all the details, but below is a simple method to create and gather a much smaller support bundle that includes the vmware.log file so you can review stun times.

You will need the long name of the directory where the VM files are located. Log on to the ESXi host where the database VM is running using ssh and use the command: vim-cmd vmsvc/getallvms  to list vmx files and the long names unique associated with them.

For example, the long name for the example database VM used in this post is output as:
26 vsan-tc2016-db1 [vsanDatastore] e2fe4e58-dbd1-5e79-e3e2-246e9613a6f0/vsan-tc2016-db1.vmx rhel7_64Guest vmx-11

Next, run the command to gather and bundle only log files:

vm-support -a VirtualMachines:logs.

The command will echo the location of the support bundle, for example:
To see the files collected, check '/vmfs/volumes/datastore1 (3)/esx-esxvsan4.iscinternal.com-2016-12-30--07.19-9235879.tgz'.

You can now use sftp to transfer the file off the host for further processing and review.

In this example, after uncompressing the support bundle navigate to the path corresponding to the database VMs long name. For example, in this case:
<bundle name>/vmfs/volumes/<host long name>/e2fe4e58-dbd1-5e79-e3e2-246e9613a6f0.

You will see several numbered log files; the most recent log file has no number, i.e. vmware.log. The log may be only a few 100 KB, but there is a lot of information; however, we care about the stun/unstun times, which are easy enough to find with grep. For example:

$ grep Unstun vmware.log
2017-01-04T21:30:19.662Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 1091706 us
--- 
2017-01-04T22:15:58.846Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 38123 us
2017-01-04T22:15:59.573Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 298346 us
2017-01-04T22:16:03.672Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 301099 us
2017-01-04T22:16:06.471Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 341616 us
2017-01-04T22:16:24.813Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 264392 us
2017-01-04T22:16:30.921Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 221633 us

We can see two groups of stun times in the example, one from snapshot creation and a second set 45 minutes later for each disk when the snapshot is deleted/consolidated (e.g. after the backup software has completed copying the read-only vmx file). The above example shows that most stun times are sub-second, although the initial stun time is just over one second.

Short stun times are not noticeable to an end user. However, system processes such as IRIS Database Mirroring continuously monitor whether an instance is ‘alive’. If the stun time exceeds the mirroring QoS timeout, the node may be considered uncontactable and ‘dead’, and a failover will be triggered.

Tip: To review all the logs or for trouble-shooting, a handy command is to grep all the vmware*.log files and look for any outliers or instances where stun time is approaching QoS timeout. The following command pipes the output to awk for formatting:

grep Unstun vmware* | awk '{ printf ("%'"'"'d", $8)} {print " ---" $0}' | sort -nr

Summary

You should monitor your system regularly during normal operations to understand stun times and how they may impact QoS timeout for HA, such as mirroring. As noted, strategies to keep stun/unstun time to a minimum include running backups when database and storage activity is low and having well-set-up storage. For constant monitoring, logs may be processed by using VMware Log Insight or other tools.

In future posts, I will revisit backup and restore operations for InterSystems Data Platforms. But for now, if you have any comments or suggestions based on the workflows of your systems, please share them via the comments sections below.

The DEV community is buzzing about OpenTelemetry standardizing LLM tracing. That is a real win. Spans, traces, semantic conventions for gen AI — all of it matters. I have been watching this space for a while.

But I want to say something that production experience has drilled into me.

Observability without correction is a dashboard full of problems you are still solving manually.

What Tracing Gives You
OpenTelemetry for LLMs gives you visibility into:

• Latency per call
• Token consumption
• Span trees across your agent chain
• Model inputs and outputs at each step

That is genuinely useful. I am not dismissing it.
But here is what it does not give you:

Detection that the output is hallucinated before it reaches your user
Automatic retry with a corrected prompt when groundedness fails
Cost circuit breakers that fire before your inference bill explodes
Safety flags that block a response instead of just logging that it was bad

You are still the correction layer. You are the human staring at a Grafana dashboard at 2am deciding whether to roll back a prompt.

What Production AI Actually Looks Like

I spent the last two years building AI systems at scale inside regulated industries. Healthcare revenue cycle. Power grid intelligence. Genomics pipelines. These are not playgrounds.

The pattern I kept seeing: teams invest heavily in logging and tracing. They build beautiful dashboards. And then when the LLM misbehaves in production, the process to correct it is still manual, slow, and incident-driven.

The gap is not observability. The gap is autonomous correction at the output layer.

Nobody had shipped that as a product. So I built it.

ARGUS: Autonomous Runtime Guardian for Unified Systems

ARGUS is an open-source LLM observability platform that goes one layer further than tracing. It evaluates six dimensions of LLM output in real time:

For agentic systems specifically, ARGUS adds three more signals:

• ASF Agent Success Fraction — what percentage of agent tasks complete successfully
• ERR Error Recovery Rate — how well the agent recovers from tool failures
• CPCS Cost Per Completed Subtask — real cost accountability at the task level

When a dimension fails a threshold, ARGUS does not just log it. It triggers a correction loop.

The Architecture in One Diagram

LLM Call → ARGUS Eval Layer → Pass/Fail per Dimension
↓
Fail Detected
↓
Autonomous Correction Loop
(prompt rewrite + retry)
↓
Corrected Output → Your App

The observability layer is the open-source core (argus-ai on PyPI). The autonomous correction loop is the proprietary layer being built for enterprise deployment.

Why OpenTelemetry + ARGUS Is the Right Stack
I am not building against OpenTelemetry. The right mental model is:

• OpenTelemetry = infrastructure observability, distributed tracing, the plumbing
• ARGUS = semantic output evaluation, correction, quality guarantee

They compose. You can pipe ARGUS evaluation results as spans into your OTel collector. You get the full picture: infrastructure health AND output quality in the same trace.

What I Learned the Hard Way

At R1 RCM I led the engineering work that contributed to a $4.1B acquisition. The AI systems underpinning that work processed millions of healthcare claims. When the LLM got it wrong, it was not just a metric. It was a denied claim, a delayed payment, a patient impact.

Tracing told us what happened. Correction prevented why it would happen again.

That difference is what drove me to build ARGUS.

Get Started
pip install argus-ai

GitHub: github.com/anilatambharii/argus-ai
PyPI: pypi.org/project/argus-ai

If you are building LLM systems in production and want to collaborate, reach out. This is open-source and I want it to be the standard evaluation layer the community builds on.

25 years of production AI. All opinions are mine. All lessons were expensive.

HumanWritten #ExpertiseFromField