Contents

Publisher’s Note (July 2026): Shortly before press, mgcvUI was consolidated into earthUI (version 0.11.0 and later), which now provides the GAM workflow described here alongside the MARS and elastic-net workflows. mgcvUI remains available in its final released form (0.3.0) at https://github.com/wcraytor/mgcvUI; future development continues in earthUI. This article applies to mgcvUI as released.

Introduction

History and Background

Generalized Additive Models

Generalized Additive Models (GAMs) were formalized by Trevor Hastie and Robert Tibshirani in their 1990 monograph. GAMs extend the linear model by replacing each linear term \(\beta_j x_j\) with a smooth function \(f_j(x_j)\), so the model becomes:

\[y = \beta_0 + f_1(x_1) + f_2(x_2) + \cdots + f_p(x_p) + \varepsilon\]

Each \(f_j\) is estimated from the data using a penalized regression spline. The smooth functions allow the model to capture nonlinear relationships without the user specifying a functional form in advance. The penalty controls the wiggliness of each smooth, preventing overfitting while allowing enough flexibility to track real patterns.

The mgcv Package

The R package mgcv (Mixed GAM Computation Vehicle) was developed by Simon Wood at the University of Bath (Wood 2017). First released in 2001, it is now the standard R implementation of GAMs and ships with every R installation. Its underlying methods are described in a series of papers covering stable multiple smoothing parameter estimation (Wood 2004), thin-plate regression splines (Wood 2003), fast stable REML estimation (Wood 2011), and general smooth model selection (Wood et al. 2016). Key features include:

• Automatic smoothness selection: The smoothing parameter for each term is estimated as part of the model fitting, using REML (Restricted Maximum Likelihood), GCV (Generalized Cross-Validation), or ML (Maximum Likelihood).

• Multiple basis types: Thin plate regression splines (tp), cubic regression splines (cr), P-splines (ps), and B-splines (bs), each with different properties.

• Tensor product smooths: te() and ti() allow smooth interactions between variables without the curse of dimensionality.

• Variable selection: The select = TRUE option adds an extra penalty that can shrink smooth terms to zero, effectively performing variable selection.

• Concurvity diagnostics: Measures the analogue of multicollinearity for smooth terms.

Comparison with MARS (earth) and Elastic Net (glmnet)

The earth application and its two Post Processing Modules (glmnet and mgcv) use different modeling engines. The following table summarizes the key differences:

MARS (Multivariate Adaptive Regression Splines) was introduced by Jerome Friedman in 1991. It builds piecewise linear models by adaptively selecting hinge functions and their knot positions from the data. The R implementation is the earth package by Stephen Milborrow.

Elastic net regression, implemented in the glmnet package by Friedman, Hastie, and Tibshirani (2010), combines Lasso (\(L_1\)) and ridge (\(L_2\)) penalties for simultaneous variable selection and coefficient shrinkage.

The earthUI–mgcvUI Pipeline

A powerful workflow combines earthUI’s automatic knot discovery with mgcvUI’s smooth estimation. When earthUI exports an .rds result file and mgcvUI imports it:

1. Earth discovers the knots — MARS finds data-driven hinge positions (change points) for each variable.

2. mgcvUI uses them as anchors — The earth knots become the basis for cubic regression splines (cr basis) in the GAM, giving the smooth terms a head start with knot positions that reflect real structure in the data.

3. mgcv refines the fit — The penalized spline estimation smooths out the piecewise linear earth fit, producing smooth partial effect curves while preserving the general shape discovered by earth.

This pipeline bridges exploratory modelling (earth) with confirmatory modelling (GAM), combining the best of both approaches.

Recommended Workflow

For real estate appraisal and similar applications requiring interpretable, defensible models:

1. Start with earthUI to discover the important variables, nonlinear relationships, and interactions in your data. Earth’s automatic basis selection and g-functions provide excellent initial insights.

2. Refine with mgcvUI if you want smoother partial effects. Import earthUI’s knots into mgcvUI for a seamless transition from piecewise linear to smooth models.

3. Use glmnetUI when you need aggressive variable selection (Lasso), when the number of predictors approaches or exceeds the number of observations, or when you want a purely linear model with regularization for defensibility.

What Is mgcvUI?

mgcvUI is a graphical user interface for the R mgcv package. It runs as a local Shiny application — there is no login, no server, and no accounts. You launch it from R, import a dataset (CSV or Excel), configure your model, and fit a GAM interactively.

As of the current release, mgcvUI is no longer a separately installed package: it ships inside the earthUI package as its GAM Post Processing Module, started with earthUI::launch_mgcv(). The application itself — and everything described in this article — is unchanged; only the installation and launch command differ. The three routines (earth, glmnet, mgcv) share one project tree, one settings database, and one copy of the supporting infrastructure.

The application provides a complete workflow: data import, variable configuration with smooth term specification, model fitting with background processing, diagnostic plots, smooth partial effect curves, and downloadable reports in Word, PDF, or HTML format.

Generalized Additive Models replace the linear relationship \(\beta x\) with a smooth function \(f(x)\) for each predictor. This means:

• No need to specify functional forms — the data determines the shape of each relationship.

• Smooth partial effects — each predictor’s contribution is a smooth curve, not a straight line, making it easy to see how a variable affects the response at different values.

• Automatic smoothness selection — mgcv determines how wiggly each curve should be via REML or GCV, balancing fit against overfitting.

Three Purpose Modes

Every project has a purpose, chosen when you create it in Section 1 of the sidebar (Project) and locked thereafter. The active project’s purpose determines which tools and interface elements appear; to work in a different mode, create or open a project with that purpose. The three modes are:

• General — GAM regression for any type of population or dataset. This is the default mode. It provides the full GAM workflow without any domain-specific additions.

• For Appraisal — GAM regression tailored for real estate appraisal. Adds features specific to single-property valuation, including subject property handling, special column designations, and Residual Constraint Approach (RCA).

• Market Area Analysis — GAM regression tailored for market area studies. Adds features for analyzing groups of properties in a defined market, with optional subject row handling.

In all three modes, the core modeling engine is identical — you are always fitting a GAM via mgcv::gam(). The purpose setting controls which additional tools and interface elements are available.

Real Estate–Specific Features

When either For Appraisal or Market Area Analysis is selected, mgcvUI activates several features designed for real estate analysis:

• Special column designations — Each predictor can be tagged with a special role such as contract_date, dom, concessions, latitude, longitude, living_area, lot_size, actual_age, effective_age, area, site_dimensions, or display_only. These designations control how the column is handled during fitting and output.

• Sale Age column — When a column is designated as contract_date and an Effective Date is provided, mgcvUI computes a sale_age column (days between sale date and effective date) and substitutes it as a predictor.

• RCA computations (appraisal only) — In appraisal mode, after fitting the model mgcvUI can compute Residual Constraint Approach (RCA) output, which produces per-comparable adjustments, net/gross adjustment summaries, and an adjusted sale price for the subject property.

Getting Started

To use mgcvUI:

1. Install earthUI in R: install.packages("earthUI") — the mgcv routine is included. (See “Installing earthUI and vProlog” in the earthUI article in this issue for the development version and optional components.)

2. Launch the application: run earthUI::launch_mgcv() in R. The app opens in your web browser on port 7880.

3. Create or open a project in Section 1 of the sidebar. A project sets the purpose (General, For Appraisal, or Market Area Analysis) and work location (country, state, county, city) and its input/output folders. Projects are stored under a shared regProj root and are shared with the sibling apps earthUI and glmnetUI. Opening a project with a different purpose is how you switch modes.

4. Import your data in Section 2 by selecting a CSV or Excel file from the active project’s input folder. Add files to that folder, then click Refresh to list them.

5. Import from earthUI (optional) — Section 3 lets you import an earthUI result .rds file to seed the GAM with earth-discovered knots. Skip this step if you do not have an earthUI result.

6. Configure variables — choose your target and predictors, set data types, assign any special column roles, and mark variables as factor or linear.

7. Set mgcv parameters — family, method, gamma, basis type, k, tensor type, interactions, and other options.

8. Fit the model — click “Fit Mgcv GAM Model” and review the results in the main panel.

9. Export — download predictions as Excel, generate and convert a Quarto report, or (in appraisal mode) compute RCA adjustments and a sales grid.

Settings are automatically persisted in a SQLite database and restored when you reload the same input file.

MLS Input Data Requirements

For real estate appraisal and market analysis workflows, your input data typically comes from a Multiple Listing Service (MLS) export. This chapter describes the expected file structure and the columns that mgcvUI can use.

File Format & Structure

mgcvUI accepts CSV and Excel (.xlsx, .xls) files. On import, column names are automatically converted to snake_case — for example, “Living SqFt” becomes living_sqft, “Contract Date” becomes contract_date, and “Sale Price” becomes sale_price. This normalization ensures consistent column references throughout the workflow. The CSV separator and decimal mark used during import are determined by the locale settings (see Chapter 3, “Locale & Regional Settings”).

Your data file should be a flat table with one row per property and one column per attribute. The first row of the file must contain column headers.

Required Columns for Appraisal Mode

While mgcvUI works with any set of columns, the full appraisal workflow benefits from having the following columns:

Spreadsheet column names can be in a foreign language — the “special” names are in English so that the R program can give them special treatment. Otherwise, the given column names show up in the regression models, graphs and, if doing appraisals, the output reports.

Not all columns are required. mgcvUI adapts — if a column is missing, the corresponding feature is simply omitted. However, for real estate pricing models certain columns are highly recommended:

1. Sale Age — the number of days between the contract sale date and the effective date. If multi-year sales history is being used, sale_age often plays a central role.

2. Living Area — also goes by names such as “Living Sqft,” “GLA” (gross living area).

3. Total Bath Count — the total number of full, quarter, half, and 3/4 bathrooms.

4. Garage Bays or Garage Area — garage spaces or square footage.

5. Lot Size — the land area of the property.

6. Longitude, Latitude, and if available Area ID for geographic price variation.

Special Column Naming Conventions

mgcvUI identifies columns by their special type designation, not by their column name. You can name your columns anything you like in the MLS export — what matters is that you assign the correct special type in the Variable Configuration table (Chapter 6).

Data Quality & Completeness

• Missing values (NA): Rows with NA values in any predictor or target column are automatically removed before fitting. mgcvUI subsets the data to only the columns used in the model, so NAs in unused columns do not cause row drops.

• Numeric columns: Sale price, living area, lot size, and similar fields must be numeric.

• Factor columns: Text, TRUE/FALSE, and R-factor columns are treated as categorical automatically — they can only enter the model as factors. A numeric column (e.g., a numeric area or style code) becomes a factor only when you tick its Factor checkbox; mgcvUI never infers categorical from a numeric’s value range, so discrete numeric predictors like bath_count stay continuous (smoothed). The Special role is independent of the factor decision.

Subject Row Placement

In Appraisal mode, row 1 must be the subject property. All remaining rows are comparable sales. The subject row is excluded from model fitting. After fitting, the model still generates predictions for the subject row.

In Market Area Analysis mode, placing the subject in row 1 is optional.

In General mode, there is no special row handling — all rows are treated equally.

General Purpose Mode

Overview

General Purpose mode is the default when you launch mgcvUI. It provides the complete GAM workflow for any dataset — not just real estate. You can use mgcvUI for scientific data, financial analysis, engineering studies, or any regression problem where smooth nonlinear relationships are expected.

In General mode, the interface omits the real estate–specific features (special columns, sale age, RCA). The sidebar is streamlined to focus on variable selection, parameter configuration, model fitting, and export.

The Sidebar Workflow

The sidebar is organized into numbered, collapsible sections that guide you from project selection through report export. Most sections appear only once an active project is open, and several are specific to the project’s purpose. The output folder is derived from the active project — there is no separate output-folder field.

1. Project — Create or open a project. A project sets the purpose and work location (country, state, county, city) and its input (<os>_in/) and output folders. Projects are stored under a shared regProj root (default ~/regProj, overridable with the REGPROJ_ROOT environment variable or the Settings dropdown) and are shared with the sibling apps earthUI and glmnetUI, so models from all three live side by side. Click Close Project to switch.

2. Import Data — Select a CSV or Excel file from the active project’s input folder. Add files to the folder via Finder/Explorer, then click Refresh to list them. For Excel files with multiple sheets, a sheet selector appears. Column names are automatically converted to snake_case.

3. Import from earthUI (optional) — Import an earthUI result .rds file to seed the GAM with earth-discovered knot positions (and to seed variable selections). This step is optional — skip it if you do not have an earthUI result to import.

4. Variable Configuration — Target variable selector, response transform (none, log, log10), predictor table with checkboxes for Include, Factor, and Linear. The Special column appears only in Appraisal and Market modes. See Chapter 6 for full details.

5. Mgcv Call Parameters — All model configuration: parameter presets, family, method, gamma, cross-validation, select, basis type, k, tensor type, interaction matrix, and advanced parameters. See Chapter 7 for the complete parameter reference.

6. Fit Mgcv GAM Model — The button that runs the model. Results appear in the tabs on the right.

7. Download Output — Exports predictions, residuals, CQA scores, and per-variable contributions as an Excel file to the project’s output folder. Available in all purpose modes once the model is fit.

8. Calculate RCA Adjustments & Download — Computes per-variable RCA adjustments for each comparable and interpolates the subject residual via CQA. Appears only in Appraisal and Market modes, after fitting.

9. Generate Sales Grid & Download — Generates the Sales Comparison Grid, recommending comparables by gross adjustment percentage. Appears only in Appraisal mode.

10. Generate Quarto Report — Writes a self-contained Quarto (.qmd) report bundle — with all plots and tables — into the project’s output folder. Available once the model is fit.

11. Convert Quarto Report — Renders the generated .qmd bundle to HTML, Word, or PDF. PDF requires a LaTeX installation; if none is detected, the PDF option is hidden. See Chapter 13.

The purpose-specific steps are omitted (and the remaining steps renumbered) in the other modes: in Market mode the Sales Grid step is absent, and in General mode both the RCA and Sales Grid steps are absent.

Import from earthUI (Optional)

Section 3 of the sidebar — Import from earthUI — lets you import an earthUI result .rds file. This enables the earth–mgcv pipeline: earth’s data-driven knot positions become anchor points for GAM smooth terms.

When an earthUI result is imported, mgcvUI:

• Displays a summary showing the target variable, R-squared, number of terms, and knot positions per variable.

• Auto-selects the “Earth Pipeline” parameter preset (cubic regression spline basis, gamma = 1.4).

• Sets the target variable to match earth’s target.

• Pre-checks Include for earth’s predictor variables.

• Pre-checks Factor for earth’s categorical variables and Linear for earth’s linear predictors.

• Pre-populates interactions detected by earth (highlighted yellow in the interaction matrix, locked from manual changes).

• Provides an Export Knots CSV button to download earth’s knots as a CSV file.

The Variable Configuration table then shows a note explaining that the variable selections (Include, Linear, Factor) and spline knots were seeded from the earth model. Unlike glmnetUI, where the glmnet fit is bound to earth’s basis, mgcvUI fits the GAM from your current selections — earth only provides a head start, and every control remains editable.

A Clear button removes the imported earth data and resets to standalone mode.

Main Panel Tabs

After data import, the main panel provides the following tabs (model-dependent tabs populate after fitting):

Settings Persistence

mgcvUI automatically saves your configuration to an SQLite database, keyed by the input filename. When you reload the same file, all settings are restored: target selection, predictor checkboxes, data types, mgcv parameters, and interaction matrix. The purpose is a property of the active project rather than a global setting, and the last-used input file is remembered per project and auto-loaded when the project is reopened.

Dark Mode

Click the moon/sun icon in the upper-right corner to toggle between Nord Light and Nord Dark themes. The theme preference is saved in localStorage and persists across sessions. All UI elements adapt to the selected theme.

Locale & Regional Settings

mgcvUI supports international number and CSV formatting conventions through a country-based locale system. The Settings dropdown in the title bar provides Country and Paper selectors for 30+ supported countries. Each preset configures:

• CSV separator — comma (,) for US/UK/Japan or semicolon (;) for most of Europe.

• Decimal mark — period (.) or comma (,).

• Thousands separator — comma (US/UK/Japan), period (Germany/Italy/Spain), space (Finland/France/Poland/Baltics), or apostrophe (Switzerland).

• Paper size — Letter (US/Canada/Mexico) or A4 (everywhere else).

Click Save to store your locale preferences globally. The Import Data section also has a per-file Locale selector for one-off imports that use different conventions. The same Settings dropdown holds the regProj root folder field, which relocates the shared project tree used by mgcvUI, earthUI, and glmnetUI.

Appraisal Mode

When the active project’s purpose is For Appraisal, mgcvUI configures itself for single-property valuation. All features described in Chapter 3 remain available; this chapter covers only the appraisal-specific additions.

Subject Row Handling

In appraisal mode, row 1 of your dataset is the subject property and all remaining rows are comparable sales. Your input file must be organized accordingly (see Chapter 2). The subject’s sale price can be left blank or set to any value — mgcvUI automatically treats it as NA during fitting.

After fitting, the model generates predictions for the subject row, which is the basis for the RCA adjustment workflow.

Effective Date & Sale Age

In appraisal and market modes, an Effective Date field appears in the Variable Configuration section (defaulting to today’s date). If you designate a column as contract_date in the Special dropdown, mgcvUI computes a sale_age column — the number of integer days between each sale’s contract date and the effective date. This column is added as a predictor.

When the Effective Date changes, sale_age is automatically recomputed.

Special Column Designations

In appraisal and market modes, a Special dropdown appears for each predictor in the Variable Configuration table. See Chapter 6 for the complete reference of special types and their effects.

RCA Adjustments Overview

The Calculate RCA Adjustments & Download button (visible in appraisal and market modes after fitting) computes market-derived adjustments for each comparable relative to the subject. The full RCA workflow is described in Chapter 11.

Market Area Analysis Mode

When the active project’s purpose is Market Area Analysis, mgcvUI provides the same real estate–specific features as appraisal mode (special columns, sale age) but is oriented toward analyzing a group of properties rather than valuing a single subject.

Differences from Appraisal Mode

• Subject row handling is flexible — in market mode, row 1 is not automatically treated as a subject property.

• No Sales Grid — the “Generate Sales Grid & Download” step is not available.

When to Use Market Mode

Market Area Analysis mode is appropriate when you are:

• Building a GAM model for a neighborhood or market area to understand nonlinear value drivers

• Analyzing how variables like square footage, age, lot size, and location affect sale prices across a group of properties

• Preparing support for a market conditions analysis or neighborhood delineation

• Working with a dataset where you want special column features (sale age) but do not need the sales grid workflow

Variable Selection

Section 4 of the sidebar — Variable Configuration — is where you choose which columns participate in the model and how they are treated.

Target Variable

The Target (response) variable dropdown at the top of Section 4 lists every numeric column in your dataset. Select one column as the response variable (e.g., sale price). The target column is automatically excluded from the predictor list.

Response Transform

Below the target selector, a Response Transform dropdown offers three options:

• None (raw) — Use the response as-is. Appropriate when the relationship between predictors and response is approximately additive on the original scale.

• Log (natural) — Apply natural log transform before fitting. When back-transformed, this makes adjustments proportional (%) rather than absolute ($). Particularly useful for sale prices, where a $10,000 adjustment means very different things at different price levels.

• Log10 — Apply base-10 log transform.

When a log transform is selected, values \(\leq 0\) in the response are automatically filtered out. All output (predictions, contributions, residuals) is automatically back-transformed to the original scale.

The Predictor Table

Below the target selector, a scrollable table lists every remaining column. Checkbox columns use rotated vertical headers for compactness. The column order is:

Factor vs. Smooth vs. Linear: By default, included numeric variables get a smooth term \(f(x)\). Checking Factor creates a categorical term (one coefficient per level). Checking Linear creates a simple linear term \(\beta x\) instead of a smooth. Variables marked as both Factor and Linear are treated as Factor.

Data Type Detection & Overrides

mgcvUI automatically detects data types on import. Numeric, integer, logical, factor, and date columns are recognized. You can override any detection by changing the Type dropdown. Changing types affects how the column is treated in the model.

Special Column Types Reference

In appraisal and market modes, the Special dropdown provides the following options:

Weighting:

• weight — Observation weight column (only one allowed; rows with weight = 0 are excluded from fitting)

Date & Time Types:

• contract_date — Triggers automatic sale_age computation from the Effective Date

• sale_age — Identifies a pre-computed sale-age column (days from the effective date), used as-is

• listing_date — Used as a fallback for computing Days on Market

• dom — Identifies the Days on Market column

Record Type:

• sale_type — Identifies the record-type column that distinguishes the subject from comparable sales

Monetary Types:

• concessions — Identifies sale concessions (seller credits, buyer incentives, etc.)

Size & Location Types:

• latitude — Values automatically rounded to 3 decimal places

• longitude — Same rounding treatment as latitude

• area — Market area or neighborhood identifier

• living_area — Enables per-square-foot residual calculations (residual_sf and cqa_sf)

• lot_size — Site size column

• site_dimensions — Grouped with lot size

Age Types:

• actual_age — Property age column

• effective_age — Effective property age

Display Types:

• display_only — Column is included in Excel exports but excluded from model fitting entirely. Use this for address fields, MLS numbers, parcel IDs, or other reference data.

Parameter Selection

Section 5 of the sidebar — Mgcv Call Parameters — provides access to all configuration options for the GAM. Each parameter has a blue help icon (?) with a tooltip explanation.

Parameter Presets

A dropdown at the top offers two presets that configure sensible defaults for common workflows:

The Earth Pipeline preset is automatically selected when earthUI knots are imported. The cubic regression spline (cr) basis is required for earth knot integration because it allows specifying exact knot positions.

Family

Choose the distribution family for your response variable:

Method

The smoothing parameter estimation method:

Gamma (Smoothing Multiplier)

The gamma parameter multiplies the effective degrees of freedom in the smoothing criterion, encouraging smoother (less wiggly) fits. Default is 1.2.

Higher gamma values guard against overfitting by penalizing complexity more heavily. The Earth Pipeline preset uses 1.4 for additional smoothing when refining earth’s knots.

Cross-Validation

When checked (default), mgcvUI runs 10-fold cross-validation after fitting to compute a CV R-squared. This provides an honest estimate of out-of-sample predictive power.

Select (Shrinkage)

When checked (default), adds an extra penalty to each smooth term that can shrink it to zero. This enables automatic variable selection — unimportant smooth terms are effectively removed from the model. Implemented via select = TRUE in gam().

Default Basis

The spline basis function type for smooth terms:

Default k (Basis Dimension)

The basis dimension \(k\) controls the maximum wiggliness of each smooth. A value of 0 (the default) means “automatic” — mgcvUI uses \(k = 10\) or the number of earth knots, whichever is appropriate.

• Small k (3–5): Very smooth curves, nearly linear.

• Medium k (8–15): Moderate flexibility (usually sufficient).

• Large k (20+): Very flexible — risk of overfitting without strong regularization.

Tensor Type

For interactions between continuous variables, two tensor product types are available:

Allowed Interactions

A collapsible Allowed Interactions section displays an upper-triangular checkbox matrix for all included smooth (non-linear, non-factor) predictors. Each checkbox enables a tensor product smooth between the corresponding variable pair.

• Allow All / Clear All — Bulk toggle buttons.

• Click a variable name — Toggle all interactions for that variable.

• Right-click a variable name — “Block from main effect” — marks the variable for interaction-only use (no univariate smooth).

When earthUI knots are imported, earth-detected interactions are pre-checked and locked (highlighted with a yellow background). If the earth model used degree = 1 (no interactions), an informational message is shown.

Factor-by-Smooth Interactions

A separate Factor-by-Smooth Interactions matrix appears when both factor and smooth variables are included. Each checkbox creates a s(x, by = factor) term — a separate smooth curve for each factor level.

Advanced Parameters

A collapsible “Advanced” section exposes additional settings:

• Optimizer — Algorithm for smoothing parameter estimation: outer/newton (default), outer/bfgs, or efs (extended Fellner-Schall).

• Scale — Scale parameter. 0 (default) = estimate from data. Set to a known value for fixed-scale models.

• Discrete covariate method — Fast discretized fitting for large datasets (discrete = TRUE).

• Threads — Number of parallel threads (1–32). Only effective when discrete = TRUE.

Fitting the Model

The Fit Button

Section 6 of the sidebar contains the Fit Mgcv GAM Model button. Clicking it runs the model with your current configuration.

What Happens During Fitting

When you click Fit, mgcvUI:

1. Validates the configuration — checks for at least one predictor and at least 10 complete rows. Warns if fewer than 50% of rows are complete (listing the columns responsible for most NAs).

2. Prepares the data — subsets to model-relevant columns only, removes rows with NAs, applies weights (if designated), excludes the subject row (in appraisal mode), applies response transforms.

3. Builds the GAM formula — constructs smooth terms s(x, bs="cr", k=10) for numeric predictors, factor terms for categoricals, linear terms for variables marked Linear, and tensor products for selected interactions. Earth knots are embedded as explicit knot positions in cr basis terms.

4. Handles edge cases — drops constant variables, converts binary predictors to factors, caps \(k\) to the number of unique values minus 1, augments earth knots with data quantiles when more knots are needed.

5. Fits the GAM — calls mgcv::gam() with the constructed formula, family, method, gamma, select, and other parameters.

6. Runs cross-validation (if enabled) — 10-fold CV computing out-of-sample R-squared.

7. Updates all result tabs — Summary, Equation, Contribution plots, Diagnostics, and all other tabs are populated.

Background Fitting & Progress

When the callr package is available, model fitting runs in a background process so the app remains responsive. A dark-themed terminal-style progress overlay shows:

• Timer displaying elapsed seconds/minutes.

• Color-coded trace log: blue for status, yellow for CV fold progress, red for errors, green for results.

• An Abort button to kill the fitting process if needed.

• A Close button that appears after completion.

If callr is not available, fitting runs synchronously (the app freezes until fitting completes).

After successful fitting, a white checkmark appears on the Fit button and a status line shows: “R-sq = X, CV R-sq = X, Dev = X%, AIC = X, n = X.”

Result Tabs

Data

Shows the imported data as an interactive DataTable with horizontal scrolling and 15 rows per page. In appraisal and market modes the tab is split into a Subject Property table (row 1) and a Comparable Sales table (the remaining rows); in General mode a single table is shown.

Each cell is kept to a single line and truncated to the column width, so wide free-text fields (such as property remarks) do not stretch a row down the page. Single-click selects a row; double-click any cell to open a pop-up showing its full, untruncated contents.

Equation

Displays the fitted GAM model in two parts:

Model equation (MathJax-rendered):

• Left-hand side shows the response with appropriate transform (log, log10) and link function.

• Right-hand side shows the intercept value, smooth terms as \(f_i(\text{variable})\), factor terms grouped by variable with level counts, and linear terms with coefficient values.

• Family and method are displayed above the equation.

Smooth Function Definitions table:

• Term formula string (e.g., s(living_sqft, bs="cr", k=8))

• Variable name

• Type (s, te, ti, or linear)

• Basis function type

• \(k\) value

• Earth knot positions (if imported)

Summary

Six metric cards displayed across the top:

• R-squared — proportion of deviance explained

• CV R-squared — cross-validation R-squared (if CV enabled)

• Dev. Explained % — percentage of deviance explained

• AIC — Akaike Information Criterion

• n — number of observations used

• Smooths — number of smooth terms

Below the cards:

• Smooth Terms table — Term, EDF (effective degrees of freedom), Ref.df, F statistic, p-value

• Parametric Terms table — Term, Estimate, Std Error, t-value, p-value

Variable Importance

A horizontal bar chart showing the relative importance of each model term:

• Smooth terms: Ranked by F-statistic (blue bars)

• Parametric terms: Ranked by \(|t|\)-value (green bars)

Below the chart, a DataTable shows Term, Type, EDF, Statistic, and p-value, sorted by statistic descending.

Contribution Plots

Every model term that contributes to the predicted value has an interactive plotly contribution plot, displayed in a bordered card. The tab shows plots for all term types:

Univariate Smooth Terms — \(s(x)\)

• Blue line with 95% confidence ribbon

• Grey scatter points showing partial residuals (for non-log-transformed models)

• Red dashed vertical lines at earth knot positions (if earth imported)

• Hover tooltip: Variable value, Contribution ($), 95% CI, Rate (slope per unit)

For log-transformed models, the y-axis is back-transformed to dollar contributions using the formula \(\bar{y} \times (e^{f(x)} - 1)\), making the curves directly interpretable in dollar terms.

For latitude/longitude variables, the slope is shown per 0.001 degrees rather than per degree, since that is a more meaningful geographic increment.

Factor-by-Smooth Terms — \(s(x, \text{by}=\text{factor})\)

One colored line per factor level, overlaid on a single plot. Useful for seeing how a predictor’s effect varies across groups (e.g., different neighborhoods).

Interaction Terms — \(ti(x_1, x_2)\) and \(te(x_1, x_2)\)

Heatmap showing the 2D contribution surface. Color scale: red (negative) \(\to\) white (zero) \(\to\) blue (positive). Hover shows both variable values and the contribution amount.

Parametric Terms — Linear and Factor

• Numeric linear terms: scatter plot of data points with the fitted line overlaid

• Factor terms: bar chart showing mean contribution per factor level

Correlation

A heatmap matrix showing Pearson correlations among all numeric predictors. Available immediately after data import — no model fitting required.

• Color gradient: blue (\(-1\)) \(\to\) white (\(0\)) \(\to\) red (\(+1\))

• Correlation values printed in each cell

• Useful for identifying concurvity (the smooth analogue of multicollinearity) before fitting

Diagnostics

Two sets of diagnostic visualizations:

Diagnostic panel (via gratia::appraise): Four plots — residuals vs fitted, Q-Q plot, histogram of residuals, and response vs fitted values.

Actual vs Predicted scatter plot: Observed vs predicted values with a 45-degree dashed reference line. For log-transformed models, values are back-transformed to the original scale.

RCA Adjustments

Three histogram plots displayed after RCA computation (appraisal/market mode only):

• Residual Adjustment % — Distribution of residual adjustments as a percentage of sale price

• Net Adjustment % — Distribution of net adjustments

• Gross Adjustment % — Distribution of gross adjustments

Each histogram shows mean, median, and standard deviation in the subtitle, with dashed reference lines for mean and median.

Anova

A combined ANOVA table merging parametric and smooth terms from the GAM summary. Shows Term, Type (parametric or smooth), and all associated statistics.

Mgcv Output

Raw text output showing:

• Fitting timing

• The GAM formula

• Model print output

• Full summary(model) output

• Family and link function

• Overall concurvity measures

Sign Check

When earthUI knots are imported, this tab compares the direction of each variable in the earth model with its direction in the GAM:

• earth_direction: increasing, decreasing, or mixed (based on hinge function signs)

• gam_direction: increasing, decreasing, or mixed (estimated from smooth derivative on a 100-point grid)

• consistent: TRUE/FALSE

• warning: Description of any mismatch

A status line shows “All directions consistent” or the number of variables with inconsistencies.

Concurvity

Concurvity is the smooth analogue of multicollinearity. This tab shows:

• Overall Concurvity table: Rows for worst/observed/estimate measures, columns for each smooth term. Color-coded: white ($<\(0.5), yellow (0.5--0.8), red (\)>$0.8).

• Pairwise Concurvity table: Worst-case pairwise matrix with the same color coding.

Values above \(\sim\)0.8 in the “worst” row indicate that a smooth can be well-approximated by the other smooths, suggesting redundancy.

Basis Check

Output from mgcv::gam.check(): basis dimension adequacy tests for each smooth term. If a smooth’s \(k\) is too small, this test will flag it with a low p-value, suggesting you should increase \(k\).

Downloading Data

After fitting, the Download Output (Excel) button (sidebar section 7) exports an Excel file with predictions and diagnostics to the active project’s mgcv output folder.

Output Columns

In appraisal mode, row 1 (subject) has residual/cqa/cqa_sf set to NA. Rows are sorted by residual_sf descending when a living_area column is designated.

A white checkmark appears on the download button after successful completion.

CQA Scores

CQA ranks each row’s residual against all others on a 0–10 scale:

• High CQA (\(\sim\)9–10): sold for much more than predicted — likely superior condition/quality/appeal

• Low CQA (\(\sim\)0–1): sold for much less than predicted — likely inferior condition or distressed sale

• CQA \(\sim\)5: near the median

RCA Calculations & Downloading

The RCA (Residual Constraint Approach) workflow is available in Appraisal and Market Area Analysis modes, after fitting the model. The subject property must be in row 1.

Opening the RCA Dialog

Click the Calculate RCA Adjustments & Download button in sidebar section 8. A modal dialog appears with:

• Score type: CQA or CQA per SF (if living_area designated). The per-SF option scales the subject’s residual by living area.

• Subject CQA Score: Accepts values from 0.00 to 10.00. The field starts blank; when an earthUI model has been imported, the score type and value you used in earthUI are carried forward from the shared project settings.

Click Generate to compute and download. The Excel file is written to the active project’s mgcv output folder (<os>_out_mgcv).

CQA Score Interpolation

1. Comparables’ CQA scores and residuals are sorted

2. Linear interpolation (approx()) maps your CQA value to a residual

3. Subject value = model prediction + interpolated residual

4. Weight-0 rows are treated as if they had the same residual as the subject

Output Columns

A white checkmark appears on the button after successful computation.

Sales Comparison Grid

The Sales Comparison Grid is available in Appraisal mode only, after computing RCA adjustments. It generates a formatted Excel workbook suitable for inclusion in appraisal reports.

Comp Selection Dialog

Clicking the Generate Sales Grid & Download button opens a modal dialog listing all comparable sales. Comps are split into two groups:

• Recommended comps (pre-checked): Gross adjustment \(<\) 25% of sale price, sorted by sale age (most recent first). Maximum 30 comps.

• Other comps (unchecked by default): Remaining comps, sorted by gross adjustment percentage.

Select your comps, then click Generate Sales Grid to create the workbook.

Required special types: contract_date and living_area must be designated. Recommended special types: latitude, longitude, and lot_size — the grid will work without them but some fields will be blank.

Workbook Layout

The generated workbook contains multiple sheets, each holding 3 comps side by side:

• Header rows: Subject and comp addresses, sale prices, sale dates, and Days on Market

• Grouped rows: Location (latitude, longitude, area), Site Size (lot size, site dimensions), and Age (actual age, effective age) with group headers and sub-items

• Model variable rows: One row per smooth term showing the subject contribution, comp contributions, and adjustment amounts

• Residual feature rows: Empty, unlocked cells for appraiser input on features not captured by the model (e.g., condition, quality, view)

• Summary rows: Net Adjustment %, Gross Adjustment %, and Adjusted Sale Price with live Excel formulas

Sheet Protection & Residual Cells

Each sheet is protected to prevent accidental edits, but the residual feature value cells are explicitly unlocked. This allows appraisers to enter values for features not in the model while preserving the formula-driven adjustment calculations.

Downloading Reports

Report generation is a two-step Quarto workflow, replacing the single-step report of earlier releases.

Step 1: Generate Quarto Report

The Generate Quarto Report button (available once the model is fit) writes a self-contained .qmd bundle to the active project’s mgcv output folder. The bundle holds the Quarto source, pre-rendered plot files (PNG and PDF), a report_data.rds payload with all tables and model metadata, and a reference.docx style template for Word output. Because the bundle is self-contained, you can edit the .qmd source or combine it with other Quarto documents before rendering.

Plots are pre-rendered when the bundle is generated; if an individual plot fails, a placeholder image is written and the rest of the bundle generates normally.

Step 2: Convert Quarto Report

The Convert Quarto Report section renders a .qmd file to one or more output formats via quarto::quarto_render(). It defaults to the most recently generated bundle, but the Browse button lets you select any .qmd file. Three formats are available:

• HTML — Self-contained HTML document. Works on all platforms with no extra software. Recommended default.

• Word (.docx) — Rendered with the bundled reference.docx style template. Suitable for editing and distribution.

• PDF — Requires a LaTeX installation (see System Requirements below). If no LaTeX is detected, the PDF option is automatically hidden from the format checkboxes.

The legacy single-step renderers, render_gam_report() and export_gam_docx(), remain available for programmatic use.

Report Contents

The generated report includes:

• Dataset & Model: Data file, sample size, family, method, and fitting information

• Results Summary: R\(^2\), CV R\(^2\), Deviance Explained, AIC, and related fit metrics

• Model Equation: Model equation with \(f_i\) notation for smooth terms and explicit coefficients for linear terms, plus the smooth function definitions. Long equations wrap across multiple lines in PDF.

• Summary tables: Smooth terms (EDF, F, p-value) and parametric terms (Estimate, Std Error, t, p-value)

• Enabled interactions: A matrix of the tensor and factor-by-smooth interaction terms included in the model (shown only when interactions are present)

• Variable Importance: Bar chart ranked by F-statistic (smooths) or \(|t|\) (parametric)

• Smooth Effects (all term types):

  • Univariate smooth plots (static ggplot2 versions of the interactive plots)

  • Factor-by-smooth multi-line plots

  • Interaction heatmaps (\(ti\) and \(te\) terms)

  • Parametric term plots (scatter for numeric, bar chart for factor)

• Predictor Correlations heatmap

• Diagnostics: Residual diagnostics and actual vs predicted scatter

• mgcv Summary: The full text output of the fitted model

• Notation appendix: An unnumbered closing section explaining the value-contribution function notation used in the report

A white checkmark appears on the Generate Quarto Report button after successful generation.

Exporting Model Functions

mgcvUI can export the fitted GAM’s smooth functions as standalone code in multiple languages. Each smooth is evaluated on a 200-point grid and exported as a lookup table with linear interpolation, allowing the model to be used outside of R without any dependency on mgcv.

Function export is programmatic in the current release. The exported helpers take a fitted result object (produced by fit_gam()) and write the files described below:

• generate_r_code(), generate_python_code(), and generate_cpp_code() — per-language source files

• export_functions_sqlite() — SQLite lookup-table database

• export_functions_zip() — bundles any combination of the four formats into a zip archive

R Functions

File: gam_functions.R

• Header comments with response variable, R\(^2\), CV R\(^2\), and the formula.

• An INTERCEPT constant.

• One approxfun() per smooth variable with pre-computed x/y vectors.

• Linear term coefficients as named constants.

• Usage: sum the intercept, each g_var(value), and each coef_var * value.

Python Functions

File: gam_functions.py

• numpy import.

• INTERCEPT constant.

• Per-variable: _x_var and _fx_var arrays, g_var(x) function using np.interp().

• Linear term coefficients as COEF_* constants.

C++ Header

File: gam_functions.hpp

• Header-only with #pragma once.

• gam_functions namespace.

• interp() helper using std::lower_bound.

• Per-variable: inline function with static const arrays.

• Linear term coefficients as constexpr double.

SQLite Database

File: gam_functions.sqlite

• Table smooth_grids: columns (variable, x, fx) — the 200-point lookup tables.

• Table model_info: response, r_squared, cv_rsq, dev_explained, aic, n_obs, family, method, formula.

• Table linear_terms: variable, coefficient.

• Table intercept: intercept value.

Comparison with earthUI and glmnetUI

The earth application and its Post Processing Modules are one toolset for regression modeling. They share the same data format, special column types, RCA workflow, and demo datasets, but use different modeling engines.

Key Differences

One earthUI-only differentiator worth noting: earthUI includes an experimental Prolog/DCG feature that extracts structured property attributes from free-text MLS remarks.

Shared Features

All three tools provide:

• A shared regProj project tree (default ~/regProj) — projects, input files, and per-method outputs are shared across the sibling apps

• CSV and Excel import with snake_case column conversion

• 30+ country locale support with CSV separator, decimal mark, and paper size

• Special column designations (contract_date, living_area, area, latitude, longitude, etc.)

• Automatic sale_age computation from effective date

• Dark/light mode toggle (Nord theme)

• Settings persistence (SQLite and/or localStorage)

• Correlation heatmap, variable importance, diagnostics

• CQA scoring and RCA adjustment workflow

• Sales Comparison Grid with comp selection and formatted Excel output

• Compatible with the same demo datasets (Appraisal_1.csv)

Recommended Workflow

• Start with earthUI to discover the important variables, nonlinear relationships, and interactions. Earth’s automatic model building provides the best starting point.

• Refine with mgcvUI if you want smoother partial effects. Import earthUI’s knots for a seamless transition.

• Use glmnetUI when you need a purely linear model with aggressive variable selection or coefficient sign constraints.

• Compare all three on the same data to assess whether the additional complexity of nonlinear models is justified.

Demo Dataset: Appraisal_1.csv

Description

The demo dataset is shared with earthUI and glmnetUI. It is located in the demo_mls/ folder of the mgcvUI source, or if earthUI is installed:

demo_file <- system.file("extdata", "Appraisal_1.csv", package = "earthUI")

The file contains roughly 1,500 residential sale records from a simulated MLS export, with the subject property in row 1 and the comparable sales in the remaining rows. The data represents single-family home sales in a multi-area market with a range of property sizes, ages, and locations.

This is not real data, but is based on a realistic neighborhood in Northern California. All identification information has been altered or removed.

Columns

Suggested Quick Start

1. Launch the mgcv routine: earthUI::launch_mgcv()

2. Create a new project (Section 1) with purpose For Appraisal

3. Place Appraisal_1.csv in the project’s input folder and select it in Section 2 (click Refresh if needed)

4. Select sale_price as the target

5. Optionally set Response Transform to Log (natural) for proportional adjustments

6. Assign special types as shown in the table above

7. Include predictors: sale_age, living_sqft, baths_total, lot_size, area_id (as factor), age, latitude, longitude, garage_spaces

8. Keep defaults: REML, gamma = 1.2, tp basis, k = auto, select = TRUE

9. Click Fit Mgcv GAM Model

10. Review Summary, Contribution (smooth plots), and Diagnostics tabs

11. Download output (Step 7), review the CQA ranking

12. Compute RCA adjustments (Step 8) with a CQA score of ~5.00

13. Generate a Sales Comparison Grid (Step 9) with recommended comps

System Requirements & Troubleshooting

Supported Platforms

mgcvUI runs on macOS, Windows, and Linux. The application is developed and primarily tested on macOS with RStudio. Platform-specific notes are provided below.

R Version

R \(\geq\) 4.1.0 is required. RStudio Desktop (2023.06 or later) is strongly recommended — it bundles Quarto and pandoc (needed for report rendering) and provides a convenient environment for launching the app.

Required R Packages

The following packages are installed automatically as dependencies:

Optional Dependencies

Platform-Specific Notes

macOS

Fewest issues. Homebrew is recommended for system libraries. If PDF reports are needed:

# In R:
install.packages("tinytex")
tinytex::install_tinytex()

Windows

Works well with RStudio Desktop. Key notes:

• Corporate/locked-down machines: If the temp directory or user data directory is restricted, settings persistence and report generation may fail. The app will display a warning in the R console but continue operating without persistence.

• LaTeX: Install TinyTeX from R (same command as macOS above) or install MiKTeX from miktex.org.

• Long file paths: Windows has a 260-character path limit. Keep your working directory path short.

Linux

Most variable across distributions. Ubuntu/Debian users may need system libraries before R packages will compile:

sudo apt install libcurl4-openssl-dev libssl-dev \
  libxml2-dev libsqlite3-dev libfontconfig1-dev

For PDF reports: tinytex::install_tinytex() or install texlive-full from the system package manager.

For headless servers (no display), plotly interactive plots render as static fallbacks in reports. The Shiny app itself requires a web browser connection.

Graceful Degradation

mgcvUI is designed to work even when optional components are missing:

Troubleshooting

“PDF option is not available”

No LaTeX installation detected. Run in R:

install.packages("tinytex")
tinytex::install_tinytex()

Then restart the app. The PDF option will appear.

“Settings will not persist”

The SQLite database directory cannot be created or written to. Check permissions on:

• macOS/Linux: ~/Library/Application Support/R/mgcvUI/ or ~/.local/share/R/mgcvUI/

• Windows: %APPDATA%/R/data/mgcvUI/

“Diagnostics plot unavailable”

The gratia::appraise() function occasionally fails on complex models (many tensor interactions, non-standard families). This is a known upstream issue. Workaround: run mgcv::gam.check(model) in the R console to see the diagnostic plots directly.

“Report error: number of items to replace…”

This was a known issue in earlier versions related to the gratia package. Ensure you have the latest version of mgcvUI installed. The current version catches this error and produces the report with a placeholder for the diagnostics section.

Port 7880 already in use

If the app fails to start because port 7880 is occupied (from a previous session), run:

# macOS/Linux:
lsof -ti:7880 | xargs kill

# Windows (PowerShell):
Stop-Process -Id (Get-NetTCPConnection -LocalPort 7880).OwningProcess

Package installation fails on Linux

Missing system libraries. Install the development headers listed in the Linux section above, then retry install.packages().

References