# NTK regime check: one CKA drift test before portrait fine-tuning

Ella Sullivan · October 20, 2026

> Run a CKA drift test before portrait fine-tuning: a score of 0.95 or higher confirms the NTK regime, so you can freeze features and retrain only the head.

| Takeaway | Detail |
| --- | --- |
| Use CKA 0.95 as the NTK-regime cutoff. | Compare penultimate-layer features before and after fine-tuning: CKA ≥ 0.95 indicates the kernel remains effectively pinned at initialization. |
| When CKA is at least 0.95, freeze features and retrain only the head. | In the NTK regime, keep the penultimate-layer representation fixed and fit a new output head rather than paying for full-network fine-tuning. |
| When CKA is below 0.95, treat training as outside the NTK regime. | A post-fine-tuning CKA score below 0.95 indicates feature drift beyond the stated initialization-pinned regime. |
| Run the kernel-drift test before launching fine-tuning runs. | The one-shot comparison of pre- and post-fine-tuning CKA determines whether to use frozen features with a retrained head or proceed with feature-changing fine-tuning. |

This guide explains why overparameterized portrait networks can behave like fixed-feature models under gradient-flow training. It provides a one-test decision rule for identifying the neural tangent kernel regime and choosing between frozen features and feature-changing fine-tuning.

![NTK regime check](https://static.mm-ais.com/article-images-ai/ntk-regime-check-one-cka-drift-test-befo-ai-29d60872.jpg)

## NTK mechanism: why wide nets freeze

The relevant mechanism is the neural tangent kernel (NTK) regime: at sufficient overparameterization, gradient-flow training of the final layer is provably equivalent to kernel regression whose kernel is fixed at initialization. In this regime, optimization updates the head weights while the learned face representation itself remains effectively fixed. This is the theoretical basis for why wide portrait generators can adapt their output layer without substantially reorganizing the penultimate-layer features.

*On the Convergence of Overparameterized Problems: Inherent Tradeoffs and the Role of Overparameterization* provides a central gradient-flow analysis. The authors interpret training of an overparameterized network with linear activations as a gradient-flow problem and show that its global convergence properties reduce to those of an associated linear model. The check for this mechanism is whether the last layer is nearly linear in its input features: if the learned head can account for the training behavior while the upstream representation remains stable, the optimization is behaving like the wide-limit linear model rather than relearning the representation.

The same fixed-feature behavior appears in the convergence analysis of regularized Newton methods for neural networks hosted by Oxford’s ORA repository. That work formalizes the overparameterized limit in which enough width makes the optimization problem behave like a convex kernel problem. A practical interpretation is that curvature corrections do not rescue substantive feature learning when the network is in this limit: they can accelerate convergence of the head, but the direction and meaning of the representation remain governed by the initialization-time kernel.

For portrait generation, the useful consequence is a representation-stability check rather than a claim that every fine-tuning run must preserve features. Before accepting a fine-tuned model, inspect whether the backbone’s learned portrait representation changed relative to initialization and whether the output-head loss decreased. If the representation is stable while the head improves, the run is consistent with the fixed-feature mechanism: the generator has mainly recalibrated how existing features are combined.

This distinction also explains why head-only training can be a sensible fallback when the upstream representation does not need to change. The convergence analyses concern the wide, overparameterized limit, so they justify a mechanism and diagnostic direction, not a blanket guarantee for every architecture, initialization, or training schedule. The defensible rule is to treat kernel-fixed behavior as an empirically testable property of the run: stable upstream features combined with a changing head indicate NTK-like behavior, whereas simultaneous representation change indicates that the run has entered feature-learning territory.

![NTK mechanism: why wide nets freeze — NTK regime check](https://static.mm-ais.com/article-images-ai/ntk-regime-check-one-cka-drift-test-befo-ai-0185d857.jpg)

## The one test: CKA drift check

Use one before/after centered kernel alignment (CKA) measurement as the go/no-go diagnostic for a portrait model. Select a fixed set of 64–256 representative input portraits, run the identical images through the model before fine-tuning and again after the candidate fine-tune, and extract the penultimate-layer activations in both runs. The samples, preprocessing, batching, and activation-extraction procedure should remain identical so that the comparison reflects representational drift rather than differences in the evaluation set.

For each portrait, flatten the penultimate activation vector, place the resulting rows into an activation matrix, and center the columns of each matrix independently. Compute linear CKA between the centered before-and-after matrices, then report the resulting similarity as a single number alongside the number of portraits used. Linear CKA is the appropriate check here because it compares the geometry of the learned feature spaces while reducing sensitivity to orthogonal rotations and other changes in feature scale.

Apply a threshold of 0.95. A CKA score of 0.95 or higher means the NTK/fixed-feature prediction held for that run: the fine-tune refit the head without materially reorganizing the backbone’s representation. In that case, keep the backbone features frozen and train only the head rather than paying for another full-backbone fine-tuning run. A score below 0.95 means the backbone genuinely moved, so the run cannot be treated as a head-only refit; inspect its loss and output quality separately before deciding whether to continue with feature-learning training.

Record the threshold as a decision rule rather than as a universal property of portrait networks. The CKA score is empirical, and its interpretation depends on using consistent activations, the same images, and a representative fine-tuning configuration. A single aggregate number is useful for triage, but it should be accompanied by the activation-extraction settings and sample count so that a later run can reproduce the check.

The convergence-rate context also favors checking early rather than assuming that ordinary training is the most efficient route. The JMLR paper *Preconditioned Gradient Descent for Overparameterized Matrix Factorization* reports that an inexpensive preconditioner can restore a linear convergence rate that vanilla gradient descent loses through a polynomial dependence on the condition number. Although that result is not itself a portrait-model benchmark, it supports a practical discipline: verify the regime of a candidate run before committing additional compute. The CKA drift check is that regime test—one score, one threshold, and a direct answer to whether the backbone remained fixed or learned new features.

## Options: frozen vs fine-tune vs precondition

Option A is the frozen-backbone approach: keep the portrait generator’s learned representation fixed and train a new head by kernel regression on those features. It is the least expensive route and is directly aligned with the convergence analysis of the regularized Newton method in the overparameterized limit reported by Oxford ORA. For compliance or crop-geometry failures, where the required output changes but identity fidelity is already adequate, this is the first option to test. The practical check is simple: train the replacement head on held-out portrait examples and confirm that it corrects the target geometry without degrading identity matching.

Option B is full fine-tuning. It becomes justified when the pilot’s CKA result falls below 0.95, because that result indicates feature movement rather than convergence within the initialization-pinned regime. If the pilot stays at or above 0.95, full fine-tuning is not supported by the observed behavior: it adds optimization and storage costs while the backbone remains effectively unchanged. For a production run, make the decision from the recorded pilot comparison rather than from the nominal learning rate, batch size, or desired output quality.

| Option | What is trained | Best use | Decision rule |
| --- | --- | --- | --- |
| Frozen backbone + new head | Head only | Crop, framing, and compliance geometry | Start here when identity fidelity is already acceptable |
| Full fine-tune | Backbone and head | Failures that require changed portrait features | Proceed only after the pilot shows CKA below 0.95 |
| Preconditioned head training | Head with a Newton-style preconditioner | Headshot compliance models | Preferred when the frozen head converges too slowly or remains poorly conditioned |

Option C is preconditioned head training. Instead of applying an undifferentiated update to every head parameter, regularized Gauss–Newton or Newton-style preconditioning can account for the local curvature of the head objective. Oxford ORA’s convergence analysis supports examining regularized Newton methods in the overparameterized setting, while the ResearchGate GGN study analyzes the two-layer network in the optimization loop of a resulting generalized Gauss–Newton method, including the relevant parameter scaling.

Use preconditioning as a targeted remedy, not as a substitute for the diagnostic: first identify whether the problem is head fitting, feature learning, or both. If the frozen features already encode identity well and only the compliance target is wrong, compare ordinary head training against a regularized Gauss–Newton head using the same data split and evaluation set. Select the winner by held-out compliance and identity metrics; the explicit winner in this comparison is preconditioned head training for headshot compliance models, especially when ordinary head optimization is slow or ill-conditioned.

## Numbers: width, steps, checkpoints

Use three observation points: initialization, 10% of the planned training steps, and the final step. At initialization, log the model width, total step budget, CKA = 1.0 baseline, and the initial head loss on your portrait validation set. The loss value is a measurement from your run, not a literature constant; keep the same validation portraits and evaluation procedure at every checkpoint so that the logged values remain comparable.

At checkpoint 2, after 10% of the planned steps, log CKA and validation loss again. If CKA is at least 0.95 at this point, the kernel-drift criterion says the run is remaining in the NTK regime. You can stop the full fine-tune and retain the learned features, using the checkpoint to retrain only the head. With the step budget used as the denominator, stopping after 10% leaves 90% of the original steps unrun and therefore avoids approximately 90% of the planned compute. Report the actual saved GPU-hours from your logs rather than converting this percentage into an estimated price or schedule.

The final checkpoint is the required completion of the measurement protocol: log both CKA and validation loss after all planned steps, whether you stopped early or ran to completion. A final CKA below 0.95 indicates feature-learning behavior, so the early-stop decision should be reversed and the additional training should not be presented as an NTK-regime result. A final CKA of at least 0.95 confirms that the full run stayed within the kernel regime. In either case, preserve the three checkpoint records so the trajectory is visible rather than collapsing the decision into a single before/after number.

The cost of being wrong runs in both directions. Continuing a full fine-tune in the NTK regime spends the entire run’s GPU-hours on head updates while the portrait features remain effectively fixed, so much of that expenditure buys little additional feature adaptation. Stopping too early in a feature-learning run can also be costly: the checkpoint may not yet contain the representation changes needed for the downstream task, and the final loss can reveal whether the early checkpoint was genuinely sufficient.

Condition numbers provide the rationale for treating optimization behavior separately from representation drift. The work on preconditioning in overparameterized low-rank matrix problems reports that vanilla gradient descent can suffer a polynomial dependence on the condition number, while preconditioning can improve the convergence rate. The JMLR analysis of preconditioned gradient descent similarly emphasizes restoring a linear convergence rate in the overparameterized setting. Accordingly, record the condition number of the fixed-feature head problem when it is available, and compare loss progress with representation drift: a large condition number can slow optimization without implying that the portrait features learned new information.

## What convergence theory does not prove

The convergence results that make the NTK argument appealing are statements about idealized regimes, not warranties for every portrait generator as deployed. The Oxford ORA analysis of Newton’s method, for example, develops convergence guarantees for the *regularized* method in the overparameterized limit. A production fine-tuning recipe may use an objective, optimizer, schedule, and regularization configuration that do not match those assumptions. The applicable check is therefore to compare the proposed training setup against the assumptions of the cited theorem, not simply to apply the theorem’s conclusion.

The Gauss-Newton evidence has a similarly bounded scope. The ResearchGate paper studies convergence for a two-layer overparameterized network under a specified parameter scaling. It does not establish that arbitrary portrait architectures, objectives, or samplers inherit the same behavior. The arXiv analysis of gradient flow for overparameterized problems with linear activations is likewise tied to its stated model class. The HAL ResNet work is valuable because it connects finite-width ResNet convergence analysis with an infinite-width perspective, but that bridge does not supply a production-width cutoff or prove that every finite portrait network lies in the regime.

| Proven boundary | What the cited convergence statements cover |
| --- | --- |
| Finite-width portrait models | Outside the proven statements unless the model and training assumptions are separately established |
| Exact width at which the NTK regime begins | Not established by the cited sources; no universal width threshold is available |
| Diffusion-sampler effects | Not covered by the cited convergence statements |

None of the grounded sources—Oxford ORA, the arXiv gradient-flow paper, the HAL ResNet analysis, or the JMLR work—provides a single “width *N* is enough” rule for portrait generators. Architecture depth, parameterization, initialization, loss, optimizer, and regularization can all change the approximation regime. The practical rule is to test each production model empirically rather than infer a safe width from a theorem for a different network family.

Preconditioning results, including the JMLR analysis of preconditioned gradient descent and the proceedings paper on preconditioned low-rank optimization, address convergence-rate questions under their own problem formulations. They should not be read as evidence that a diffusion portrait pipeline receives equivalent guarantees. Before committing to fine-tuning runs, verify the theorem-to-system match, and use the kernel-drift diagnostic as the empirical arbiter: if the evidence indicates an NTK-like regime, freeze the learned features and retrain only the head; if feature movement is substantial, treat the model as feature-learning and choose a different adaptation strategy.

## And drift checklist

In a worked run, evaluate 128 sample AI headshots through the portrait model at initialization, after 200 fine-tuning steps, and after the full 2,000 steps. The 10% checkpoint is 200 steps. If CKA is 0.97 at that checkpoint and 0.96 at the end, while the head loss falls and the backbone loss remains flat, keep the backbone frozen and retrain only the head. The falling final CKA is still above the 0.95 operating threshold, so the run remains in the designated NTK regime for this decision.

For the counterfactual run, suppose CKA at the 10% checkpoint is 0.88. That result indicates that the learned representation has moved, so budget the full fine-tune rather than treating a frozen-feature head as an equivalent substitute. In this case, retraining only the head would not close the identity-fidelity gap, because the needed representation change is not present in the frozen backbone. The practical rule is binary: CKA at or above 0.95 supports freezing the backbone and retraining the head; CKA below 0.95 supports continuing the full fine-tuning plan.

Use this five-row checklist for each portrait-model experiment, filling each bracket with the value observed in your own run:

Save the checklist beside the run configuration, including the checkpoint schedule, evaluation-set identifier, and loss definitions. This makes the decision auditable: one threshold, one early checkpoint, and one recorded action. The broader convergence motivation is consistent with the Oxford ORA analysis “Convergence analysis of Newton’s method for neural networks” and the arXiv paper “On the Convergence of Overparameterized Problems: Inherent Tradeoffs and the Role of Overparameterization,” but the checklist itself records observed measurements rather than treating them as universal guarantees.

## Decision rules for portrait pipelines

**Decision rule:** Before committing to a full portrait-model fine-tune, run a short pilot and evaluate the model at the 10% training checkpoint. If the pilot’s CKA score is at least 0.95, terminate the full run and retrain only the prediction head on frozen penultimate-layer features. If the score is below 0.95, continue with the planned full fine-tune. This two-branch rule turns the kernel-drift measurement into an operational go/no-go decision rather than an advisory diagnostic.

**If CKA ≥ 0.95 at the 10% checkpoint:** kill the full fine-tune immediately and switch to the frozen-feature pipeline. The measured stability means that continuing to update the portrait backbone is unlikely to provide a meaningful benefit for the target task, while the head can still be trained to satisfy the required output mapping. Preserve the checkpoint and backbone configuration so that the pilot and the final head-only run use the same feature extractor.

**If CKA < 0.95 at the 10% checkpoint:** proceed with the full fine-tune. The convergence analysis in *On the Convergence of Overparameterized Problems: Inherent…* provides the relevant theoretical setting for distinguishing optimization inside the wide regime from behavior outside it. In this pipeline, the pilot measurement supplies the practical evidence that feature learning is occurring; the 10% checkpoint is therefore a stopping check for deciding whether to keep investing in backbone updates.

**Compliance-specific trigger:** apply the same rule whenever the fine-tune targets identity-document portrait compliance, including ICAO or passport-style outputs. If the observed failure is geometric—incorrect crop, framing, head-to-image ratio, or document-layout alignment—rather than a failure of facial photorealism, use frozen features and retrain only the head. In that case, the head can directly learn the required geometry decision without spending a full fine-tune on backbone changes that the task does not need.

For every portrait pipeline, record the 10% checkpoint result, the resulting branch, and the identity-fidelity rationale so the decision can be audited later.

eason for any head-only decision. Treat CKA ≥ 0.95 as a stop signal for backbone training, CKA < 0.95 as authorization to continue the full fine-tune, and geometry-only identity-document failures as an additional reason to prefer the head-only path. This keeps the decision tied to one measurable checkpoint instead of requiring separate exploratory runs for every pipeline variant.

## What to do next

| Step | Action | Why it matters |
| --- | --- | --- |
| 1 | Run the one-shot kernel-drift test on the portrait model before paying for fine-tuning runs. | This test determines whether the network remains effectively pinned at initialization or whether fine-tuning changes its internal features. |
| 2 | Extract the penultimate-layer features from the same portrait inputs before and after fine-tuning. | A matched before-and-after comparison isolates changes in the representation used by the classifier. |
| 3 | Compute CKA between the pre-fine-tuning and post-fine-tuning penultimate-layer features, using 0.95 as the NTK-regime cutoff. | CKA measures representational stability and provides the decision threshold for distinguishing the NTK regime from feature drift. |
| 4 | If CKA is at least 0.95, freeze the portrait model’s features and retrain only the output head. | The kernel remains effectively pinned at initialization, so a new head can exploit the fixed representation without full-network fine-tuning. |
| 5 | If CKA is below 0.95, proceed with full-network fine-tuning. | Features moved beyond the stated initialization-pinned regime, so full fine-tuning is buying representational change rather than merely replacing the output head. |
| 6 | Record the one-shot CKA result and apply the corresponding training decision before launching additional runs. | Using the kernel-drift result prevents unnecessary full-network runs and connects the next training choice directly to measured feature stability. |

## Frequently Asked Questions

**What comparison should be made to check whether a portrait network is in the NTK regime?**

Compare penultimate-layer features before and after fine-tuning using CKA.

**What does a CKA score at or above the stated cutoff imply about the kernel?**

It indicates that the kernel remains effectively pinned at initialization.

**What training approach is recommended when the CKA score is at or above the cutoff?**

Freeze the features and retrain only the output head.

**What should be done if the post-fine-tuning CKA score is below the stated cutoff?**

Treat the training as outside the NTK regime because it indicates feature drift beyond the initialization-pinned regime.

**When should the kernel-drift test be run relative to launching fine-tuning runs?**

Run it before launching fine-tuning runs.

**Why can an overparameterized portrait network behave like a fixed-feature model during gradient-flow training?**

At sufficient overparameterization, gradient-flow training of the final layer is provably equivalent to kernel regression whose kernel is fixed at initialization.

## Quick answers

| When should the kernel-drift test be run? | Run the kernel-drift test before launching fine-tuning runs. |
| --- | --- |
| What does a post-fine-tuning CKA score below the NTK-regime cutoff indicate? | It indicates feature drift beyond the stated initialization-pinned regime. |
| What training approach is recommended in the NTK regime? | Keep the penultimate-layer representation fixed and fit a new output head rather than paying for full-network fine-tuning. |
| How does the one-shot pre- and post-fine-tuning CKA comparison determine the training approach? | It determines whether to use frozen features with a retrained head or proceed with feature-changing fine-tuning. |
| Why can overparameterized portrait networks behave like fixed-feature models under gradient-flow training? | At sufficient overparameterization, gradient-flow training of the final layer is provably equivalent to kernel regression whose kernel is fixed at initialization. |

Also worth reading: **Why linear regression remains the essential foundation for complex neural networks**: [Why linear regression remains the](https://kahma.io/blog/why-linear-regression-remains-the-essential-foundation-for-complex-neural-networks.php) · **Why the Tesla love affair is cooling across the political spectrum**: [Why the Tesla love affair](https://kahma.io/blog/why-the-tesla-love-affair-is-cooling-across-the-political-spectrum.php) · **Prompt like a human for smarter AI**: [Prompt like a human for](https://kahma.io/blog/prompt-like-a-human-for-smarter-ai.php)

### Related reading

- [ICAO 9303 Part 5: How Portrait Geometry Affects Verification](https://kahma.io/blog/icao-9303-part-5-how-portrait-geometry-affects-verification.php)
- [Remaker AI Portrait Generator: Free Online Headshots in Seconds](https://kahma.io/blog/remaker_ai_portrait_generator_free_online_headshots_in_seconds.php)
- [7 Innovative AI Portrait Techniques Transforming Photography in Late 2024](https://kahma.io/blog/7_innovative_ai_portrait_techniques_transforming_photography.php)
- [The Evolution of AI-Powered Color Alteration in Portrait Photography A 2024 Update](https://kahma.io/blog/the_evolution_of_ai_powered_color_alteration_in_portrait_pho.php)
- [Building Mental Resilience Through Portrait Photography A 7-Step Approach to Overcoming Self-Image Barriers](https://kahma.io/blog/building_mental_resilience_through_portrait_photography_a_7.php)
- [AI-Enhanced Portrait Photography How Cutting-Edge Technology is Revolutionizing the Industry](https://kahma.io/blog/ai_enhanced_portrait_photography_how_cutting_edge_technology.php)

### Latest

- [Passport photo compliance: how to benchmark face-landmark verifiers](https://kahma.io/blog/passport-photo-compliance-how-to-benchmark-face-landmark-verifiers.php)
- [AI Passport Photos Rejected: UN Aviation (ICAO) 9303 Fails 12% — Retouch, Not...](https://kahma.io/blog/ai-passport-photos-rejected-un-aviation-icao-9303-fails-12-retouch-not-retake.php)
- [TAAFI AND CASO ANNOUNCE 2018 ANIMATION/VFX JOB FAIR](https://kahma.io/blog/taafi-and-caso-announce-2018-animationvfx-job-fair.php)

Canonical: https://kahma.io/blog/ntk-regime-check-one-cka-drift-test-before-portrait-fine-tuning.php
Markdown: https://kahma.io/blog/ntk-regime-check-one-cka-drift-test-before-portrait-fine-tuning.php/index.md