Refactoring Python ML Code With Pydantic, Modules, and Dependency Injection
From Notebook Experiments to Clean CodeComing from a background in structured languages (like Go) 2026-11-17 16:19:5 Author: hackernoon.com(查看原文) 阅读量:6 收藏

From Notebook Experiments to Clean Code

Coming from a background in structured languages (like Go), I was shocked by the state of many AI code examples. Often, tutorials and prototypes live in Jupyter Notebooks — full of global variables, one-off code cells, and hidden state. This notebook style might be fine for quick experiments, but it falls apart in production. Code written in a linear, ad-hoc way is hard to debug, tough to reuse, and nearly impossible to maintain over time. In fact, seasoned practitioners warn that data scientists’ first draft notebooks are typically “messy and ad hoc”, and should never be handed off to engineers without cleanup .

Whether you’re a junior ML engineer, a data scientist transitioning to software engineering, or an experienced developer new to AI, the goal is the same: write cleaner, more maintainable AI code. In this guide, we’ll explore how to refactor a “messy” notebook into a production-grade Python project. We’ll cover moving from .ipynb scripts to proper .py modules, adding strict data validation with Pydantic, and using dependency injection to manage heavy resources like ML models. By applying these techniques, you’ll bring clean code principles into your ML projects — bridging the gap between exploratory analysis and reliable production systems.

Viktor Tr's image-68b1d8

The Problem with Notebook-Style Code

Jupyter notebooks are fantastic for exploration, but they encourage some bad habits that don’t translate well to production. Here are a few common issues when code is left in notebook form:

  • Global state and side effects: Notebooks often use global variables to pass data between cells. This makes it hard to track where a variable was set or mutated, leading to hidden couplings in your code. As a result, the code is fragile and can break in unexpected ways. In production, relying on implicit global state is a recipe for bugs and confusion.
  • Lack of structure: In a notebook, you might load data, preprocess, train a model, and make plots all in one file. There are no clear boundaries between steps. Such entangled code is difficult to test or reuse. If one piece fails, you can’t easily isolate it. For example, imagine a single giant notebook that does data cleaning, feature engineering, model training, and evaluation all in sequence — it’s basically impossible to import just the training logic without running everything else.
  • Poor reusability and modularity: Functions are often optional in notebook-land; one might just copy-paste code between cells. This violates the DRY (Don’t Repeat Yourself) principle. When you later need the same preprocessing in a new project, there’s no clean function to call — you end up copying code again, risking inconsistencies. It’s no surprise that experts emphasize modularizing such code into “bite-sized chunks” (functions) to facilitate debugging, testing, and maintenance.
  • Hard to debug or extend: A notebook’s execution order can be non-linear (you can run cells out of order). This can mask bugs — code might work accidentally because you ran a certain cell first. In a production script, everything runs top-to-bottom, so these hidden dependencies break. Additionally, notebooks don’t enforce type checks or clear interfaces between components, making it harder for others (and your future self) to understand or extend the code.

In short, notebook code is like a rough draft of an essay — good for getting ideas down, bad for publishing. Just as an author must edit and restructure a draft before publishing, a data scientist or ML engineer should refactor and structure their code before deploying it. The good news: by adopting some software engineering best practices, you can dramatically improve your AI code’s quality without losing the agility of notebooks during development.

Viktor Tr's image-7ab738

Step 1: Refactor Your Notebook into Modules and Functions

The first step toward clean, production-grade code is moving from notebooks to Python modules. Instead of one giant .ipynb file that does everything, you’ll create a proper project structure with .py files. This encourages you to encapsulate logic into functions and classes, making your code easier to read and reuse.

How to get started? Begin by identifying pieces of code in your notebook that represent independent tasks — for example, data loading, data preprocessing, model training, and inference. For each of these, do the following:

  1. Wrap logic into functions: Take the code that performs one logical task and put it inside a function with a clear name. For instance, if you have a sequence of pandas operations preparing a DataFrame, turn it into a function def load_and_preprocess_data(path): …. This simple step of adding functions immediately imposes structure. One guide concisely advises: “turn stable code into functions and move them to a dedicated module” to make notebooks more readable and easier to productionize. Instead of a dozen copy-pasted lines, you get a single function call that clearly states its purpose.
  2. Create dedicated modules: Place related functions into separate Python files (modules). You might have a data_utils.py for data cleaning functions, a model.py for model creation and inference functions, etc. This separation follows the single responsibility principle — each module handles a distinct aspect of the workflow. Later, your main script or notebook can import these modules. This approach was highlighted as a best practice in Jupyter: outsourcing code to modules not only improves readability, but is “incredibly helpful when productionizing your workflow”.
  3. Use a main script or entry point: In a production setting, you won’t run cells interactively. Instead, you might have an entry-point script (e.g. train.py or app.py) that orchestrates the process by calling those functions in order. For example, train.py could load config, call load_and_preprocess_data, then train the model and save it. Using the if name == “main”: guard in Python is useful here — you can ensure certain code (like loading a large dataset or model) runs only when the script is executed directly, not on import.
  4. Apply coding standards: As you refactor, adhere to general clean coding practices — name your functions and variables clearly, avoid overly long functions, and add documentation (docstrings) to explain non-obvious code. Remember, clean code principles and PEP8 style aren’t exclusive to web developers; your ML code benefits from them too. In one summary: “Use meaningful variable and function names, comment sensibly, modularize your code, and don’t be too lazy to refactor.” This will make your code more accessible to teammates and to yourself in a few months.

By the end of this step, your single notebook should evolve into a small Python package or set of scripts. For example, you might end up with a structure like:

my_project/
├── data_utils.py    # data loading & preprocessing functions
├── model_utils.py   # model definition, training, inference functions
├── main.py          # orchestrates the workflow (could be training or inference pipeline)
└── config.yaml      # (optional) configuration for the run

This modularization pays off immediately. If you need to update how data is loaded, you know to go to data_utils.py. If you want to reuse the model inference in a web service, you can import model_utils.py there. The code becomes testable — you can write unit tests for individual functions. And for collaboration, handing a fellow developer a well-structured codebase is far kinder than giving them an unwieldy notebook. As one ML engineering guide put it, cleaning up code and breaking it into modules makes it far more understandable and portable for others who need to work with it .

Step 2: Embrace Pydantic for Data Validation and Typing

Production-grade AI code not only needs structure; it needs reliability. A common source of bugs in ML projects is invalid or unexpected data — maybe a field is missing, a string is in place of a number, or a value is out of range. In notebooks, we often assume our data is fine (or we hack around issues on the fly). In production, we should never assume: we should validate. This is where Pydantic comes in.

What is Pydantic? Pydantic is a Python library that uses the power of type hints (Python’s typing annotations) to validate data at runtime. You define Python classes that describe the schema of your data (by extending BaseModel), and Pydantic automatically checks that any input data matches that schema — throwing clear errors if not. It’s widely used in FastAPI for validating request/response data, but it’s just as useful in standalone ML code for ensuring data integrity.

Here’s how Pydantic can elevate your AI code quality:

  • Explicit data schemas: Instead of passing around raw dictionaries or Pandas DataFrames without checks, you define Pydantic models for the inputs (or outputs) of your functions. For example, if your model expects a person with an age and salary, you can create a class Person(BaseModel) with those fields and types. By doing so, you make an executable contract — if someone tries to call your code with a wrong type (say, age as a string “forty” instead of an int), Pydantic will raise an error before it breaks your model logic. This is safer and easier to debug.
  • Automatic type conversion and validation: Pydantic will try to coerce types when possible (e.g. a JSON number into a Python int) and will enforce constraints. You can add rules like “age must be >= 0” or “name must match a regex” using Pydantic’s Field definitions. For instance, one can declare age: int = Field(…, ge=0, lt=120) to ensure no negative ages and no unrealistic ages. These kinds of checks catch mistakes early — “Pydantic models can validate types and ranges, so you don’t try to train with train_size=1.5 by mistake.” Instead of your code failing deep inside a training loop or (worse) producing a bad model silently, you get an immediate, clear exception pinpointing the issue.
  • Clarity and self-documentation: A Pydantic model serves as documentation for your data. Reading the class definition tells you exactly what fields are expected and of what type. This is great for maintenance and for new team members. It brings some of the strictness of statically-typed languages to Python in a very developer-friendly way. As a bonus, the error messages from Pydantic are quite descriptive (e.g., if a field is the wrong type or missing, it will list it out), which saves you from writing boilerplate checks.
  • Reusable validation logic: If both your training code and your inference code need to enforce the same data schema, you can use the same Pydantic model in both places. For example, you might validate your training dataset with a Pydantic model (to ensure no column is out-of-bounds or wrongly formatted) and use that exact model class in a FastAPI endpoint to validate incoming inference requests. This means consistency across your pipeline — you define “what is a valid input” once.

Quick Example: Suppose we expect an input dictionary for a prediction with keys “user_id” (int) and “features” (list of floats). Using Pydantic:

from pydantic import BaseModel, conlist

class PredictionInput(BaseModel):
    user_id: int
    features: conlist(float, min_items=3, max_items=3)

This defines that features must be a list of 3 floats (maybe representing a fixed-size feature vector). If someone passes 2 or 4 features, or a non-float, Pydantic will raise a validation error. In our code, we can do:

data = PredictionInput(**raw_data)  # raw_data could be a dict from JSON or elsewhere
# If this succeeds, we know data.user_id is an int and data.features is a list[float] of length 3.
model.predict(data.features)  # safe to use

In just a few lines, we gained a lot of confidence about the input. You can imagine how this scales: define schemas for your model’s input, output, config files, etc., and you drastically cut down on the “unknowns” your code has to handle. As one ML ops author notes, using Pydantic for configs brings safety (catching errors like a string where a float should be) and team-friendliness (others can easily see and tweak what’s expected) .

In summary, Pydantic helps you write AI code that fails fast and clearly when given bad data, rather than producing puzzling errors down the line. It’s a way of bringing strong typing and validation to a dynamic language like Python, resulting in more robust pipelines. If you’ve ever spent hours debugging why your model crashed only to find an unexpected null or string in your input, you’ll appreciate how Pydantic could have caught the issue in milliseconds.

Viktor Tr's image-658aa8

Step 3: Use Dependency Injection to Handle Models and Resources

One of the biggest differences between a throwaway prototype and a production system is how they manage expensive resources. In AI applications, a prime example of an expensive resource is your trained model object. Often, the model can be large (hundreds of MBs or a 10GB model in extreme cases) and loading it from disk or initializing it can be slow. Yet, in many beginner code examples, you’ll see something like this in a function:

def predict_one(sample):
    model = load_model("model.pkl")   # loads model every time
    return model.predict(sample)

If load_model takes 5 seconds (or 5 minutes!), calling predict_one repeatedly will reload the model each time — an obviously terrible idea for production. Unfortunately, this pattern arises when data scientists wrap notebook code into functions without restructuring how the model is accessed. The notebook might have loaded the model once at top, but if you turn a cell into a function naively, you get re-loading on each call.

The solution is to apply a bit of dependency injection (DI). In plain terms, dependency injection means giving a function or class the things it needs, instead of having it acquire them itself . We want to load the model once, and then inject it into any function or component that uses it, rather than loading it inside those functions every time. This approach has two big benefits: it improves performance (load once, reuse in memory) and improves testability (you can inject a dummy or smaller model for testing).

Here are some ways to implement this in your ML code:

  • Module or application scope loading: The simplest method, if you’re writing a script or building a web service, is to load the model at startup and keep it in a variable that everyone can access. For example, you might have at the top of your script model = load_model(“model.pkl”) once, and then your predict(sample) function simply uses the already-loaded model. This is effectively using a global for the model (which we generally minimize, but for a single immutable model object it’s acceptable). Many web frameworks facilitate this pattern. For instance, in FastAPI you can use the startup event or a dependency that yields a loaded model. One approach shown in practice is using a startup context to load the ML model and store it, then each request function uses that stored model. The key is: don’t load on every request — load once and reuse.
  • Function parameters/arguments: A more explicit form of injection is to pass the model as an argument to functions that need it. For example, instead of a method that internally does model.predict, write it as def predict(sample, model): return model.predict(sample). Then manage the model at a higher level (maybe load it in main and pass it in). This makes the dependency obvious and allows swapping the model easily. It also avoids hidden global state. In fact, avoiding globals by passing data as parameters is a fundamental clean coding practice.
  • Classes or singletons: You can encapsulate the model in a class. For example, define a class MyModelPredictor where the init loads the model (once), and a method predict(sample) uses self.model. Then you instantiate predictor = MyModelPredictor() once, and call predictor.predict(x) as needed. This is essentially the object-oriented way to inject the model dependency — the class carries the model around for you. It’s similar to the approach suggested in some Q&A discussions for not reloading models . The first call creates the heavy model, subsequent calls reuse it via self.model.
  • Dependency injection frameworks: In larger applications, you might use a formal DI framework or container (common in languages like Java, but libraries exist in Python too). These allow you to declare that, say, Model is a singleton resource, and any component that needs it will be provided that one instance. This is perhaps overkill for most small ML projects, but it’s good to know the concept scales. Even without a framework, you can achieve the same by designing your code to accept dependencies (via parameters or initializers) rather than hard-coding them.

The overarching principle is to avoid redundant work and shared mutable state. By loading the model once, you treat it as a dependency that you provide to your functions or endpoints. This not only speeds up your code, it also makes it more reliable. You won’t accidentally run out of memory by loading the model 100 times, and you won’t be surprised by differences between model instances (because there’s just one in use).

Another advantage of dependency injection is in testing. Suppose you wrote a function classify_image(img, model). In production, you pass in the real heavy model. In a unit test, you could pass a lightweight fake model (with the same interface) to quickly test that classify_image calls the model correctly. This is much harder if classify_image always reads the model from disk internally — in tests, you’d have to actually have a model file around. DI enables loose coupling of components, meaning you can swap out pieces easily . This is a hallmark of good software engineering and is very applicable to ML code as well.

In practice, implementing DI for your model might be as simple as:

# At startup:
MODEL = load_model("big_model.bin")  # global constant or module-level variable

# In your prediction function/file:
def predict(inputs):
    # use the already loaded MODEL
    return MODEL.predict(inputs)

Or using FastAPI as an example:

app = FastAPI()

@app.on_event("startup")
def load_model():
    app.state.model = load_model("big_model.bin")

@app.post("/predict")
def predict(request: InputSchema):
    # model is loaded once at startup and stored in app.state
    return app.state.model.predict(request.features)

However you do it, the idea is to initialize expensive resources once, and inject them wherever needed. This prevents the nightmare scenario of re-loading a 10GB model for every single inference call. It also connects back to our earlier theme: structuring code. Rather than calling heavy operations in the middle of business logic, you structure your code to separate initialization from execution.

Viktor Tr's image-da0fd8

Conclusion: Clean Code, Better AI Systems

Writing AI/ML code “like a software engineer” might sound tedious at first, especially if you come from a purely data science background. But adopting these clean code practices will pay dividends as your projects grow. By refactoring notebooks into modular code, you make your work reproducible and maintainable. By adding data validation with Pydantic, you catch errors early and build confidence in your pipeline’s outputs. By handling models and other resources with dependency injection, you avoid inefficiencies and create flexible, testable code.

In my own journey, this approach felt like connecting two worlds: my past experience writing structured, type-safe Go code and my present work in the Python AI realm. I realized that the discipline and quality mindset from traditional software development can (and should) be applied to machine learning projects. The result is code that not only accomplishes the task at hand but is also easier to debug, extend, and collaborate on.

No matter if you’re a junior ML engineer just starting out, a data scientist learning to productionize models, or an experienced developer dipping into AI — you can write clean, production-grade code. Treat your code as an asset, not just a byproduct of research. Structure it well, validate everything, manage your dependencies smartly. Your future self (and your teammates and users) will thank you when the “AI code” runs as smoothly and reliably as any other well-engineered software component. Quality code is the foundation of deploying successful AI solutions at scale, so don’t hesitate to bring some engineering rigor into your data science work. In the end, clean code isn’t just about pleasing your inner perfectionist — it’s about building AI systems that are robust and deliver value in the real world.

Happy coding, and happy refactoring!


文章来源: https://hackernoon.com/refactoring-python-ml-code-with-pydantic-modules-and-dependency-injection?source=rss
如有侵权请联系:admin#unsafe.sh