Chapter 3 - Testing out in the wild

Deep Learning
Why should we test our model with random real-world data
Author

Satyabrata pal

Why to test machine learning model?

Why do we need to test our machine learning model? Didn’t we test it during the training time using the validation set? Well! we did. Kind of, but that validation set was created by us from the data that we had created. Even when you do test your model on a test set, it might still have the same biases as the validation set and training set. To make sure that your model behaves correctly, we will have to test the trained model on a variety of data taken from the wild.

Re-creating the model

I will have to re-create the datablock and the model code here as well. Let’s do that. We will use the functions from my library fastaibreadcrumbs and some fucntions that we used in chapter 2.

Import the required libraries

Code
try: import gradio as gr
except ModuleNotFoundError:
    !pip install -Uq gradio
Code
# install fastkaggle if not available
try: from fastaibreadcrumbs.core import *
except ModuleNotFoundError:
    !pip install -Uq fastaibreadcrumbs
Code
import gc
from fastai.vision.all import *
from fastai.vision.gan import *
from fastkaggle import *
from fastaibreadcrumbs.core import *
Code
config = {'path_orig':Path('test_860'), 'path_crappy': Path('crappy')}

Prepare the test data

I will use a picture taken from my mobile phone for this. The reason is that the target users may use a deblurring app to deblurify the pictures they take on their mobile phones. One reason for this assumption is that people would want to deblurify images which are blurred, right and most of the times we take images with our mobile deives.

Code
config['test_path'] = Path('test')

for your test you can click any picture from your phone and put in the above “test” path or replace the “test” path in the above code with your path.

Code
test_img = get_image_files(config['test_path'])
test_img
(#4) [Path('test/IMG20230128074559.jpg'),Path('test/IMG20230128074749.jpg'),Path('test/IMG20230128074758.jpg'),Path('test/IMG20230128074733.jpg')]

ReCreating the learner and dataloader

Just like chapter2, we will recreate our datalaoders and learner.

Code
def get_dls(sz:int,bs:int, src):
    item_tfms = Resize(sz)
    batch_tfms = [Normalize.from_stats(*imagenet_stats)]
    get_y = lambda x: x.relative_to(config['path_crappy'])
    files_crappy = get_image_files(src)
    
    dls= get_unet_dls(bs, source = files_crappy, get_y = get_y, 
                     splitter = RandomSplitter(), item_tfms = item_tfms,
                     batch_tfms = batch_tfms)
    
    return dls

def get_inf_model(dls, model:str):
    unet_learn = unet_learner(dls, models.resnet34, loss_func=F.l1_loss,
                     blur=True, norm_type=NormType.Weight).load(model)
    
    return unet_learn

def save_pred(path:str, dest:str,learner):
    path = Path(path)
    dest = Path(dest)
    preds = learner.predict(path)
    arr = preds[0].numpy().transpose(1,2,0).astype(np.uint8)
    dest.mkdir(parents=True, exist_ok=True)
    Image.fromarray(arr).save(dest/path.name)
    return dest/path.name
Code
dls = get_dls(480, 8, config['path_crappy'])
Code
learner = get_inf_model(dls, './model_256')

Like last time let’s also create the fucntion to save our predictions.

Code
def save_pred(path:str, dest:str,learner):
    path = Path(path)
    dest = Path(dest)
    preds = learner.predict(path)
    arr = preds[0].numpy().transpose(1,2,0).astype(np.uint8)
    dest.mkdir(parents=True, exist_ok=True)
    Image.fromarray(arr).save(dest/path.name)
    return dest/path.name
Code
def display_result(path):
    dest = save_pred(path,"gen_imgs",learner)
    return dest

Creating the gradio UI

We will create a gradio Ui to visually test our results. This is the same UI as chapter 2.

Code
with gr.Blocks() as demo:
    with gr.Row():
        image_input = gr.Image(type="filepath")
        image_output = gr.Image()
    deblur_btn = gr.Button("Deblurrify")
    deblur_btn.click(fn=display_result,
                     inputs=image_input, outputs=image_output)

demo.launch()

Observations

  • The blur is not removed.
  • The noise in the image is also amplified.

Reasons for the bad performance

This is a typical example of model performing well on training data but not performing good on real world data.

  • I trained the model on data where the motion blur was simulated only in a single direction but in real world camera shake/motion blur in an image can be in any direction (horizontal, vertical etc.). So while training the model, this thing should be kept in mind.

  • Secondly, my training data only contained high quality images and thus there was no sample which would tell the model what todo in case noise is available in the image (this is something that I can look into in later iterations.).

  • The model that I am using is trained on 256*256 px images. I think I should train it on a bit higher size images as well.

Next steps

I will go back to my training and then train it with these corrections in training data and I will take a look into some new architectures which can predict the noise and motion blurs better then the present architecture.

Reference

  • Chapter 3 code can be found here.
  • I have created a dataset of high resolution images that I collected from the free stock photography website pexels.com. During the training cycle of my model I combined my data with another dataset which also had some more high resolution images.