Python 2 and 3 compatible pickle save and load

Python's pickle is great and convenient to use, however python 2 and python 3 differing in unicode handling is making pickled files quite incompatible to load.

Compatibility Issue

Use python 2 save a pickle

import pickle

with open('test.pickle', 'wb') as f:
    pickle.dump(my_object)

and afterward loading in python 3

import pickle

with open('test.pickle', 'rb') as f:
    pickle.load(f)

might result in a UnicodeDecodeError issue

UnicodeDecodeError: 'ascii' codec can't decode byte 0xfe in position 0: ordinal not in range(128)

Solution

Use python pickle's encoding argument, ref:

https://docs.python.org/3/library/pickle.html#pickle.Unpickler

def load_pickle(pickle_file):
    try:
        with open(pickle_file, 'rb') as f:
            pickle_data = pickle.load(f)
    except UnicodeDecodeError as e:
        with open(pickle_file, 'rb') as f:
            pickle_data = pickle.load(f, encoding='latin1')
    except Exception as e:
        print('Unable to load data ', pickle_file, ':', e)
        raise
    return pickle_data