Tutorial 3. Loading data into the Layer object

This section will cover how to properly subclass DataLoader class if specialized loader is required.

First, let’s load the required data.

# Load EBSD data
import numpy as np

EBSD = np.genfromtxt(
    "./data/SiC_in_NiSA.ctf", dtype=float, skip_header=15, delimiter="\t", names=True
)

Boilerplate

We will deal with the low-level API first. This pattern will be used repetitively. It is possible to wrap this boilerplate to your own function for the enhanced convenience.

# Load data into the layer
from pyxc.core.layer import Layer
from pyxc.core.processor.arrays import column_parser
from pyxc.core.container import Container2D
from pyxc.core.loader import ImageLoader, XYDLoader
from pyxc.transform.homography import Homography

layer_ebsd = Layer(
    data=column_parser(EBSD, format_string="dxydddddddd"),
    container=Container2D,
    dataloader=XYDLoader,
    transformer=Homography,
)

Using XYDLoader

The XYDLoader object is useful to load 2-dimensional array-like data or structured arrays. It is required that the array’s first and second columns contain numeral values of X and Y data. Therefore, to use XYDLoader, preparation of data to a proper format is important.

Using column_parser function

The column parser function is the utility function to refine data. It reorders columns based on the provided format string. x and y means columns that contain x and y information, while _ means ignore. All other chracters are regarded data.

2-dimensional array-like

Let’s continue with an example. First, 2-dimensional array-like.

arr = np.random.random((3, 4))
arr

array([[0.50735678, 0.03827714, 0.88425536, 0.68960317],
       [0.79059358, 0.26352113, 0.04189033, 0.92274307],
       [0.73868396, 0.69864402, 0.00887661, 0.77108242]])

Let’s assume we are setting the second and third columns as x and y, while the first column remains as data column. You can see, the x and y columns are moved to the first and second columns.

column_parser(arr, "dxy")

array([[0.03827714, 0.88425536, 0.50735678, 0.68960317],
       [0.26352113, 0.04189033, 0.79059358, 0.92274307],
       [0.69864402, 0.00887661, 0.73868396, 0.77108242]])

or, you can exclude some columns by specifying _ or explicitly set return_unspecified to False.

column_parser(arr, "_dxy")

array([[0.88425536, 0.68960317, 0.03827714],
       [0.04189033, 0.92274307, 0.26352113],
       [0.00887661, 0.77108242, 0.69864402]])

column_parser(arr, "dxy", return_unspecified=False)

array([[0.03827714, 0.88425536, 0.50735678],
       [0.26352113, 0.04189033, 0.79059358],
       [0.69864402, 0.00887661, 0.73868396]])

Structured array

For a structured array is works exactly same. Let’s use ebsd data we’ve previously loaded.

EBSD

array([(2.,  0.    ,  0.   , 11., 0., 160.45, 47.733, 233.82, 1.0211, 160., 255.),
       (2.,  0.2776,  0.   , 10., 0., 160.15, 47.888, 233.74, 1.3246, 161., 255.),
       (2.,  0.5553,  0.   , 10., 0., 160.14, 47.928, 234.  , 1.3319, 161., 255.),
       ...,
       (2., 26.932 , 20.546, 11., 0., 159.7 , 48.268, 235.35, 1.2136, 154., 255.),
       (2., 27.21  , 20.546, 10., 0., 159.24, 48.137, 234.97, 0.861 , 159., 255.),
       (2., 27.487 , 20.546, 10., 0., 158.98, 48.32 , 235.21, 1.125 , 162., 255.)],
      dtype=[('Phase', '<f8'), ('X', '<f8'), ('Y', '<f8'), ('Bands', '<f8'), ('Error', '<f8'), ('Euler1', '<f8'), ('Euler2', '<f8'), ('Euler3', '<f8'), ('MAD', '<f8'), ('BC', '<f8'), ('BS', '<f8')])

Let’s say, we need X, Y, Phase, Euler1, Euler2, Euler3. Then the format string should be dxy__ddd. Also we don’t want to retrieve trailing MAD, BC, and BS, so we will explicitly specify ‘return_unspecified’ to False.

column_parser(EBSD, "dxy__ddd", return_unspecified=False)

array([( 0.    ,  0.   , 2., 160.45, 47.733, 233.82),
       ( 0.2776,  0.   , 2., 160.15, 47.888, 233.74),
       ( 0.5553,  0.   , 2., 160.14, 47.928, 234.  ), ...,
       (26.932 , 20.546, 2., 159.7 , 48.268, 235.35),
       (27.21  , 20.546, 2., 159.24, 48.137, 234.97),
       (27.487 , 20.546, 2., 158.98, 48.32 , 235.21)],
      dtype={'names': ['X', 'Y', 'Phase', 'Euler1', 'Euler2', 'Euler3'], 'formats': ['<f8', '<f8', '<f8', '<f8', '<f8', '<f8'], 'offsets': [8, 16, 0, 40, 48, 56], 'itemsize': 88})

Correctly processed data (with proper format string) is compatible with XYDLoader. You can load the data to the Layer.

# Load data into the layer
from pyxc.core.layer import Layer
from pyxc.core.processor.arrays import column_parser
from pyxc.core.container import Container2D
from pyxc.core.loader import ImageLoader, XYDLoader
from pyxc.transform.homography import Homography

layer_ebsd = Layer(
    data=column_parser(EBSD, format_string="dxy__ddd", return_unspecified=False),
    container=Container2D,
    dataloader=XYDLoader,
    transformer=Homography,
)
layer_ebsd.container

Container2D([(   0, nan, nan,  0.    ,  0.   , 2., 160.45, 47.733, 233.82),
             (   1, nan, nan,  0.2776,  0.   , 2., 160.15, 47.888, 233.74),
             (   2, nan, nan,  0.5553,  0.   , 2., 160.14, 47.928, 234.  ),
             ...,
             (7497, nan, nan, 26.932 , 20.546, 2., 159.7 , 48.268, 235.35),
             (7498, nan, nan, 27.21  , 20.546, 2., 159.24, 48.137, 234.97),
             (7499, nan, nan, 27.487 , 20.546, 2., 158.98, 48.32 , 235.21)],
            dtype=[('row', '<i4'), ('x', '<f4'), ('y', '<f4'), ('x_raw', '<f4'), ('y_raw', '<f4'), ('Phase', '<f8'), ('Euler1', '<f8'), ('Euler2', '<f8'), ('Euler3', '<f8')])

Loading image data

Loading image data is very straightforward. Image data are 2- or 3-dimensional array-like objects with the shape of (i, j, k). Each channel of the image will be stored as serialized form, with the column name of Channel_{integer}. Let’s make sample image data. Just use ImageLoader.

im3channel = np.random.random((4, 4, 3))

Then, we can easily load the data into the array.

# Load data into the layer
from pyxc.core.layer import Layer
from pyxc.core.processor.arrays import column_parser
from pyxc.core.container import Container2D
from pyxc.core.loader import ImageLoader, XYDLoader
from pyxc.transform.homography import Homography

layer_im3c = Layer(
    data=im3channel,
    container=Container2D,
    dataloader=ImageLoader,
    transformer=Homography,
)
layer_im3c.container

Container2D([( 0, nan, nan, 0., 0., 0.96937499, 0.13936242, 0.72364392),
             ( 1, nan, nan, 1., 0., 0.35975408, 0.10091848, 0.95045094),
             ( 2, nan, nan, 2., 0., 0.40054063, 0.33450998, 0.58534131),
             ( 3, nan, nan, 3., 0., 0.95742908, 0.62253418, 0.42588247),
             ( 4, nan, nan, 0., 1., 0.76316555, 0.53874036, 0.6482441 ),
             ( 5, nan, nan, 1., 1., 0.82320666, 0.79070129, 0.95406686),
             ( 6, nan, nan, 2., 1., 0.51433208, 0.32939568, 0.25830387),
             ( 7, nan, nan, 3., 1., 0.5546261 , 0.84218189, 0.273518  ),
             ( 8, nan, nan, 0., 2., 0.88380361, 0.04694765, 0.09171057),
             ( 9, nan, nan, 1., 2., 0.80741769, 0.16209902, 0.95504076),
             (10, nan, nan, 2., 2., 0.62918872, 0.88079637, 0.46776218),
             (11, nan, nan, 3., 2., 0.38724316, 0.93526209, 0.83616115),
             (12, nan, nan, 0., 3., 0.30222166, 0.94326678, 0.90913035),
             (13, nan, nan, 1., 3., 0.54921236, 0.96669931, 0.93282651),
             (14, nan, nan, 2., 3., 0.80842747, 0.2044053 , 0.58574304),
             (15, nan, nan, 3., 3., 0.78077453, 0.74936394, 0.22605188)],
            dtype=[('row', '<i4'), ('x', '<f4'), ('y', '<f4'), ('x_raw', '<f4'), ('y_raw', '<f4'), ('Channel_0', '<f8'), ('Channel_1', '<f8'), ('Channel_2', '<f8')])

x = np.linspace(0, 1, 10)
y = np.linspace(2, 10, 10)
data = np.random.random((10, 3))
example_container = Container2D(x_raw=x, y_raw=y, data=data)

As you can see, an example_container is now initialized correctly.

example_container

Container2D([(0, nan, nan, 0.        ,  2.       , 0.17951544, 0.57240073, 0.82072686),
             (1, nan, nan, 0.11111111,  2.8888888, 0.94635903, 0.66720142, 0.28918972),
             (2, nan, nan, 0.22222222,  3.7777777, 0.99414624, 0.95613272, 0.12769116),
             (3, nan, nan, 0.33333334,  4.6666665, 0.39385997, 0.50770747, 0.07395108),
             (4, nan, nan, 0.44444445,  5.5555553, 0.71376294, 0.66972877, 0.98793257),
             (5, nan, nan, 0.5555556 ,  6.4444447, 0.47548626, 0.31919555, 0.19173375),
             (6, nan, nan, 0.6666667 ,  7.3333335, 0.11570966, 0.57009082, 0.15563965),
             (7, nan, nan, 0.7777778 ,  8.222222 , 0.80489317, 0.85835354, 0.34290033),
             (8, nan, nan, 0.8888889 ,  9.111111 , 0.54964601, 0.60756746, 0.93553461),
             (9, nan, nan, 1.        , 10.       , 0.25828889, 0.9817641 , 0.48121845)],
            dtype=[('row', '<i4'), ('x', '<f4'), ('y', '<f4'), ('x_raw', '<f4'), ('y_raw', '<f4'), ('Channel_0', '<f8'), ('Channel_1', '<f8'), ('Channel_2', '<f8')])

We didn’t provide the structured array. Therefore, the column names for the data are automatically determined such as Channel_0, Channel_1, and Channel_2.

example_container.dtype.names

('row', 'x', 'y', 'x_raw', 'y_raw', 'Channel_0', 'Channel_1', 'Channel_2')