
The idea would be to have a python module with the following structure:
```
- mapies/__init__.py => main file
- mapies/generic_classes.py => module that defines generic template for mapies preprocessors and grids (can probably be split in several modules)
- mapies/preprocessors/*.py => modules containing the actual preprocessors
- mapies/grids/*.py => modules containing the actual grids
```
where a `Preprocessor` is an object that contains import (`load`, `from_raw`) and export (`to_providentia`, `to_da`, `to_file`) methods, and some methods to convert between gridded and non-gridded data.
The generic classes make use of abstract base classes (`abc.ABC`), i.e. they enforce a specific template for the derived classes: as long as the derived class follows the template (which it must, otherwise an error will be raised), then they will be compatible with the library. This makes it really easy for other people (or us in the future) to write new pre-processor classes, when the need comes, but it makes it very important to define these generic classes correctly.

See the bottom of this file for an example of how that could be integrated to various workflows.
## `mapies/__init__.py`
```python
from omegaconf import DictConfig

# mapies/__init__.py
from mapies.preprocessors.generic_classes import GriddedMetaData, GenericPreprocessor
```

## `mapies/generic_classes.py`
Template for `Grid` and `Preprocessor` classes:
- Template method to read from one data source
- Template method to write to an internal format (whatever makes most sense given the data type)
- Real methods to write to DA, PROVIDENTIA, or any other user-specified format
- Real methods to do some basic filtering and regridding (to be developed)

The "in-memory" data format should be common to all classes (although they're allowed to change it, for instance if in one specific case it really doesn't fit in memory, or doesn't make any sense). Can be based on pandas, xarray, polars, plain old numpy ... whatever is the most efficient

Parallelization can be implemented for some methods (when and where it makes sense)

```python
from dataclasses import dataclass                     # just a cleaner way to create a class
from abc import ABC                                   # Abstract Base Classes, for imposing a structure
from mapies.grids import RotatedGrid, GenericGrid     # Mapies "grids" module

# for type annotations:
from omegaconf import DictConfig                      # library to handle yaml files (or another equivalent library)
from pathlib import Path
from typing import List, Callable
from pandas import DataFrame, Timestamp
from datetime import datetime


# The following one should be in mapies.grids:
class GenericGrid(ABC):
	@abstractmethod
	def aggregate(self, preprocessor) -> "GenericPreprocessor":
		"""
		Returns an aggregated version of the preprocessed data (following some aggregation method)
		"""


@dataclass
class GenericMetaData(ABC):
	is_gridded : bool
	is_griddable : bool
	

@dataclass
class GenericPreprocessor(ABC):
	data: DataFrame
	metadata: GenericMetaData

	@abstractmethod
	def store(self) -> Path | List[Path]:
	    """
	    Function used internally to write the pre-processed data on file. The file format might have to be different depending on the data type, so this is just a placeholder that the different classes (one for each data type) will have to implement themselves.
	    The function should return a single path or a list of paths, pointing to where the data has been stored
	    """

	@abstractmethod
	@classmethod
	def load(cls, paths : Path | List[Path]) -> "GenericPreprocessor":
		"""
		Reverse of "store". Creates a "GenericPreprocessor" instance (or one of its derived classes) based on a file or a list of files
		"""

	@abstractmethod
	@classmethod
	def from_raw(cls, conf: DictConfig) -> "GenericPreprocessor"
		"""
		Import RAW data
		"""

	def to(self, cls : GenericPreprocessor) -> "GenericPreprocessor"
		"""Concert to a different preprocessor, to be able to use its methods"""
	    return cls(self.data, self.metadata)

	def to_providentia(self, conf: DictConfig) -> Path | List[Path]:
		"""
		Returns data in a file format prepared for PROVIDENTIA.
		This can be implemented in a generic way (so it shouldn't be an abstracmethod)
		"""
		raise NotImplementedError

	def to_DA(self, conf: DictConfig) -> Path:
		"""
		Write the data to a format readable by the DA
		"""

	def to_file(self, method: Callable) -> Path:
		"""
		Generic interface to write data in a user-specified format.
		The user needs to provide a function that will take the whole object as single argument and return the path to a file
		"""
		return method(self)

	def to_gridded(self, grid: GenericGrid) -> "GenericPreprocessor":
		"""
		Return a gridded version of the object. Some preprocessors may not allow gridding, in this case they should return an error (or handle the exception in a nice way).
		"""
		if not self.conf.griddable:
			raise RuntimeError("This data cannot be gridded")
		return grid.aggregate(self)

	def crop(
			self, 
			start: Timestamp | datetime | str = None, 
			end | Timestamp | datetime | str = None, 
			west : float = -180, 
			east : float = 180,
			south : float = -90,
			north : float = 90
		) -> "GenericPreprocessor":
		"""
		Return a copy of the object, limited to a specific temporal and/or spatial extent. 
		I wrote it assuming "pandas" as a backend, but it's just for the example.
		"""
		if start is None :
			start = self.start
		if end is None :
			end = self.end

		data = self.data.loc[
			(self.data.time >= start) & (self.data.time < end) & (self.data.lon >= west) & (self.data.lon < east) & (self.data.lat > south) & (self.data.lat < north)
		]

        # Return a new object, of the same (derived) class as "self"
		return self.__class__(data, self.metadata)

	@property
	def start(self) -> Timestamp:
		return Timestamp(self.data.time.min())

	@property
	def end(self) -> Timestamp:
		return Timestamp(self.data.time.max())
```

## `mapies/preprocessors/da.py`
Example pre-processor. Inherits from the generic class and provides concrete implementations of the abstract methods that it defines (+ any additional method that may be relevant to the data type)
```python
from mapies import GenericMetaData, GenericPreprocessor


@dataclass
class DAMetaData(GenericMetadata):
	is_gridded : bool = False        # DA data is by default not gridded, but can be in some instances
	is_griddable : bool = True       #
	coordinates : GenericGrid        # Here "GenericGrid" would rather be a "RotatedGrid" instance
	instrument : str                 # We can have many application-specific attributes. "GenericMetaData" should tell which ones are mandatory 
	author : str                     #


class DAPreprocessor(GenericPreprocessor):
	def store(self) -> Path | List[Path]:
		data = self.data.to_xarray()
		for k, v in self.metadata.items():
			data.attrs[k] = v
		data.to_netCDF(pth)
		return Path(pth)

	@classmethod
	def load(cls, paths: Path | List[Path]) -> "DAPreprocessor":
		data = xr.open_mfdataset(paths)
		return cls(
			data = data.to_dataframe().reset_index(),
			metadata = DAMetaData(
				is_gridded = data.gridded,
				coordinates = RotatedGrid.from_str(data.coordinates)
				experiment = data.expid
			)
		)

	@classmethod
	def from_raw(cls, *args, **kwargs) -> "DAPreprocessor"
		# Another copy of load (since "RAW" doesn't mean much for DA data)
	    return cls.load(*args, **kwargs)

```

## Example workflows
Some example on how this can be used:
- preprocessing raw data
- writing data for a specific project (e.g. PROVIDENTIA or DA)
- include a grid aggregation step in the process
- write to a custom file, using a user-provided writing method
```python
from mapies.preprocessors import VIIRSPreprocessor
from omegaconf import OmegaConf
from mapies.grids import RotatedGrid, MyClimGrid
```
* Preprocess raw VIIRS data:
```python
conf = OmegaConf.read('viirs.yaml')
data = VIIRSPreprocessor.from_raw(conf.paths.viirs_raw, conf.start, conf.end)
data.store(conf.paths.preprocessed)
```
* write data for a specific project (e.g. PROVIDENTIA)
```python
conf = OmegaConf.read('providentia.yaml')
data = VIIRSPreprocessor.load(conf.paths.preprocessed)
data.to_providentia(conf.paths.providentia, conf.start, conf.end)
```
* Include an aggregation step in the process (aggregation is not necessarily identical as regridding, i.e. the result may not be on a regular grid)
```python
conf = OmegaConf.read('da.yaml')
data = VIIRSPreprocessor.load(conf.paths.preprocessed)
grid = RotatedGrid(**conf.grid)         # This should construct a "RotatedGrid" object that will handle the rotation and aggregation the way we want it in DA:
aggregated_data = grid.aggregate(data)
aggregated_data.to_da(conf.paths.obs_DA)
```
* preprocess data using a user-specified method to define the format:
```python
conf = OmegaConf.read('clim.yaml')
data = VIIRSPreprocessor.load(conf.paths.preprocessed)
data.to_grid(MyClimGrid(conf)).to_file(conf.paths.preprocessed_files)
```
* Convert data preprocessed in one format into another one (e.g. to use PROVIDENTIA to plot data used in DA, or to use a method from another preprocessor
```python
# Converting from one class to another:
DAPreprocessor.load(conf.path.preprocessed).to_providentia(conf.paths.providentia)

# Apply filters implemented in "VIIRSPreprocessor" to data that has already been written out to DA format:
DAPreprocessor.load(conf.path.preprocessed).to(VIIRSPreprocessor).apply_filters(conf.viirs.filters).to_da(conf.path.preprocessed)