refactor: adds _compliant sub-package

[dangotbanned]

Closes #2044

What type of PR is this? (check all applicable)

• 💾 Refactor
• ✨ Feature
• 🐛 Bug Fix
• 🔧 Optimization
• 📝 Documentation
• ✅ Test
• 🐳 Other

Related issues

• Closes: fix TypeVar used in (SparkLike|DuckDB)Expr  #2044 (comment)
• feat: Spec'd-out CompliantExpr #2119 (comment)
• Supersedes (refactor: adding Compliant* sub-protocols #2055)
• refactor: adds _Stores(Native|Compliant) #2130
• refactor: Generic CompliantSelector #2064

Checklist

• Code follows style guide (ruff)
• Tests added
• Documented the changes

If you have comments or can explain your changes, please do so below

Direct follow-up to (#2119), aiming to make space for similar changes for other Compliant* protocols.

This PR spiralled a bit, but tackles a few quite-related issues.

Lazy-only typing

#2044 has been resolved through the use of the new protocol NativeExpr.

narwhals/narwhals/_compliant/expr.py

Lines 65 to 73 in b3bdb3b

	class NativeExpr(Protocol):
	"""An `Expr`-like object from a package with [Lazy-only support](https://narwhals-dev.github.io/narwhals/extending/#levels-of-support).
	Protocol members are chosen *purely* for matching statically - as they
	are common to all currently supported packages.
	"""
	def between(self, *args: Any, **kwds: Any) -> Any: ...
	def isin(self, *args: Any, **kwds: Any) -> Any: ...

(Arrow|PandasLike)* deduplication

All of the new classes prefixed with Eager now provide the common functionality between the two Eager-only backends.
The big one is EagerExpr

narwhals/narwhals/_compliant/expr.py

Lines 257 to 260 in b3bdb3b

	class EagerExpr(
	CompliantExpr[EagerDataFrameT, EagerSeriesT],
	Protocol38[EagerDataFrameT, EagerSeriesT],
	):

A very unexpected part of that was that they could also share *ExprNamespace implementations

• (5a13b63)
• (2777822)

*Namespace protocols

Working on the Eager namespaces led to the new generic protocols for both (expr|series)_* namespaces https://github.com/narwhals-dev/narwhals/blob/b3bdb3b3ba52dada9cadde3c50d68520f1270a09/narwhals/_compliant/any_namespace.py

These can be re-used for all backends 😄

Misc

I followed up on (#2055 (comment)) and converted some methods to simpler .from_ variants.
The only remaining one (_create_compliant_series) is discussed in (#2149 (comment))

I also opened (#2169) for following up some strange pandas behavior

Note

There might be some other bits in commit descriptions I've missed

[dangotbanned]

Really want something that rolls off the tongue better than this 🤔

[MarcoGorelli]

thanks Dan! From a quick look I think this looks great, feel free to ping me for review when you're happy with it

[dangotbanned]

thanks Dan! From a quick look I think this looks great, feel free to ping me for review when you're happy with it

Thanks @MarcoGorelli, will do!

[dangotbanned]

I feel like this is fine in terms of a pure refactoring of:

narwhals/narwhals/_expression_parsing.py

Lines 160 to 172 in b508e63

	# For PyArrow.Series, we return Python Scalars (like Polars does) instead of PyArrow Scalars.
	# However, when working with expressions, we keep everything PyArrow-native.
	extra_kwargs = (
	{"_return_py_scalar": False}
	if returns_scalar and expr._implementation is Implementation.PYARROW
	else {}
	)
	out: list[CompliantSeries] = [
	plx._create_series_from_scalar(
	getattr(series, attr)(**extra_kwargs, **_kwargs),
	reference_series=series, # type: ignore[arg-type]
	)

Into this override:

narwhals/narwhals/_arrow/expr.py

Lines 138 to 141 in e64626a

	def _reuse_series_extra_kwargs(
	self, *, returns_scalar: bool = False
	) -> dict[str, Any]:
	return {"_return_py_scalar": False} if returns_scalar else {}

---

But I really can't shake wanting to handle this differently somehow.
Haven't spent too long thinking about it - maybe take another look before review

[dangotbanned]

Took a while to fully unravel.

Tip

See this code search and my earlier note to understand existing usage.

Very rough notes

IGNORE PLS

I'm just sticking this here to get it out of my changes:

    # NOTE: Calling from an instance of `Namespace` (provides `context`)
    # NOTE: All usage within `*Expr.map_batches`
    #   - `PandasLikeExpr` uses that **once**
    #   - `ArrowExpr` uses **twice**
    #   - Expr -> Namespace -> Series
    #   - But handling `numpy`
    # NOTE: `PandasLikeDataFrame.with_row_index` uses the wrapped `utils` function once
    # NOTE External
    # - `_expression_parsing.extract_compliant` (numpy array)
    # - `dataframe.DataFrame._extract_compliant` (numpy array)
    # NOTE General
    # - Most similar to `EagerSeries._from_iterable`

I'm starting to think the broader replacement for _create_compliant_series will be:

Collapsed imports

from typing import TYPE_CHECKING
from typing import Any
from typing import Protocol
from typing import overload

from narwhals._compliant.typing import EagerDataFrameT
from narwhals._compliant.typing import EagerSeriesT

if TYPE_CHECKING:
    import numpy as np
    from typing_extensions import TypeAlias

    from narwhals.typing import _1DArray
    from narwhals.typing import _2DArray

_NumpyScalar: TypeAlias = "np.generic[Any]"


class EagerNamespace(
    CompliantNamespace[EagerDataFrameT, EagerSeriesT],
    Protocol[EagerDataFrameT, EagerSeriesT],
):
    @overload
    def from_numpy(self, value: _NumpyScalar | _1DArray, /) -> EagerSeriesT: ...

    @overload
    def from_numpy(self, value: _2DArray, /) -> EagerDataFrameT: ...

    def from_numpy(self, value: _NumpyScalar | _1DArray | _2DArray, /) -> Any: ...

_NumpyScalar is handled by:

narwhals/narwhals/_compliant/series.py

Lines 54 to 55 in 067252b

	def _from_scalar(self, value: Any) -> Self:
	return self._from_iterable([value], name=self.name, context=self)

_1DArray is handled by:

narwhals/narwhals/_compliant/series.py

Lines 57 to 59 in 067252b

	@classmethod
	def _from_iterable(
	cls: type[Self], data: Iterable[Any], name: str, *, context: _FullContext

_2DArray would be each of the implementations in:

nw.functions.from_numpy

narwhals/narwhals/functions.py

Lines 435 to 561 in c223138

	def from_numpy(
	data: _2DArray,
	schema: Mapping[str, DType] | Schema | Sequence[str] | None = None,
	*,
	native_namespace: ModuleType,
	) -> DataFrame[Any]:
	"""Construct a DataFrame from a NumPy ndarray.
	Notes:
	Only row orientation is currently supported.
	For pandas-like dataframes, conversion to schema is applied after dataframe
	creation.
	Arguments:
	data: Two-dimensional data represented as a NumPy ndarray.
	schema: The DataFrame schema as Schema, dict of {name: type}, or a sequence of str.
	native_namespace: The native library to use for DataFrame creation.
	Returns:
	A new DataFrame.
	Examples:
	>>> import numpy as np
	>>> import pyarrow as pa
	>>> import narwhals as nw
	>>>
	>>> arr = np.array([[5, 2, 1], [1, 4, 3]])
	>>> schema = {"c": nw.Int16(), "d": nw.Float32(), "e": nw.Int8()}
	>>> nw.from_numpy(arr, schema=schema, native_namespace=pa)
	┌──────────────────┐
	|Narwhals DataFrame|
	|------------------|
	| pyarrow.Table |
	| c: int16 |
	| d: float |
	| e: int8 |
	| ---- |
	| c: [[5,1]] |
	| d: [[2,4]] |
	| e: [[1,3]] |
	└──────────────────┘
	"""
	return _from_numpy_impl(data, schema, native_namespace=native_namespace)
	def _from_numpy_impl(
	data: _2DArray,
	schema: Mapping[str, DType] | Schema | Sequence[str] | None = None,
	*,
	native_namespace: ModuleType,
	) -> DataFrame[Any]:
	from narwhals.schema import Schema
	if not is_numpy_array_2d(data):
	msg = "`from_numpy` only accepts 2D numpy arrays"
	raise ValueError(msg)
	implementation = Implementation.from_native_namespace(native_namespace)
	if implementation is Implementation.POLARS:
	if isinstance(schema, (Mapping, Schema)):
	schema_pl: pl.Schema | Sequence[str] | None = Schema(schema).to_polars()
	elif is_sequence_but_not_str(schema) or schema is None:
	schema_pl = schema
	else:
	msg = (
	"`schema` is expected to be one of the following types: "
	"Mapping[str, DType] | Schema | Sequence[str]. "
	f"Got {type(schema)}."
	)
	raise TypeError(msg)
	native_frame = native_namespace.from_numpy(data, schema=schema_pl)
	elif implementation.is_pandas_like():
	if isinstance(schema, (Mapping, Schema)):
	from narwhals._pandas_like.utils import get_dtype_backend
	it: Iterable[DTypeBackend] = (
	get_dtype_backend(native_type, implementation)
	for native_type in schema.values()
	)
	native_frame = native_namespace.DataFrame(data, columns=schema.keys()).astype(
	Schema(schema).to_pandas(it)
	)
	elif is_sequence_but_not_str(schema):
	native_frame = native_namespace.DataFrame(data, columns=list(schema))
	elif schema is None:
	native_frame = native_namespace.DataFrame(
	data, columns=[f"column_{x}" for x in range(data.shape[1])]
	)
	else:
	msg = (
	"`schema` is expected to be one of the following types: "
	"Mapping[str, DType] | Schema | Sequence[str]. "
	f"Got {type(schema)}."
	)
	raise TypeError(msg)
	elif implementation is Implementation.PYARROW:
	pa_arrays = [native_namespace.array(val) for val in data.T]
	if isinstance(schema, (Mapping, Schema)):
	schema_pa = Schema(schema).to_arrow()
	native_frame = native_namespace.Table.from_arrays(pa_arrays, schema=schema_pa)
	elif is_sequence_but_not_str(schema):
	native_frame = native_namespace.Table.from_arrays(
	pa_arrays, names=list(schema)
	)
	elif schema is None:
	native_frame = native_namespace.Table.from_arrays(
	pa_arrays, names=[f"column_{x}" for x in range(data.shape[1])]
	)
	else:
	msg = (
	"`schema` is expected to be one of the following types: "
	"Mapping[str, DType] | Schema | Sequence[str]. "
	f"Got {type(schema)}."
	)
	raise TypeError(msg)
	else: # pragma: no cover
	try:
	# implementation is UNKNOWN, Narwhals extension using this feature should
	# implement `from_numpy` function in the top-level namespace.
	native_frame = native_namespace.from_numpy(data, schema=schema)
	except AttributeError as e:
	msg = "Unknown namespace is expected to implement `from_numpy` function."
	raise AttributeError(msg) from e
	return from_native(native_frame, eager_only=True)

The _2DArray case is really interesting to me, since it has already carved out space for us (albeit DataFrame only):

narwhals/narwhals/functions.py

Lines 554 to 560 in c223138

	try:
	# implementation is UNKNOWN, Narwhals extension using this feature should
	# implement `from_numpy` function in the top-level namespace.
	native_frame = native_namespace.from_numpy(data, schema=schema)
	except AttributeError as e:
	msg = "Unknown namespace is expected to implement `from_numpy` function."
	raise AttributeError(msg) from e

[dangotbanned]

If we look ahead to v2, the particular names will change but direction is the same:

• API: io functions for v2 #2116
• API: io functions for v2 #2116 (comment)

[dangotbanned]

Here's what I think the generics need to be:

from typing import Protocol
from typing_extensions import TypeVar

from narwhals.typing import _1DArray
from narwhals.typing import _2DArray

if TYPE_CHECKING:
    import numpy as np
    from typing_extensions import Self
    from typing_extensions import TypeAlias

    from narwhals.utils import _FullContext


_NumpyScalar: TypeAlias = np.generic[Any]

ToNumpyT_co = TypeVar("ToNumpyT_co", covariant=True)
FromNumpyT_contra = TypeVar("FromNumpyT_contra", contravariant=True, default=ToNumpyT_co)


class NumpyConvertible(Protocol[ToNumpyT_co, FromNumpyT_contra]):
    def to_numpy(self) -> ToNumpyT_co: ...
    @classmethod
    def from_numpy(cls, value: FromNumpyT_contra, *, context: _FullContext) -> Self: ...


class SeriesImpl(NumpyConvertible[_1DArray, _NumpyScalar | _1DArray]): ...

class DataFrameImpl(NumpyConvertible[_2DArray]): ...

Notes

• To get TypeVar defaults, I'll use the same trick as (feat(typing): Backport generic Series to v1 #2110 (comment))
  • Probably will want to move that into a new module nw._typing_compat
    • Can also be the home for Protocol38 and nw.utils.deprecated
  • Using https://docs.astral.sh/ruff/settings/#lint_typing-modules
• I'm not using quoted annotations/forward refs in this example
  • Will in the real thing
  • Just trying to make it easier to read on GitHub 🙂

Prior Art

• https://github.com/zen-xu/pyarrow-stubs/blob/e9b6405cd151b38e2c6ea8e759e089ce795e4630/pyarrow-stubs/__lib_pxi/array.pyi#L1031-L1052

[dangotbanned]

thanks Dan! From a quick look I think this looks great, feel free to ping me for review when you're happy with it

@MarcoGorelli I'm gonna have to wrap this up soon - just to avoid spending too long on resolving conflicts.

Still somewhat manageable atm - but fully a self-inflicted chore I want to avoid now 😅

Will do my best to get it in shape today - really happy with the rabbit holes I've gone down though 😉

Update

This'll have to do for now - ready as I'll ever be @MarcoGorelli