FrançaisGet started

Analysis tools: pandas and scikit-learn

The toolkit of a Python analysis: Jupyter notebook, NumPy and pandas for tables, scikit-learn and its fit, transform, predict, score contract, splitting the set, the pipeline; R, Orange and KNIME only situated.

  • 26 cards
  • 3 topics
  • Free
Study this deck

The outline, topic by topic

Each topic is weighted the way the exam weighs it.

  1. Concepts
  2. Questions
  3. Sort and link

A few cards from the deck

  • NumPy: what is it?

    Show the answer

    A Python library of multi-dimensional numerical arrays, whose operations apply to all elements at once.

    Computations are vectorised and fast, far more than a Python loop. Almost the whole ecosystem, pandas and scikit-learn included, rests on it.

  • The iris dataset has 150 rows and 4 features. What is X_test.shape after this code?

    ```python
    from sklearn.datasets import load_iris
    from sklearn.model_selection import train_test_split

    X, y = load_iris(return_X_y=True)
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
    `

    • (30, 4)
    • (120, 4)
    • (30, 150)
    • (4, 30)
    Show the answer

    (30, 4)

    test_size=0.2 keeps 20% of 150 rows for the test, that is 30; the 4 features stay. (120, 4) is the shape of X_train.

  • A scikit-learn model learns with …(X, y), predicts with …(X) and is scored with …(X, y).

    Show the answer

    A scikit-learn model learns with fit(X, y), predicts with predict(X) and is scored with score(X, y).

    This contract is shared by all estimators. Calling predict before fit raises NotFittedError: the model has not learned anything yet.

  • pandas: what is it?

    Show the answer

    A Python library of labelled tables: reading a CSV, filtering, grouping, joining and cleaning data in named columns.

    Its central object is the DataFrame. Each column is a Series; df.head() shows five rows, df.describe() summarises the numerical columns.

  • Why is this line, placed before the split, a mistake?

    ```python
    X = StandardScaler().fit_transform(X)
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
    `

    • fit_transform returns a NumPy array, which train_test_split cannot split
    • The test set's mean and standard deviation are used to transform the training data
    • StandardScaler applies only to targets, never to the feature matrix X
    • The split must always precede reading the data, not follow its transformation
    Show the answer

    The test set's mean and standard deviation are used to transform the training data

    The scaler computed its parameters on all rows, test included: the test set is no longer untouched. Fix: fit the scaler on X_train alone, then transform on X_test, or put everything in a Pipeline.

Study this deck