Zepworks – Blog

Fixing the blank login screen after upgrading Ubuntu 24.04 to 26.04

Thu, 08 Oct 2026 00:00:00 -0700

TLDR

TLDR; I upgraded from Ubuntu 24.04 LTS to 26.04 LTS and ended up with a login screen that had no users and no password box. The top bar worked, but there was no way to log in. After spending too much time reinstalling GNOME and GDM packages, I found that an extra AppArmor profile for bwrap was preventing GNOME from loading images. Moving that profile out of the way and rebooting fixed the problem.

If you have the same symptoms, don’t disable AppArmor or delete profiles blindly. Check whether you have a conflicting /etc/apparmor.d/usr.bin.bwrap first.

The problem

After upgrading Ubuntu from 24.04 to 26.04, I was greeted with a mostly blank login screen.

Interestingly, the entire UI wasn’t broken. The top bar was there. I could see the date in the center and open the menu in the top-right corner. Brightness controls and the power/reboot options worked too.

What was missing was the important part: the list of users and the password input. I couldn’t log in.

The machine uses Intel integrated graphics, and Ubuntu was using Wayland. Since part of GNOME was rendering correctly, I wasn’t convinced this was a graphics driver problem.

Fortunately, I could still get to a terminal with Ctrl+Alt+F3.

Things that didn’t fix it

I tried quite a few things before finding the actual issue:

Disabled Wayland in /etc/gdm3/custom.conf by setting WaylandEnable=false. The problem was exactly the same with X11.
Ran sudo dpkg --configure -a, updated the package lists, and reinstalled ubuntu-desktop, gdm3, and ubuntu-session.
Reinstalled accountsservice, thinking GDM might not be getting the user list.
Cleared GDM’s cache and configuration under /var/lib/gdm3/.
Restarted GDM and eventually rebooted the machine.

None of these made any difference.

The error in the logs

The useful information was in the system journal. From TTY3, I ran:

sudo journalctl -b | grep -iE 'gdm|gnome-shell|greeter'

There was a JavaScript error in GNOME Shell’s login dialog:

JS ERROR: Error: Argument child may not be null
_updateLogoTexture@resource:///org/gnome/shell/gdm/loginDialog.js:1086:27
_updateLogo@resource:///org/gnome/shell/gdm/loginDialog.js:1094:14
_init@resource:///org/gnome/shell/gdm/loginDialog.js:679:14
LoginDialog@resource:///org/gnome/shell/gdm/loginDialog.js:537:4

There were also errors about failing to load a pixbuf, along with AppArmor messages involving bwrap and glycin:

This may indicate that pixbuf loaders or the mime database could not be found.

apparmor="DENIED" ... comm="bwrap" ... capname="setpcap"

The loginDialog.js stack trace was the key. The login dialog was failing while trying to load its logo. Apparently, a missing image was enough to make the entire login dialog disappear!

This is also described in Ubuntu bug #2169691. The bug is about GDM failing completely when its logo cannot be loaded, rather than simply displaying the login screen without the logo.

Why was an image breaking the login screen?

Ubuntu 26.04 uses glycin for loading images. Glycin runs image loaders in a sandbox using Bubblewrap (bwrap). AppArmor controls what these processes are allowed to do.

On my machine, AppArmor was reporting a conflict involving profiles attached to /usr/bin/bwrap. As a result, bwrap wasn’t getting the permissions it needed, the image loader failed, and GNOME couldn’t load the logo for the login dialog.

So the chain of failures was roughly:

Conflicting AppArmor profiles for bwrap.
bwrap fails to create the sandbox needed by glycin.
Glycin fails to load images.
GDM fails to load its logo.
loginDialog.js throws an exception, and the user selection/password UI never appears.

Not exactly where I expected a missing login screen to lead me.

Finding the conflicting AppArmor profile

I checked the AppArmor files related to bwrap:

ls -l /etc/apparmor.d/*bwrap*

On my machine, there were three:

/etc/apparmor.d/bwrap-userns-restrict
/etc/apparmor.d/glycin.bwrap
/etc/apparmor.d/usr.bin.bwrap

The first two were expected. The third one was interesting, so I looked at it:

cat /etc/apparmor.d/usr.bin.bwrap

It was a tiny profile containing this rule (excerpt):

/usr/bin/bwrap flags=(unconfined) {
    userns,
    ...
}

Ubuntu already had bwrap-userns-restrict, which also provides an AppArmor profile for Bubblewrap. The additional usr.bin.bwrap profile was competing for the same executable.

Before changing anything, I checked whether the extra file belonged to an installed package:

dpkg -S /etc/apparmor.d/usr.bin.bwrap

The result was:

dpkg-query: no path found matching pattern

So it wasn’t owned by an installed Debian package. I don’t know what originally created it, but it wasn’t something I needed to keep active alongside Ubuntu’s bwrap profile.

The fix

I moved the conflicting profile to /root instead of deleting it:

sudo mv /etc/apparmor.d/usr.bin.bwrap /root/usr.bin.bwrap.apparmor-backup

Then rebooted:

sudo reboot

That was it. The login screen came back, with the user list and password input working normally.

A reboot matters here because removing a profile file doesn’t necessarily remove the profile already loaded in the kernel. Rebooting ensures AppArmor starts with the remaining configuration.

What about security?

I didn’t disable AppArmor. I only moved one extra, non-package-owned profile that conflicted with the profiles Ubuntu provides.

If you want to check AppArmor on your machine after the fix, run:

sudo aa-status

You can also look for Bubblewrap and glycin profiles:

sudo aa-status | grep -Ei 'bwrap|glycin'

It’s worth checking the output rather than assuming your system has exactly the same configuration as mine. And if dpkg -S says your usr.bin.bwrap belongs to a package, investigate that before moving it.

Conclusion

The actual fix was moving one AppArmor profile and rebooting. Getting to that point took a lot longer because everything initially looked like a broken GDM or GNOME upgrade.

What I hadn’t expected was that GDM could lose its entire login dialog because it couldn’t load a logo, or that the real problem would be a leftover AppArmor profile for an image loader’s sandbox.

If you’re upgrading from Ubuntu 24.04 to 26.04 and end up with a blank login screen but a working top bar, check the loginDialog.js errors and the AppArmor profiles for bwrap before you start reinstalling the desktop.

DeepDiff 9.0.0 Is Released

Sun, 29 Mar 2026 00:00:00 -0700

I’m happy to announce that DeepDiff 9.0.0 is released.

Major releases are always a little tricky. On one hand, they are exciting because you get to clean up behavior, fix long-standing issues, and move the project forward. On the other hand, even small backward incompatible changes can surprise people.

This release has a few of those changes, especially around cases where ignore_order=True is enabled. The results can be slightly different than before, and in this case that is a good thing because the new behavior is more correct.

If you are upgrading, please take a quick look at the mini migration notes in the changelog at /deepdiff/current/changelog.html.

The main migration note

The biggest migration item in 9.0.0 is around to_dict() and to_json().

These methods now accept a verbose_level parameter and always return a usable text-view dict. When the original view is 'tree', they now default to verbose_level=2 so you get the full details. The old view_override parameter is removed.

In plain English, if you were relying on the previous output from to_dict() or to_json() while using tree view, you may need to explicitly pass the verbose_level you want in order to get the same kind of output as before.

It is a small change, but it is exactly the kind of thing that is worth calling out in a major release.

What is new in 9.0.0

There are a few headline changes in this release.

First, DeepDiff is dropping support for Python 3.9 and adding support for Python 3.14.

Second, DeepDiff now supports a callable group_by, which gives you more flexibility when grouping records before comparing them.

Third, to_flat_dicts now has a FlatDeltaDict TypedDict return type, which should make typing support cleaner for people working with static analysis and editors that understand type hints.

And then there are a bunch of bug fixes and correctness fixes that I am very happy to get in.

The short English version

If I had to summarize DeepDiff 9.0.0, I would put it like this:

DeepDiff is more correct now, especially when ignore_order=True
the serialization output is more consistent
type hints are a bit better
edge cases around objects, big integers, and flattened output got cleaned up
and the project is moving forward with newer Python versions

That is really what this release is about.

A closer look at the changes

One important fix is that DeepDiff no longer misses some int-versus-float type changes when ignore_order=True is enabled. That kind of thing can lead to very confusing results because the whole point of a diff tool is to be precise when it matters.

Another nice improvement is that reporting now always uses the t1 path. That makes the output more consistent and easier to reason about.

There was also a fix for the colored view when all list items are removed. That was one of those bugs that only shows up in a specific situation, but once you hit it, it is annoying.

There was a fix for hasattr() swallowing AttributeError in __slots__ handling for objects that also define __getattr__. This is exactly the sort of Python edge case that is easy to miss and painful to debug.

DeepDiff also now handles Fraction as a numeric type, which is a nice improvement for users who work with more precise numeric representations.

Some serialization-related bugs got fixed too:

_convert_oversized_ints failing on NamedTuple
an orjson TypeError when integers exceed the 64-bit range
and a bug in to_flat_dicts where include_action_in_path and report_type_changes were not being passed through correctly

There were also fixes for ignore_keys in detailed__dict__ and a type hint issue around logarithmic similarity.

So overall, this is not a flashy release. It is a solid release. It makes DeepDiff more correct, more consistent, and less surprising in edge cases.

Thank you to the contributors

I also want to thank the people who contributed to this release.

Thank you to echan5 for adding support for callable group_by.

Thank you to Yann Rouillard for fixing the colored view display when all list items are removed.

Thank you to Tomas Pereira de Vasconcelos for fixing the hasattr() and __slots__ handling issue for objects with __getattr__.

Thank you to devin13cox for fixing Delta producing phantom entries when items both move and change values with iterable_compare_func.

Thank you to vitalis89 for fixing the ignore_keys issue in detailed__dict__.

Thank you to ljames8 for fixing the logarithmic similarity type hint.

And thank you to akshat62 for adding Fraction numeric support.

Open source gets better one contribution at a time, and I really appreciate everybody who took the time to help improve DeepDiff.

Another milestone

Another important milestone is that DeepDiff has now reached 72 million monthly downloads on PyPI.

That number is honestly a bit surreal to me.

When you maintain a project for a long time, most of the work does not feel glamorous. It is bug reports, edge cases, documentation, testing, release work, and trying very hard not to break people’s systems. So it is exciting to see DeepDiff being used at this scale.

DeepDiff is also used by major companies from Netflix to Google, Uber, and many others. On GitHub alone, around 20,000 open source repositories depend on it.

That is both exciting and humbling.

It is exciting because it is satisfying to know that something I built is useful to so many people.

It is humbling because once a project reaches this kind of usage, every change matters more.

Moving DeepDiff under Qluster

I have also decided to move DeepDiff under my Qluster brand.

The users of Qluster and the users of DeepDiff are not exactly the same, so this is not about trying to make them into the same thing.

It is really about brand recognition and trust.

I want Qluster to stand for software that is practical, battle-tested, and built with care. DeepDiff has earned trust over the years, and I think bringing it under the Qluster brand helps strengthen that story.

In other words, if I am going to keep building tools, I want people to be able to recognize the name behind them.

Pypi Download Stats

Pypi provides stats on DeepDiff downloads:

last_day	last_month	last_week
1,562,245	72,748,471	19,070,482

Python Version	percent	downloads
3.12	34.53%	118,790,753
3.11	32.38%	111,409,067
3.10	14.86%	51,112,548
3.13	5.66%	19,460,560
3.9	5.61%	19,298,453
null	2.95%	10,158,686
3.8	2.78%	9,570,766
3.14	0.66%	2,259,586
3.7	0.30%	1,020,215
3.6	0.20%	679,827
2.7	0.08%	278,586
3.5	0.01%	25,873
3.15	0.00%	1,658
3.4	0.00%	266
Total		344,066,844

OS	percent	downloads
Linux	97.91%	336,878,293
Windows	1.04%	3,593,485
Darwin	0.66%	2,267,059
null	0.39%	1,327,074
other	0.00%	933
Total		344,066,844

A couple of the PyPI numbers stood out to me. First, DeepDiff is overwhelmingly used on Linux, which probably means a lot of those downloads are coming from CI/CD, containers, servers, and other backend environments. I was also surprised to see Windows ahead of macOS. And on the Python side, most usage is still on 3.11 and 3.12, while 3.14 adoption is still pretty early. We are already using 3.14 in Qluster, so that puts us a bit ahead of the curve there.

Final thoughts

It is exciting to be a maintainer of something that is used by so many people.

It is also a responsibility that I do not take lightly.

DeepDiff ends up in test suites, data pipelines, internal tools, debugging workflows, and production systems that I will never personally see. That is part of what makes maintaining it meaningful to me.

DeepDiff 9.0.0 is a major release, but it is a very DeepDiff kind of major release. It is about correctness, consistency, and moving the project forward carefully.

I hope you enjoy it.

Have fun!

Sep Dehpour

How to persist pod logs on Minikube

Tue, 17 Sep 2024 01:21:14 -0700

TLDR

TLDR; I had a brilliant idea to write logs to a file on Minikube and have those logs persist between the restarts. I had to jump through some hoops. The main one was realizing that I had to change the folder ownership of where the files will persist within Minikube itself and use hostPath volume type on the pod so that I could persist the logs in Minikube.

Why you don’t need this on a managed k8s cluster

Normally, on a managed Kubernetes cluster, you write the logs to stdout/stderr. The logs automatically end up in a logs bucket in the case of Google Cloud, and you can view them on Cloud logging. You don’t “lose” the logs. But with Minikube, you will.

Overview of the problem

Minikube persists data that is written to specific addresses including /data/*
We can use Kubernetes PersistentVolume to mount a volume at /data/logs-data
Minikube will create a /data/logs-data that is owned by root.
If you are running docker as a non-root user, it can’t write to /data/logs-data

The docker image

In the Dockerfile, we want to explicitly assign a user_id to the user we want to run inside Docker. In this case, we create a nonroot user:

FROM ...
RUN adduser --disabled-password --gecos "" --uid 1000 --shell /bin/bash nonroot 

RUN mkdir -p /var/logs-data && \
    chown nonroot:nonroot /var/logs-data && \
    chmod 755 /var/logs-data
USER nonroot
...
CMD ["sleep", "infinity"]

The key here is assigning an explicit ID to the nonroot user by passing –uid 1000. The nonroot user won’t need a password to log in, hence the --disable-password flag. We won’t be passing name and other information when creating the user so we do --gecos "". While we are at it let’s assign a default shell to the user too: --shell /bin/bash

Minikube and file permissions

minikube ssh "sudo mkdir -p /data/logs;sudo chown 1000:1000 /data/logs-data"

We create the /data/logs folder inside minikube proactively and assign the user_id 1000 as the owner. Note that we are using the same ID that we explicitly added in the Dockerfile.

Kubernetes yaml file

We will use a hostPath volume type. A hostPath volume mounts a file or directory from the host node’s filesystem into your Pod. In this case directly from Minikube.

We will explicitly tell Kubernetes to use user_id 1000 for all operations in the docker image by passing fsGroup, runAsGroup, and runAsUser.

Example:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: logs-data
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
  hostPath:
    path: /data/logs-data  # Host path for Minikube node
  storageClassName: standard
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: logs-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  volumeName: logs-data
---
apiVersion: batch/v1
kind: Job
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  template:
    spec:
      securityContext:
        fsGroup: 1000  # Group ID from atlas docker image nonroot user
        runAsGroup: 1000  # Group ID from atlas docker image nonroot user
        runAsUser: 1000  # User ID from atlas docker image nonroot user
      containers:
      - name: main
        image: path_to_image
        command:
          - sleep
          - infinity
        volumeMounts:
          - name: logs-data
            mountPath: /var/logs-data
      restartPolicy: Never
      volumes:
        - name: logs-data
          persistentVolumeClaim:
            claimName: logs-data

When we apply the above yaml file, we can start writing logs to files inside /var/logs-data which will end up inside /data/logs-data on Minikube. These will persist when the pods restart, killed, and even when Minikube restarts!

If you want to know what fsGroup, runAsGroup, and runAsUser are, here is some more material to read:

Bonus material: fsGroup, runAsGroup, and runAsUser

In Kubernetes, fsGroup, runAsGroup, and runAsUser are part of the securityContext for pods and containers. These fields are used to control the ownership and permissions for processes and files within the containers, helping you run containers with non-root users and ensuring proper access control. Here’s an explanation of each:

1. `runAsUser`:

Definition: Specifies the user ID (UID) that the container’s process should run as.
Purpose: Ensures that processes inside the container run as a non-root user, improving security by avoiding the use of the default root user in the container.
Effect: All processes in the container will run as this specified user. If a container image defines a specific user, runAsUser will override it.

Example:

securityContext:
  runAsUser: 1000  # Process will run as user with UID 1000

Use Case: If your container image is based on a Linux system, you can create users inside the image with specific UIDs, such as 1000 for a non-root user, and then use runAsUser to run the container process under this user. This is useful for avoiding privilege escalation or preventing containerized processes from running as root.

2. `runAsGroup`:

Definition: Specifies the group ID (GID) that the container’s process should run as.
Purpose: Similar to runAsUser, but it controls the primary group under which the process will run.
Effect: All processes in the container will run under this group. If the image specifies a group, runAsGroup will override it.

Example:

securityContext:
  runAsGroup: 1000  # Process will run with GID 1000

Use Case: You can ensure that the process in the container has the appropriate group access to certain files or directories. For instance, if your container process needs access to files owned by a particular group (like group 1000), runAsGroup ensures the process has the correct group permissions.

3. `fsGroup`:

Definition: Specifies the group ID (GID) for file system access control.
Purpose: Allows you to set a group that will own the files created by the container and control access to shared volumes. This ensures proper group-based access to shared resources like volumes.
Effect:
- When a volume is mounted inside the container, Kubernetes changes the group ownership of the volume and its files to the fsGroup specified.
- New files created inside the mounted volumes will have this group ID.
- This applies only to writable volumes (like PersistentVolume or EmptyDir), not to hostPath volumes.

Example:

securityContext:
  fsGroup: 2000  # Files on mounted volumes will have GID 2000

Use Case: When using shared storage (like PersistentVolume), fsGroup helps ensure that files created by different containers or users can be accessed by other containers in the same group. It’s especially useful in scenarios where multiple containers need to write to the same volume but are running under different users.

How They Work Together:

When you set runAsUser, runAsGroup, and fsGroup, these values control how processes and files are managed inside the container:

runAsUser determines the user ID of the processes running inside the container.
runAsGroup sets the primary group of those processes.
fsGroup sets the group ownership of mounted volumes, ensuring that the container has access to shared files through group permissions.

Example Scenario:

Let’s say you have a container running a process as runAsUser: 1000 and runAsGroup: 1000, with fsGroup: 2000. The container’s process will:

Run as user 1000 and group 1000 (i.e., the process will execute as a non-root user and group).
Any volumes mounted in the container will have their group ownership changed to 2000 (as specified by fsGroup), ensuring that files written to those volumes are accessible by other containers or users in group 2000.

Conclusion

This was a few lines of configurations at the end. What took too much time for me to realize was that the hostPath volumes in Kubernetes do not respect the fsGroup, runAsGroup, or runAsUser settings when mounted. The Kubernetes securityContext, including fsGroup, does not change the ownership or permissions of files on hostPath volumes. This is because hostPath volumes directly mount directories from the host node’s filesystem, and Kubernetes does not modify the file ownership or permissions of the host’s file system when doing so.

Upgrading to Pydantic 2: More quirky than you would expect.

Mon, 27 Nov 2023 01:21:14 -0700

Bye bye class Config

The Pydantic V1 behavior to create a class called Config in the namespace of the parent BaseModel. Now, you need to add a class attribute call model_config.

class Schema(BaseModel):
    class Config:
        orm_mode = True

becomes

class Schema(BaseModel):
    model_config = ConfigDict(from_attributes=True)

Even the names of the attributes have changed. orm_mode is renamed to from_attributes.

Pydantic 2 does not automatically coerce to the type you want to

In Pydantic 1, it tried to coerce the type when it could. For example:

from pydantic import BaseModel


class Schema(BaseModel):
    path: str

m = Schema(path=123)
>>> m.path
'123'

In Pydantic 2:

>>> m.path
Traceback (most recent call last):
pydantic_core._pydantic_core.ValidationError: 1 validation error for Schema
path
  Input should be a valid string [type=string_type, input_value=123, input_type=int]
    For further information visit https://errors.pydantic.dev/2.5/v/string_type

You have to explicitly allow specific type conversions:

from pydantic import BaseModel, ConfigDict


class Schema(BaseModel):
    path: str
    model_config = ConfigDict(coerce_numbers_to_str=True)


m = Schema(path=123)
>>> m.path
'123'

Private field names don’t work with Pydantic 2

If you have field names in pydantic 1 that started with _, you will have to rename them to be public names.

For example I had private field names in our data such as _id, and _history that worked perfectly fine with Pydantic 1. Once upgraded to Pydantic 2, I tried to monkey patch Pydantic 2 so it reads those fields.

import pydantic

def is_valid_field_name(name: str) -> bool:
    """
    Monkey patching Pydantic's check for valid field names because it is a breaking change
    that is causing our code from Pydantic 1 not to work.
    """
    return not name.startswith('__')

pydantic._internal._fields.is_valid_field_name = is_valid_field_name
pydantic._internal._model_construction.is_valid_field_name = is_valid_field_name

However, patching worked for simple cases but there were edge cases that Pydantic was still throwing exceptions. The Pydantic writers recommend you to use the new PrivateAttr feature.

I found it to be useless. You can’t still set those attributes as you would with the rest of your model:

from datetime import datetime
from pydantic import BaseModel, PrivateAttr


class Schema(BaseModel):
    _processed_at: datetime


m = Schema(_processed_at=datetime.now())

>>> m._processed_at
KeyError: '_processed_at'
AttributeError: 'Schema' object has no attribute '_processed_at'

You have the option to use alias

from datetime import datetime
from pydantic import BaseModel, PrivateAttr, Field


class Schema(BaseModel):
    processed_at: datetime = Field(alias='_processed_at')


m = Schema(_processed_at=datetime.now())

In [15]: m.processed_at
Out[15]: datetime.datetime(2023, 11, 27, 12, 49, 4, 557375)

In [16]: m._processed_at
AttributeError: 'Schema' object has no attribute '_processed_at'

That means your original data can have a private field name. Once it is converted to the Pydantic model, the field name will change. That was not what exactly I wanted. What I needed was to keep the field names intact between our data and Pydantic objects.

After wrestling too many hours on this issue, I decided to change all private keys in my data to have the underline as a suffix instead of prefix. So in this case, the _processed_at became processed_at_.

Pydantic 2 is tricky when multiple types are allowed.

from pydantic import BaseModel
from typing import List, Union


class Schema(BaseModel):
    path: List[Union[int, str]]

In Pydantic 1:

>>> obj = Schema(path=['0', 'item'])
>>> obj
Schema(path=[0, 'item'])

In Pydantic 2, even though int is the first allowed type, it goes for the str type. Even if you set the coerce_numbers_to_str=True in the model_config.

>>> obj = Schema(path=['0', 'item'])
>>> obj
Schema(path=['0', 'item'])

Pydantic 2 Optional fields are defined very differently

In Pydantic 1, if you provided a default value for a field, it would become an optional field.

from pydantic import BaseModel, Field, ConfigDict


class Schema(BaseModel):
    history_id: int = None


m = Schema()
>>> m.id is None
True

In Pydantic 2:

Introducing Qluster - ETL As Spreadsheet

Tue, 17 Oct 2023 01:21:14 -0700

💥 Qluster

Qluster recently graduated from the USC Marshall School Of Business Incubator and the Columbia Business School Accelerator (CAVA).

Where it came from?

The insight came when I was running the Fair Search Team (RIP). We had just raised over $400 million in equity, burning cash like no other -WeWork excluded. The global search bar was the main interface to Fair’s car buying experience. There was a lot of pressure to deliver. Execs would put meetings on the calendar titled “Search is broken”. And I had to tell them: “Your search results are only as good as your data. Your data is broken, and that’s why your search is broken”. Since the search team was getting the heat, I had to fix data downstream in search when it should have been cleaned upstream at the ingestion point before we even had a chance to use it.

The core problem

The core problem was that those who had the context of the data, like the operations folks, didn’t have the tools to fix the issues themselves. They had to rely on the engineers to do their job for them.

Introducing Qluster

Fast forward to 2023, we built Qluster to allow non-engineers to fix moving data at scale. Fixing data at the point of ingestion - before it is used and consumed - dramatically decreases the cost to the business.

I am working with select number of clients as design partners. If you are interested in trying Qluster, please let me know!

DeepDiff Tutorial: Deep Commandline

Fri, 01 Jan 2021 11:21:14 -0700

DeepDiff 5.2.1 comes with the new commandline feature!

Get sample data

Let’s start by looking at the Google Analytics Sample dataset on Kaggle.

Here is a sample query taken from paultimothymooney’s notebook

Let’s run this for a few different months in 2017:

2017 04
2017 05
2017 06
2017 07

Let’s export the output of these queries for each month into individual csv files from Kaggle.

What has changed from month to month?

We can use the deep diff command to get the difference between any 2 files.

$ deep diff google_store_bounce_rate_2017_06.csv google_store_bounce_rate_2017_07.csv

{ 'values_changed': { "root[0]['bounce_rate']": { 'new_value': 51.55729166666667,
                                                  'old_value': 46.51783094609373},
                      "root[0]['source']": { 'new_value': 'google',
                                             'old_value': '(direct)'},
                      "root[0]['total_no_of_bounces']": { 'new_value': 19798.0,
                                                          'old_value': 17440.0},
                      "root[0]['total_visits']": { 'new_value': 38400,
                                                   'old_value': 37491},
                      "root[1]['bounce_rate']": { 'new_value': 43.265798602382986,
                                                  'old_value': 52.884508297137764},
                      "root[1]['source']": { 'new_value': '(direct)',
                                             'old_value': 'google'},
                      "root[1]['total_no_of_bounces']": { 'new_value': 8606.0,
                                                          'old_value': 9497.0},
                      "root[1]['total_visits']": { 'new_value': 19891,
                                                   'old_value': 17958},
                      "root[2]['bounce_rate']": { 'new_value': 66.72964887419303,
                                                  'old_value': 67.13819368879217},
                      "root[2]['total_no_of_bounces']": { 'new_value': 4238.0,
                                                          'old_value': 1234.0},
                      "root[2]['total_visits']": { 'new_value': 6351,
                                                   'old_value': 1838},
                      "root[3]['bounce_rate']": { 'new_value': 53.9553752535497,
                                                  'old_value': 57.738435179897195},
                        ...

Group by

Ok the above results are obviously not very useful. DeepDiff is comparing items line by line. Instead what we care is to compare the items based on the “source” field in the csv. We can use the new group_by

$ deep diff google_store_bounce_rate_2017_04.csv google_store_bounce_rate_2017_07.csv
--group-by source

{ 'dictionary_item_added': [root['facebook.com'], root['sites.google.com'], root['dfa']],
  'dictionary_item_removed': [root['t.co'], root['yahoo'], root['reddit.com']],
  'values_changed': { "root['(direct)']['bounce_rate']": { 'new_value': 43.265798602382986,
                                                           'old_value': 43.71677762295505},
                      "root['(direct)']['total_no_of_bounces']": { 'new_value': 8606.0,
                                                                   'old_value': 8471.0},
                      "root['(direct)']['total_visits']": { 'new_value': 19891,
                                                            'old_value': 19377},
                      "root['Partners']['bounce_rate']": { 'new_value': 52.348993288590606,
                                                           'old_value': 58.888076079005124},
                                                           ...

exclude-regex-paths

We are only interested in the bounce rate and rank changes in the above results. How can we limit the results to only what we want? One way of doing it is to use the exclude-regex-paths

$ deep diff google_store_bounce_rate_2017_04.csv 
google_store_bounce_rate_2017_07.csv --group-by source
--exclude-regex-paths total_visits --exclude-regex-paths total_no_of_bounces

{ 'dictionary_item_added': [root['dfa'], root['facebook.com'], root['sites.google.com']],
  'dictionary_item_removed': [root['t.co'], root['reddit.com'], root['yahoo']],
  'values_changed': { "root['(direct)']['bounce_rate']": { 'new_value': 43.265798602382986,
                                                           'old_value': 43.71677762295505},
                      "root['Partners']['bounce_rate']": { 'new_value': 52.348993288590606,
                                                           'old_value': 58.888076079005124},
                                                            ...

Great, the results are more useful now.

significant-digits

Some of these numbers are too close to each other. Maybe we don’t care if the bounce rate of (direct) changed from 43.71677762295505 to 43.265798602382986:

"root['(direct)']['bounce_rate']": {
    'new_value': 43.265798602382986,
    'old_value': 43.71677762295505}

But we care about Partners bounce rate changing almost 6%:

"root['Partners']['bounce_rate']": {
    'new_value': 52.348993288590606,
    'old_value': 58.888076079005124}

DeepDiff provides the significant-digits and also the new Math Epsilon features to solve this problem.

$ deep diff google_store_bounce_rate_2017_04.csv google_store_bounce_rate_2017_07.csv
--group-by source --significant-digits 0 --number-format-notation e 
--exclude-regex-paths total_visits --exclude-regex-paths total_no_of_bounces

{ 'dictionary_item_added': [root['sites.google.com'], root['dfa'], root['facebook.com']],
  'dictionary_item_removed': [root['reddit.com'], root['t.co'], root['yahoo']],
  'values_changed': { "root['Partners']['bounce_rate']": { 'new_value': 52.348993288590606,
                                                           'old_value': 58.888076079005124},
                      "root['google.com']['']": { 'new_value': 6,
                                                  'old_value': 7},
                      "root['google.com']['bounce_rate']": { 'new_value': 49.72826086956522,
                                                             'old_value': 43.90243902439025},
                      "root['m.facebook.com']['']": { 'new_value': 5,
                                                      'old_value': 6}}}

Great, we see only the most important changes. Google.com’s rank has changed from 7 to 6 in one month, the partners bounce rate has decreased 6% and m.facebook.com has improved in rank too!

significant-digits behind the scene

In order to understand how significant-digits work, we need to take a look at the number_to_string function.

Remember the (direct) bounce rate that was removed when we used --significant-digits 0 --number-format-notation e:

"root['(direct)']['bounce_rate']": {
    'new_value': 43.265798602382986,
    'old_value': 43.71677762295505}

>>> from deepdiff.helper import number_to_string
>>> number_to_string(43.265798602382986, significant_digits=1)
'43.3'
>>> number_to_string(43.265798602382986, significant_digits=0)
'43'
>>> number_to_string(43.71677762295505, significant_digits=1)
'43.7'
>>> number_to_string(43.71677762295505, significant_digits=0)
'44'

This function converts numbers into strings and “rounds” them based on the significant_digits that were passed. The significant_digits are the digits after the decimal point.

Note that with significant_digits=0, the 2 floating point numbers for (direct) bounce rate were rounded to 43 and 44 which means DeepDiff would have reported them in the diff results.

significant-digits + number-format-notation

That’s where the number-format-notation comes to play.

"root['(direct)']['bounce_rate']": {
    'new_value': 43.265798602382986,
    'old_value': 43.71677762295505}

>>> number_to_string(43.71677762295505, significant_digits=1)
'43.7'
>>> number_to_string(43.71677762295505, significant_digits=0,
number_format_notation='e')
'4e+1'
>>> number_to_string(43.71677762295505, significant_digits=1,
number_format_notation='e')
'4.4e+1'

By switching to the number_format_notation='e' we are switching to the “scientific notation”. And by setting the significant_digits=0, number_format_notation='e' we are removing 43.265798602382986 vs. 43.71677762295505 from the diff results:

$ deep diff google_store_bounce_rate_2017_04.csv google_store_bounce_rate_2017_07.csv
--group-by source --significant-digits 0 --number-format-notation e 
--exclude-regex-paths total_visits --exclude-regex-paths total_no_of_bounces

{ 'dictionary_item_added': [root['sites.google.com'], root['dfa'], root['facebook.com']],
  'dictionary_item_removed': [root['reddit.com'], root['t.co'], root['yahoo']],
  'values_changed': { "root['Partners']['bounce_rate']": { 'new_value': 52.348993288590606,
                                                           'old_value': 58.888076079005124},
                      "root['google.com']['']": { 'new_value': 6,
                                                  'old_value': 7},
                      "root['google.com']['bounce_rate']": { 'new_value': 49.72826086956522,
                                                             'old_value': 43.90243902439025},
                      "root['m.facebook.com']['']": { 'new_value': 5,
                                                      'old_value': 6}}}

Which months are the most similar?

Now that we know how to diff the files to get the results we need, let’s ask ourselves the next interesting questions. Which months are the most similar?

DeepDiff provides a solution to this question via Deep Distance

Deep Distance is the distance between 2 objects. It is a floating point number between 0 and 1. Deep Distance in concept is inspired by Levenshtein Edit Distance.

At its core, the Deep Distance is the number of operations needed to convert one object to the other divided by the sum of the sizes of the 2 objects capped at 1. Note that unlike Levensthtein Distance, the Deep Distance is based on the number of operations and NOT the “minimum” number of operations to convert one object to the other. The number is highly dependent on the granularity of the diff results. And the granularity is controlled by the parameters passed to DeepDiff.

The deep diff commandline gives you the deep distance when you pass the --get-deep-distance flag. For example let’s get the deep distance between months 4 and 7:

$ deep diff google_store_bounce_rate_2017_04.csv google_store_bounce_rate_2017_07.csv
--group-by source --significant-digits 0 --number-format-notation e 
--exclude-regex-paths total_visits --exclude-regex-paths total_no_of_bounces
--get-deep-distance

{ 'deep_distance': 0.16831683168316833,
  'dictionary_item_added': [root['facebook.com'], root['dfa'], root['sites.google.com']],
  'dictionary_item_removed': [root['reddit.com'], root['t.co'], root['yahoo']],
  'values_changed': { "root['Partners']['bounce_rate']": { 'new_value': 52.348993288590606,
                                                           'old_value': 58.888076079005124},
                      "root['google.com']['']": { 'new_value': 6,
                                                  'old_value': 7},
                      "root['google.com']['bounce_rate']": { 'new_value': 49.72826086956522,
                                                             'old_value': 43.90243902439025},
                      "root['m.facebook.com']['']": { 'new_value': 5,
                                                      'old_value': 6}}}

month 04 distance from month 07 is 0.168

What about month 05 from 07?

$ deep diff google_store_bounce_rate_2017_05.csv google_store_bounce_rate_2017_07.csv
--group-by source --significant-digits 0 --number-format-notation e 
--exclude-regex-paths total_visits --exclude-regex-paths total_no_of_bounces
--get-deep-distance

{ 'deep_distance': 0.1188118811881188,
  'dictionary_item_added': [root['facebook.com'], root['sites.google.com']],
  'dictionary_item_removed': [root['groups.google.com'], root['t.co']],
  'values_changed': { "root['Partners']['bounce_rate']": { 'new_value': 52.348993288590606,
                                                           'old_value': 56.18830675778283},
                      "root['analytics.google.com']['']": { 'new_value': 3,
                                                            'old_value': 2},
                      "root['m.facebook.com']['bounce_rate']": { 'new_value': 64.27503736920778,
                                                                 'old_value': 69.74789915966386},
                      "root['youtube.com']['']": { 'new_value': 2,
                                                   'old_value': 3}}}

month 05 distance from month 07 is 0.118

And month 06 from 07?

$ deep diff google_store_bounce_rate_2017_06.csv google_store_bounce_rate_2017_07.csv
--group-by source --significant-digits 0 --number-format-notation e 
--exclude-regex-paths total_visits --exclude-regex-paths total_no_of_bounces
--get-deep-distance

{ 'deep_distance': 0.08415841584158416,
  'dictionary_item_added': [root['facebook.com']],
  'dictionary_item_removed': [root['baidu']],
  'values_changed': { "root['(direct)']['']": {'new_value': 1, 'old_value': 0},
                      "root['(direct)']['bounce_rate']": { 'new_value': 43.265798602382986,
                                                           'old_value': 46.51783094609373},
                      "root['Partners']['bounce_rate']": { 'new_value': 52.348993288590606,
                                                           'old_value': 61.23381568926123},
                      "root['analytics.google.com']['bounce_rate']": { 'new_value': 53.9553752535497,
                                                                       'old_value': 57.738435179897195},
                      "root['google']['']": {'new_value': 0, 'old_value': 1},
                      "root['google.com']['']": { 'new_value': 6,
                                                  'old_value': 5},
                      "root['m.facebook.com']['']": { 'new_value': 5,
                                                      'old_value': 6}}}

month 06 distance from month 07 is 0.084

So the most similar month to 07 is the month before it!

Example 2

Let’s take a look at another sample data. We have the following csv files:

t1.csv

first_name	last_name	zip
Joe	Nobody	90011
Jimmy	Brown	90007
Sara	Smith	90007

t2.csv

first_name	last_name	zip
Joe	Nobody	90011
Jimmy	Brown	90404
Sara	Smith	90007

$ deep diff t1.csv t2.csv
{ 'values_changed': { "root[1]['zip']": { 'new_value': '90404',
                                          'old_value': '90007'}}}

As you can see here, Jimmy has moved from downtown LA (90007) to Santa Monica (90404).

Let’s take a look at t3.csv now:

first_name	last_name	zip
Joe	Nobody	90011
Sara	Smith	90007
Jimmy	Brown	90404

When we run the diff command:

$ deep diff t1.csv t3.csv
{ 'values_changed': { "root[1]['first_name']": { 'new_value': 'Sara',
                                                 'old_value': 'Jimmy'},
                      "root[1]['last_name']": { 'new_value': 'Smith',
                                                'old_value': 'Brown'},
                      "root[1]['zip']": { 'new_value': '90007',
                                          'old_value': '90007'},
                      "root[2]['first_name']": { 'new_value': 'Jimmy',
                                                 'old_value': 'Sara'},
                      "root[2]['last_name']": { 'new_value': 'Brown',
                                                'old_value': 'Smith'},
                      "root[2]['zip']": { 'new_value': '90404',
                                          'old_value': '90007'}}}

The results don’t look very useful. It has literally compared the rows line by line.

Ignore Order

Perhaps if we don’t care about the order of the rows we can get more useful results? We can do that by passing --ignore-order.

$ deep diff t1.csv t3.csv --ignore-order
{ 'values_changed': { "root[1]['zip']": { 'new_value': '90404',
                                          'old_value': '90007'}}}

Repeats and ignore order

Here we have a repeated line in t4.csv

first_name	last_name	zip
Joe	Nobody	90011
Jimmy	Brown	90404
Sara	Smith	90007
Jimmy	Brown	90404

Note that the repeat won’t make it into the diff results when passing --ignore-order.

$ deep diff t1.csv t4.csv --ignore-order
{ 'values_changed': { "root[1]['zip']": { 'new_value': '90404',
                                          'old_value': '90007'}}}

Patch

DeepDiff allows you to create “patches” just like Linux patch command. Note that the commandline “patch” is only a subset of what DeepDiff’s Delta’s Python API provides.

We create the patch by passing --create-patch. Also since we are ignoring the order of the rows, we need to pass --report-repetition to makes sure the repetitions are part of the “diff” results in the patch.

$ deep diff t1.csv t4.csv --ignore-order --report-repetition --create-patch > patch1

Remember that the diff between those 2 files was that row one’s zip has changed to 90404.

As an example let’s have t1new.csv

first_name	last_name	zip
Joe	Nobody	90011
Jimmy	Brown	90007
Sara	Smith	90007
John	Doe	90001
Max	Foo	23232

Let’s create one more patch file!

$ deep diff t1.csv t1new.csv --ignore-order --report-repetition --create-patch > patch2

So we have:

patch1 is the delta of t1.csv and t4.csv
patch2 is the delta of t1.csv and t1new.csv

Applying the patches

The deep patch command has a very similar interface to Linux’s patch command:

$ deep patch t1new.csv patch1 --backup

first_name	last_name	zip
Joe	Nobody	90011
Jimmy	Brown	90404
Sara	Smith	90007
John	Doe	90001
Max	Foo	23232

Awesome, it updated Jimmy’s zip code to be 90404 in this t1new.csv file.

We can reuse the same patch file on t1:

$ deep patch t1.csv patch1 --backup

t1.csv after applying patch1: Jimmy’s zipcode in this file is updated too!

first_name	last_name	zip
Joe	Nobody	90011
Jimmy	Brown	90404
Sara	Smith	90007

$ deep patch t1.csv patch2 --backup

t1.csv is now effectively the same as t1new.csv!

first_name	last_name	zip
Joe	Nobody	90011
Jimmy	Brown	90404
Sara	Smith	90007
John	Doe	90001
Max	Foo	23232

Patches can be applied to any objects!

some.yaml

---

-
    first_name: Joe
    last_name: Nobody
    address: 3232 Main st.
    phone: 323-123-2345
    zip: 90011
-
    first_name: Jimmy
    last_name: Brown
    address: 11th There sq.
    phone: 111-123-9911
    zip: 90002
-
    first_name: Sara
    last_name: Smith
    address: Downtown LA
    phone: 818-113-2005
    zip: 90007

Yes we can patch this yaml file from the patch file we made between 2 csv files!

$ deep patch some.yaml patch1 --backup

- address: 3232 Main st.
  first_name: Joe
  last_name: Nobody
  phone: 323-123-2345
  zip: 90011
- address: 11th There sq.
  first_name: Jimmy
  last_name: Brown
  phone: 111-123-9911
  zip: 90404
- address: Downtown LA
  first_name: Sara
  last_name: Smith
  phone: 818-113-2005
  zip: 90007

As you can see the formatting of the yaml file is changed. This is due to the fact that DeepDiff loads the file into a Python dictionary, modifies it and then writes it back to disk. During this operation, the file loses its original formatting.

$ deep patch some.yaml patch2 --backup

The patch2 adds new lines to the yaml file:

- address: 3232 Main st.
  first_name: Joe
  last_name: Nobody
  phone: 323-123-2345
  zip: 90011
- address: 11th There sq.
  first_name: Jimmy
  last_name: Brown
  phone: 111-123-9911
  zip: 90404
- address: Downtown LA
  first_name: Sara
  last_name: Smith
  phone: 818-113-2005
  zip: 90007
- first_name: John
  last_name: Doe
  zip: '90001'
- first_name: Max
  last_name: Foo
  zip: '23232'

What about applying to a python object?

As noted, the deep patch command is the commandline interface for what DeepDiff calls Delta.

from deepdiff import DeepDiff, Delta


class CustomClass:
    def __init__(self, a, b=None):
        self.a = a
        self.b = b

    def __str__(self):
        return "Custom({}, {})".format(self.a, self.b)
    __repr__ = __str__

t1 = CustomClass(a=10, b=10)
t2 = CustomClass(a=10, b=12)

The Delta object can be applied to Python objects:

>>> diff = DeepDiff(t1, t2)
>>>
>>> delta = Delta(diff)
>>>
>>> t1
Custom(10, 10)
>>> t2
Custom(10, 12)
>>> t3 = t1 + delta
>>> t3
Custom(10, 12)

The “patch” files are essentially Delta dictionaries that are serialized via Python’s pickle. You can apply those patches to any objects! From python built-in types to custom classes, Numpy arrays and more. The commandline interface let’s you create and use patch from csv, json, toml, yaml and pickle files!

Looking for something in the file?

The deep grep is used to grep through the files. For example let’s look for sara in the above yaml file:

$ deep grep sara some.yaml -i
{'matched_values': ["root[2]['first_name']"]}

We get the exact path to the item[s] that have sara in them instead of just the line number[s]!

Extracting a value of a path?

The deep extract allows you to extract values from path’s in a file:

$ deep grep sara some.yaml -i
{'matched_values': ["root[2]['first_name']"]}

$ deep extract "root[2]" some.yaml
{ 'address': 'Downtown LA',
  'first_name': 'Sara',
  'last_name': 'Smith',
  'phone': '818-113-2005',
  'zip': 90007}

Conclusion

The new commandline interface to DeepDiff provides more accessible ways to directly deal with files from within the shell:

DeepDiff
- $ deep diff --help
Delta
- $ deep patch --help
grep
- $ deep grep --help
extract
- $ deep extract --help

Hope you will enjoy using the deep commandline!

Sep Dehpour

A How To Guide: Remotely Accessing Minikube Kubernetes on KVM

Fri, 25 Sep 2020 01:21:14 -0700

Note: After writing this article I talked to the maintainer of Minikube and turns out the recommended way to remotely access Minikube is to do SSH tunneling. In fact kubectl port forwarding and everything works fine over the ssh tunnel. For more details visit https://github.com/kubernetes/minikube/issues/4733

If you are like me, you have some older hardware sitting around catching dust. The other day I decided to install a lightweight Kubernetes cluster on an Ubuntu 20 box.

There are a few options when it comes to choosing a lightweight Kubernetes distributions:

Minikube You will have many virtualization options.
kind Similar to Minikube but only runs on Docker.
k3s built for IoT & Edge computing.
k3d K3d is K3s in Docker.

I’m sure there are even more options. These all sound fantastic. However, most of them run on Docker. The problem with Docker on Linux is that you have to run it as the root user (at least that is true at the time of writing this article.). That exposes you to a lot of security risks.

Out of all these options, Minikube supports many drivers from Virtualbox, KVM to Docker. You can even run Minikube without a driver, skipping the Virtual Machine creation.

The None driver obviously could cause a lot of damage and should be avoided. That leaves us with KVM and Virtualbox.

I have to tell you that you will have a much smoother experience with Virtualbox when it comes to remotely accessing Minikube. But Virtualbox and Minikube were not playing nicely together on my Ubuntu box. There were times that Virtualbox was using 200% CPU while just sitting idle. See here for an example of somebody complaining about this issue.

Our Setup

Our setup will be the Ubuntu 20 LTS box, pc1 which will host a Minikube cluster and a separate computer called the pc2. We want to use kubectl from pc2 to hit the cluster on pc1 just as if we were using kubectl to hit a Kubernetes cluster on the cloud.

1. Install Minikube, kubectl, and KVM on the host

Follow the official docs

2. Run Minikube with explicit apiserver ip on the host

Find your computer’s ip address. Here 192.168.1.121 is the ip of the host that is running minikube and we are giving 8GB memory and 8 CPUs to the kubernetes cluster.

minikube start --vm-driver kvm2 --memory 8112 --cpus 8 --apiserver-ips=192.168.1.121

3. Port Forward

Note: All these steps were needed because I was using KVM as the Minikube driver. If you are using Virtualbox, it is just a click of the button to add the port forwarding in the network section of the settings.

Unless you understand libvirt, forget about port forwarding via kvm/iptables. I will never get back the hours I wasted trying. I ended up using the good old Nginx for port forwarding.

3.a Install Nginx

$ sudo apt update
$ sudo apt install nginx

3.b Reverse Proxy via NGINX

Make NGINX do the reverse proxy. The important part here is that Minikube has its own SSL certificates. So we want NGINX to only pass through the traffic and not deal with SSL and the certs. Fortunately the more recent versions of NGINX allow you to do that via the stream directive.

Unlike the server directive, I could only add the stream directive directly to the nginx.conf

stream {
  server {
      listen 192.168.1.121:51999;
      #TCP traffic will be forwarded to the specified server
      proxy_pass 192.168.39.131:8443;       
  }
}

So in this example the ip of the host of minikube is 192.168.1.121 and the port we want to expose to the world is 51999.

You can find the ip of the Minikube instance in your ~/.kube/config file. The port should be 8443.

Before we do anything, let’s test the Nginx settings:

$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

And restart the nginx:

$ sudo systemctl restart nginx

3.c Open the port on the firewall

My Ubuntu box’s IP address on the local network is 192.168.1.121. We want to let the local network be able to reach this box on port 51999.

We are gonna run:

$ sudo ufw allow from 192.168.1.0/24 to any port 51999

And then run $ sudo ufw status to make sure that UFW (Ubuntu Firewall) is enabled. If not, please enable it UFW! You will probably want to add some rules to allow ssh to the host.

3.d Test from local

Let’s first of all make sure that we have access to minikube.

$ kubectl get pods
No resources found in default namespace.

In this case I don’t have any pods already running. But we know that kubectl is working fine just from that response. Let’s load the ~/.kube/config and make sure that the Nginx reverse proxy works.

Change the server line to use the host ip and the port we just opened:

apiVersion: v1
clusters:
- cluster:
    certificate-authority: ../.minikube/ca.crt
    # server: https://192.168.39.131:8443  <-- this was the previous value there
    server: https://192.168.1.121:51999
  name: minikube
contexts:
- context:
    cluster: minikube
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate: ../.minikube/profiles/minikube/client.crt
    client-key: ../.minikube/profiles/minikube/client.key

Now run $ kubectl get pods again. You should get exact the same response as before.

4. Certs

Create a ~/.minikube_remote on pc2 and run the following commands. This is using scp which assumes the ssh configs are setup for your remote machine to access the host. If you don’t want to use scp, you can just use a USB to copy these files.

mkdir ~/.minikube_remote
cd ~/.minikube_remote
scp pc1:.minikube/ca.crt .
scp pc1:.minikube/profiles/minikube/client.crt .
scp pc1:.minikube/profiles/minikube/client.key .

cat <<EOF > ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority: ../.minikube_remote/ca.crt
    server: https://192.168.1.121:51999
  name: minikube
contexts:
- context:
    cluster: minikube
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate: ../.minikube_remote/client.crt
    client-key: ../.minikube_remote/client.key
EOF

5. Remote Access

Now you should be able to use the kubectl on the pc2 to remote access the minikube on pc1! The best part about this is that kubectl port-forward should work as normal too in case you need to access services in your cluster.

Conclusion

It is always fun to distribute the load from your machine to other pieces of hardware you may have. If you want to even get more serious, you can deploy a “real” kubernetes cluster on your machines. You may want to use a bare metal hypervisor, learn kops or even use a few old machines.

Have fun!

DeepDiff: 1.7 Million Downloads A Month

Thu, 25 Jun 2020 01:21:14 -0700

It is June 2020 and I have some exciting news to share.

DeepDiff has surpassed 1.7 million downloads a month!

Notable DeepDiff Users

The percentage of downloads per Python version

Python version	percent	downloads
3.8	38.99%	667,561
3.6	29.98%	513,385
3.7	11.49%	196,773
2.7	9.78%	167,391
3.5	9.26%	158,587
Total		1,712,177

Date range: 2020-05-01 - 2020-05-31

Python 3 downloads of DeepDiff have taken over

(Click on any image to zoom in)

DeepDiff had a sudden drop in Python 2 users

DeepDiff 4 stopped Python 2 support and we had a sudden dip in Python 2 downloads.

The majority of downloads are in Linux

Windows is still a small chunk of downloads:

DeepDiff 5 Is Here!

Mon, 22 Jun 2020 11:21:14 -0700

DeepDiff 5 is finally here!

Delta Object

The Delta object is introduced.

DeepDiff Delta is a directed delta that when applied to t1 can yield t2 where delta is the difference between t1 and t2. The Delta objects are like git commits but for structured data. You can convert the diff results into Delta objects, store the deltas, and later apply to other objects.

Example:


>>> t1 = [1, 2, [3, 5, 6]]
>>> t2 = [2, 3, [3, 6, 8]]

>>> diff = DeepDiff(t1, t2, ignore_order=True, report_repetition=True)
>>> diff
{'values_changed': {'root[0]': {'new_value': 3, 'old_value': 1}, 'root[2][1]': {'new_value': 8, 'old_value': 5}}}
>>> delta = Delta(diff)
>>> delta
<Delta: {'values_changed': {'root[0]': {'new_value': 3}, 'root[2][1]': {'new_value': 8}}}>

>>> t1 + delta == t2
True

Note that we can apply delta to objects different than the original objects they were made from:

>>> t3 = ["a", 2, [3, "b", "c"]]
>>> t3 + delta
[3, 2, [3, 8, 'c']]

And it comes with Numpy support:

>>> from deepdiff import DeepDiff, Delta
>>> import numpy as np
>>> t1 = np.array([1, 2, 3, 5])
>>> t2 = np.array([2, 2, 7, 5])
>>> diff = DeepDiff(t1, t2)
>>> delta = Delta(diff)
>>> delta + t1
array([2, 2, 7, 5])
>>> delta + t2 == t2
array([ True,  True,  True,  True])

There is way more to Delta from serialization for storing Delta to other details.

Deep Distance

The concept of Deep Distance is introduced.

Deep Distance is the distance between 2 objects. It is a floating point number between 0 and 1. Deep Distance in concept is inspired by Levenshtein Edit Distance.

>>> from deepdiff import DeepDiff
>>> DeepDiff(10.0, 10.1, get_deep_distance=True)
{'values_changed': {'root': {'new_value': 10.1, 'old_value': 10.0}}, 'deep_distance': 0.0014925373134328302}
>>> DeepDiff(10.0, 100.1, get_deep_distance=True)
{'values_changed': {'root': {'new_value': 100.1, 'old_value': 10.0}}, 'deep_distance': 0.24550408719346048}
>>> DeepDiff(10.0, 1000.1, get_deep_distance=True)
{'values_changed': {'root': {'new_value': 1000.1, 'old_value': 10.0}}, 'deep_distance': 0.29405999405999406}
>>> DeepDiff([1], [1], get_deep_distance=True)
{}
>>> DeepDiff([1], [1, 2], get_deep_distance=True)
{'iterable_item_added': {'root[1]': 2}, 'deep_distance': 0.2}
>>> DeepDiff([1], [1, 2, 3], get_deep_distance=True)
{'iterable_item_added': {'root[1]': 2, 'root[2]': 3}, 'deep_distance': 0.3333333333333333}
>>> DeepDiff([[2, 1]], [[1, 2, 3]], ignore_order=True, get_deep_distance=True)
{'iterable_item_added': {'root[0][2]': 3}, 'deep_distance': 0.1111111111111111}

Improved granularity of results when ignore_order=True

>>> from pprint import pprint
>>> from deepdiff import DeepDiff
>>> t1 = [
...     {
...         'key3': [[[[[1, 2, 4, 5]]]]],
...         'key4': [7, 8],
...     },
...     {
...         'key5': 'val5',
...         'key6': 'val6',
...     },
... ]
>>> 
>>> t2 = [
...     {
...         'key5': 'CHANGE',
...         'key6': 'val6',
...     },
...     {
...         'key3': [[[[[1, 3, 5, 4]]]]],
...         'key4': [7, 8],
...     },
... ]

In DeepDiff 4:

>>> pprint(DeepDiff(t1, t2, ignore_order=True))
{'iterable_item_added': {'root[0]': {'key5': 'CHANGE', 'key6': 'val6'},
                         'root[1]': {'key3': [[[[[1, 3, 5, 4]]]]],
                                     'key4': [7, 8]}},
 'iterable_item_removed': {'root[0]': {'key3': [[[[[1, 2, 4, 5]]]]],
                                       'key4': [7, 8]},
                           'root[1]': {'key5': 'val5', 'key6': 'val6'}}}

In DeepDiff 5:

>>> pprint(DeepDiff(t1, t2, ignore_order=True, cache_size=5000, cutoff_intersection_for_pairs=1))
{'values_changed': {"root[0]['key3'][0][0][0][0][1]": {'new_value': 3,
                                                       'old_value': 2},
                    "root[1]['key5']": {'new_value': 'CHANGE',
                                        'old_value': 'val5'}}}

Pretty print

Use the pretty method for human readable output regardless of what view you have used to generate the results.

>>> from deepdiff import DeepDiff
>>> t1={1,2,4}
>>> t2={2,3}
>>> print(DeepDiff(t1, t2).pretty())
Item root[3] added to set.
Item root[4] removed from set.
Item root[1] removed from set.

New Optimizations

Many new optimizations are introduced, especially when dealing with nested data structures, numeric lists and, Numpy arrays.

Read about optimizations here.

Caching

Caching can dramatically improve the performance for nested objects especially when ignore_order=True.

For example, lets take a look at the performance of the benchmark_deeply_nested_a in the DeepDiff-Benchmark repo.

Without any caching it takes 10 seconds to do the diff!

And with caching it takes under a second:

Improved Numpy Support

Previously, DeepDiff barely supported Numpy. DeepDiff 5 comes with a much more comprehensive support of Numpy.

For example, a sample diff with numbers took up to 30 seconds without the optimizations:

And 5 seconds with Numpy optimizations:

Conclusion

DeepDiff 5 comes with many new features and improvements. Please star it on github if you find it useful.

I would like to thank everybody who helped this release possible from creating PR’s, to beta testings, and providing feedback.

Nathaniel Brown Adds support for datetime.time
Michał Karaś for the pretty view
Christian Kothe for the basic support for diffing numpy arrays
Timothy for truncate_datetime
laike9m for beta testing
David Herman for beta testing
Wlad for beta testing

Faster, cheaper, and better: A story of breaking a monolith

Wed, 17 Jul 2019 01:21:14 -0700

Intro

Disclaimer: Fair is Sep Dehpour’s employer at the time of writing this article.

If you don’t know about Fair, we are a vehicle subscription app. So when it came to naming the monolith that was dealing with vehicles, it was a no-brainer to call it vehicle-service. As the team grew, soon the on-boarding, testing, and development became slow and painful. We needed to release search updates independently and changes were frequent. But we could only move as fast as the slowest gear, the monolith itself.

TL;DR:

On May 21st 2019, we broke search service out of vehicle-service. It was faster, cheaper, and better!

Faster

The same API in the new service is around 4x times faster. It is so much faster that it caused the anomaly detector to trigger alerts!

Not to mention, it is faster to setup, develop, and run tests: tests run 10x faster!

Cheaper

It is also cheaper to run. We estimate that it is going to save us hundreds of thousands of dollars³ annually in infrastructure costs!

Better

Last but not the least, it resulted in better code quality, monitoring, and bug fixing. All issues are now immediately noticed and resolved.

Below is an example of errors in the span of 24 hours in vehicle-service on the left vs. search-service on the right. Search errors on the monolith could easily get lost in the sea of other errors.

Monolithic vs. Micro-Services Architecture

There are trade-offs between using a monolithic ¹ vs. a micro-services architecture.

The main benefits of a monolithic architecture are:

Shared source code
Shared responsibility
Simple architecture
Simple to deploy

The main benefits of a micro-services architecture are:

Team ownership of source code (assuming not monolithic repo)
Independent releases
Easier for on-boarding new engineers
Scale micro-services independently

Some companies do really well with monoliths. Some don’t. Almost everybody starts with the monolithic architecture though. At the early stages of a project, the team is small and the problem domains may not be clear yet. So it is usually over-engineering to start with micro-services. But at some point the monolith becomes too big and slow to change, especially if the culture of the company rewards doing shortcuts that bring immediate value to the business. The code quality starts to degrade at the expense of long-term sustainability. Technical debt sky rockets. That’s when engineers start complaining about the monolith and propose projects to “break the monolith”. However, there is a non-trivial amount of work to separate a monolith into micro-services. It is usually underestimated.

What makes a good break

After going through a number of monolith breakup projects at various places, I have come to the conclusion that you need to:

Have the right reasons to break the monolith. Constantly evaluate the cost of the separation against “fixing” the monolith.

We decided to move into micro-services architecture at Fair in order to give each team the “ownership” of the service they were working on, reduce blockers, move faster, and release independently.
Decouple the right functionalities: what changes frequently and needs to move faster.

We started breaking out the 2 major user facing functionalities: search and auto-complete. The rest of this article focuses on search. You can read about auto-complete here.
Be committed to decoupling. Don’t share a database between services even if it means some duplicate data.
While it is generally a bad idea to increase the scope of a project that is already non-trivial, don’t just move the problems from one code base to another. The new service needs to have an immediate tangible impact both for the end user and the business in order to pave the way for other breakouts from the monolith.

Different ways to break

There are generally 4 ways to break:

Rewrite a new service from scratch
Clone the original monolith and trim the code.
Grow the new service inside the monolith and then separate.
Copy paste all the relevant pieces of code + tests and refactor.

As an engineer, it is always intriguing to start a new service from scratch. We did actually spend a few weeks down this road before coming to realization that it was impractical. As Joel Spolsky puts it, starting from scratch is the single worst strategic mistake that any software company can make.

The monolith was too big. Cloning and trimming it would have taken too long. What we ended up doing was option 4: Copy pasting all the relevant pieces of code and refactor.

Hint: make sure you have good test coverage in the monolith before you start. Once you copy the tests to the new service, resist all urges to modify the tests. You can either refactor the code or the tests at a time but not both!

Part A: Re-architecture

Monolith Architecture

Here is a simplified diagram of the parts of the monolith related to search:

Notice how RDS and Elasticsearch were intertwined for all apis. Any performance issues of the APIs or the Relational Database (RDS) would propagate to search. If the RDS timed out, if the IDs were running out, or if our own inventory processing workers had scaled up beyond the capacity of the RDS, the end user would have experienced a slow search.

Search Service Architecture

Since the RDS was one of the big obstacles in scaling search, we removed it in the search service. The Elasticsearch data needed to have everything that search would have utilized. That effectively made Elasticsearch the source of truth.

Monolith to publish, Search to subscribe

In order to make sure the Elasticsearch in search had all the data needed while giving us the flexibility to iterate on the new service, turn the new service off, turn it back on, and still catch up with the changes, we went with the Pubsub pattern. That way the monolith and search service did not even need to know about each other and we could achieve true decoupling.

We used the AWS SQS (unordered queue) but even better if you use something like Kafka.

Part B: Catching up with a moving target

The monolith is a moving target. New features are constantly being developed and we could not slow down the development just so that the search service could catch up.

One could say, just develop the micro service as fast as you can to catch-up. That’s easy in theory. Maybe you are lucky and can have “code freezes”. We don’t have that luxury.

The following helped with catching search up to the monolith functionalities:

Traffic Mirroring/Shadowing

We had all search traffic mirrored to hit both the monolith and the micro-service in all environments (even prod). That way we could develop, deploy, monitor, and measure search service’s performance and functionalities. Most importantly, doing so would build the confidence that the micro-service works as well if not better than the monolith.

The prerequisites to traffic mirroring are:

Making sure that the data storages are truly separate and there is no chance of impacting the critical path of the monolith.
Keeping the APIs consistent between the monolith and the micro-service. From the perspective of the client of the API, only the service name and nothing else should change.
Have a way to mirror the traffic.

Traffic Mirroring via Service Mesh

A great way to do traffic mirroring is to use a service mesh layer. For example, mirroring in Istio.

However, at the time of separating the search service, we did not have a service mesh layer in our Kubernetes cluster. Therefore mirroring via service mesh was not a feasible option.

Traffic Mirroring via UDP + Queue

Since we did not have the option of using the service mesh at the time, we resolved to creating a server to handle traffic mirroring. Once a request was received by the monolith, the request plus the monolith’s response was sent to this simple server via UDP. Then this information was put on a queue. A queue worker would pop the request and make the same request to the micro-service and compare the results between the monolith and the micro-service:

Why UDP?

Acknowledge packet (ACK) does not exist in UDP. We did not need acknowledgments when mirroring.
We did not care if some UDP packets were dropped.
Most importantly, under no circumstances should the performance of the “receiver” (simple mirroing service)have affected the monolith.

UDP and network congestion

Using UDP had its own slight risk of UDP taking over the network traffic if we ran into network congestion. We decided that the risk was minimal and moved forward with using UDP. (UDP lacks the TCP congestion control.)

New features developed in the micro-service before the monolith

It also helped to develop new features and business requirements first in the micro-service and then in the monolith. This way both the monolith and the micro-service needed to catch up with each other instead of one always chasing the other.

Part C: Refactor and Optimize before switching all the traffic

In our case we focused on 2 areas to deliver value:

A. Elasticsearch Optimizations

The main areas of focus for the refactor were:

1. Flatten the data

The lack of limitation is the enemy of data structure²

Arguably one of the strengths and at the same time a weakness of Elasticsearch is that you don’t have to design a schema. You can literally shove in anything in json format and let Elasticsearch handle it. That’s great for something like holding the logs in the ELK stack. But when it comes to using Elasticsearch in an application, it is a different story. That lack of limitation gives you a lot of freedom in the short-term, but it will bite you when you need to scale your application.

Thanks to this freedom, all sorts of data was shoved into Elasticsearch in our monolith, whether it was used for querying or not. Worst of all the data was deeply nested.

Elasticsearch under the hood is using Lucene to index data. Lucene indexes are flat. So when you have a nested data structure, it creates hidden separate indexes per level of the nested data. If you are doing a nested query, it will quickly get ugly and slow.

Flattening the data structure arguably contributed the most to reducing our APIs latency by 4x.

In order to flatten the data, we used “name spacing” to move the nested fields to the root of the document. For example:

{
    "foo": {
        "bar": ""
    }
}

became: {"foo__bar": ""}

We went from a nested data-structure with a max depth of 5 to a flat data-structure with a depth of one.

This is what the nested data-structure in the monolith’s Elasticsearch looked like:

(A screen-shot of the Elasticsearch mapping rotated 90 degrees to fit this page horizontally.)

And once it was flattened:

Do you see how simpler it looks? And there are less fields. Which brings us to point 2:

2. Keep superfluous information in binary form

Remove any fields that don’t need to be searchable. Put them all into one binary blob field.

For example, in our case the “single vehicle detail page” had way more fields such as image links and other information that did not need to be searchable or visible in the “list view”. By putting all those fields into a binary blob, we could efficiently store and retrieve them just for the “detail view”.

3. Don’t leave doc_values and index values to defaults

Use the correct Elasticsearch attributes for the field. Ask yourself the following questions:

Do I ever need to query this field?

If yes, then keep it index=true. Otherwise index=false. By having it as index=false, it won’t be added to Lucene’s inverted index.
Do I ever need to sort by this field or run aggregations on it, or access it through a script?

If yes, make the doc_value=true. Otherwise doc_value=false. This one specifically will save you disk space by setting it to false.

4. Index only if new or changed.

Avoid indexing something that is already indexed. If nothing has changed, don’t index it again. This is where we used DeepDiff’s DeepHash to generate a Murmur3 hash of the Elasticsearch payloads. Before inserting any new document, we could compare the hash of the old vs. new document along with the timestamp of the document to determine if anything needed to be indexed. You might wonder why “timestamp” is needed in addition to the hash. The reason is that we use the un-ordered AWS SQS queue. So messages will not be received in the order they were sent. You only want the latest “state”.

5. Remove unnecessary abstractions

We used the Elasticsearch-DSL library to give us a nice object oriented interface into Elasticsearch. While this library is written with good intentions, if you are doing anything serious with Elasticsearch, I highly discourage you from using this library. Both at my current job at Fair and at my previous job at Chownow, I got rid of this library and switched to Elasticsearch-Py which is a lower level library. Both libraries are brought to you by Elastic, the company behind Elasticsearch. But the unnecessary abstractions that the higher level library creates to shove things into an object oriented interface comes at a high cost: Bugs are prevalent, many Elasticsearch features are unsupported, and you need to learn a new object oriented query language when dealing with Elasticsearch.

Switching to the lower level library also meant our code became more “functional” which made testing way easier.

6. Optimize the number of shards, replicas and indices

Depending on your use case, you may want to optimize these settings for reads, writes, memory, cpu usage or a combination of them. In Elasticsearch, pay attention to the number of shards, replicas, and indices.

Aiming for faster reads in the micro-service, we reduced the number of shards per index to one, increased the number of replicas and more importantly reduced the number of indices by 1000x relative to the monolith.

B. Unittest optimizations

One of the bottlenecks in the monolith dev work was how slow tests were running. While you don’t want to refactor the tests themselves during the breaking, you may find ways to increase the tests’ speed. In our case, we were loading fixtures into Elasticsearch and RDS more often than necessary. As an optimization, when tests were not modifying the state and could use the same data in Elasticsearch, we grouped them together and loaded the data once. That alone made the tests run 10x faster.

Part D: Final Architecture: KISS

And finally, this is our simplified high level architecture that is relevant to this article. As they say, Keep It Simple Stupid!

Part E: Stay Committed and finish the breakout!

Separating a monolith into micro-services is non-trivial. No matter what path you take, expect to discover issues and dependencies that will increase the scope of the project. Throughout the breakout, there will be questions about whether to continue with the break or focus resources elsewhere to bring immediate value to the company. Don’t forget: It’s a Marathon not a sprint. Be prepared. Start by breaking smaller user facing functionalities and make them clear winners on all fronts first. That paves the way to break more complicated components later on.

Good luck!

Sep Dehpour

To discuss this article please visit Hacker News

Footnotes

1: Monolith Service
We are talking about monolith service not monolith code as they do in Google
2: The lack of limitation is the enemy of data structure
That’s originally a quote by Orson Welles that I took the liberty to modify. https://www.brainyquote.com/quotes/orson_welles_109697
3: save us hundreds of thousands of dollars
A lot of the cost savings were because we could decrease the size of our elasticsearch cluster. This was mostly due to the flattening of the elasticsearch document schema, which improved performance dramatically but also allowed us to run significantly fewer ES nodes.

DeepDiff Tutorial: Comparing Numbers

Fri, 12 Apr 2019 01:21:14 -0700

This tutorial is written based on DeepDiff 4.0.6.

One of the features of DeepDiff that comes very handy is comparing nested data structures that include numbers. There are times that we do care about the exact numbers and want it to be reported if anything slightly changed.

from pprint import pprint
from deepdiff import DeepDiff


t1 = {"key": [1.2, 1.5]}
t2 = {"key": [1.20, 1.50]}

>>> pprint(DeepDiff(t1, t2))
{}

Let’s say the numbers gets more precise:

t1 = {"key": [1.21, 1.5]}
t2 = {"key": [1.2100000000001, 1.50]}

>>> pprint(DeepDiff(t1, t2))
{'values_changed': {"root['key'][0]": {'new_value': 1.2100000000001,
                                       'old_value': 1.21}}

significant_digits

Do we really care about this change? Perhaps we don’t. In that case we have a few options. The first option is to pass significant_digits that we care about. By default the significant_digits sets how many digits after the decimal point to be considered when comparing numbers.

t1 = {"key": [1.21, 1.5]}
t2 = {"key": [1.2100000000001, 1.50]}

>>> pprint(DeepDiff(t1, t2, significant_digits=5))
{}

So if we care only about 5 digits of accuracy after the decimal points, we set the significant_digits=5 like the above example.

What if we care about the difference of numbers in the way that the difference is relative to the size of the number?

For example between 2.0001 and 2.0002 we may care about the difference of 0.001 but the difference between 20000.0001 and 20000.0002 is too small compared to the actual numbers that are being compared.

Is we don’t set the significant_digits, everything will be reported in the results:

t1 = {"key": [2.0001, 20000.0001]}
t2 = {"key": [2.0002, 20000.0002]}

>>> pprint(DeepDiff(t1, t2))
{'values_changed': {"root['key'][0]": {'new_value': 2.0002,
                                       'old_value': 2.0001},
                    "root['key'][1]": {'new_value': 20000.0002,
                                       'old_value': 20000.0001}}}

And if we set the significant_digits=3, both values disappear.

>>> pprint(DeepDiff(t1, t2, significant_digits=3))
{}

That’s where number_format_notation comes to play:

number_format_notation

To make DeeoDiff to consider diffs based on the ratio of diff to the original numbers, we can set the number_format_notation parameter. The number_format_notation is by default set to “f” meaning fixed point. However setting it to “e” which stands for the exponent notation or scientific notation, gives us what we want:

>>> pprint(DeepDiff(t1, t2, significant_digits=4, number_format_notation="e"))
{'values_changed': {"root['key'][0]": {'new_value': 2.0002,
                                       'old_value': 2.0001}}}

Basically in the above diff we are saying that we care about 4 significant digits in the scientific notation which automatically makes the diff relative to the size of the number.

ignore_numeric_type_changes

So far so good. What if we have type changes in our numbers? For example you loaded a json file that has floats but the Python object you have includes decimal types.

from decimal import Decimal

t1 = {"key": [Decimal('2.0001')]}
t2 = {"key": [2.0001]}

>>> pprint(DeepDiff(t1, t2))
{'type_changes': {"root['key'][0]": {'new_type': <class 'float'>,
                                     'new_value': 2.0001,
                                     'old_type': <class 'decimal.Decimal'>,
                                     'old_value': Decimal('2.0001')}}}

To solve this problem, DeepDiff provides the ignore_numeric_type_changes parameter:

t1 = {"key": [Decimal('2.0001')]}
t2 = {"key": [2.0001]}

>>> pprint(DeepDiff(t1, t2, ignore_numeric_type_changes=True))
{}

Behind the scene, DeepDiff converts both of the numbers into string representation of them with the accuracy of 12 significant digits by default. You can again overwrite the significant_digits with passing the parameter. Let’s set that to a higher number:

t1 = {"key": [Decimal('2.0001')]}
t2 = {"key": [2.0001]}

>>> pprint(DeepDiff(t1, t2, ignore_numeric_type_changes=True, significant_digits=18))
{'values_changed': {"root['key'][0]": {'new_value': 2.0001,
                                       'old_value': Decimal('2.0001')}}}

In other words, 2.0001 == Decimal('2.0001') when significant_digits=12 (default) but not when we increase the significant_digits to 18.

This is due to floating point arithmetic issues. A good resource to take a look at is located at https://docs.python.org/3/tutorial/floatingpoint.html

To understand what happens, behind the scene DeepDiff converts the numbers into strings whenever the ignore_numeric_type_changes=True. In such casesm by defeault it uses number_format_notation="f" which stands for fixed point notation but again we can use the number_format_notation to change that behaviour.

When you don’t pass the significant_digits, the default value of 12 is used behind the scene:

>>> '{:.12f}'.format(2.0001)
'2.000100000000'
>>> '{:.12f}'.format(Decimal('2.0001'))
'2.000100000000'

But when you use significant_digits=18

>>> '{:.18f}'.format(2.0001)
'2.000100000000000211'
>>> '{:.18f}'.format(Decimal('2.0001'))
'2.000100000000000000'

As you can see the float and decimal won’t match anymore! You can use the significant_digits and number_format_notation to have granular control over how numbers are compared when ignore_numeric_type_changes=True

Just like what we did with the number_format_notation, we can limit the reported diff to be limited to numbers that their diff is big enough compared to their size:

t1 = {"key": [Decimal('2.0001'), Decimal('20000.0001')]}
t2 = {"key": [2.0002, 20000.0002]}

>>> pprint(DeepDiff(t1, t2, ignore_numeric_type_changes=True, significant_digits=4, number_format_notation="e"))
{'values_changed': {"root['key'][0]": {'new_value': 2.0002,
                                       'old_value': Decimal('2.0001')}}}

number_to_string_func

For the power users who want more granular control over how numbers are compared, you can pass a custom function that converts numbers to strings.

The original function that converts numbers to strings resides in the helper.py module.

Here is its current implementation at the time of writing of this article:

from decimal import Decimal, localcontext

ZERO_DECIMAL_CHARACTERS = set("-0.")

number_formatting = {
    "f": r'{:.%sf}',
    "e": r'{:.%se}',
}


def number_to_string(number, significant_digits, number_format_notation="f"):
    """
    Convert numbers to string considering significant digits.
    """
    try:
        using = number_formatting[number_format_notation]
    except KeyError:
        raise ValueError("number_format_notation got invalid value of {}. The valid values are 'f' and 'e'".format(number_format_notation)) from None
    if isinstance(number, Decimal):
        tup = number.as_tuple()
        with localcontext() as ctx:
            ctx.prec = len(tup.digits) + tup.exponent + significant_digits
            number = number.quantize(Decimal('0.' + '0' * significant_digits))
    result = (using % significant_digits).format(number)
    # Special case for 0: "-0.00" should compare equal to "0.00"
    if set(result) <= ZERO_DECIMAL_CHARACTERS:
        result = "0.00"
    # https://bugs.python.org/issue36622
    if number_format_notation == 'e' and isinstance(number, float):
        result = result.replace('+0', '+')
    return result

All that this function does is to convert the numbers into strings based on the significant_digits and formatting notation (“f” for fixed point or “e” for scientific.)

You can modify this function or its results and pass it to DeepDiff as the number_to_string_func.

For a silly example let’s say you don’t care if numbers below 100 have changed. You only care if numbers above 100 have changed. Then you do:

from deepdiff.helper import number_to_string


def custom_number_to_string(number, *args, **kwargs):
    number = 100 if number < 100 else number
    return number_to_string(number, *args, **kwargs)

t1 = [10, 12, 100000]
t2 = [20, 22, 100000]

ddiff = DeepDiff(t1, t2, significant_digits=3, number_format_notation="e",
                 number_to_string_func=custom_number_to_string)
>>> ddiff
{}

Note: number_to_string_func is only used when either the significant_digits is set or ignore_numeric_type_changes is set.

Learn More!

This was the first in a series of tutorials I will be writing for DeepDiff. Hope you enjoyed it!

To learn more about DeepDiff please take a look at the documentation or the source code and give it a star on github if you find it useful!

You AutoComplete Me

Thu, 28 Feb 2019 01:21:14 -0700

Part 1: Into

Auto Complete: Summary

Back in 2018, I was given the task to improve our autocomplete experience. We already had an Autocomplete backend solution using Elasticsearch, but what we needed was something that was faster, more customizable which could handle complex queries such as red bmw x5 in napa 2019. This is the story of how I approached it, what I learnt, and the final product that it came to be, using a pure Python solution! In this talk the user will get familiar with various data structures in Python, from the built-in deque to creating Trie-tree and Directed Acyclic Word Graph (DAWG) and even fuzzy matching via phonetic algorithms and Levenshtein edit distance. We also explore the grey area between incremental search and delimited search. Finally, a package called fast-autocomplete is introduced which we already use in production and are very excited to get it into the wild.

What is Autocomplete

Did you know we can categorize search into 2 types?

Delimited Search: You type something, hit enter and get results.
Incremental Search AKA Auto Complete AKA Auto Suggestion: As you type text, possible matches are found and immediately presented to the user.

It gets even more interesting. Jef Raskin, who conceived the Macintosh project at Apple, wrote in his The Humane Interface book that I can see almost no occasions when a delimited search would be preferred [over autocomplete].

I mean, let’s be honest. Unless you really know what you are searching for, you expect the autocomplete to guide you these days, no?

Part 2: Autocomplete: How?

1. Trie-Tree (Radix Tree, Prefix Tree)

https://en.wikipedia.org/wiki/Trie

Trie-Tree is one of the most common ways to do autocomplete. The graph is structured such that a node is a word or a flag that means “add up the edges”, an edge is a letter of a word.

Here is an example:

As the user types each letter, the algorithm traverses through the graph, moving from node to node. At each node, all descendants of that node are displayed. For example, if the user typed bm, the descendants bmw, bmw x5, and bmw x3 are returned.

Trie-Tree in Python

So how would we do this in Python? What about making a node class?

class Node:
    def __init__(self):
        self.children = ?

Hmm, the children are the other nodes. And what connects one node to the other? Yes, the letters are the edges!

        a letter
node 1 -----------> node 2

A letter maps one node to another. Yes, mapping, or what Python calls a dictionary!

class Node:
    def __init__(self):
        self.children = {}

While we are here, let’s add an attribute to hold the optional full word for the node too.

class Node:
    def __init__(self):
        self.children = {}
        self.word = None

Inserting into the tree

When we insert a word, we want to insert the letters one by one and create a node if we don’t already have it:

class Node:
    def __init__(self):
        self.children = {}
        self.word = None

    def insert(self, word):
        node = self
        for letter in word:
            if letter not in node.children:
                node.children[letter] = Node()
            node = node.children[letter]
        node.word = word
        return node

Traversing the tree

We can follow the same logic as above for traversing. We traverse starting from the current node until we have exhausted all of the letters in the query or we reach a dead-end in our graph.

    def traverse(self, query):
        node = self
        for letter in query:
            child = node.children.get(letter)
            if child:
                node = child
            else:
                break

        return node

Now let’s add some __repr__ to the node for easier debugging. Here is the whole thing together:

class Node:

    def __init__(self):
        self.children = {}
        self.word = None

    def insert(self, word):
        node = self
        for letter in word:
            if letter not in node.children:
                node.children[letter] = Node()
            node = node.children[letter]
        node.word = word
        return node

    def traverse(self, query):
        node = self
        for letter in query:
            child = node.children.get(letter)
            if child:
                node = child
            else:
                break

        return node

    def __repr__(self):
        return f'< children: {list(self.children.keys())}, word: {self.word} >'

Let’s look at the example graph again:

Let’s traverse now:

WORDS = ['bmw', '2018', 'x3', 'x5', 'bmw x5', 'bmw x3', 'napa']

root = Node()

for word in WORDS:
    root.insert(word)

>>> root.traverse(query="bm")
< children: ['w'], word: None >

So far so good: we got to the node which has the w edge which links to the bmw node. What if we type something that is not in the graph?

>>> root.traverse(query="bmq")
< children: ['w'], word: None >

The traverse function stops when it can no longer find an edge with the requested letter (“q” in the above example). And if we type a full word that is in the graph:

>>> root.traverse(query="bmw")
< children: [' '], word: 'bmw' >

Nice! However, this has not “completed” anything for us yet. It has only traversed to a node. In order for it to make a “completion”, we want to get all of the descendants of the current node.

How do we get all the descendant nodes of bm? Using recursion we get bmw x5, then bmw x3, and finally bmw. But it makes more sense if the nodes closer to the root of the tree are returned first. Implementing a breadth-first strategy can be useful for that.

Breadth-first

Queue.Queue vs. collections.deque

We can employ breath-first search by using a queue. Python comes with the Queue.Queue module but it is not useful in this context: Queue was made to enable threads to communicate using queued messages. What we need to use is collections.deque, which is just a data-structure. It has O(1) complexity in appending and popping from both ends.

    def get_descendant_nodes(self):
        que = collections.deque()
        for letter, child_node in self.children.items():
            que.append((letter, child_node))
        while que:
            letter, child_node = que.popleft()
            if child_node.word:
                yield child_node
            for letter, grand_child_node in child_node.children.items():
                que.append((letter, grand_child_node))

Great! Now we can return actual autocomplete results!

>>> found_node = root.traverse(query='bm')
>>> list(found_node.get_descendants_nodes())
[< children: [' '], word: bmw >, < children: [], word: bmw x5 >, < children: [], word: bmw x3 >]

Synonyms

It turns out that there are people out there who call bmw, bimmer. By using Trie-Tree, we can have completely separate branches for the same thing: bmw and x5.

It seems like a waste of space to have bmw and x5 at 2 different branches in the trie. What if we could combine these 2 branches into one? That’s where DAWG comes in.

Before adding synonyms:

After adding synonyms:

And this is just the tip of the iceberg. What if we join branches where the words mean the same thing?

2. Directed Acyclic Word Graph (DAWG, Deterministic Acyclic Finite State Automaton)

A DAWG is very similar to a trie-tree except that the branches can merge back to each other! So it is not a trie anymore: it is an acyclic graph. Take a look at how much space we can save by merging branches that mean the same thing:

To change our trie-tree into a DAWG, we will give the insert method the ability to set the word attribute of the node:

    def insert(self, word, add_word=True):
        node = self
        for letter in word:
            if letter not in node.children:
                node.children[letter] = Node()

            node = node.children[letter]

        if add_word:
            node.word = word
        return node

Now when we add words we can manually merge branches when there are synonyms:

SYNONUMS = {'bmw': 'beemer'}
root = Node()

for word in WORDS:
    synonym = SYNONUMS.get(word)
    if synonym:
        inserted_node = root.insert(word)
        inserted_synonym = root.insert(synonym[:-1], add_word=False)
        inserted_synonym.children[synonym[-1]] = inserted_node
    else:
        root.insert(word)

Great! Now we can return actual autocomplete results for synonyms:

>>> found_node = root.traverse(query='beem')
>>> list(found_node.get_descendants_nodes())
[< children: [' '], word: bmw >, < children: [], word: bmw x5 >, < children: [], word: bmw x3 >]

DAWGs are very useful to eliminate suffix redundancy. However, one of the drawbacks of DAWG is that there can be several paths to get to the same node. So if it is important how you get to a node, it could cause some problems.

What we are doing here is just touching the surface of what DAWGs can offer and how much space they can save. For example, the DAWG package written by Mikhail Korobov can reduce memory usage for a list of 3 million words by a factor of more than one hundred percent!!

Python list: about 300M
dawg.DAWG: 2M

You can read more about DAWG here: https://en.wikipedia.org/wiki/Deterministic_acyclic_finite_state_automaton

Now that you are familiar with Trie and DAWG, let’s step back and think about them for a second. When the user for example types bmw x5 na, we get to the leaf node that has bmw x5, but there is nothing else left in the branch to traverse. Now what? How can we recommend napa to them? One way to solve it is using Markov chains:

3. Markov Chain

Another interesting way to create autocomplete is using Markov chains:

Imagine that we feed a corpus of text into a system where it keep tracks of how many times any two words follow each other. Based on that we come up with the probability of which word follows which. Using these probabilities, we can give autocomplete suggestions to users based on the most historically common co-occurring words.

There is a Python library that does this for you by Rodrigo Palacios: https://github.com/rodricios/autocomplete

pip install autocomplete

autocomplete.predict('the','b')

[('blood', 204),
 ('battle', 185),
 ('bone', 175),
 ('best', 149),
 ('body', 149),
 ...]

To read more about Markov chains take a look at: https://en.wikipedia.org/wiki/Markov_chain

Summary of methods

Before we continue, let’s summarize what we have gone through.

Trie-Tree: Simple but not space efficient
Directed Acyclic Word Graph (DAWG): More complicated, very space efficient
Markov Chain: Based on probabilistic relationships between words, not character to character.

Part 3: Dealing with mis-spellings

Option 1: Fat-Finger Error

Let’s assume the person knows what they want but presses the wrong key next to the key they meant to press. The Python Autocomplete package by Rodrigo Palacios (the one that uses Markov Chains) addresses this scenario. All you need is a dictionary that maps each character with all of its neighboring characters on a typical keyboard.

NEARBY_KEYS = {
    'a': 'qwsz',
    'b': 'vghn',
    'c': 'xdfv',
    'd': 'erfcxs',
    ...
}

You can then traverse whichever search algorithm you are using with each of these “fat-finger error” alternatives.

Option 2: Instead of using the original characters as the edges of the graph, use phonetic algorithms

Metaphone or Soundex:

These are phonetic algorithms for indexing words by their sound. Words that sound the same will have the same code!

For example there is a jellyfish package.

In [1]: import jellyfish

In [3]: jellyfish.metaphone(u'bmw 3 series')
Out[3]: 'BM  SRS'

In [4]: jellyfish.metaphone(u'bmw 3 seris')
Out[4]: 'BM  SRS'

If what user enters and what is in the graph sound the same, it works perfectly!

In this case we would have put BM SRS as an edge in the graph and the word at that node would have been bmw 3 series.

The problem with this approach is that:

In [5]: jellyfish.metaphone(u'bmw 4 seris')
Out[5]: 'BM  SRS'

In our case, those are two different models!

Option 3: Levenshtein Edit Distance

Levenshtein Edit Distance gives you the minimum number of operations required to transform one word into another:

kitten → sitten (substitution of “s” for “k”)
sitten → sittin (substitution of “i” for “e”)
sittin → sitting (insertion of “g” at the end).

Photo courtesy of Wikipedia

And there is a Python Levenshtein package that can calculate the distance for you. The jellyfish package has its own implementation of the algorithm too.

Part 4: Switching our Autocomplete from Elasticsearch to Fast-Autocomplete

At this point this was an experimental/research project. We had already an autocomplete service that was using Elasticsearch but…

Issue: speed and errors on Elasticsearch

Take a look at this diagram to see what was going on. There were errors from time to time, and there were times that the latency was above a second!

And the service couldn’t handle complex queries. In order to have red bmw x5 2019 in Napa you had to put the whole sentence as an option in Elasticsearch.

What else is out there?

At this point the main contender was what Elasticsearch is built on: Lucene. Lucene is the Java library that has a Python wrapper called PyLucene.

Lucene gives you 3 options for Autocomplete: http://lucene.apache.org/core/7_2_0/suggest/index.html

JaSpell-based autosuggest, which is based on Java Spelling Checking Package.
Ternary Search Tree based autosuggest. This is something we didn’t discuss here but is more efficient than Trie-Tree. Basically it is a Tree within a Trie. Here is an interesting post about it by Igor Ostrovsky: http://igoro.com/archive/efficient-auto-complete-with-a-ternary-search-tree/
And finally Finite-state based autosuggest. Yes, Lucene implemented DAWG for autocomplete too!

Both Lucene and PyLucene are Apache projects. Surprisingly I could not find good documentation for how to use PyLucene. Plus I had to run a JVM to run Lucene itself and deal with managing the JVM.

That was the point that decided to take matters into my own hands.

The answer is: Build from scratch

Introducing `fast-autocomplete`

Using the DAWG data structure, Levenshtein Edit Distance, and LFU caching.

pip install fast-autocomplete

https://github.com/seperman/fast-autocomplete

Thanks to graphviz, it can even create an animation of words insertion!

# word = dictionary of words to their contexts
# synonyms = dictionary of words to their synonyms
from fast_autocomplete import demo, AutoComplete
autocomplete = AutoComplete(words=words, synonyms=synonyms)
demo(autocomplete, max_cost=3, size=5)

You might ask what did you decide to do when the algorithm ends up on the leaf of a branch? How do you get from bmw x5 to bmw x5 in napa ?

Part 5: Dealing with sentences

Markov Chains vs. Looping through the graph

Coming back to our problem of how to recommend Napa once we reached the leaf node in the graph while searching for bmw x5, Markov Chains could have been a great solution. I said they “could” if and only if we had a corpus of text to extract all sorts of words related to cars, which we could use to get the co-occurring probabilities of which words follow which. However since I did not have such a corpus of data, I decided not to use Markov Chains. Instead, the approach I took was to go back to the root node and continue traversing from the root any time we have reached a dead-end on a leaf node.

And here we go!

part 6: Demo

demo 1

The incremental AND delimited search

Remember that we had two types of search:

Delimited Search: You type something, hit enter, get results.
Incremental Search AKA Auto Complete AKA Auto Suggestion.

That’s where I thought, why not turn the DAWG into a FST (Finite-State Transducer) that translates the user entry into Lucene syntax. By doing that the same graph that provides suggestions to the user as they type could also translate the user’s query for a delimited search!

Demo 2

You might be asking, did you not decide to not use Lucene? The answer is I didn’t use it for Autocomplete but we still decided to use it for “delimited search”.

Part 7: The Outcome

Once we switched to fast-autocomplete, here is what we observed:

No more errors and 3-4x performance improvement in autocomplete

Ok I’m cheating here a little bit. There are still some errors that happen randomly due to network failures and other factors. Unfortunately it is not easy to reproduce those errors. Currently our error rate is 0.0004 errors per each request. The performance boost though has been very tangible.

More user engagement with Autocomplete

Complex queries

Now we can have user’s type more complex queries!

If you go to fair.com you can see the autocomplete for yourself. Here are 2 snapshots of the autocomplete in action:

Part 8: Conclusion

Writing our own autocomplete was fun, and ended up with performance gains, customizability and better user engagement. So overall it was a big win and I’m happy that I had the opportunity to pursue it. I hope you learnt something by reading this article too.

Please checkout Fast Autocomplete and if you find it useful, please give it a star. https://github.com/seperman/fast-autocomplete

You can visit fair.com to see the autocomplete in action and maybe even do some car shopping!

Finally I would like to say a big thank you to my colleague John Case who took the time to edit this article and provide constructive feedback.

Magic Method, on the wall, who, now, is the fairest one of all?

Fri, 19 May 2017 15:34:03 -0700

Update May 22: Here is the video of the talk:

The Proposal

This was my talk proposal for Pycon 2017 which got accepted. The proposal is slightly modified to match the final talk better. For example originally I was going to talk about writing a Redis Client too but I ended up removing that from the final talk. I will be giving this talk tomorrow on Saturday May 20th!

The code samples used in this talk can be found at: https://github.com/seperman/bad-ideas

Or simply: pip install bad-ideas

This talk is lightly inspired by my other talk: Harness the power of Python magic methods and lazy objects.

Summary

Magic methods are a very powerful feature of Python and can open a whole new door for you. However, with great power comes great responsibility.

In this talk we explore magic method’s capabilities by first experimenting with recreating echo, grep, and pipe bash command syntaxes as valid Python syntaxes. And finally we learn about reference counting and the garbage collector by creating undeletable objects.

Once you see what magic methods can bring to the table; the limit is only your imagination!

Who and Why

This talk is mainly geared towards novice Python developers and is about Python’s magic methods. However, the ideas brought up in the experiments can be very interesting for more experienced Python developers as well. The audience is expected to have limited or even no exposure to the magic methods.

The talk is expected to make the audience excited about what magic methods can bring to the table and demystify certain syntaxes that they might have seen and used in certain libraries, for example the Django queries or SQLAlchemy queries from chaining filters and operators.

Magic Method, on the wall, who, now, is the `fairest` one of all?

Intro

1.5 min

What are Python’s magic methods?

Special methods that you can define to add “magic” to your classes.
They all look like: __something__.

Example: __init__

So what can we do with them?

Let’s do some experiments and learn by examples!

Disclaimer

Magic methods are a very powerful feature of Python and can open a whole new door for you. However with great power comes great responsibility.

The following experiments are solely for educational purposes and NOT for production code.

Experiment: Type Less

4.5 min

I don’t know about you but I don’t like typing too much. Every keystroke is a stress on your fingers and over time it adds up.

There are times that you need to type something like:

a = 10
a = a + 20

or as you know the shortcut that is:

a += 20

Since we are all about typing less. What if we just do:

a+20 and that does the job for us? We save one keystroke of =.

>>> a = 10
>>> a + 20
>>> print(a)
30

Let’s look at the full list of magic methods.

`add` and `sub`

We can do that using __add__ and __sub__ magic methods.

class Num:

    def __init__(self, value):
        self.value = value

    def __add__(self, other):
        self.value += other
        return self.value

    def __sub__(self, other):
        self.value -= other
        return self.value

    def __repr__(self):
        return str(self.value)

    __str__ = __repr__

>>> a = Num(10)
>>> a
10
>>> a + 20
>>> a
30
>>> a - 5
>>> a
25

Yay, we removed the need to type = which saves a couple million keystrokes a year.

What do you think is gonna happen if we do:

20 + a

Oops! We get:

TypeError: unsupported operand type(s) for +: 'int' and 'Num'

Let’s look at __add__ again:

    def __add__(self, other):
        self.value += other
        return self.value

We are adding the “other” to the current value. When we run: a + 20 it runs a.__add__(a, 20) however when we do 20 + a it runs int.__add__(20, a) and then it freaks out!

`radd` and `rsub`

That’s where the reversed add and sub come to play.

What happens is that when it runs int.__add__(20, a) and gets a TypeError, then it tries the reverse add which is a.__radd__(a, 20).

    def __rsub__(self, other):
        self.value = other - self.value
        return self.value

    __radd__ = __add__

Here is the full implementation:

class Num:

    def __init__(self, value):
        self.value = value

    def __add__(self, other):
        self.value += other
        return self.value

    def __sub__(self, other):
        self.value -= other
        return self.value

    def __rsub__(self, other):
        self.value = other - self.value
        return self.value

    def __repr__(self):
        return str(self.value)

    __str__ = __repr__
    __radd__ = __add__

>>> a = Num(10)
>>> a + 20
>>> a
30

>>> a - 5
>>> a
25

>>> 40 + a
>>> a
65

>>> 20 - a
>>> a
-45

Disclaimer: That’s a bad idea. Don’t try this at home. I mean at work.

Experiment: Print Filter

2 min

Filter is a built-in function that does filtering on iterables.

Here is an example in Python 3:

foo = [1, 2, 3, 5, 6, 7]
bar = filter(lambda x: x % 3 == 0, foo)

We are filtering the list to have only elements that are divisible by 3.

If you print bar, what do you think you are gonna get?

>>> print(bar)
<filter at 0x119151d18>

That’s right, bar is a generator (in Python3).

How do you get the filtered list printed?

One way is to convert the generator to a list:

>>> print(list(bar))
[3, 6]

But sometimes that is too much work if you want to keep printing the object and you know you don’t want it as a generator when printing.

How can we modify the built-in filter so it converts itself into a list when printed?

Everything is an object

Did you know that everything is an object in Python?

def func(x):
    previous_x = getattr(func, "_x", "Not set")
    print("new value: {}, previous value: {}".format(x, previous_x))
    func._x = x

>>> func(10)
new value: 10, previous value: Not set
>>> func(20)
new value: 20, previous value: 10
>>> func(30)
new value: 30, previous value: 20

Even built-in functions are object! Yes, even built-in functions. Let’s subclass the filter builtin function and add some __str__ and __repr__ to filter:

class Filter(filter):
    def __str__(self):
        return str(list(self))

    __repr__ = __str__


bar = Filter(lambda x: x % 3 == 0, foo)
print(bar)  # prints [3, 6]

Now you can use Filter instead of filter and printing will give you the filtered results. No worries!

Experiment: Echo to file

1.5 min

Here is one of my favorites in bash:

echo "hello" >> foo.txt

And sometimes that syntax is too good not to use in Python.

The trick is that Python 2 used to have something like this:

print >> myfile, "Hello World!\n"
print >> myfile, "I want a burrito."

But you can’t do that in Py 3 anymore since print is a function now.

Hmm, what have we got for >> operator?

operator	method
»	Binary operation of `__rshift__`

Awesome! Lets get to work.

myfile = open("hello.txt", "w")

class Echo:
    def __init__(self, text):
        self.text = text

    def __rshift__(self, other):
        other.seek(0, 2)
        other.write(self.text)

# Writes to the end of the file!
>>> Echo("Hello World!\n") >> myfile
>>> Echo("I want a burrito.") >> myfile

Experiment: Pipe and Grep

3.5 min

here is another favorite from bash: pipe and grep. I use it all the time.

command | grep something

First of all, what can we use for pipe | operator?

Let’s see.

operator	method
`	`

aha!

Let’s say we define a grep that uses a | (binary or) operator. This is one way we can define it:

class Grep:
    def __or__(self, other):
        ...

Note the self and other in the arguments. It means that in order to use | with this grep, it first needs to __init__ the grep and then do the pipe. Which means the order we write things are gonna be different than what we are used to see in bash:

instead of text | grep something, it is gonna be grep(something) | text. But wait a second. There is reverse or: __ror__ too. That can let us write text | grep(seomthing)!

class Grep:

    def __init__(self, item):
        self.item = item.lower()

    def thefilter(self, line):
        return self.item in line

    def __ror__(self, other):
        if isinstance(other, str):
            other = other.lower().split('\n')
        return Filter(self.thefilter, other)


lines = """
Whether you're new to programming or
an experienced developer, it's easy
to learn and use Python.
Checkout jobs.python.org
for Python jobs.
"""

>>> lines | Grep('Python')
['to learn and use Python.',
'checkout jobs.python.org',
'for python jobs.']

Awesome! You can even even chain the greps!

>>> found = lines | Grep('Python') | Grep('jobs')
>>> print(found)
['checkout jobs.python.org', 'for python jobs.']

Experiment: The undeletable

5.5

Let’s say you run del obj. Normally that would delete the object but we want to make it undeletable!

>>> del obj
<obj: I'm still here. You CAN NOT delete me!>
>>> obj
<Yes I'm still here!>

Did you know what happens when you delete an object?

del obj

Garbage Collector

Hmm, ok let’s review how deleting works in Python.

cPython specifically keeps track of number references to the object. This is called reference counting. When you do del obj, it sets the number of references to the object to zero.

Then the Garbage collector goes and deallocates the object. However if your object has a finalizer, then things can get tricky.

You might ask what is a finalizer? objects with finalizers are objects with a __del__ method and generators with a finally block.

And if your object has a finalizer, the garbage collector will run the finalizer only at that moment.

The important thing to keep in mind is that it is not guaranteed that __del__ will run immediately after you run del since it is up to the garbage collector to run the __del__. The __del__ might never run. So you can’t ever depend on it. But for the sake of this experiment, we will use __del__ into our advantage.

Again, when the __del__ is run, the reference count to the object has already been set to zero and it is literally removed from the name space that it existed before.

So how can we resurrect the object once the __del__ is running? Maybe we can raise some exception in __del__ so it can’t be successfully run and the GC aborts deleting it?

The answer is no. cPython will abort running __del__ but it will still deallocate the object.

Here is another idea: we still have access to self inside __del__. After all it is

def __del__(self):
   ...

How can we use this to our advantage? Maybe we put the object back in the name-space it was deleted from?

class Obj:

    def __del__(self):
        global obj
        obj = self
        print("You can't delete me!")

    def __str__(self):
        return "<obj:{}>".format(id(self))


>>> obj = Obj()

>>> print(obj)
<obj:123123>
>>> del obj
You can't delete me!
>>> print(obj)
<obj:123123>

Del didn’t delete the object! It is the same object with the same id!

So what happened here again?

del obj sets the number of references to obj to zero and removes it from the globals name space in this case.
The garbage collector tries to deallocate the object and free the memory.
The garbage collector sees that the obj has __del__ finalizer method and runs it.
We define a new global variable called obj. I know I cheated here and I already knew the object’s name was obj. There are ways to find the name but that would have made the code way longer. You can see the full version implemented here.
We set the global obj to be self which is the current object that is being deleted.
The object is resurrected! It is the same object with the same ID as before!

Pep 442

Pep 442 was introduced in Python 3.4 and made some backward incompatible changes into how the finalizer is called by the garbage collector.

We were running the above code:

print(obj)
del obj
print(obj)

And in Python 3.4+ what we get is something like:

<obj:4482786192>
You can't delete me!
<obj:4482786192>

But in Python 2 to 3.3 you get:

<obj:4549435760>
You can't delete me!
<obj:4549435760>
You can't delete me!
You can't delete me!

Basically in Python 3.4+ __del__ methods will be executed at most once by the garbage collector and it will no longer matter whether an object with a finalizer is a part of cyclic trash.

Conclusion

1 min

We explored magic method’s capabilities by first experimenting with

Making a number object so we type less
Subclassed filter so we print the filtered and see the results
Echo to file
Pipe and grep
Undeletable object

As we saw, the magic methods can bring a lot to the table; the limit is only your imagination!

Hope you learnt something from these bad ideas! https://github.com/seperman/bad-ideas

Don’t forget to pip install bad-ideas and play around with the code!

Harness the power of Python magic methods and lazy objects.

Sat, 10 Sep 2016 15:34:03 -0700

This is based on a talk I gave at SoCal Python meetup. The PDF of this talk is available here.

All the code is run in Python 3.5 but it works with Python 2 too.

Overview

When it comes to performance, lazy loading can help a lot. I remember when I first started using Django and Django’s lazy queries were magic to me.

In this article I will try to dimistify lazy objects in Python, mainly focusing on lazy loading aspect but not lazy writing. However in order to do lazy loading, we need to have a good understanding of some of the Python’s magic methods. After that we try to learn writing lazy objects by doing an example: Writing a Redis client.

In the end, the complete version of that Redis client is presented in case you want to study its code and/or find it useful!

Lazy Loading

Defer initialization of an object until the point at which it is needed.

There are arguments both pro and against lazy design:

Why lazy?

Better performace (depending on your design)
Better illusion of performance (when dealing with heavy objects)
Less hits to your database (depending on your design)

Why not lazy?

Inconsistent state.
Code complexity.
More hits to your database (depending on your design).

Why Lazy?

Better performance, less hits to database

Example: Django Queries

The act of creating a QuerySet does not involve any database activity. You can stack filters together all day long, and Django will not actually run the query until the QuerySet is evaluated

>>> q = Entry.objects.filter(headline__startswith="What")
>>> q = q.filter(pub_date__lte=datetime.date.today())
>>> q = q.exclude(body_text__icontains="food")
>>> print(q) # <-- evaluated here

Example: Redis get vs mget

Let’s benchmark hitting redis 100 times with get requests or combining 100 queries into one query and then getting all 100 items at once.

Basically the idea is that just like Django queries, you combine all queries into one big query and hit the db.

https://github.com/seperman/benchmark/blob/master/pyredis_benchmark.py

This is what I get on my machine hitting Redis local.

Fetching 100 keys from Redis
100 x Get one key: 4.84 miliseconds
1 x Mget 100 keys*: 0.93 miliseconds

In other words mget was 5 times faster.

*: mget is redis command for getting a bunch of string keys instead of get which gets one key.

Better illusion of performance

Example:

Unlimited scroll. Faster page load time.
Load chunks of big graph as you need.

Less memory usage

Load chunks of big graph as you need.

Why not lazy?

Inconsistent state

Adds another layer of abstraction thus more difficult to keep the states consistent.

More hits to your database

(with a bad design)

for obj in all_lazy_objects:
    print(obj)  # evaluates one by one in a bad design.

Lazy loading is a design pattern. It fits certain use cases. Lets start with an example:

Code maintainability

It can make the code slower to write or harder to maintain.

Case Study: Redis Client

Learn by doing: Let’s create a Redis client that can simply fill up the values in a html template.

What we will learn:

Why lazy loading is a good choice for writing a Redis client.
How to write lazy objects in Python, which requires a good understanding of:
Python Magic methods.

option 1: multiple get

The first option is to have multiple get requests to Redis and then pass them through the context to the html template.

context = dict(
    title = redis.get('root.homepage.title'),
    x = redis.get('root.things.x'),
    y = redis.get('root.things.y'),
    z = redis.get('root.things.z'),
    footer = redis.get('root.footer')
)

<html>
<head>
<title>{{ title }}</title>
</head>
<body>
Hello,
We do {{ x }}, {{ y }} and {{ z }}.
{{ footer }}
</body>
</html>

We can do it better!

option 2: mget

Mget is the redis command for multi get. It makes one request to Redis to get multiple strings. It can run way faster than making several get requests as we saw.

context = dict(
    title, x, y, z, footer = redis.mget('root.homepage.title',
                                        'root.things.x',
                                        'root.things.y',
                                        'root.things.z',
                                        'root.footer')
)

<html>
<head>
<title>{{ title }}</title>
</head>
<body>
Hello,
We do {{ x }}, {{ y }} and {{ z }}.
{{ footer }}
</body>
</html>

Note that get is for getting strings from Redis. For other types of data we need to use other commands. And in order to combine all these commands into one query, we can use the Redis’s pipleline feature.

option 3: lazy

We will just pass a root object in the context to the template renderer. Then frontend guys decide which parts of the root children they need. Think about root as a huge dictionary that has every content you wanted to be cached in Redis in the first place. By passing the root to the html, the frontend can choose keys in the root to be used and only those keys will be loaded. This way you don’t have to think about what variables to pass into your context beforehand. All you need is passing a root object and later filling it up as you go.

context = {'root': root}

<html>
<head>
<title>{{ root.homepage.title }}</title>
</head>
<body>
Hello,
We do {{ root.things.x }}, {{ root.things.y }} and {{ root.things.z }}.

{{ root.footer }}
</body>
</html>

Redis data types overview

If you are familiar with Redis data types and commands, just jump to the next section. Otherwise here is an overview of Redis data types and specific Redis commands for those types.

String get, mget, set, mset (m = multi)
List lrange (get a range), rpush (append), rpop (pop)
Hash (Dictionary) hset (set a key), hsetall (set a whole dict), hgetall (get a whole dict)
Set sadd, smembers

Our goal is to write a Redis client that automatically uses the correct Redis data type and command for the data we are assigning in Python.

In other words, typing root.var1="some text" should automatically use set and typing root.var2=[1,2,3] should automatically use rpush and etc.

Create a Redis client

Step 1: Load strings

Our goals:

Create a root object that print(root.something) in Python gets root.something value from Redis.
something could be any valid attribute name in Python

>>> print(root.something)
value of root.something in Redis

But before we can write this Redis client we need to have a better understanding of Python magic methods:

Python magic methods

What are Python’s magic methods?

Special methods that you can define to add “magic” to your classes.
They all look like: __something__ and are pronounced dunder something.

`init`

(pronounced dunder init)

The first method to get called in an object’s instantiation? wrong!

`new`

The first method to get called in an object’s instantiation. It takes the class, then any other arguments that it will pass along to __init__.

`del`

Now you created object. What is run when you delete the object? You might think it is __del__ but that’s wrong. __del__ is run when garbage collected.

Here is a table of the more important ones:

Magic Methods in a glance

Protocol for containers (to define containers like lists,…)	Descriptor (custom class attributes that we fully control)	Called when no attribute found	Called whether the attribute is found or not	Called when garbage collecting
`__getitem__(self, key)`	`__get__(self, obj, cls=None)`	`__getattr__(self, name)`
`__setitem__(self, key, value)`	`__set__(self, obj, val)`		`__setattr__(self, name, value)`
`__delitem__(self, key)`	`__delete__(self, obj)`		`__delattr__(self, name)`	`__del__(self)` Not recommended. Instead use `__exit__` (context manager)

Create Redis Client

Now that we had an overview of Python magic methods, let’s get back to writing our client:

Step 1: load

Goal:

>>> print(root.something)
value of root.something in Redis

`getattr` to rescue

Assuming we have the following 2 keys of root.something and root.anotherthing already set in Redis:

class Root:
    def __getattr__(self, key):
        class_name = self.__class__.__name__.lower() 
        return redis.get("{}.{}".format(class_name, key))

>>> print(root.something)
value of root.something in Redis

>>> print(root.anotherthing)
value of root.anotherthing in Redis

And it works!

Step 2: save

Goal:

>>> root.something = "value"

And then we can verify it saved to Redis by using redis-cli:

$ redis-cli
127.0.0.1:6379> get root.something
"value"

First try:

class Root:
    def __init__(self):
        self.class_name = self.__class__.__name__.lower()

    def __getattr__(self, key):
        return redis.get("{}.{}".format(self.class_name, key))

    def __setattr__(self, key, value):
        redis.set(key, value)
        
root = Root()
root.something = "value"
print(root.something)

When we run print(root.something) we get:

    return redis.get("{}.{}".format(self.class_name, key))
RecursionError: maximum recursion depth exceeded while calling a Python object

Why do you think we are running into this error?

Hints:

When the maximum depth recursion error happens we have these values: key = "class_name" and there are 2 keys in Redis: class_nameand something
__getattr__ Called when no attribute found
__setattr__ Called whether the attribute is found or not

Ok let’s go over what happens step by step:

class Root:
    def __init__(self):
        self.class_name = self.__class__.__name__.lower()

    def __getattr__(self, key):
        return redis.get("{}.{}".format(self.class_name, key))

    def __setattr__(self, key, value):
        redis.set(key, value)
        
root = Root()
root.something = "value"
print(root.something)

1. __init__ runs.
2. self.class_name needs to be set.
3. __setattr__ is run to set self.class_name.
4. The key `class_name` is saved into Redis.
5. root.something = "value" sets key `something` into redis too.
6. print(root.something)
7. __getattr__ for `something` is run.
8. in order to get the key from Redis, it needs to get self.class_name.
9. But self.class_name was never set on the object. It was saved into Redis.
10. Tries to get self.class_name which needs self.class_name itself.

Save: The Solution

Keep track of native attributes.
Save to and read from __dict__

You might ask what are native attributes? It is just a variable name that keeps track of what is native to our library vs. should be read or written to Redis. For example to make sure if root.classname is called, it should not save/read it to/from Redis but if root.somethingelse is called, it is not one of the native attributes and can be saved or read from Redis!

And here is an implementation that works:

NATIVE_ATTRIBUTES = {'class_name'}

class Root:
    def __init__(self):
        self.class_name = self.__class__.__name__.lower()

    def get_redis_key_path(self, key):
        return "{}.{}".format(self.class_name, key)

    def __getattr__(self, key):
        if key in NATIVE_ATTRIBUTES:
            return self.__dict__[key]
        else:
            key = self.get_redis_key_path(key)
            return redis.get(key)

    def __setattr__(self, key, value):
        if key in NATIVE_ATTRIBUTES:
            self.__dict__[key] = value
        else:
            key = self.get_redis_key_path(key)
            redis.set(key, value)

root = Root()
root.something = 10
print(root.something)  # b"10"

Step 3: Make it lazy

Now that we have something that reads and write to Redis, our goal is to only evaluate the object when needed aka lazy load. In this case we want to evaluate when printing which means running the __str__ method of our object.

>>> obj = root.something
>>> obj  # runs __repr__
<Lazy root.something>
>>> print(root.something)  # runs __str__
b"10"

Implementation with lazy object:

NATIVE_ATTRIBUTES = {'class_name'}

class Root:
    def __init__(self):
        self.class_name = self.__class__.__name__.lower()

    def get_redis_key_path(self, key):
        return "{}.{}".format(self.class_name, key)

    def __getattr__(self, key):
        if key in NATIVE_ATTRIBUTES:
            return self.__dict__[key]
        else:
            key = self.get_redis_key_path(key)
            return Lazy(key)

    def __setattr__(self, key, value):
        if key in NATIVE_ATTRIBUTES:
            self.__dict__[key] = value
        else:
            key = self.get_redis_key_path(key)
            redis.set(key, value)

class Lazy:
    def __init__(self, key):
        self.key = key

    @property
    def value(self):
        # Redis keeps strings as bytes
        return redis.get(self.key).decode('utf-8')

    def __repr__(self):
        return "<Lazy {}>".format(self.key)

    def __str__(self):
        return self.value

And when we run it:

>>> root = Root()
>>> root.something = 10

>>> root.something  # does not evaluate the object yet since it runs __repr__
<Lazy root.something>
>>> print(root.something)  # evaluates since it runs __str__
b"10"

Now that we can set root.something to 10 what do you think happens when we run this?

>>> root = Root()
>>> root.something = 10
>>> root.something > 8

This is what we get:

>>> root = Root()
>>> root.something = 10
>>> root.something > 8
TypeError: unorderable types: Lazy() > int()

What magic method can help us make this lazy object comparable to numbers? Look at the full list here.

…

Yes, you guessed right: __gt__ (greater than) and __lt__ (less than):

class Lazy:
    def __init__(self, key):
        self.key = key

    @property
    def value(self):
        # Redis keeps strings as bytes
        return redis.get(self.key).decode('utf-8')

    def __repr__(self):
        return "<Lazy {}>".format(self.key)

    def __str__(self):
        return self.value

    def __gt__(self, other):
        return float(self.value) > other

    def __lt__(self, other):
        return float(self.value) < other

>>> root.something > 8
True
>>> root.something < 11
True

Great! Now what do you think happens if we run this? Is root.something equal to 10?

>>> root.something = 10
>>> root.something == 10

This is what we get:

>>> root.something = 10
>>> root.something == 10
False

In order to make the lazy object handle equality, we need to use __eq__. Using __lt__ and __gt__ is NOT enough to make Python do equality comparison!

class Lazy:
    def __init__(self, key):
        self.key = key

    @property
    def value(self):
        return redis.get(self.key).decode('utf-8')

    def __repr__(self):
        return "<Lazy {}>".format(self.key)

    def __str__(self):
        return self.value

    def __gt__(self, other):
        return float(self.value) > other

    def __lt__(self, other):
        return float(self.value) < other

    def __eq__(self, other):
        return float(self.value) == other

>>> root.something = 10
>>> root.something == 10
True

Deep thoughts: Equality vs. Identity

Ok, so what do you think about this now? root.something is equal to 10. Is it 10?

>>> root.something = 10
>>> root.something == 10
True
>>> root.something is 10

And this is what we get:

>>> root.something = 10
>>> root.something == 10
True
>>> root.something is 10
False

The reason is that the is keyword deals with identity NOT equality. In python, 2 objects are identical only if their IDs are the same. In other words if they are actually the same object in memory.

So >>> root.something is 10 is equal to >>> id(root.something) == id(10)

Setting attributes of attributes

Here is another interesting case. What if the key name in Redis is going to have more than one dot in it?

>>> root.something.another = 10
# No error!
>>> print(root.something.another)
# What do you think it is gonna print here?

>>> root.something.another = 10
# No error!
>>> print(root.something.another)
AttributeError: 'Lazy' object has no attribute 'another'

Solution:

root.something is the lazy object
root.something should return self for any non-native attribute
Then update self.key from root.something to root.something.another

So basically if you ask for attribute of attribute of Root, we keep returning the same lazy object to you but internally change the key of that lazy object to have the new attribute in it.

Example:

root.something will return a Lazy object. The lazy object’s internal key value is root.something at this point.
root.something.another will return the same lazy object. The lazy object’s internal key value is root.something.another at this point.

If you are interested to actually see how setting/getting attributes of attributes works, I invite you to take a look at the dot module: https://github.com/seperman/dotobject.

Step 4: save lists

Now that we can read and save strings to Redis, let’s try another data type: lists.

Redis has specific commands for specific data types. We need to use the right command to save lists.

Our goal is to simply set a list in Python and have it saved as a list in Redis.

root.sides = ["fries", "salad"]

This is our current implementation that doesn’t handle lists:

class Root:
    def __init__(self):
        self.class_name = self.__class__.__name__.lower()

    def get_redis_key_path(self, key):
        return "{}.{}".format(self.class_name, key)

    def __getattr__(self, key):
        if key in NATIVE_ATTRIBUTES:
            return self.__dict__[key]
        else:
            key = self.get_redis_key_path(key)
            return Lazy(key)

    def __setattr__(self, key, value):
        if key in NATIVE_ATTRIBUTES:
            self.__dict__[key] = value
        else:
            key = self.get_redis_key_path(key)
            redis.set(key, value)

And we add a couple of lines in __setattr__ to handle Iterables (lists are Iterables):

from collections import Iterable

class Root:
    def __init__(self):
        self.class_name = self.__class__.__name__.lower()

    def get_redis_key_path(self, key):
        return "{}.{}".format(self.class_name, key)

    def __getattr__(self, key):
        if key in NATIVE_ATTRIBUTES:
            return self.__dict__[key]
        else:
            key = self.get_redis_key_path(key)
            return Lazy(key)

    def __setattr__(self, key, value):
        if key in NATIVE_ATTRIBUTES:
            self.__dict__[key] = value
        else:
            key = self.get_redis_key_path(key)
            if isinstance(value, strings):
                redis.set(key, value)
            elif isinstance(value, Iterable):
                redis.delete(key)
                redis.rpush(key, *value)

And it works!

>>> root.sides = ["fries", "salad"]

$ redis-cli
127.0.0.1:6379> lrange root.sides 0 -1
1) "fries"
2) "salad"

Step 5: load lists

Now that we can save lists to Redis, how about reading lists from Redis?

root.sides = ["fries", "salad"]
print(root.sides)
["fries", "salad"]

Our current implementation of lazy objects:

class Lazy:
    def __init__(self, key):
        self.key = key

    @property
    def value(self):
        return redis.get(self.key).decode('utf-8')

In order to read lists, we get the datatype from Redis and if it is list, then we use the proper command:

class Lazy:
    def __init__(self, key):
        self.key = key

    @property
    def value(self):
        thetype = redis.type(self.key)
        if thetype == b'string':
            result = redis.get(self.key).decode('utf-8')
        elif thetype == b'list':
            result = redis.lrange(self.key, 0, -1)
            result = [i.decode('utf-8') for i in result]
        return result

Yay! It works!

>>> root.sides = ["fries", "salad"]
print(root.sides)
"fries"

Let’s get some item from the list then.

>>> root.sides = ["fries", "salad"]
print(root.sides[0])
["fries", "salad"]
>>> print(root.sides[0])

Ooops!

>>> root.sides = ["fries", "salad"]
print(root.sides[0])
["fries", "salad"]
>>> print(root.sides[0])
TypeError: 'Lazy' object does not support indexing

Let’s look at the magic methods table to see how we can solve this problem:

Magic methods: setters and getters

Protocol for containers (to define containers like lists,…)	Descriptor (custom class attributes that we fully control)	Called when no attribute found
`__getitem__(self, key)`	`__get__(self, obj, cls=None)`	`__getattr__(self, name)`
`__setitem__(self, key, value)`	`__set__(self, obj, val)`
`__delitem__(self, key)`	`__delete__(self, obj)`

Our current implementation:

class Lazy:
    def __init__(self, key):
        self.key = key

    @property
    def value(self):
        thetype = redis.type(self.key)
        if thetype == b'string':
            result = redis.get(self.key).decode('utf-8')
        elif thetype == b'list':
            result = redis.lrange(self.key, 0, -1)
            result = [i.decode('utf-8') for i in result]
        return result

Seems like what we need is using __getitem__:

class Lazy:
    def __init__(self, key):
        self.key = key

    @property
    def value(self):
        thetype = redis.type(self.key)
        if thetype == b'string':
            result = redis.get(self.key).decode('utf-8')
        elif thetype == b'list':
            result = redis.lrange(self.key, 0, -1)
            result = [i.decode('utf-8') for i in result]
        return result

    def __getitem__(self, key):
        return self.value[key]

Let’s run this:

>>> root.sides = ["fries", "salad"]
print(root.sides[0])
["fries", "salad"]
>>> print(root.sides[0])
fries

Yes!

RedisWorks

As you can see it can get complicated. And I don’t want to bother you with every edge case and data type that is out there. That’s why I wrote Redisworks and it readu to use. Redisworks is basically what we created through this article plus it handles other data types and edge cases. It should be relatively easy to read the source code for you now that you understand basically what it does!

Based on DotObject and PyRedis.

pip install redisworks

https://github.com/seperman/redisworks

Lazy Redis Queries
Multi Query evaluation
Dynamic Typing
Ease of use

Here are some examples of how dynamic typing makes life easier:

Strings

PyRedis

>>> from redis import StrictRedis
>>> redis = StrictRedis()
>>> redis.set("root.something", "value")

RedisWorks

>>> from redisworks import Root
>>> root=Root()
>>> root.something = "value"

Lists with different data types

PyRedis

In PyRedis you get back everything as string:

>>> redis.rpush("root.sides", 10, "root.something", "value")
>>> values = redis.lrange("root.sides", 0, -1)
>>> values
[b'10', b'root.something', b'value']

RedisWorks

>>> root.sides = [10, "fries", "coke"]
>>> root.sides[1]
'fries'
>>> "fries" in root.sides
True
>>> type(root.sides[0])
int

Nested lists

PyRedis

In PyRedis the nested list is returned serialized.

>>> values = [10, [1, 2]]
>>> redis.rpush("root.sides", *values)
2
>>> redis.lrange("root.sides", 0, -1)
[b'10', b'[1, 2]']

RedisWorks

>>> root.sides = [10, [1, 2]]
>>> root.sides
[10, [1, 2]]
>>> type(root.sides[1])
<class 'list'>

Dictionaries

PyRedis

Do you really want to remember hmset, hgetall?

>>> redis.hmset("root.something", {1:"a", "b": {2: 2}})
>>> val = redis.hgetall("root.something")
>>> val
{b'b': b'{2: 2}', b'1': b'a'}
>>> val[b'b']
b'{2: 2}'

RedisWorks

>>> root.something = {1:"a", "b": {2: 2}}
>>> root.something
{'b': {2: 2}, 1: 'a'}
>>> root.something['b'][2]
2

Bottom-line

Lazy loading is a design pattern that fits certain use cases. We studied how writing a Redis client can fit a good use case for lazy objects and we learned how to write lazy objects. In order to write lazy objects, you need to understand Python magic methods. These methods might be magic to the users of your code but it should not be magic to you.

Why did I write this article?

This is basically a talk I gave at SoCal Python. I used the opportunity to write a Redis client that I wanted to use it myself and gave a talk about what I learnt writing it.

pip install redisworks

https://github.com/seperman/redisworks

Hope you enjoyed reading this.

The PDF of this talk is available here.

RedisWorks, the Pythonic Redis Client.

Wed, 10 Aug 2016 15:34:03 -0700

As a person who loves dynamic typing, I have always been frustrated with Py-Redis. It is a great library, but it is too low level. It maps the actual Redis commands directly.

I always wanted to just say the.path = "something" and have it magically saved into Redis. Then somewhere else say print(the.path) and get the value from Redis. As simple as that! And here it is: RedisWorks

Install

pip install redisworks

Note that RedisWorks needs Redis server 2.4+.

Setup

let’s say if you want all the keys in Redis to start with the word root. Then you:

root = Root()  # connects to Redis on local host by default

Or if you want to be more specific:

root = Root(host='localhost', port=6379, db=0)

Saving to Redis

Saving to Redis is as simple as assigning objects to attributes of root or attributes of attributes of root (you can go as deep as you want.) Make sure you are not using any Python’s reserved words in the key’s name.

Example:

>>> from redisworks import Root
>>> import datetime
>>> root = Root()
>>> root.my.list = [1, 3, 4]
>>> root.my.other.list = [1, [2, 2]]
>>> 
>>> some_date = datetime.datetime(2016, 8, 22, 10, 3, 19)
>>> root.time = some_date
>>> 
>>> root.the.mapping.example = {1:1, "a": {"b": 10}}

Redis works will automatically convert your object to the proper Redis type and immediately write it to Redis as soon as you assign an element!

The respective keys for the above items will be just like what you type: root.my.list, root.time, root.the.mapping.example:

If you use redis-cli, you will notice that the data is saved in the proper Redis data type:

127.0.0.1:6379> scan 0
1) "0"
2) 1) "root.the.mapping.example"
   2) "root.time"
   3) "root.my.list"
127.0.0.1:6379> type root.the.mapping.example
hash
127.0.0.1:6379> type root.time
string
127.0.0.1:6379> type root.my.list
list

Reading from Redis

Reading the data is as simple as if it was just saved in Python memory!

Redis works returns Lazy queries just like how Django returns lazy queries. In fact the lazy objects code is borrowed from Django!

If you ran the example from Saving to Redis, run a flush root.flush() to empty Redisworks Cache. This is so it goes and gets the objects from Redis instead of reading its own current copy of data:

>>> from redisworks import Root
>>> import datetime
>>> root = Root()
>>> thetime = root.time
>>> thelist = root.my.list
>>> mydict = root.the.mapping.example
>>> mydict  # is not evalurated yet!
<Lazy object: root.the.mapping.example>
>>> print(mydict)
{1:1, "a": {"b": 10}}  # Now all the 3 objects are read from Redis!
>>> mydict
{1:1, "a": {"b": 10}}
>>> root.my.list
[1, 3, 4]
>>> root.my.other.list
[1, [2, 2]]
>>> root.time
2016-08-22 10:03:19

Changing root key name

Every key name by default starts with the word root. If you want to use another name, simple subclass Root:

>>> from redisworks import Root
>>> post=Post()
>>> post.item1 = "something"  # saves to Redis
...
>>> print(post.item1)  # loads from Redis
something

Numbers as attribute names

Let’s say you want root.1 as a key name. Python does not allow attribute names start with numbers.

All you need to do is start the number with the character i so Redisworks takes care of it for you:

>>> root.i1 = 10
>>> print(root.i1)
10

The actual key in Redis will be root.1

Use it, let me know how it goes!

I’m pretty excited about this library. I learned a lot while making it and I will be speaking about it in the SoCal Python meetup on Aug 30 2016. Check it out if you are around!

Sep

Amazon S3 made simple for Python apps.

Sun, 10 Apr 2016 15:34:03 -0700

From time to time I need to do some simple file/folder transfer/query from a Python app to a S3 bucket. There is a great Python library called Boto that offers a comprehensive low level interface to AWS. I consider it too low level for simple operations on S3. On the other hand there is a fantastic command line library called s3cmd written in Python but it is designed to be used from command line and not from Python apps. Every other Python library I found to be dealing with S3 buckets, seemed unmaintained.

That’s why I came up with S3utils. To make life easier for simple S3 operations from Python apps.

S3utils

https://github.com/seperman/s3utils A user friendly interface to deal with Amazon S3 bucket and Cloud Front in Python.

The s3utils methods are made to be just like Linux commands so it is easy to remember and use.

Behind the scene it is a light Python wrapper around Amazon Boto library.

Installation

Install from PyPi:

pip install s3utils

Setup

Normal Setup

>>> from s3utils import S3utils
>>> s3utils = S3utils(
... AWS_ACCESS_KEY_ID = 'your access key',
... AWS_SECRET_ACCESS_KEY = 'your secret key',
... AWS_STORAGE_BUCKET_NAME = 'your bucket name',
... S3UTILS_DEBUG_LEVEL = 1,  # change it to 0 for less verbose
... )

Django Setup

If you are using s3utils in Django, follow this:

in your settings file:

S3UTILS_DEBUG_LEVEL = 1  # change it to 0 for less verbose
AWS_ACCESS_KEY_ID = 'your access key'
AWS_SECRET_ACCESS_KEY = 'your secret key'
AWS_STORAGE_BUCKET_NAME = 'your bucket name'

in your code:

>>> from s3utils import S3utils
>>> s3utils = S3utils()

Later in your code if you want to overwrite your bucket name in your code from what it is in the Django settings:

>>> from s3utils import S3utils
>>> s3utils = S3utils(AWS_STORAGE_BUCKET_NAME='some other bucket')

Commands

Mkdir

So let’s start with a simple command: mkdir

Just what you would expect, it creates a folder on S3. Example:

>>> from s3utils import S3utils
>>> s3utils = S3utils(
... AWS_ACCESS_KEY_ID = 'your access key',
... AWS_SECRET_ACCESS_KEY = 'your secret key',
... AWS_STORAGE_BUCKET_NAME = 'your bucket name',
... S3UTILS_DEBUG_LEVEL = 1,  #change it to 0 for less verbose
... )
>>> s3utils.mkdir("path/to/my_folder")
Making directory: path/to/my_folder

Cp

Copy a file or folder from local to S3.

Note: As of April 2016, S3utils does not currently support S3 to local direction. I plan to add it in future.

Cp Parameters

local_path : string

Path to file or folder. Or if you want to copy only the contents of folder, add /* at the end of folder name. Example: s3utils.cp("/copy/contents/of/*", "/somewhere/on/s3/")
target_path : string

Target path on S3 bucket.
acl : string, optional

File permissions on S3. Default is public-read

acl options:
- private: Owner gets FULL_CONTROL. No one else has any access rights.
- public-read: Owners gets FULL_CONTROL and the anonymous principal is granted READ access.
- public-read-write: Owner gets FULL_CONTROL and the anonymous principal is granted READ and WRITE access.
- authenticated-read: Owner gets FULL_CONTROL and any principal authenticated as a registered Amazon S3 user is granted READ access
del_after_upload : boolean, optional

delete the local file after uploading. This is effectively like moving the file. You can use s3utils.mv instead of s3utils.cp to move files from local to S3. It basically sets this flag to True. default = False
overwrite : boolean, optional

overwrites files on S3 if set to True. Default is True
invalidate : boolean, optional

invalidates the CDN (a.k.a Distribution) cache if the file already exists on S3

default = False

Note that invalidation might take up to 15 minutes to take place. It is easier and faster to use cache buster to grab latest version of your file on CDN than invalidation.

Examples

>>> s3utils.cp("path/to/folder","/test/")
copying /path/to/myfolder/test2.txt to test/myfolder/test2.txt
copying /path/to/myfolder/test.txt to test/myfolder/test.txt
copying /path/to/myfolder/hoho/photo.JPG to test/myfolder/hoho/photo.JPG
copying /path/to/myfolder/hoho/haha/ff to test/myfolder/hoho/haha/ff

>>> # When overwrite is set to False:
>>> s3utils.cp("path/to/folder","/test/", overwrite=False)
test/myfolder/test2.txt already exist. Not overwriting.
test/myfolder/test.txt already exist. Not overwriting.
test/myfolder/hoho/photo.JPG already exist. Not overwriting.
test/myfolder/hoho/haha/ff already exist. Not overwriting.

>>> # To overwrite the files on S3 and invalidate the CDN (cloudfront) cache so the new file goes on CDN:
>>> s3utils.cp("path/to/folder","/test/", overwrite=True, invalidate=True)
copying /path/to/myfolder/test2.txt to test/myfolder/test2.txt
copying /path/to/myfolder/test.txt to test/myfolder/test.txt
copying /path/to/myfolder/hoho/photo.JPG to test/myfolder/hoho/photo.JPG
copying /path/to/myfolder/hoho/haha/ff to test/myfolder/hoho/haha/ff

Mv

Move the file to the S3 and deletes the local copy

It is basically s3utils.cp that has del_after_upload=True

Examples

>>> s3utils.mv("path/to/folder","/test/")
moving /path/to/myfolder/test2.txt to test/myfolder/test2.txt
moving /path/to/myfolder/test.txt to test/myfolder/test.txt
moving /path/to/myfolder/hoho/photo.JPG to test/myfolder/hoho/photo.JPG
moving /path/to/myfolder/hoho/haha/ff to test/myfolder/hoho/haha/ff

Chmod

sets permissions (acl) for a file on S3

Chmod Parameters

target_file : string

Path to file on S3
acl : string, optional

File permissions on S3. Default is public-read

options:

private: Owner gets FULL_CONTROL. No one else has any access rights.
public-read: Owners gets FULL_CONTROL and the anonymous principal is granted READ access.
public-read-write: Owner gets FULL_CONTROL and the anonymous principal is granted READ and WRITE access.
authenticated-read: Owner gets FULL_CONTROL and any principal authenticated as a registered Amazon S3 user is granted READ access

Examples

>>> s3utils.chmod("path/to/file","private")

Ls

gets the list of file names (keys) in a s3 folder

Ls Parameters

folder : string

Path to file on S3
num: integer, optional

number of results to return, by default it returns all results.
begin_from_file: string, optional

which file to start from on S3. This is useful in case you are iterating over lists of files and you need to page the result by starting listing from a certain file and fetching certain num (number) of files.

Examples

>>> from s3utils import S3utils
>>> s3utils = S3utils(
... AWS_ACCESS_KEY_ID = 'your access key',
... AWS_SECRET_ACCESS_KEY = 'your secret key',
... AWS_STORAGE_BUCKET_NAME = 'your bucket name',
... S3UTILS_DEBUG_LEVEL = 1,  #change it to 0 for less verbose
... )
>>> print s3utils.ls("test/")
[u'test/myfolder/', u'test/myfolder/em/', u'test/myfolder/hoho/', u'test/myfolder/hoho/.DS_Store', u'test/myfolder/hoho/haha/', u'test/myfolder/hoho/haha/ff', u'test/myfolder/hoho/haha/photo.JPG']

LL

Get the list of files and permissions from S3

LL Parameters

folder : string

Path to file on S3
num: integer, optional

number of results to return, by default it returns all results.
begin_from_file : string, optional

which file to start from on S3.

This is useful in case you are iterating over lists of files and you need to page the result by starting listing from a certain file and fetching certain num (number) of files.
all_grant_data : Boolean, optional

More detailed file permission data will be returned.

Examples

>>> from pprint import pprint
>>> my_folder_stuff = s3utils.ll("/test/")
>>> pprint(my_folder_stuff, indent=2)
{
  "test/myfolder/": [
    {
      "name": "owner's name",
      "permission": "FULL_CONTROL"
    }
  ],
  "test/myfolder/em/": [
    {
      "name": "owner's name",
      "permission": "FULL_CONTROL"
    }
  ],
  "test/myfolder/hoho/": [
    {
      "name": "owner's name",
      "permission": "FULL_CONTROL"
    }
  ],
  "test/myfolder/hoho/.DS_Store": [
    {
      "name": "owner's name",
      "permission": "FULL_CONTROL"
    },
    {
      "name": null,
      "permission": "READ"
    }
  ],
  "test/myfolder/hoho/haha/": [
    {
      "name": "owner's name",
      "permission": "FULL_CONTROL"
    }
  ],
  "test/myfolder/hoho/haha/ff": [
    {
      "name": "owner's name",
      "permission": "FULL_CONTROL"
    },
    {
      "name": null,
      "permission": "READ"
    }
  ],
  "test/myfolder/hoho/photo.JPG": [
    {
      "name": "owner's name",
      "permission": "FULL_CONTROL"
    },
    {
      "name": null,
      "permission": "READ"
    }
  ],
}

Invalidate CDN

Invalidate the CDN (distribution) cache for a certain file of files. This might take up to 15 minutes to be effective.

You can check for the invalidation status using check_invalidation_request.

Examples

>>> aa = s3utils.invalidate("test/myfolder/hoho/photo.JPG")
>>> print aa
('your distro id', u'your request id')
>>> invalidation_request_id = aa[1]
>>> bb = s3utils.check_invalidation_request(*aa)
>>> for inval in bb:
...     print('Object: %s, ID: %s, Status: %s' % (inval, inval.id, inval.status))

Conclusion

S3 Utils aims to give Python app developers, an easy to use, high level interface to Amazon S3 buckets. If you do find Sutils useful, please give it a star on github: https://github.com/seperman/s3utils

Diff It To Digg It

Thu, 07 Apr 2016 15:34:03 -0700

The Talk

Here is the actual talk!

I gave a talk at Pycon 2016 about how DeepDiff does what it does.

The proposal

This is my Python talk proposal for 2016. It is about how my DeepDiff does what it does. I will be presenting this talk on June 1st at Pycon. https://us.pycon.org/2016/schedule/presentation/1862/

Edit (June 2016): read more about how the presentation went at Pycon presentation after thoughts

Summary

Anybody who has used git diff will know that your life is not the same once you start diffing. When you get to the habit, there is no going back. In this talk we will see how to diff python objects, and the different scenarios that you can bump into while diffing: from diffing built-in datatypes, nested objects, slots, unhashables to objects with loops and more. Diffing python objects can give you a good insight into Python data types. Finally a package called DeepDiff is introduced which makes diffing easy.

Description

Intro

Once upon a time in land a far away, I had to write a test that would check a system’s output to the expected one. This didn’t sound like a lot of work, except when there was a change anywhere in the dictionary, I had to dig it to find what exactly changed. (show 2 big nested dictionaries that are very different and the result of diffing.)

So the question was:

how do I feed in 2 objects (t1, t2) and get their diff in a way that it handles:

Diff nested dictionaries and custom objects
Get the exact path and value of the item that had been changed, added or removed
Ignore an iterable’s order when required
Work with both Python 2 and 3

So I thought, ok cool let’s start simple and build upon that. First we categories the objects depending on both the approach we want to take to diff them and the format of the output we want to get:

Objects categories

Text Sequences aka strings (basestring, str, bytes)
Numerics (int, float, long, complex, datetime.datetime, Decimal)
Mappings (Dict, OrderedDict, Defaultdict)
Set, Frozen set
Other Iterables (List, Generator, Deque, Tuple, Custom Iterable)
Custom Objects

Text Sequences aka Strings

basestring, str, bytes

py3	example	py2	example
str	“hello”	unicode	u"hello"
bytes	b"hello"	str	“hello”

The best tool for multi-line strings diffing in Python is the difflib:

import difflib
difflib.unified_diff(t1.splitlines(), t2.splitlines(), lineterm='')

Example:

>>> import difflib
>>> t1="""
... Hello World!
... """
>>> t2="""
... Hello World!
... It is Ice-cream time.
... """
>>> mygenerator = difflib.unified_diff(t1.splitlines(), t2.splitlines(), lineterm='')
>>> print '\n'.join(list(mygenerator))
---
+++
@@ -1,2 +1,3 @@

 Hello World!
+It is Ice-cream time.

Numerics

Py3	int, float, complex, Decimal
Py2	int, float, long, complex, Decimal

Set, Frozenset

A set object is an unordered collection of distinct hashable objects. Frozenset is immutable set.

>>> t1 = {1,2,3}
>>> t2 = {3,4,5}
>>> items_added = t2 - t1
>>> items_removed = t1 - t2
>>> items_added
set([4, 5])
>>> items_removed
set([1, 2])

Mappings

Dict, OrderedDict, Defaultdict

When dealing with dictionaries, first we need to find the mutual keys, keys added and removed:

t1_keys= set(t1.keys())
t2_keys= set(t2.keys())

same_keys = t2_keys.intersection(t1_keys)

added = t2_keys - same_keys
removed = t1_keys - same_keys

Then we want to diff the values of common keys between t1 and t2. We will do this recursively.

Diff other Sequences

List, Tuple and other iterables

What is an iterable? any object that has __iter__ so you can iterate over it. For example a list, tuple or a generator.

Diff sequence when order is important

>>> from itertools import zip_longest
>>> 
>>> t1=[1,2,3]
>>> t2=[1,2,5,6]
>>> 
>>> class ListItemRemovedOrAdded(object):
...     pass
... 
>>> items_removed = []
>>> items_added = []
>>> items_modified = []
>>> 
>>> for (x, y) in zip_longest(t1, t2, fillvalue=ListItemRemovedOrAdded):
...     if y is ListItemRemovedOrAdded:
...         items_removed.append(x)
...     elif x is ListItemRemovedOrAdded:
...         items_added.append(y)
...     else:
...         items_modified.append("{} changed to {}".format(x, y))
... 
>>> 
>>> items_removed
[]
>>> items_added
[6]
>>> items_modified
['1 changed to 1', '2 changed to 2', '3 changed to 5']

Diff sequence when we don not care about the order

Convert the sequence to set and then diff it!

>>> t1=[1,2]
>>> t2=[1,3,4]
>>> t1set=set(t1)
>>> t2set=set(t2)
>>> t1set-t2set
{2}
>>> t2set-t1set
{3, 4}

But…

>>> t1=[1, 2, {3:3}]
>>> t2=[1]
>>> t1set = set(t1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'dict'

Let’s read the definition of set again:

A set object is an unordered collection of distinct hashable objects.

If your iterable contains unhashable item, we can’t convert it to a set. What is the difference of Unhashable and Mutable?

Let’s review this again:

Mutable vs. Immutable

Mutable

Mutable objects can change their value but keep their id().
Immutable

An object with a fixed value. Immutable objects include numbers, strings and tuples. Such an object cannot be altered. A new object has to be created if a different value has to be stored. They play an important role in places where a constant hash value is needed, for example as a key in a dictionary.

Hashable

https://docs.python.org/3/glossary.html#term-hashable

An object is hashable if it has a hash value which never changes during its lifetime (it needs a __hash__() method), and can be compared to other objects (it needs an __eq__() method).

All of Python’s immutable built-in objects are hashable, while no mutable containers (such as lists or dictionaries) are. Objects which are instances of user-defined classes are hashable by default; they all compare unequal (except with themselves), and their hash value is derived from their id().

Unhashable vs. Mutable

A lot of people confuse these 2 concepts with each other.

Example of Hashable object that is Mutable:

>>> class A:
...     aa=1
...
>>> hash(A)
2857987
>>> A.aa=2
>>> hash(A)
2857987

Objects which are instances of user-defined classes are hashable by default; they all compare unequal (except with themselves), and their hash value is derived from their id().

Diff sequence containing unhashable ignoring the order

So how do we diff iterable that contains unhashable? And we don’t care about the order or items?

Approach 1: Sort Iterable and then compare objects one by one

The first approach I took was to convert iterable to list, then sort it and then compare objects one by one. The irony here is we don’t care about the order of items in the original iterables but in order to compare them side by side we order them:

Python 2:

>>> t1=[{1:1}, {3:3}, {4:4}]
>>> t2=[{3:3}, {1:1}, {4:4}]
>>> t1.sort()
>>> t1
[{1: 1}, {3: 3}, {4: 4}]
>>> t2.sort()
>>> t2
[{1: 1}, {3: 3}, {4: 4}]

Ok great, now we can iterate through both lists and compare each item in t1 to the corresponding one in t2:

>>> [(a, b) for a, b in zip(t1,t2) if a != b]
[]

Python 3:

>>> t1=[{1:1}, {3:3}, {4:4}]
>>> t2=[{3:3}, {1:1}, {4:4}]
>>> t1.sort()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unorderable types: dict() < dict()

Ok, so this is weird. It worked in Python 2 but NOT python 3. Obviously the sorting algorithm has changed from Py2 to 3.

What is the solution?

Sort key

https://wiki.python.org/moin/HowTo/Sorting

There is a key parameter to specify a function to be called on each list element prior to making comparisons. The output of key function is used for comparison and sorting. Think about it as a function that creates hooks that represent the object for the sort algorithm. Then the sort algorithm grabs these hooks and orders the hooks and thus sorts the actual object.

>>> student_tuples = [
        ('john', 'A', 15),
        ('jane', 'B', 12),
        ('dave', 'B', 10),
]
>>> sorted(student_tuples, key=lambda student: student[2])   # sort by age
[('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]

So what can we use for sorting unhashables?

The idea that came to me was to use the hash of objects as the key parameter.

But the problem was dictionaries were unhashable so how could I create a hash for them? Hmm, what about serializing the dictionary and then use the hash of the serialized object for sorting. The best thing about this idea was that if 2 dictionaries are the same, their serialized version should be the same and thus their hash should be the same:

>>> import json
>>> t1=[{1:1}, {3:3}, {4:4}]
>>> t2=[{3:3}, {1:1}, {4:4}]
>>> t1.sort(key=lambda x: hash(json.dumps(x)))
>>> t2.sort(key=lambda x: hash(json.dumps(x)))
>>> t1
[{1: 1}, {3: 3}, {4: 4}]
>>> t2
[{1: 1}, {3: 3}, {4: 4}]
>>> [(a, b) for a, b in zip(t1,t2) if a != b]
[]

yay!

Diffing iterables with different lengths

>>> from itertools import zip_longest
>>> import json
>>>
>>> t1=[10, {1:1}, {3:3}, {4:4}]
>>> t1.sort(key=lambda x: hash(json.dumps(x)))
>>> t1
[{1: 1}, {3: 3}, {4: 4}, 10]
>>>
>>> t2=[{3:3}, {1:1}, {4:4}]
>>> t2.sort(key=lambda x: hash(json.dumps(x)))
>>> t2
[{1: 1}, {3: 3}, {4: 4}]
>>>
>>>
>>> class ListItemRemovedOrAdded(object):
...     pass
...
>>> items_removed = []
>>> items_added = []
>>>
>>> for (x, y) in zip_longest(t1, t2, fillvalue=ListItemRemovedOrAdded):
...     if y is ListItemRemovedOrAdded:
...         items_removed.append(x)
...     elif x is ListItemRemovedOrAdded:
...         items_added.append(y)
...     # else:
...     #     keep_diffing(x, y)
...
>>> items_removed
[10]
>>> items_added
[]

Now what about adding another item to t1:

>>> t1=[10, "a", {1:1}, {3:3}, {4:4}]
>>> t1.sort(key=lambda x: hash(json.dumps(x)))
>>> t1
['a', {1: 1}, {3: 3}, {4: 4}, 10]
>>> t2
[{1: 1}, {3: 3}, {4: 4}]

Ooops, the items with the same hash do not align with each other anymore!

If I run our diff now, we get the wrong result:

>>> items_removed = []
>>> items_added = []
>>> items_modified = []
>>>
>>> for (x, y) in zip_longest(t1, t2, fillvalue=ListItemRemovedOrAdded):
...     if y is ListItemRemovedOrAdded:
...         items_removed.append(x)
...     elif x is ListItemRemovedOrAdded:
...         items_added.append(y)
...     else:
...         items_modified.append("{} changed to {}".format(x, y))
...
>>> items_removed
[{4: 4}, 10]
>>> items_added
[]
>>> items_modified
['a changed to {1: 1}', '{1: 1} changed to {3: 3}', '{3: 3} changed to {4: 4}']

So ordering items based on their hashes was a wrong idea to begin with. But at least we learnt about the key parameter for sorting.

Approach 2: Put items in a dictionary of {item_hash: item}

>>> import json
>>>
>>> t1 = [10, "a", {1:1}, {3:3}, {4:4}]
>>> t2 = [{3:3}, {1:1}, {4:4}, "b"]
>>>
>>>
>>> def create_hashtable(t):
...     '''
...     Creates hashtable of {item_hash: item}
...     '''
...     hashes = {}
...     for item in t:
...         try:
...             item_hash = hash(item)
...         except TypeError:
...             try:
...                 item_hash = hash(json.dumps(item))
...             except:
...                 print ("Warning: Can not produce a hash for %s item" % (item))
...             else:
...                 hashes[item_hash] = item
...         else:
...             hashes[item_hash] = item
...     return hashes
...
>>> t1_hashtable = create_hashtable(t1)
>>> t2_hashtable = create_hashtable(t2)
>>>
>>> items_added = [t2_hashtable[i] for i in t2_hashtable if i not in t1_hashtable]
>>> items_removed = [t1_hashtable[i] for i in t1_hashtable if i not in t2_hashtable]
>>>
>>> items_added
['b']
>>> items_removed
['a', 10]

This approach worked!

Now what if the object is not serializable? Ok in that case we just have to iterate over every object in t1 to see if it exists in t2. And do the same thing for t2 objects comparing to t1. Yes, you got it right as simple as that.

>>> t1 = [1, {1:1}]
>>> t2 = [2, {1:1}]
>>> id(t2)
4566093704
>>> id(t2[1])
4566093512
>>> id(t1[1])
4564810504
>>> t1[1] in t2
True

When it is not Json serializable

What if the object is not json serializable?
What if json serializable version of 2 different objects are the same?

For example Decimals are not Json serializable:

>>> from decimal import Decimal
>>> num = Decimal(10.0)
>>> num
Decimal('10')
>>> from json import dumps
>>> dumps(num)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ProgramsB\Python27\lib\json\__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "C:\ProgramsB\Python27\lib\json\encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "C:\ProgramsB\Python27\lib\json\encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "C:\ProgramsB\Python27\lib\json\encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: Decimal('10') is not JSON serializable
>>>

And if we convert Decimal(10) to integer 10, then the serialization of the two will be the same. Thus the hash will be the same. And when we diff it will think these 2 objects are the same.

Here is how another serialization method comes in:

Pickle

Pickle can serialze most Python objects. It is rare that you bump into objects that can’t be pickled. However the important thing for us to create serialization of content in a way that the serialization does not change if object’s contents are the same.

Let’s see how Pickle’s serialization result looks like for 2 objects that have the same content:

>>> from pickle import dumps
>>> t = ({1: 1, 2: 4, 3: 6, 4: 8, 5: 10},
'Hello World', (1, 2, 3, 4, 5), [1, 2, 3, 4, 5])
>>> dumps(t)
"((dp0\nI1\nI1\nsI2\nI4\nsI3\nI6\nsI4\nI8\nsI5\
nI10\nsS'Hello World'\np1\n(I1\nI2\nI3\nI4\nI5\
ntp2\n(lp3\nI1\naI2\naI3\naI4\naI5\natp4\n."
>>> dumps(({1: 1, 2: 4, 3: 6, 4: 8, 5: 10},
           'Hello World', (1, 2, 3, 4, 5),
            [1, 2, 3, 4, 5]))
"((dp0\nI1\nI1\nsI2\nI4\nsI3\nI6\nsI4\nI8\nsI5\
nI10\nsS'Hello World'\np1\n(I1\nI2\nI3\nI4\nI5
\ntp2\n(lp3\nI1\naI2\naI3\naI4\naI5\natp4\n."

The serilzation of both objects look the same. Yes!

Cpickle

What about cPIckle? It is faster than Pickle! It is written in C. And supposed to be 1000 times faster than Pickle.

>>> from cPickle import dumps
>>> t = ({1: 1, 2: 4, 3: 6, 4: 8, 5: 10},
'Hello World', (1, 2, 3, 4, 5), [1, 2, 3, 4, 5])
>>> dumps(t)
"((dp1\nI1\nI1\nsI2\nI4\nsI3\nI6\nsI4\nI8\nsI5\n
I10\nsS'Hello World'\np2\n(I1\nI2\nI3\nI4\nI5\n
tp3\n(lp4\nI1\naI2\naI3\naI4\naI5\nat."
>>> dumps(({1: 1, 2: 4, 3: 6, 4: 8, 5: 10},
           'Hello World', (1, 2, 3, 4, 5),
            [1, 2, 3, 4, 5]))
"((dp1\nI1\nI1\nsI2\nI4\nsI3\nI6\nsI4\nI8\nsI5\n
I10\nsS'Hello World'\n(I1\nI2\nI3\nI4\nI5\nt(lp2
\nI1\naI2\naI3\naI4\naI5\natp3\n."

Uh oh! The result serializations of the same objects are not the same using cPickle!.

Why? cPickle tries to optimize the serialization by including if the object is referenced once or more than once! For a discussion around this look at http://stackoverflow.com/q/7501577/1497443

Why cPickle works for our case

I ended up using cPickle to create the serilzation to generate the {item_hash: item} hash table in DeepDiff. The reason it worked for DeepDiff was that we were creating references to the objects inside the libaray while diffing. So the cPickle of objects with equal content was always the same during the program run. Again cPickle ONLY produces different results for the same content if there is only one reference to the object vs. more than one reference.

What did we learn from diffing iterables

Difference of unhashable and mutable
Sets can only contain hashable
Create hash for dictionary
Custom sorting with a key function
Converting a squence into hashtable

Diff Custom objects

Let’s use __dict__ attribute to get the instance’s attributes. By default, instances of classes have a dictionary for attribute storage called __dict__:

>>> class CL:
...     attr1 = 0
...     def __init__(self, thing):
...         self.thing = thing

>>> obj1 = CL(1)
>>> obj2 = CL(2)
>>> obj2.attr1 = 10
>>> obj1.__dict__
{'thing': 1}  # Notice that att1 is not here
>>> obj2.__dict__
{'attr1': 10, 'thing': 2}

And we deal with it as if we were diffing 2 dictionaries.

`slots` aka When there is no `dict`

If the class is using __slots__ to have more control over the object attributes, there won’t be a __dict__:

Example:

>>> class ClassA(object):
...     __slots__ = ['x', 'y']
...     def __init__(self, x, y):
...         self.x = x
...         self.y = y
...
>>> t1 = ClassA(1, 1)
>>> t2 = ClassA(1, 2)
>>>
>>> t1.d = 10
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'ClassA' object has no attribute 'd'

As you can see with __slots__ we can’t dynamically add attributes. You can think of __slots__ as a bridge towards static typing in python.

And unlike __dict__, __slots__ is NOT a dictionary:

>>> t1.__slots__
['x', 'y']

So in order to create a dictionary out of the attributes:

>>> t1 = {i: getattr(t1, i) for i in t1.__slots__}
>>> t2 = {i: getattr(t2, i) for i in t2.__slots__}
>>> t1
{'x': 1, 'y': 1}
>>> t2
{'x': 1, 'y': 2}

As far as diffing goes, the rest is just like comparing 2 dictionaries.

loops

Did you know that Python objects can have attributes pointing to themselves!

>>> class LoopTest(object):
...     def __init__(self, a):
...         self.loop = self
...         self.a = a
...
>>> t1 = LoopTest(1)
>>> t2 = LoopTest(2)
>>> t1
<__main__.LoopTest object at 0x02B9A910>
>>> t1.__dict__
{'a': 1, 'loop': <__main__.LoopTest object at 0x02B9A910>}

Let’s step back and think about it for a sec:

For custom objects, we are creating a dictionary out of instance attributes. Then we are creating 3 lists: attributes added, attributes removed and common attributes (common keys of the dictionaries). Then we are recursively diffing the common attributes values. So once we hit the same object again, we get stuck in a never ending loop!

Detect loops with ID

As you know every object in python has a unique ID. And if 2 objects have the same ID, they are the same object.

Let’s say A is parent of B and B is parent of C and C is parent of A:

A --> B --> C --> A

object	A	B	C	A
id	1	2	3	1

If we keep the ID of every object and every parent of the object, once we start processing an object, we can check to see if its ID exists in the list of IDs of its parents. And in that case, we get out of the loop!

Example:

object	ID List
A	1
B	1, 2
C	1, 2, 3
A	1, 2, 3, 1 (Loop detected, get me out of here!)

Since we will be creating this list of IDs for every object in the nested structure that we process recursively, we have to make sure the ID list is immutable.

def diff_common_children_of_dictionary(t1, t2, t_keys_intersect, parents_ids):
    ''' difference between common attributes of objects or values of common keys of dictionaries '''
    for item_key in t_keys_intersect:

        t1_child = t1[item_key]
        t2_child = t2[item_key]

        item_id = id(t1_child)

        if parents_ids and item_id in parents_ids:
            print ("Warning, a loop is detected.")
            continue

        parents_added = set(parents_ids)
        parents_added.add(item_id)
        parents_added = frozenset(parents_added)

        diff(t1_child, t2_child, parents_ids=parents_added)

Why it has to be immutable? because otherwise the same parents_ids object is gonna be passed around the recursion (passed by reference) instead of a new parents_ids object per object.

What did we learn about diffing custom objects

We try to convert every object into a dictionary. Either using its __dict__ or __slots__ when diffing.
Once objects are dictionaries, we will process it recursively to diff all their common attributes.
Python objects can have attributes pointing to themselves or any of their parents!
In order to detect loops in objects, we can use an immutable container of object IDs to know all the ancestors of an object.

Conclusion

Diffing python objects can give you a good insight into Python data types. I find diffing very useful in debugging and testing. For example I have made all my assertEqual calls to diff the test result object and the expected result if the assertion fails. This helps a lot to understand how exactly your expected result and test result are different.

To make things easier, I have made a package called DeepDiff that you can easily install from Pypi and use:

pip install DeepDiff

https://github.com/seperman/deepdiff

DeepDiff makes it super easy to diff complex objects:

Examples:

List difference

>>> t1 = {1:1, 2:2, 3:3, 4:{"a":"hello", "b":[1, 2, 3]}}
>>> t2 = {1:1, 2:2, 3:3, 4:{"a":"hello", "b":[1, 3, 2, 3]}}
>>> ddiff = DeepDiff(t1, t2)
>>> pprint (ddiff, indent = 2)
{ 'iterable_item_added': ["root[4]['b']: [3]"],
  'values_changed': ["root[4]['b'][1]: 2 ===> 3", "root[4]['b'][2]: 3 ===> 2"]}

List that contains dictionary:

>>> t1 = {1:1, 2:2, 3:3, 4:{"a":"hello", "b":[1, 2, {1:1, 2:2}]}}
>>> t2 = {1:1, 2:2, 3:3, 4:{"a":"hello", "b":[1, 2, {1:3}]}}
>>> ddiff = DeepDiff(t1, t2)
>>> pprint (ddiff, indent = 2)
{ 'dic_item_removed': ["root[4]['b'][2][2]"],
  'values_changed': ["root[4]['b'][2][1]: 1 ===> 3"]}

Named Tuples:

>>> from collections import namedtuple
>>> Point = namedtuple('Point', ['x', 'y'])
>>> t1 = Point(x=11, y=22)
>>> t2 = Point(x=11, y=23)
>>> print (DeepDiff(t1, t2))
{'values_changed': ['root.y: 22 ===> 23']}

Ignoring order:

>>> t1 = [{"a": 2}, {"b": [3, 4, {1: 1}]}]
>>> t2 = [{"b": [3, 4, {1: 1}]}, {"a": 2}]
ddiff = DeepDiff(t1, t2, ignore_order=True)
>>>
>>> print(DeepDiff(t1, t2))
{}

And more: https://github.com/seperman/deepdiff

Celery: Get Your Task Together

Mon, 07 Mar 2016 15:34:03 -0700

Why Distributed Task Queues?

Offload long jobs to background processes example: video conversion
Offload too many [small] jobs to background processes example: commenting system
Keep track of jobs, monitor, auto-restart
Schedule jobs
Replace cron jobs. Cron jobs can run maximum once a minute. That is a limitation. And they don’t have the best interface.

Message Queue vs. Task Queue

Message Queue are the basic functionality of passing, holding, and delivering messages Example: Redis, RabbitMQ
Tasks Queue manage work to be done and is considered a type of message queue Example: Celery

Distributed Task Queus

in Python

Based on Popularity

Solution	Stars on git	Downloads/mo
Celery	4600	400,000
RQ (Redis Queue)	2600	40,000
⤿ Django RQ	428	13,000
Huey	824	3,000
MrQ (Mr. Q)	340	5,000
Taskmaster	346	1,000

Who uses them? Everyone.

Solution	User
Celery	Instagram, Mozilla, Truecar
RQ (Redis Queue)	?
⤿ Django RQ	?
Huey	?
MrQ (Mr. Q)	Pricing Assistant (creator)
Taskmaster	Disqus (creator)

Celery Architecture

Image from parallel programming in Python book

RQ Architecture

Client ⥂ Redis ⥄ Worker

Celery Task Example

from celery import Celery
app = Celery('tasks', broker='amqp://guest@localhost//')
@app.task
def add(x, y):
    import time; sleep(5*60)
    return x + y

------------------

>>> from tasks import add
>>> result = add.delay(4, 4)
>>> result.ready()
False
5 minutes later:
>>> result.ready()
True

RQ Task example

from rq import Queue
from redis import Redis
from somewhere import count_words_at_url

# Tell RQ what Redis connection to use
redis_conn = Redis()
# no args implies the default queue
q = Queue(connection=redis_conn)

# Delay execution of count_words_at_url('http://nvie.com')
job = q.enqueue(count_words_at_url, 'http://nvie.com')
print job.result   # => None

# Now, wait a while, until the worker is finished
time.sleep(2)
print job.result   # => 889

Monitoring

Celery - Flower
RQ - Dashboard

Celery monitoring: Flower

Workers

Tasks

CPU usage

RQ Monitoring: RQ Dashboard

MRQ Monitoring: MRQ Dashboard

Celery vs. RQ - overview

	Celery	RQ
Complexity of code	Very complicated	Easy to understand
Documentation	Take a while to read	Simple
Monitoring	Flower	RQ Dashboard
message brokers	RabbitMQ, Redis, MongoDB	Redis
result backends	RabbitMQ, Redis, Memcached, MongoDB, Cassandra,…	Redis
Concurrency	Master-Slave processes	supervisord() + fork
Scheduler	Celerybeats	3rd party
Language	Can send tasks from one language to another	Only Python
Subtasks	Can create tasks within tasks	Nope
Django support	Built-in	Django-rq

Why use RQ?

It all comes down to simplicity. and… MEMORY LEAK

Redis Memory Leak

Celery has Memory Leak issues
Some memory leak can happen with older broker libraries. i.e. librabbitmq
Celery monitor (Flower) has huge memory leak.
RQ offers less but its memory leak should be much smaller than Celery (not verified it myself)

Why use Celery?

Use Celery when:

RQ limits you to use Redis both as message broker and result backend. If you need another broker/result backend.
Redis can drop messages. But it will pick it up later. If that bothers you, you can’t use RQ.
Celery is way more feature rich and flexible than RQ.
If you don’t ever really need to know the magic behind the scene in Celery.

Complexity

How complex is RQ:

How complex is Celery:

And in case you were wondering…

How complex is Django:

So if you can understand Django source code, you should be able to understand Celery’s too.

Zepworks – Blog

Fixing the blank login screen after upgrading Ubuntu 24.04 to 26.04

TLDR

The problem

Things that didn’t fix it

The error in the logs

Why was an image breaking the login screen?

Finding the conflicting AppArmor profile

The fix

What about security?

Conclusion

DeepDiff 9.0.0 Is Released

The main migration note

What is new in 9.0.0

The short English version

A closer look at the changes

Thank you to the contributors

Another milestone

Moving DeepDiff under Qluster

Pypi Download Stats

Final thoughts

How to persist pod logs on Minikube

TLDR

Why you don’t need this on a managed k8s cluster

Overview of the problem

The docker image

Minikube and file permissions

Kubernetes yaml file

Bonus material: fsGroup, runAsGroup, and runAsUser

1. runAsUser:

2. runAsGroup:

3. fsGroup:

How They Work Together:

Example Scenario:

Conclusion

Upgrading to Pydantic 2: More quirky than you would expect.

Bye bye class Config

Pydantic 2 does not automatically coerce to the type you want to

Private field names don’t work with Pydantic 2

Pydantic 2 is tricky when multiple types are allowed.

Pydantic 2 Optional fields are defined very differently

Introducing Qluster - ETL As Spreadsheet

💥 Qluster

Where it came from?

The core problem

Introducing Qluster

DeepDiff Tutorial: Deep Commandline

Get sample data

What has changed from month to month?

Group by

exclude-regex-paths

significant-digits

significant-digits behind the scene

significant-digits + number-format-notation

Which months are the most similar?

Example 2

Ignore Order

Repeats and ignore order

Patch

Applying the patches

Patches can be applied to any objects!

What about applying to a python object?

Looking for something in the file?

Extracting a value of a path?

Conclusion

A How To Guide: Remotely Accessing Minikube Kubernetes on KVM

Our Setup

1. Install Minikube, kubectl, and KVM on the host

2. Run Minikube with explicit apiserver ip on the host

3. Port Forward

3.a Install Nginx

3.b Reverse Proxy via NGINX

3.c Open the port on the firewall

3.d Test from local

4. Certs

5. Remote Access

Conclusion

DeepDiff: 1.7 Million Downloads A Month

DeepDiff has surpassed 1.7 million downloads a month!

Notable DeepDiff Users

1. `runAsUser`:

2. `runAsGroup`:

3. `fsGroup`: